Serialization & Wire Formats

How It Works

Every time one service sends data to another, that data must be serialized — converted from in-memory objects to bytes on the wire. The choice of serialization format affects payload size, CPU consumption, schema management complexity, and the ability to evolve data structures over years without breaking consumers.

This is not an academic exercise. At scale, serialization format is a systems engineering decision with measurable cost implications.

JSON: The Universal Default

JSON is the lingua franca of web APIs. Human-readable, self-describing, supported by every language, and trivially debuggable with curl and jq. For public APIs and low-throughput internal services, JSON is the pragmatic choice.

But JSON has structural inefficiencies that compound at scale:

{
  "userId": 12345,
  "firstName": "Alice",
  "lastName": "Smith",
  "email": "alice@example.com",
  "isActive": true,
  "loginCount": 4287
}

This payload is 142 bytes. The field names (userId, firstName, etc.) consume 55 bytes — nearly 40% of the payload is schema information repeated in every single message. Multiply by 10 billion messages per day and that is 550 GB of redundant field names daily.

JSON parsing cost: JSON is text-based, requiring character-by-character parsing with string allocation for every field name and string value. Libraries like simdjson use SIMD instructions to accelerate parsing to 2-3 GB/sec, but even optimized JSON parsing is 5-10x slower than binary format deserialization.

Protocol Buffers: Schema-First Binary Encoding

Protobuf eliminates field name repetition by assigning field numbers in the schema. The wire format uses these numbers (1-2 bytes each) instead of strings.

// user.proto
syntax = "proto3";

message User {
  int32 user_id = 1;
  string first_name = 2;
  string last_name = 3;
  string email = 4;
  bool is_active = 5;
  int32 login_count = 6;
}

The same data serializes to approximately 52 bytes in Protobuf — 63% smaller than JSON. The encoding:

• Varints for integers: Small numbers use fewer bytes. user_id: 12345 uses 2 bytes instead of 5 ASCII characters.
• Field tag: Each field is prefixed with (field_number << 3) | wire_type — a single byte for fields 1-15.
• No field names: The reader uses the .proto schema to know that field 1 is user_id. The wire format contains only numbers.

Schema evolution rules: Protobuf's compatibility model is strict but effective:

• Adding a new optional field is always safe — old readers ignore unknown fields.
• Removing a field is safe if the field number is reserved (preventing accidental reuse).
• Changing a field type is unsafe unless the types are wire-compatible (e.g., int32 ↔ int64).
• Never reuse a field number. This is the cardinal rule. If field 3 was a string and is reassigned to an int32, old readers applying the string wire type to integer bytes produce garbage.

message User {
  int32 user_id = 1;
  string first_name = 2;
  reserved 3;              // last_name removed — number 3 is retired forever
  string email = 4;
  bool is_active = 5;
  int32 login_count = 6;
  string department = 7;   // new field — old readers silently ignore it
}

Apache Avro: Schema-on-Read for Event Streaming

Avro takes a fundamentally different approach. Instead of compiling the schema into both sides at build time, Avro includes or references the writer's schema with the data. The reader uses both the writer's schema and its own reader's schema to deserialize.

{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "userId", "type": "int"},
    {"name": "firstName", "type": "string"},
    {"name": "lastName", "type": "string"},
    {"name": "email", "type": "string"},
    {"name": "isActive", "type": "boolean", "default": true},
    {"name": "loginCount", "type": "int", "default": 0}
  ]
}

This design is optimized for data pipelines where:

• Producers and consumers are deployed independently
• Messages written months ago must be readable by today's consumers
• The same topic might contain messages written with different schema versions

Avro + Schema Registry: In Kafka deployments, Avro messages contain a 5-byte header (magic byte + 4-byte schema ID) instead of the full schema. The consumer fetches the writer's schema from the registry by ID and uses it for deserialization. The registry enforces compatibility rules:

• BACKWARD: New schema can read data written with the old schema.
• FORWARD: Old schema can read data written with the new schema.
• FULL: Both backward and forward compatible.
• NONE: No compatibility check (dangerous — breaks consumers silently).

MessagePack: Binary JSON Without Schemas

MessagePack occupies a unique middle ground. It is semantically identical to JSON — maps, arrays, strings, integers, booleans, null — but uses binary encoding instead of text.

JSON:    {"compact":true,"schema":0}     → 27 bytes
MsgPack: \x82\xa7compact\xc3\xa6schema\x00  → 18 bytes (33% smaller)

No schema definition. No code generation. No registry. Existing JSON APIs can switch to MessagePack by changing the serializer/deserializer and the Content-Type header (application/msgpack). This makes it the lowest-friction binary format.

The tradeoff: MessagePack still includes field names in every message (like JSON), so it does not achieve Protobuf-level compression. It is also not self-describing enough for schema evolution — there is no type system enforcing compatibility.

Performance Comparison

Real-world benchmarks serializing a 50-field message with nested objects (numbers represent single-core throughput):

Format	Serialize	Deserialize	Payload Size	Schema Required
JSON	150K msg/s	120K msg/s	1000 bytes	No
Protobuf	800K msg/s	900K msg/s	350 bytes	Yes (.proto)
Avro	600K msg/s	500K msg/s	380 bytes	Yes (.avsc)
MessagePack	400K msg/s	350K msg/s	600 bytes	No
FlatBuffers	1.2M msg/s	Zero-copy	400 bytes	Yes (.fbs)

These numbers vary by language, payload structure, and library implementation, but the relative ordering is consistent. Protobuf and FlatBuffers dominate on throughput. Avro trades some speed for schema-on-read flexibility. MessagePack is faster than JSON with smaller payloads. JSON is last on every metric except debuggability.

Zero-Copy Formats: FlatBuffers and Cap'n Proto

Standard serialization requires two steps: parse the bytes into an in-memory object, then access fields on that object. FlatBuffers and Cap'n Proto eliminate the first step. The serialized bytes are the in-memory representation. Accessing a field is a pointer offset into the buffer.

// FlatBuffers: no deserialization step
auto user = GetUser(buffer);       // just a pointer cast
auto name = user->first_name();    // pointer arithmetic into the buffer
auto count = user->login_count();  // another offset read

The advantages are significant for latency-critical paths:

• No allocation: No intermediate objects created on the heap.
• No parsing time: First access is instantaneous regardless of payload size.
• Cache-friendly: Data is accessed in wire order, which maps well to CPU cache lines.

The disadvantage: the data is read-only in its serialized form. Modifying a field requires building a new buffer. FlatBuffers is optimized for read-heavy patterns (read many fields, write once) — exactly the pattern of mobile apps receiving server responses and game engines processing network updates.

Schema Registries: The Operational Backbone

For any binary format with schemas, the Schema Registry is the critical operational component. Without it, schema management devolves into:

• Shared git repos with .proto files that fall out of sync
• Copy-pasted schema definitions across services
• Broken consumers when a producer deploys a schema change

The Confluent Schema Registry (used with Avro and Protobuf in Kafka) provides:

1. Versioned schema storage: Every schema version is immutable and addressable by ID.
2. Compatibility enforcement: Before a new schema version is registered, the registry checks it against the configured compatibility level. Incompatible schemas are rejected.
3. Schema resolution: Consumers fetch the writer's schema by ID embedded in the message, enabling deserialization of messages written with any historical schema version.

# Register a new schema version
curl -X POST http://registry:8081/subjects/user-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[...]}"}'

# Check compatibility before deploying
curl -X POST http://registry:8081/compatibility/subjects/user-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema": "{...new schema...}"}'

Decision Framework

The choice comes down to four factors:

Factor	JSON	Protobuf	Avro	MessagePack	FlatBuffers
Human readability	Excellent	None	None	None	None
Payload size	Large	Small	Small	Medium	Small
Schema evolution	Manual	Good	Excellent	Manual	Good
Integration friction	Zero	Medium (codegen)	Medium (registry)	Low	Medium (codegen)
Deserialization speed	Slow	Fast	Medium	Medium	Zero-copy

For greenfield systems: Protobuf for synchronous RPCs, Avro for event streaming with Kafka, JSON for public-facing APIs. For systems already on JSON that need better performance without operational disruption: MessagePack. For extreme latency requirements with read-heavy access patterns: FlatBuffers.

The worst outcome is not choosing the wrong format — it is having no schema discipline. A system using Protobuf with sloppy field number management is worse than a system using JSON with documented, versioned API contracts. The format is a tool; the schema governance process is the strategy.