What is Avro?

Key Takeaways

• Apache Avro is a language-neutral, schema-based serialization system for exchanging structured records.
• An Avro object container file embeds its writer schema in its header. Raw Avro binary data does not carry a schema by itself.
• Schema evolution depends on writer–reader compatibility rules, not simply on making fields nullable.
• Avro container files organize data as records in blocks; Parquet organizes data by columns for analytical access.
• Binary encoding and compression are separate decisions. Benchmark representative data rather than assuming Avro is always smaller or faster.

Apache Avro is a data serialization system that turns structured values into bytes using an explicit schema. Its language-neutral format makes it useful when services and data pipelines need to exchange records without sharing a programming language or application class model.

Avro's practical strengths are compact binary encoding and defined rules for reading data after its schema changes. Whether you should choose it over JSON or Parquet depends on how you exchange, inspect, and query those records.

What is Avro?

Data Serialization

Serialization converts an application's values into a representation you can store or transmit; deserialization reconstructs those values. For example, a user record might contain a string username and an integer age.

With Avro binary encoding, the schema supplies the structure, so field names and much of the type information don't need to repeat in every record. Records encode field values in the order declared by the writer schema. That reduces structural overhead, but it also means you need that schema to interpret the bytes correctly.

The Apache Avro introduction describes its binary format, container files, and optional code generation. Avro is not the same mechanism as Java object serialization: your shared contract is an Avro schema, not a Java class.

Data Exchange

Consider Avro for event streams, change-data-capture pipelines, service-to-service messages, and sequential data exports. Apache publishes implementations including Java, Python, C++, and C#; check the selected libraries' feature support before deploying a cross-language pipeline.

Avro defines the record representation, not your delivery guarantees, retries, or consumer deployment strategy. For event pipelines, understanding Kafka architecture remains a separate concern.

How Avro Works

The Writer Schema and Reader Schema

The writer schema describes the data that was encoded. The reader schema describes the structure your consuming application expects. They can be identical, but Avro can resolve certain differences between them.

1. The producer selects a writer schema and serializes a matching record.
2. The transport or storage arrangement preserves access to that writer schema.
3. The consumer obtains the writer schema and optionally supplies a different reader schema.
4. The decoder applies schema resolution: matching fields, ignoring unneeded writer fields, filling eligible reader fields with defaults, or rejecting incompatible differences.

Never substitute your newest schema for an older record's writer schema. Evolution works by using both schemas, not by guessing what the historical bytes mean. Apache's Avro specification defines the encoding and resolution rules.

Raw Binary Data Versus Object Container Files

Raw datum encoding represents an individual value without embedding its schema. Your application must arrange schema access separately.

An Avro object container file is a storage wrapper containing a header and record blocks. The header's required avro.schema metadata contains the writer schema as JSON—not as a special binary schema representation. All records in that file use its writer schema.

Object container file
+--------------------------------------------------+
| Header: magic bytes, writer schema JSON, metadata |
|         codec metadata, synchronization marker   |
+--------------------------------------------------+
| Block: record count, encoded byte size, records   |
|        synchronization marker                    |
+--------------------------------------------------+
| More record blocks ...                           |
+--------------------------------------------------+

Compression, when configured, applies to the record data within blocks. Synchronization markers support splitting files for parallel processing.

Where Schema Registries Fit

In messaging systems, a schema registry stores schemas separately from messages. A registry-aware serializer associates each payload with a schema identifier; the consumer uses that identifier to obtain the writer schema.

For example, Confluent documents a serializer wire format that can carry schema identification in a payload prefix or message headers. Those conventions belong to the integration, not to raw Avro datum encoding. A registry service is not required by the Avro specification.

Avro Schema Example

Save this compact record schema as user.avsc:

{
  "type": "record",
  "namespace": "com.example.events",
  "name": "User",
  "fields": [
    {"name": "username", "type": "string"},
    {"name": "age", "type": ["null", "int"], "default": null}
  ]
}

The name and namespace give this named type the full name com.example.events.User. A namespace contributes to type identity; it isn't merely documentation or a directory hint.

Types, Unions, and Defaults

The official Python getting-started guide introduces the schema notation and its primitive and complex types:

• Primitive types: null, boolean, int, long, float, double, bytes, and string.
• Named types: records group fields, enums define named symbols, and fixed types hold a declared number of bytes.
• Collections: arrays declare an item schema; maps have string keys and declare a value schema.
• Unions: a JSON array of schemas allows one of several types. Here, age accepts either null or an Avro integer.

Nullability and defaults solve different problems. The union permits a null value; "default": null supplies a value when a reader encounters older writer data whose schema lacks age. A nullable field does not inherently require a default.

Avro Schema Evolution

Compatibility has a direction. A new consumer reading old events and an old consumer reading new events are different requirements.

Confluent Schema Registry's compatibility modes distinguish checks against the latest registered version from transitive checks against all previous versions. Use the history your consumers may actually replay when choosing a policy.

Compatible and Incompatible Changes

• Add a field with a valid default: a new reader can fill it when reading older data. An old reader ignores the extra field in newer writer data.
• Add a field without a default: reading older data with the new schema fails because the reader cannot supply the missing value.
• Remove a field: the new reader can ignore it in old data. An old reader needs a default for that field to read newer data that omits it.
• Change int to long: an integer writer can resolve to a long reader, but the reverse direction is not supported.
• Change int to string: this is not a supported promotion. A default does not repair an incompatible field that exists in both schemas.

Schema compatibility also isn't business compatibility. Changing a field's meaning from cents to dollars can pass structural checks while breaking every consumer. Keep semantic changes in your migration review.

Runnable Python Serialization and Compatibility Test

This example uses Apache's avro 1.12.2 package, which requires Python 3.10 or newer. In a POSIX shell, create an environment and install it:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install avro==1.12.2

Keep user.avsc in the same directory and save this as avro_example.py. The original writer schema has only username; the revised reader schema adds the defaulted age field.

import io
import json
from pathlib import Path

import avro.schema
from avro.compatibility import (
    ReaderWriterCompatibilityChecker,
    SchemaCompatibilityType,
)
from avro.io import BinaryDecoder, BinaryEncoder, DatumReader, DatumWriter

writer_schema = avro.schema.parse(json.dumps({
    "type": "record",
    "namespace": "com.example.events",
    "name": "User",
    "fields": [{"name": "username", "type": "string"}],
}))
reader_schema = avro.schema.parse(
    Path("user.avsc").read_text(encoding="utf-8")
)

# Check the new reader against the old writer.
result = ReaderWriterCompatibilityChecker().get_compatibility(
    reader_schema, writer_schema
)
assert result.compatibility is SchemaCompatibilityType.compatible

# Serialize raw datum bytes: no container header or embedded schema.
with io.BytesIO() as buffer:
    DatumWriter(writer_schema).write(
        {"username": "ada"}, BinaryEncoder(buffer)
    )
    payload = buffer.getvalue()

# Decode with the actual writer schema and the revised reader schema.
with io.BytesIO(payload) as buffer:
    decoded = DatumReader(writer_schema, reader_schema).read(
        BinaryDecoder(buffer)
    )

assert decoded == {"username": "ada", "age": None}
print(decoded)

python avro_example.py

{'username': 'ada', 'age': None}

The assertions check backward compatibility and replay one old record through the new reader. As a negative test, remove "default": null from age in user.avsc: the compatibility assertion fails. The package's compatibility checker implementation documents the reader-first argument order.

This is a demonstration, so parsing and decoding errors propagate. In a consumer, route invalid records to an explicit failure path, preserve their schema identifiers, and avoid silently acknowledging data you couldn't decode.

Avro vs JSON

JSON is a text interchange format; an Avro schema is a contract used to interpret encoded values. JSON documents don't contain an implicit JSON Schema: JSON Schema is a separate specification you can adopt for validation.

Binary encoding is not compression. An Avro container with no codec metadata uses the uncompressed null codec. Container implementations must support null and deflate; optional codecs include Snappy and Zstandard, subject to implementation and dependency support.

JSON can also be compressed. Compare equivalent records, batch sizes, and compression settings; measure total bytes, serialization CPU, and decoding latency. Compressed JSON can be competitive, and neither format guarantees the best end-to-end size or speed.

Choose Avro when explicit contracts and schema-managed record exchange justify the tooling. Choose JSON when easy inspection and integration are more valuable than reducing per-record structural overhead.

Avro vs Parquet

The central distinction is record-oriented exchange versus column-oriented analytical access. Avro object container files store records in blocks. Parquet divides rows into row groups, with a separate column chunk for each column inside a row group.

The Parquet file format lets readers locate the column chunks they need. A query reading three columns from a wide table can therefore avoid fetching the other column chunks; an Avro reader still traverses the record-oriented encoding to reach the fields it needs.

Don't reduce this to “Avro writes fast; Parquet reads fast.” Batching, codecs, file sizes, query shape, and the surrounding system determine the result. A reasonable architecture can use Avro for event transport and Parquet for analytical storage.

For more storage context, see Avro vs Parquet in data pipelines and the Apache Parquet introduction.

Production Checklist

• Preserve writer schemas for as long as their data may be replayed; include registry metadata in backup and recovery planning.
• Run compatibility checks and replay fixtures from historical writer schemas in CI. Test both directions when old consumers remain deployed.
• Test cross-language null handling, numeric boundaries, and codec availability with your actual libraries.
• Define how consumers handle unavailable schemas, malformed payloads, and decoding failures.
• Use patched implementations and bound input sizes. Apache's 1.12.2 release notes describe Java and Python hardening against oversized allocations and decompression-related attacks.

FAQ

Does an Avro file contain its schema?

An Avro object container file embeds its writer schema as JSON in the header. Raw Avro datum bytes do not embed a schema, so messages need an external schema agreement or a schema-identification mechanism.

How does Avro schema evolution work?

Avro resolves the writer schema against the reader schema. A reader can ignore extra writer fields or supply defaults for fields absent from the writer schema, but incompatible changes can fail.

Is Avro compressed by default?

Avro binary encoding is not compression. An object container file without codec metadata uses the uncompressed null codec; compression must be selected through the implementation's configuration.

What is the difference between Avro and Parquet?

Avro container files store records in blocks for record-oriented processing. Parquet groups data into column chunks within row groups, making it a better starting point for analytical queries that scan selected columns.

How do you define a nullable field in an Avro schema?

Use a union containing null, such as ["null", "int"]. Add a valid default when a reader must supply that field for older data; nullability alone does not provide a default.

Further Reading

• Apache Avro specification — schemas, encodings, container files, and resolution rules.
• Apache Avro Python guide — schema notation and serialization basics.
• Apache Avro Python compatibility checker — implementation-level compatibility behavior.
• Confluent schema evolution and compatibility — backward, forward, full, and transitive policies.
• Confluent serializers and wire formats — schema identification in messages.
• What is JSON Schema? — explicit validation contracts for JSON.
• Apache Parquet file format — column chunks, row groups, and metadata.
• Apache Avro 1.12.2 release notes — security fixes and implementation changes.