Testing Async Messaging and Schema Evolution_

This article is part of the series From Code to Production: A Quality Framework for Multi-Service Platforms. The previous deep dive, Contract Testing and Version Compatibility, covered synchronous calls between services. This one covers the other kind of boundary: events.

In the series’ running example, the Orders API publishes OrderPlaced when a customer places an order. Payments publishes PaymentFailed when a charge is declined. Inventory consumes OrderPlaced to reserve stock, and Notifications consumes both events to email the customer. Event schemas are stored in a schema registry.

The examples use Apache Kafka and Confluent Schema Registry, because they are common and well documented. The failure modes and testing techniques apply to other brokers too, although the exact guarantees differ.

Series status. The other deep-dive articles referenced below are still being written. Their titles appear in italics, without links, until they are published.

Why events fail differently

A synchronous call fails loudly. The caller gets an error or a timeout, and it happens now. An event can fail silently, later, and somewhere else. The producer published it successfully, and the damage appears minutes afterwards in a service owned by another team.

Four failures in the example platform show the range.

A duplicate. The broker redelivers an OrderPlaced event after a consumer crash, and Inventory reserves the stock twice. This is Failure 3 from the hub article.

A lost event. Orders saves the order to its database, then crashes before publishing OrderPlaced. The order exists, but Inventory never hears about it.

The wrong order. A customer places an order and cancels it a second later. Inventory receives OrderCancelled before OrderPlaced, ignores the cancellation for an order it has never seen, and then reserves stock for a cancelled order.

An incompatible schema. Orders removes the sku field from OrderPlaced and replaces it with an items list. Orders deploys first. Inventory, still on the old schema, cannot read the new events. This is Failure 2 from the hub article.

None of these shows up in a test that publishes one well-formed event, once, to a consumer that is already running the right version. Testing event-driven systems means deliberately creating the conditions that a happy-path test avoids.

Part 1: What the broker actually guarantees

Tests should be written against the guarantees the system really has, not the ones people assume.

Delivery is at least once

Kafka producers can be configured so that retries do not create duplicate entries in the log. The broker assigns each producer an ID and deduplicates writes using sequence numbers.¹ Transactions extend this to atomic writes across partitions.²

These guarantees cover writes inside Kafka. They do not make a consumer’s side effects happen exactly once. A consumer that reads an event, updates its database, and crashes before committing its offset will read the same event again after restarting. Confluent’s documentation describes exactly this sequence as at-least-once processing.¹ Any consumer that writes to a database, calls an API or sends an email must assume it will sometimes see the same event more than once.

Ordering is per partition

Kafka preserves order within a partition, not across a topic.³ Events that must be processed in order need the same partition key. In the example, every order event uses order_id as its key, so OrderPlaced and OrderCancelled for the same order always land on the same partition, in the order they were written.

That fixes ordering within one topic. It does not help when related events travel on different topics. OrderPlaced from Orders and PaymentFailed from Payments can reach Notifications in either order, whatever keys they use. Consumers of more than one topic have to handle events arriving in an unexpected order.

What this means for tests

Part 2: Designing consumers and producers you can test

Most of these problems are solved in the design, and the tests then prove the design holds. Two patterns do most of the work.

Idempotent consumers

An idempotent consumer produces the same outcome whether it processes an event once or many times. Chris Richardson’s pattern description gives the canonical approach: record the ID of each processed message in the database, in the same transaction as the business change, and skip messages whose ID is already recorded.⁴

In Inventory’s database:

CREATE TABLE processed_events (
  consumer_name TEXT NOT NULL,
  event_id      UUID NOT NULL,
  processed_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer_name, event_id)
);

This requires every event to carry a stable, unique event_id set by the producer. An ID generated by the consumer on receipt defeats the purpose, because a redelivered event would get a new one.

Some consumers are naturally idempotent. Setting an order’s status to cancelled gives the same result no matter how many times it runs. Reserving one more unit of stock does not, so Inventory needs the explicit check.

The transactional outbox

The lost event comes from a dual write: Orders writes to its database and to the broker, and nothing makes the two succeed or fail together.

The transactional outbox pattern removes the dual write. The service inserts the event into an outbox table in the same database transaction as the business change. A separate relay process reads the outbox and publishes to the broker.⁵ The relay can poll the table, or read the database’s change log with a tool such as Debezium’s outbox event router.⁶

The relay itself delivers at least once. If it publishes an event and crashes before marking the row as sent, it publishes the event again on restart. The outbox therefore guarantees that no event is lost. Idempotent consumers guarantee that the duplicates it may create are harmless. The two patterns depend on each other.

Part 3: Testing the failure modes

Test the handler with hostile delivery

Most coverage belongs at the component level. Run the consumer’s real handler against its real database, usually in a container, and call it directly with the delivery patterns the broker can produce. No broker is needed for these tests, so they are fast.

describe("OrderPlaced handler", () => {
  it("reserves stock once when the same event is delivered twice", async () => {
    const event = orderPlaced({ eventId: "e-1", orderId: "42", sku: "MUG", quantity: 1 });
 
    await handleOrderPlaced(event);
    await handleOrderPlaced(event);
 
    expect(await reservedUnits("MUG")).toBe(1);
  });
 
  it("does not record the event when the reservation fails", async () => {
    const event = orderPlaced({ eventId: "e-2", orderId: "43", sku: "MUG", quantity: 1 });
    failNextStockUpdate();
 
    await expect(handleOrderPlaced(event)).rejects.toThrow();
    await handleOrderPlaced(event); // redelivery after the failure
 
    expect(await reservedUnits("MUG")).toBe(1);
  });
 
  it("ignores OrderPlaced for an order that was already cancelled", async () => {
    await handleOrderCancelled(orderCancelled({ eventId: "e-3", orderId: "44" }));
    await handleOrderPlaced(orderPlaced({ eventId: "e-4", orderId: "44", sku: "MUG", quantity: 1 }));
 
    expect(await reservedUnits("MUG")).toBe(0);
  });
});

The second test matters as much as the first. A common bug records the event as processed in one transaction and does the work in another. If the work fails, the redelivered event is skipped as a duplicate, and the reservation is lost for good.

The third test covers events arriving in the wrong order. Handling it requires the consumer to remember cancellations for orders it has not seen yet, or to compare version numbers carried in the events. Which approach you choose is a design decision. The test makes sure the decision is actually implemented.

Assert eventual consistency by polling, never by sleeping

Once a test crosses the broker, the result is not immediate. The tempting fix is to wait:

await publish(orderPlaced(/* ... */));
await sleep(5000); // do not do this
expect(await reservedUnits("MUG")).toBe(1);

A fixed sleep fails in both directions. On a slow CI runner, five seconds is not enough and the test fails intermittently. On a fast machine, the test wastes most of those five seconds, and hundreds of such tests add minutes to every pipeline. Poll for the expected state instead, with a timeout that fails the test clearly:

async function eventually<T>(
  check: () => Promise<T>,
  { timeoutMs = 10_000, intervalMs = 100 } = {},
): Promise<T> {
  const deadline = Date.now() + timeoutMs;
  let lastError: unknown;
  while (Date.now() < deadline) {
    try {
      return await check();
    } catch (error) {
      lastError = error;
      await new Promise((resolve) => setTimeout(resolve, intervalMs));
    }
  }
  throw lastError;
}
 
await publish(orderPlaced({ eventId: "e-5", orderId: "45", sku: "MUG", quantity: 1 }));
 
await eventually(async () => {
  expect(await reservedUnits("MUG")).toBe(1);
});

The test finishes as soon as the condition holds, and it fails with the last assertion error rather than a vague timeout. On the JVM, Awaitility provides the same behavior as a library.⁷

Polling also has a trap for negative assertions. “No stock is reserved” is true immediately, before the consumer has even received the event, so polling for it proves nothing. Publish a second, later event on the same partition as a marker, wait until the consumer has processed the marker, and then check that the first event had no effect.

Integration tests with a real broker

Component tests cannot show how the service behaves with a real broker: serialization through the schema registry, offset commits, consumer group rebalancing and restarts. Those need a real Kafka instance, which Testcontainers can start for each test run.⁸

Worth testing at this level:

• Serialization end to end. The producer serializes with the registry, and the consumer deserializes the same bytes into the expected object.
• Restart mid-stream. Stop the consumer after it processes some events but before it commits offsets, restart it, and check the final state is still correct.
• Rebalancing. Start a second consumer instance in the same group while events are flowing, and check that no event is lost and duplicates are handled.
• Outbox atomicity. Roll back an Orders transaction and check that neither the order nor the outbox row exists. Stop the relay after it publishes but before it marks the row as sent, and check that the event is published again.

Keep these tests few and focused. They are slower, and they exercise infrastructure behavior, which does not change with every feature.

Retries and dead-letter queues

Some events can never be processed: a malformed payload, a reference to a product that does not exist, or a bug in the consumer. If the consumer retries such an event forever, it blocks every event behind it on that partition. The standard answer is a dead-letter queue, where messages that cannot be delivered are moved for later inspection. The Enterprise Integration Patterns catalog calls this a Dead Letter Channel.⁹

Tests should prove three things:

1. Transient failures are retried and eventually succeed. Make the database unavailable for the first two attempts, then check that the event is processed.
2. Permanent failures go to the dead-letter topic quickly with enough context to diagnose them: the original payload, headers, error message and the number of attempts. They should not be retried until retries run out.
3. Dead-lettered events can be replayed. After fixing the cause, republishing an event from the dead-letter topic must produce the correct result. That depends on the consumer being idempotent, because some of these events may have been partially processed.

A dead-letter topic that nobody watches is a place where data disappears quietly. Alert on its size, and give it an owner.

Replaying history

Kafka topics retain events, and teams replay them to rebuild a consumer’s state, backfill a new service or recover from a bug. A replay sends a consumer running today’s code events that were written with every schema version the topic has ever had. That affects the choice of compatibility mode, covered in the next part.

Part 4: Schema evolution

The compatibility modes

A schema registry checks each new version of a schema against earlier versions before accepting it. The check depends on the subject’s compatibility mode. Confluent Schema Registry supports these modes, and its default is BACKWARD.¹⁰

The two directions are easy to confuse, so it helps to state them in terms of readers and writers:

• Backward compatibility means a consumer on the new schema can read events written with the old one.
• Forward compatibility means a consumer on the old schema can read events written with the new one.

Whether a field can be deleted safely also depends on how it was defined. In Avro, a field is optional only if it has a default value, commonly a union with null and a default of null.¹⁰

The default mode assumes a deploy order

BACKWARD protects new consumers reading old events. It makes no promise about old consumers reading new events. Its safety depends on consumers upgrading before producers.

That is the opposite of how event-driven platforms usually work. Orders owns OrderPlaced and deploys when it is ready. Inventory, Notifications and any future consumer are owned by other teams and upgrade on their own schedules. Removing the sku field is a deletion, which BACKWARD allows, so the registry accepts it. Orders deploys first, and Inventory breaks. The schema check passed, and the platform still failed, because the compatibility mode did not match the real deploy order.

Two recommendations follow for most event topics:

1. Choose the mode from the deploy order, not from the default. For a topic owned by one producer and read by independent consumers, use FORWARD_TRANSITIVE or FULL_TRANSITIVE.
2. Prefer the transitive variants when topics are retained and replayed. A non-transitive mode checks only against the latest version. A consumer replaying a year of history reads events written with every version, so compatibility has to hold across all of them.

FULL_TRANSITIVE is the most restrictive and the easiest to reason about: fields can only be added or removed if they are optional, and producers and consumers can deploy in any order.

Check compatibility on the pull request

The registry rejects an incompatible schema when a producer tries to register it. By then the change is already merged and deployed, which is too late. Run the same check in CI, against the registry the production producers use, before merging.

Confluent Schema Registry exposes a compatibility endpoint for this:¹¹

curl --fail-with-body -s \
  -X POST \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data "$(jq -n --rawfile schema schemas/order-placed.avsc '{schema: $schema}')" \
  "$SCHEMA_REGISTRY_URL/compatibility/subjects/orders.order-placed-value/versions/latest?verbose=true"

The response includes is_compatible and, with verbose=true, the reasons a schema is incompatible. Fail the build when it is false. The check uses the compatibility mode configured on the subject, so the mode itself belongs in version control and code review, not only in the registry’s settings.

Also turn off automatic schema registration in production producers, so that only CI can register schemas. Confluent’s best-practices guide recommends this, because otherwise any producer can register a new version at runtime, and the pull request check is bypassed.¹²

Changes a schema check cannot see

A compatibility check sees structure. It does not see meaning. If Orders keeps a field called total but changes its unit from cents to a decimal currency amount, the schema is identical and fully compatible, and every consumer now reads the wrong value.

Two practices cover this gap:

• Message contracts. Consumer-driven contract testing works for events as well as for requests. In a message pact, the consumer describes the event it expects, including example values, and the producer verifies that its real code produces a matching event.¹³ This catches semantic changes that the registry cannot, and it records which consumers depend on which fields. The workflow and deploy gate are the same as in Contract Testing and Version Compatibility.
• Treat a change in meaning as a new field. Add total_amount with the new meaning, keep total unchanged until no consumer reads it, and only then remove it.

When a change has to break

Some changes cannot be made compatible, such as changing the partition key or restructuring an event completely. For those, apply the same parallel change pattern used for APIs and databases, with a new topic:¹⁴

Publish both versions from the same outbox transaction, so the two topics cannot diverge. Step 3 must be checked, not assumed: the broker knows which consumer groups are still reading v1.

Where each check runs

This table narrows the hub’s stage-by-stage gate policy to event-driven boundaries.

Consumer lag is the production signal that matters most here. A consumer that cannot read new events usually does not crash. It falls behind, retries, or sends everything to the dead-letter topic. Lag and dead-letter volume show the problem long before customers do.

Costs and trade-offs

• Idempotency adds storage and a write per event. The processed_events table grows forever unless it is pruned. Keep IDs for longer than the longest possible redelivery window, including replays, and prune after that.
• The outbox adds a moving part. The relay has to be deployed, monitored and scaled. A relay that falls behind delays every event, so its lag needs an alert too.
• Strict compatibility modes slow some changes down. FULL_TRANSITIVE turns a one-step change into several deploys. That is the cost of letting teams deploy in any order, and it is usually cheaper than coordinating releases across teams.
• Integration tests with a real broker are slow and can be flaky. Most coverage should stay in handler tests. Use broker tests for behavior that only a real broker can show.
• Message contracts need both sides to participate, exactly as synchronous contracts do. Start with the event whose consumers have broken most often.

Tooling

Related articles

• From Code to Production, the hub for this series.
• Contract Testing and Version Compatibility: contracts, can-i-deploy and deploy order for synchronous APIs.
• Database Migrations, Rollback, and Roll-Forward: why published events cannot be rolled back, and how to recover instead.
• Test Data and Environment Parity: isolating event data between parallel test runs.

References