From Code to Production: A Quality Framework for Multi-Service Platforms_

Every service dashboard is green. Every pipeline passed. And checkout is broken.

If you have worked on a platform made of independently deployed services, you have probably lived some version of that sentence. Each team tested its own service, and each service does exactly what its tests say it does. The failure lives somewhere none of those tests look: between services, between versions, or between the moment a change is deployed and the moment it meets real traffic.

This article is the hub of a series about closing that gap. Its thesis is simple:

Quality is not a testing phase at the end of delivery. It is a system of continuous verification that runs from design to production, and it is owned by the engineering organization as a whole, not by QA alone.

The article explains the whole framework at a level you can act on. Each topic then has its own deep-dive article, linked where the topic appears.

Who this is for. Engineers and tech leads who design or improve testing and CI/CD for a backend platform made of multiple services.

Out of scope. Frontend and mobile testing, and evaluation of ML models. Many principles carry over, but the specifics differ.

Series status. The deep-dive articles referenced below are still being written. Their titles appear in italics, without links, until they are published.

The running example

Every article in this series uses the same small platform, so each practice can be shown against a concrete failure.

• Orders API accepts orders from clients and calls Payments synchronously.
• Payments is a gRPC service. It publishes PaymentFailed when a charge is declined.
• Inventory consumes OrderPlaced events and reserves stock.
• Notifications consumes OrderPlaced and PaymentFailed and emails the customer.
• Each service owns its own Postgres database. Event schemas are managed in a schema registry.

Four services, one synchronous boundary, two event types and three databases. That is small enough to hold in your head and large enough to fail in every interesting way.

Terms used in this series

These definitions are fixed across every article in the series.

1. Why multi-service systems break classic testing

Start with three failures in the example platform. In each case, every service passes its own test suite.

Failure 1: a replaced field

The Payments team replaces the integer amount_cents field (field number 3) in its gRPC response with a new decimal amount field (field number 7), deletes field 3 and updates its own tests. Payments is green. Orders still reads field 3, which is no longer sent, so protobuf returns the default value of zero and Orders records every order as free. Orders is also green, because its tests use a mock of Payments that still returns the old field.

The mock described what Orders believed Payments does. Nothing verified what Payments had agreed to do.

Failure 2: the wrong deployment order

Orders changes OrderPlaced to support orders with several products: it removes the single sku and quantity fields and adds an items list. The Orders team deploys first. Inventory is still running the previous version, which requires sku and cannot read the new event, so stock stops being reserved. In staging, both services had been deployed together from main, so the combination that actually ran in production was never tested.

The problem is version skew. In a platform with independent deployments, the system under test is not “the latest version of everything”. It is whatever combination of versions exists in the environment at that moment.

Failure 3: a duplicate event

The broker delivers messages at least once. During a broker failover, one OrderPlaced event is delivered twice, and Inventory reserves stock twice. Every Inventory test sends each event exactly once, so all of them pass.

What the three failures share

None of these defects exists inside a single service. They exist at boundaries: between a consumer and a provider, between two versions, and between a producer and an unreliable delivery channel. A strategy built only from unit tests inside services plus a handful of E2E tests across the whole platform has almost nothing aimed at those boundaries.

Which testing model fits

The test pyramid recommends many fast unit tests, fewer integration tests and very few UI or E2E tests. Mike Cohn introduced it in Succeeding with Agile (2009), and it remains the most common reference point.¹² Its core insight still holds: prefer fast, focused tests, and push tests down to the lowest level that can catch the defect.

The testing trophy puts most of the weight on integration tests and adds static analysis as its base. Kent C. Dodds proposed it from his experience writing JavaScript applications that run in the browser, and he has said he never considered whether it applied to microservices or backends.³ It is a useful model, but it was not designed for this problem.

The testing honeycomb was proposed by Spotify for microservices. It argues that in a small service, most of the complexity is in how the service interacts with others, so the bulk of testing should be integration tests that focus on those interaction points. Tests of implementation details should be few, and tests that pass or fail based on the correctness of another system should be fewer still, ideally none.⁴

For a multi-service platform, the honeycomb’s diagnosis is the right one. Boundaries need proportionally more testing than the classic pyramid suggests. That does not make unit tests less valuable for complex logic. It means the budget you would otherwise spend on large, fragile, cross-platform E2E suites is usually better spent on contract tests, service-level integration tests and compatibility checks.

2. The framework at a glance

The framework has one spine and three groups of practices.

The spine is the delivery path. A change moves through stages, and each transition is a quality gate that decides whether the change may continue.

The practices do not map one-to-one onto stages. Most kinds of verification run at several stages, at different depths. This matrix shows where each concern runs.

The practices fall into three groups, covered in the next three sections:

1. Testing the system. What to verify and at which layer.
2. Delivering safely. How changes reach users without betting the platform on each deploy.
3. Sustaining it. How the test system itself stays fast, trusted and relevant.

3. Testing the system

Risk and test-impact analysis

Testing starts before code exists. For every meaningful change, ask a short, fixed set of questions: which behavior changes, which services and contracts are affected, whether data or schemas change, which version combinations must keep working, how the change is rolled back or rolled forward, which new failure modes appear, and how you will observe the change in production. The answers decide which tests the change needs. In Failure 2, the question “which versions of Inventory will read this event?” would have exposed the problem before a line of code was written.

This is also where you design for testability: seams for replacing dependencies, and an injectable clock and source of randomness, so behavior can be controlled in tests.

Deep dive: Risk and Test-Impact Analysis

Testing a single service

A service should be verifiable without booting the whole platform. Unit tests cover logic. Component tests exercise the service through its real API, with its own database running in a container and its external dependencies replaced by stubs, fakes or emulators. Property-based tests are well suited to pricing, validation and state transitions, where hand-picked examples miss edge cases.

The risk in this layer is mocks. A mock encodes an assumption about another service, and assumptions go stale silently, as Failure 1 shows. Toby Clemson’s catalog of microservice testing strategies is still a good map of these choices.⁵

Deep dive: Testing a Single Service: Mocks, Fakes, and Real Dependencies

Contracts and version compatibility

Contract tests replace assumptions with agreements. In consumer-driven contract testing, the consumer’s tests generate a contract and publish it to a broker. The provider verifies that contract in its own pipeline and publishes the verification result back to the broker. Before any deploy, a gate asks the broker whether the version being deployed has been verified against the versions currently deployed in the target environment.⁶

That final check is what makes independent deployment safe, because it tests against real version combinations instead of against main. Consumer-driven contracts fit internal APIs with a known set of consumers. Public APIs and events with many consumers are usually better served by provider-owned schemas plus automated breaking-change detection, such as oasdiff for OpenAPI or buf breaking for Protobuf.⁷⁸

Deep dive: Contract Testing and Version Compatibility

Async messaging and schema evolution

Event-driven boundaries fail differently from synchronous ones. Messages can be duplicated, reordered, delayed or sent to a dead-letter queue, and state becomes eventually consistent. Consumers must be idempotent, which is the fix for Failure 3. Producers need a reliable way to update their database and publish an event together, which is what the transactional outbox pattern provides.⁹ Tests for eventual consistency should poll for the expected state with a timeout, never sleep for a fixed time.

Schema evolution is governed by compatibility modes. In Confluent Schema Registry, the default mode is BACKWARD. Backward-compatible changes require consumers to be upgraded before producers start writing the new schema. Forward-compatible changes require producers to be upgraded first. Full compatibility allows either order.¹⁰¹¹ Failure 2 was a forward-incompatible change deployed producer-first. A registry check at PR time would have blocked it only if the topic used FORWARD or FULL compatibility. Under the default BACKWARD mode, deleting a field is permitted, so the change would have passed the check and still broken Inventory.

Deep dive: Testing Async Messaging and Schema Evolution

Test data and environments

Unreliable integration suites are often a data problem rather than a code problem. Tests need deterministic seeding, unique data per test so parallel runs do not collide, and clear rules about production-derived data and personal information.

Environments need the same discipline. Ephemeral environments created per PR give isolated, disposable integration testing. A long-lived staging environment gives weak assurance if it drifts from production in configuration, infrastructure versions, feature-flag state or network topology.

Deep dive: Test Data and Environment Parity

Integration, E2E, and exploratory testing

Integration testing grows in scope as a change moves along the path: targeted on the PR, broad on main, and full in pre-production. E2E tests are expensive and, as Google’s data shows, larger tests are more likely to be flaky.¹² Keep them to the critical business journeys, such as “place an order and receive confirmation”, and give each journey one accountable owning team. Shared ownership of E2E suites tends to mean no ownership.

Automated tests only check what someone thought to check. Exploratory testing by people who understand the product finds the rest.

Deep dive: Integration, E2E, and Exploratory Testing

Non-functional testing

Correctness is not the only property that matters.

• Performance: latency regressions on main, plus load, stress, spike and soak tests before release.
• Security: dependency and secret scanning on every PR, container and infrastructure-as-code scanning, dynamic testing on a schedule, and periodic penetration tests. Scanning must also continue after release, because new vulnerabilities are disclosed against artifacts that already passed their PR scan.
• Resilience: what happens when Payments is slow, the broker is unavailable or a node dies. Chaos engineering extends this into production by running controlled experiments with a limited blast radius.¹³

Deep dive: Non-Functional Testing: Performance, Security, and Resilience

4. Delivering safely

CI/CD quality gates

A pipeline that produces reports without making decisions is not a quality system. Every stage needs an explicit gate: which checks block, which only alert, and who acts when a check fails.

On a pull request, the build runs first and produces the artifact. The checks that depend on it then run in parallel, while checks that only need source code can start immediately.

The full stage-by-stage gate policy is in When tests run and what blocks delivery below.

Deep dive: Designing CI/CD Quality Gates

Deploy vs release

Build the artifact once and promote the same immutable artifact through every environment, so what you tested is exactly what you run.¹⁴ Then separate deploying from releasing:

• Blue/green deployment switches traffic between two complete environments, which makes cutover and switch-back fast.¹⁵
• Canary deployment sends a small, increasing share of real production traffic to the new version and compares its health with the current version before continuing.¹⁶
• Feature flags deploy code with the new behavior switched off, and release it separately, gradually or per audience.¹⁷ Flags add their own risks: untested flag combinations and stale flags that nobody removes.

Deep dive: Deploy vs Release: Canary, Blue/Green, and Feature Flags

Migrations, rollback, and roll-forward

Automatic rollback is often presented as a safety net. It frequently is not one. After a destructive schema migration, the previous version may no longer be able to read the database. After events are published, other services have already acted on them, and those events cannot be recalled.

The standard technique is parallel change, also called expand and contract. It splits a backward-incompatible change into phases that are each safe on their own: expand, migrate, contract.¹⁸¹⁹ Applied to renaming a column in the Orders database:

Rolling back to v1 stays safe until step 6. After the contract step, roll-forward is the only recovery path, so the contract step should wait until v1 is gone everywhere and the rollback window has closed.

Deep dive: Database Migrations, Rollback, and Roll-Forward

Production verification

Production is where verification finishes, not where it stops. Canary analysis compares error rates, latency and business signals, such as orders completed per minute, between the new version and the current one. Synthetic checks exercise critical journeys continuously. Service level objectives define what “healthy” means.²⁰

The observability setup needs testing too. An alert that never fires and a trace that loses its context at the broker both fail silently. Every production incident should end with a new protection: a test, an alert or a design change.

Deep dive: Production Verification and Testing Your Observability

5. Sustaining it

Evolving the test suite

A test suite is not append-only. Tests are added, modified, replaced and deleted as the product changes.

• New feature: add tests according to the risk the feature introduces. New logic needs unit or component tests. A new interaction needs a contract. A new infrastructure dependency needs an integration test. A new critical journey needs an E2E test. A feature does not automatically need every kind of test.
• Bug fix: reproduce the bug with an automated test that fails, fix the bug, confirm the test passes, and keep the test. Put the test at the lowest layer that catches the bug reliably.
• Architecture change: identify which assumptions no longer hold, and delete tests that verify behavior that no longer exists. Obsolete tests cost run time and erode trust.

Flaky tests deserve particular attention. Google reported that about 1.5% of its test runs produced flaky results and that almost 16% of its tests showed some flakiness.²¹ Treat a flaky test as a defect: measure the flake rate, quarantine the test with a named owner and a deadline, and never let an automatic retry turn a failure green without leaving a visible record.

Deep dive: Evolving the Test Suite: Features, Bugs, Architecture Changes, and Flaky Tests

Measuring the quality system

The test system is a product, and it needs its own metrics. DORA’s research provides the delivery measures. DORA replaced “mean time to recover” with failed deployment recovery time in 2023, so that failures caused by changes are measured separately from external outages. In 2024 it added deployment rework rate as a fifth metric.²²²³ Alongside those, track time to developer feedback, flake rate, escaped defects and how long tests stay in quarantine.

Code coverage is useful for finding untested areas. It is a poor target, because a suite can reach high coverage while asserting nothing important.

Deep dive: Measuring the Quality System

6. When tests run and what blocks delivery

This is the reference policy for the whole series. Other articles link here instead of repeating it.

Two rules make the table work. First, a check either blocks progress or alerts a named owner. A check that does neither is noise. Second, the slowest blocking stage sets the pace of delivery, so move checks into nightly or pre-deploy runs only when they are too slow for the PR, not because they are inconvenient.

7. Principles

1. Test behavior, not implementation. Tests coupled to internals break during refactoring without catching real defects. (Testing a Single Service)
2. Catch each defect at the lowest layer that detects it reliably. Earlier feedback is generally easier to act on, because the change is still fresh in the author’s mind. (Evolving the Test Suite)
3. Test boundaries explicitly. Most multi-service failures happen between services, not inside them. (Contract Testing)
4. Test against the versions that actually run. Compatibility with main does not guarantee compatibility with production. (Contract Testing)
5. Treat async behavior as eventually consistent and at-least-once. Poll with timeouts, and make consumers idempotent. (Async Messaging)
6. Keep E2E tests few, critical and owned. (Integration, E2E, and Exploratory Testing)
7. Treat test data and environments as infrastructure. (Test Data and Environment Parity)
8. Build once and promote the same artifact. (Deploy vs Release)
9. Separate deploying from releasing. (Deploy vs Release)
10. Assume rollback may be impossible, and design migrations for it. (Migrations, Rollback, and Roll-Forward)
11. Verify in production under controlled exposure. (Production Verification)
12. Delete obsolete tests and treat flaky tests as defects. (Evolving the Test Suite)
13. Measure the test system itself, and do not make coverage the target. (Measuring the Quality System)

A final note on a claim you will see in almost every article about testing: that a bug costs 100 times more to fix in production than in design. The study usually cited for that figure has never been found, and its supposed source appears to have been an internal IBM training program.²⁴ The practical argument for early testing does not depend on that number. Feedback that arrives while a change is small and fresh is easier to act on, and a defect that never reaches users never harms them.

References