Designing CI/CD Quality Gates_

This article is part of the series From Code to Production: A Quality Framework for Multi-Service Platforms. The hub lists which checks run at each stage and what each stage blocks, in When tests run and what blocks delivery. This article is about building that policy: turning checks into decisions that a team can trust.

It uses the series’ running example: the Orders API, Payments, Inventory and Notifications services, each with its own repository and pipeline. The code examples use GitHub Actions and Argo Rollouts, because their behavior is well documented. The design applies to any CI system and deployment controller.

Series status. The other deep-dive articles referenced below are still being written. Their titles appear in italics, without links, until they are published.

A pipeline full of checks is not a quality system

Four things went wrong in the example platform over one quarter. Every one of them happened in a pipeline that had plenty of tests.

A green check that ran nothing. The Payments team refactored its workflow and added a condition to the contract verification job so it would only run on certain branches. The condition was wrong, so the job was skipped on every pull request. The job was listed as a required check, and the pull requests merged anyway. In GitHub Actions, a job skipped by a condition reports its status as success and does not block a pull request, even when it is a required check.¹ For six weeks, nothing verified Payments against its consumers.

Two green pull requests that broke main. One Orders pull request renamed a helper function. Another, opened the same day, added a new call to the old name. Each passed its checks against the version of main it was branched from. Together they did not compile. Martin Fowler calls this a semantic conflict: a change that merges cleanly as text but breaks behavior.²

A red check that nobody owned. The nightly load test for Inventory failed for three weeks. It posted to a shared chat channel that everyone had muted. The latency regression it had detected reached production during a sale.

A tested artifact that was not the deployed artifact. The Notifications pipeline rebuilt its container image in each deploy job. A dependency with a loose version range published a new release between the staging build and the production build. Production ran code that had never been tested.

None of these was a missing test. Each was a failure of the gate: the part of the pipeline that turns a check’s result into a decision, and makes sure that decision is correct, acted on and hard to get around by accident.

What a gate is

A quality gate has five parts. When any of them is missing, the gate stops working, usually without anyone noticing.

• Signal. The input: test results, scanner findings, verification results, production metrics.
• Policy. The rule. “All required jobs succeeded.” “No critical vulnerabilities without an approved exception.” “Canary error rate no more than 20% above the stable version.”
• Decision and action. What happens on failure. Every gate should do one of three things, and only one.
• Owner. A named team that responds when the gate fails. A channel is not an owner.
• Record. Which checks ran, on which commit or artifact, with what result. Without it, nobody can later answer why a change was allowed through.

The three kinds of action:

There is a fourth kind that pipelines often have and should not: the advisory check, which reports a result and expects someone to look. Advisory checks train teams to ignore red results. Either give a check a real action, or remove it.

Principles of a good gate

Fail closed. If a gate cannot reach a verdict because a job was skipped, a service was unreachable or a metric returned no data, the answer is “no”. The first failure in the opening section happened because the gate treated “did not run” as “passed”.

Be trusted. A gate that blocks for reasons unrelated to the change, such as flaky tests or unstable infrastructure, teaches developers to rerun it until it passes, and then to push for exceptions. Track how often a gate blocks and the cause turns out not to be the change. That false-block rate is the most important health measure a gate has.

Be fast where it blocks. The slowest blocking gate sets the pace of delivery. Martin Fowler’s description of continuous integration recommends keeping the commit build fast, citing the Extreme Programming guideline of a ten-minute build.³ Slow checks belong in later stages, where they alert or block less often, not in the pull request gate.

Explain the failure. “Gate failed” is not an explanation. A gate’s output should say which check failed, why, and where to look, directly in the pull request or deployment log.

Live in version control. Required checks, thresholds, exceptions and canary analysis templates are code. They should change through review, like the application they protect.

Make bypasses visible. Every organization needs an emergency path. It should be explicit, recorded and followed by the checks that were skipped. The section on overrides below covers this.

The shape of the pipeline

The gates sit at the transitions between stages. The stage-by-stage policy for the series is in the hub; this diagram shows where each gate in this article sits.

The dotted path matters. An alerting gate that is never allowed to block anything is an advisory check with extra steps. Unresolved alerts past their deadline should eventually stop releases.

The pull request gate

Build first, then fan out

Checks that need the build artifact should run after it and in parallel. Checks that only need source code, such as static analysis and secret scanning, can start immediately.

Require one gate job, not a list of checks

Branch protection usually lists required checks by name. That list goes stale: a job is renamed, split or given a condition, and the protection rule silently stops matching reality. It also has the two traps GitHub documents:

• A job skipped by a condition reports success and does not block merging, even when it is required.¹
• A workflow skipped by path filtering leaves its checks pending, so a pull request that requires them is blocked forever.⁴

A more robust pattern is to make one aggregating job the only required check, and have it fail unless every job it depends on succeeded. It must run even when a dependency failed, otherwise it is skipped too, and a skipped job reports success.

name: pull-request
 
on:
  pull_request:
  merge_group:
 
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./ci/build-and-push.sh
 
  static-analysis:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./ci/lint.sh
 
  unit:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./ci/unit-tests.sh
 
  contract-verification:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./ci/verify-pacts.sh
 
  gate:
    if: always()
    needs: [build, static-analysis, unit, contract-verification]
    runs-on: ubuntu-latest
    steps:
      - name: Fail unless every required job succeeded
        env:
          RESULTS: ${{ toJSON(needs) }}
        run: |
          echo "$RESULTS" | jq -r 'to_entries[] | "\(.key): \(.value.result)"'
          echo "$RESULTS" | jq -e 'all(.[]; .result == "success")' > /dev/null

Three details make this work:

1. if: always() keeps the gate job running when a dependency fails or is skipped.
2. The policy is “success only”. A skipped or cancelled dependency fails the gate. That is failing closed.
3. Adding a check means adding it to needs, in the same pull request, reviewed like any other change. The branch protection rule never changes.

Running fewer tests without trusting less

In a large repository, running every test on every pull request becomes too slow. Selecting only the tests affected by a change is a legitimate way to keep the gate fast. Google’s continuous testing system is the best-known example at scale.⁵

Selection must never become the only line of defense. Dependency graphs miss things: configuration files, shared fixtures, generated code. Two safeguards keep selection honest:

• When the selection logic does not recognize a changed file, run everything. An unknown change should cost CI time, not safety.
• Run the full suite on main, after merge. If it finds a failure the pull request gate missed, that is a bug in the selection rules, and it gets fixed like one.

The merge gate

A pull request gate tests a branch against the version of main it started from. By the time the pull request merges, main has moved. The second failure in the opening section lives in that gap.

A merge queue closes it. Instead of merging directly, a pull request joins a queue. The queue creates a temporary branch containing the latest main, the pull requests ahead in the queue, and this pull request, and runs the required checks on that combination. GitHub’s merge queue ensures a pull request’s changes pass all required checks when applied to the latest target branch and any pull requests already queued.⁶

On GitHub, workflows that report required checks must also trigger on the merge_group event. Otherwise the checks never run for the queue, the required status is never reported, and the merge fails.⁶ That is why the workflow above lists both pull_request and merge_group.

A merge queue costs CI capacity, because every pull request’s checks run at least twice. For a repository where several people merge each day, it is usually cheaper than the time spent repairing a broken main.

The promotion gate: build once, then prove it

The fourth failure happened because the artifact was rebuilt. Continuous Delivery by Jez Humble and David Farley makes building binaries only once a core rule of the deployment pipeline: the same artifact moves through every stage, so the thing you tested is the thing you release.⁷

Two practices make that rule enforceable rather than hopeful:

• Promote by digest, not by tag. A tag like orders:1.4.2 can be overwritten. A digest like orders@sha256:3f1c... identifies exactly one image. Every deploy job takes the digest produced by the build, never rebuilds and never resolves a tag.
• Attach provenance at build time and verify it before deploying. Build provenance records which source commit and which workflow produced an artifact. The SLSA framework defines what provenance should contain.⁸ A deploy gate that verifies provenance rejects any image that did not come from the trusted build.

In GitHub Actions, the deploy job for production reuses the digest from the build and checks it before deploying:

deploy-production:
  needs: [build, deploy-staging]
  environment: production
  runs-on: ubuntu-latest
  env:
    IMAGE: registry.example.com/orders@${{ needs.build.outputs.digest }}
  steps:
    - name: Verify the image came from this repository's build
      run: gh attestation verify "oci://$IMAGE" --owner example-org
      env:
        GH_TOKEN: ${{ github.token }}
 
    - name: Check compatibility with what is running in production
      run: |
        pact-broker can-i-deploy \
          --pacticipant orders-api --version "$GITHUB_SHA" \
          --to-environment production
 
    - name: Deploy
      run: ./ci/deploy.sh production "$IMAGE"
 
    - name: Record the deployment
      run: |
        pact-broker record-deployment \
          --pacticipant orders-api --version "$GITHUB_SHA" \
          --environment production

The can-i-deploy step is covered in depth in Contract Testing and Version Compatibility.

Scheduled checks: alerting gates that work

Load tests, soak tests, dynamic security scans and long end-to-end suites are too slow for the pull request gate. They run on a schedule, and a failure raises an alert instead of blocking. The third failure in the opening section shows how that goes wrong: an alert with no owner is the same as no check.

An alerting gate needs the same five parts as a blocking one, with the action spelled out:

• Route to a team, not a channel. The check’s configuration names the owning team, and the failure becomes an assigned issue.
• Deduplicate. A check that fails every night should update one issue, not open thirty.
• Set a deadline and enforce it. Past the deadline, the pre-deploy gate for that service blocks until the issue is closed or someone with the authority to accept the risk approves an exception. This is what separates an alerting gate from an advisory check.
• Compare against a baseline, not only a fixed number. A load test that fails when p95 latency exceeds 800 ms will stay green while latency climbs from 300 ms to 790 ms over two months. Also compare each run against the recent trend, and alert on a sustained regression.

The deploy gate

The last gate before production combines several independent policies. Each is fast, because it evaluates results that already exist rather than running tests.

Exceptions deserve care. An exception to the vulnerability policy should name the finding, the person who accepted the risk, the reason, and an expiry date. Exceptions without an expiry turn into permanent holes in the policy.

Policies like these are easier to review and test when they are written as code, with a policy engine such as Open Policy Agent, than when they are spread across shell scripts in deploy jobs.⁹

The canary gate

The canary gate is the only gate that makes its decision with real production traffic, and the only one that has to reverse a change rather than stop it. It must be automatic. A human watching a dashboard for thirty minutes is not a gate.

Argo Rollouts implements this with analysis templates. An analysis runs queries against a metrics provider such as Prometheus at an interval. A failure or error in the analysis aborts the rollout and sets the canary’s traffic weight back to zero.¹⁰

The most useful analysis compares the canary with the stable version running at the same time, rather than with a fixed threshold. Both versions see the same traffic, the same time of day and the same state of their dependencies, so differences are much more likely to come from the change.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate-vs-stable
spec:
  args:
    - name: service
  metrics:
    - name: canary-error-rate-relative-to-stable
      interval: 1m
      count: 10
      failureLimit: 2
      successCondition: len(result) > 0 && result[0] <= 1.2
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            (
              sum(rate(http_requests_total{service="{{args.service}}", track="canary", code=~"5.."}[5m]))
              /
              sum(rate(http_requests_total{service="{{args.service}}", track="canary"}[5m]))
            )
            /
            clamp_min(
              sum(rate(http_requests_total{service="{{args.service}}", track="stable", code=~"5.."}[5m]))
              /
              sum(rate(http_requests_total{service="{{args.service}}", track="stable"}[5m])),
              0.001
            )

The metric and label names depend on how your services are instrumented. The design choices carry over:

• The canary may have at most 20% more errors than the stable version, measured ten times, one minute apart, with at most two failed measurements tolerated.
• clamp_min sets a floor on the stable error rate. Without it, a stable version with zero errors makes the ratio undefined.
• An empty result fails the condition. If the canary receives too little traffic to measure, the gate does not pass by default. Make sure each canary step receives enough traffic for the analysis to mean something.
• Combine a relative check with an absolute ceiling. A relative check alone passes a canary that is exactly as bad as a stable version that is already failing.

Google’s SRE workbook discusses these trade-offs, including how canary duration and traffic share affect how quickly and reliably a problem is detected.¹¹ The rollout strategy itself is covered in Deploy vs Release: Canary, Blue/Green, and Feature Flags.

Choosing thresholds

Most gates need two of these together. An absolute limit alone lets things slowly get worse. A relative limit alone lets a bad baseline pass.

Measurements are noisy, and a threshold set right at the edge of normal variation will block constantly. Before making a check blocking, run it in alert-only mode for a few weeks, look at the distribution of results, and set the threshold where real regressions separate from noise.

Flaky checks at a gate

A flaky check is one that passes and fails on the same code. At a gate, it is worse than no check: it blocks good changes, and it teaches people to rerun until green. Google reported that about 1.5% of all its test runs were flaky, and that almost 16% of its tests had some level of flakiness.¹²

Three rules keep flakiness from eroding a gate:

• Retries must be visible. A test that passes only on retry is recorded as flaky in the results, not reported as a plain pass.
• Quarantine removes a test from the gate, not from the pipeline. A quarantined test keeps running and reporting, has an owner and a deadline, and returns to the gate when it is fixed.
• Infrastructure failures are not test failures. A runner that loses its network connection should be retried by the CI system and reported as an infrastructure problem, so it does not count against the tests or the change.

The full treatment is in Evolving the Test Suite.

Overrides and emergencies

Sometimes a fix has to reach production faster than the pipeline allows. If there is no official way to do that, people find unofficial ones: disabling a required check, pushing directly to a protected branch, or deploying by hand. Each of those leaves no record and turns the gate off for everyone.

A deliberate emergency path is safer:

The line is drawn around what the override may skip. Slow checks can be deferred. The checks that protect against making the incident worse, such as verifying the artifact, checking compatibility and watching the canary, stay on. Count overrides over time: a team that needs one every week does not have an emergency problem, it has a pipeline that is too slow.

Measuring the gates themselves

These measures complement DORA’s delivery metrics, which the hub covers. Change failure rate shows whether changes reach production broken. The measures above show which gate let them through.

Costs and trade-offs

• More gates means more waiting. Every blocking gate adds time between writing a change and seeing it in production. Add a blocking gate only when its failures are frequent or severe enough to justify that delay, and alert instead when they are not.
• Merge queues and full runs on main cost CI capacity. Budget for it explicitly rather than discovering it in the monthly bill.
• Provenance and signing add infrastructure. Key or identity management, attestation storage and verification in every deploy job all need maintenance.
• Canary analysis needs good metrics. Services need consistent instrumentation, separate labels for canary and stable traffic, and enough traffic to measure. Low-traffic services may need longer canary steps or synthetic traffic.
• Policies need owners too. Thresholds drift out of date as services change. Review them when a gate’s false-block rate rises or when a defect escapes past it.

Tooling

Related articles

• From Code to Production, the hub for this series, including the stage-by-stage gate policy.
• Contract Testing and Version Compatibility: the compatibility check behind the deploy gate.
• Deploy vs Release: Canary, Blue/Green, and Feature Flags: the rollout strategies the canary gate controls.
• Evolving the Test Suite: flaky tests, quarantine and retries in depth.

References