CI/CD and deployments

An agent added a file called .github/workflows/deploy.yml to your repo, and now pushing to main puts code on Azure. That is genuinely useful and also slightly alarming: a file you didn’t write has permission to change production. This article is about reading that file critically — what each part does, which parts are load-bearing, and where the version an agent typically emits is quietly unsafe.

Build once, release many times

A pipeline has two jobs with different purposes. The build stage turns source into one immutable artefact — a zip, a wheel, a container image — and runs the tests against it. The release stage takes that exact artefact and places it somewhere.

The distinction is not bureaucracy. If staging and production each build from source, they can produce different bytes: a floating dependency version resolves differently, a runner image updates, a registry hiccups. You then test one thing and ship another. Build once, and a rollback becomes “redeploy artefact 47” rather than “rebuild a three-week-old commit and hope”.

# Before — an agent favourite: build inside each deploy job
jobs:
  deploy-staging:
    steps:
      - run: pip install -r requirements.txt && pytest
      - uses: azure/webapps-deploy@v3
        with: { app-name: iff-api, slot-name: staging, package: . }
  deploy-prod:
    steps:
      - run: pip install -r requirements.txt   # resolved again, possibly differently
      - uses: azure/webapps-deploy@v3
        with: { app-name: iff-api, package: . }

# After — one build, one artefact, two releases
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt
      - run: pytest -q
      - run: zip -r app.zip . -x '.git/*' '.venv/*'
      - uses: actions/upload-artifact@v4          # the thing that gets shipped
        with: { name: app, path: app.zip }

OIDC login: no stored credential

The older pattern put a service principal’s client secret in a GitHub secret called AZURE_CREDENTIALS. It works, and it means a long-lived key to your subscription lives in a repository setting forever.

OpenID Connect (OIDC) removes it. GitHub mints a short-lived, signed token describing which repository, branch and environment is running. Azure trusts that token through a federated identity credential configured on a Microsoft Entra application or user-assigned managed identity, and exchanges it for an access token good for minutes.

release:
  needs: build
  runs-on: ubuntu-latest
  environment: production            # gates on a required reviewer
  permissions:
    id-token: write                  # WITHOUT this, azure/login has no token to exchange
    contents: read
  steps:
    - uses: actions/download-artifact@v4
      with: { name: app }
    - uses: azure/login@v2
      with:
        client-id: ${{ secrets.AZURE_CLIENT_ID }}        # identifiers, not secrets
        tenant-id: ${{ secrets.AZURE_TENANT_ID }}
        subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

Two smells worth recognising. If you see creds: ${{ secrets.AZURE_CREDENTIALS }}, the workflow is on the old stored-secret path. And if permissions: id-token: write is missing, OIDC login fails with a confusing “no token” error — that missing line is the single most common cause.

Environments and approvals

A GitHub environment is more than a label. It carries its own secrets and variables, can require named reviewers, can impose a wait timer, and can be restricted to specific branches. Microsoft’s own guidance for public repositories is to use environment secrets rather than repository secrets, precisely because a job cannot read an environment’s secrets until a required reviewer approves it.

That is what makes environment: production a real gate rather than a comment. Scope your federated credential to that environment too, and a token minted by a staging run is not usable against production.

Slots and swap

A deployment slot is a second live copy of your App Service app, with its own hostname, sharing the plan. You deploy to staging, check it, then swap.

What a swap actually does, in order: apply the target slot’s slot-specific settings to the source slot (which restarts its instances), wait for every instance to come back, warm them with an HTTP ping, and only then switch the routing rules. Production stays online the whole time; if warm-up fails, the swap aborts without touching it.

    - uses: azure/webapps-deploy@v3
      with:
        app-name: iff-api
        slot-name: staging            # deploy to the slot, never straight to production
        package: app.zip
    - run: |
        # migrations first (see below), then hand traffic over
        az webapp deployment slot swap -g iff-rg -n iff-api \
          --slot staging --target-slot production

Two things bite people here.

Which settings travel. App settings and connection strings are swapped with the code unless you tick “deployment slot setting”. So a staging database connection string will follow your code into production unless pinned. Custom domains, TLS settings, IP restrictions, scale settings, Always On, CORS and managed identities stay with the slot.

# Pin the connection string to the slot so it never swaps
az webapp config appsettings set -g iff-rg -n iff-api --slot staging \
  --settings DATABASE_URL="$STAGING_DB" --slot-settings DATABASE_URL

Warm-up. By default the swap pings / and accepts any response code. A FastAPI app can answer / before its database pool, model client or cache is ready. Point the ping at a real readiness path:

az webapp config appsettings set -g iff-rg -n iff-api --slot staging \
  --settings WEBSITE_SWAP_WARMUP_PING_PATH=/health \
             WEBSITE_SWAP_WARMUP_PING_STATUSES=200

Migrations: the part that can’t be rolled back

Everything above is reversible because code is reversible. Database schema is not.

During a swap — and during any deploy across multiple instances — old and new code run against the same database at the same time. A migration that drops or renames a column breaks the version still serving traffic, and redeploying the old artefact does not bring the data back.

The pattern that makes this safe is expand and contract: split one breaking change into three deploys.

-- Deploy 1 (expand): additive only, old code unaffected
ALTER TABLE responses ADD COLUMN sentiment_score double precision;

-- Deploy 2 (migrate): backfill, new code writes both, reads the new column
UPDATE responses SET sentiment_score = legacy_score WHERE sentiment_score IS NULL;

-- Deploy 3 (contract): only once nothing live reads it any more
ALTER TABLE responses DROP COLUMN legacy_score;

Where the migration step goes matters as much as what it contains. Run it in the release job, once, not in application startup — on a scaled-out app, startup means every instance racing to run the same migration.

    - run: alembic upgrade head          # one job, one runner, before the swap
      env:
        DATABASE_URL: ${{ secrets.PROD_DATABASE_URL }}

Add a concurrency block at workflow level so two pipeline runs cannot migrate at once:

concurrency:
  group: deploy-production
  cancel-in-progress: false            # queue, never overlap, for deploys

Rollback, in order of preference

1. Swap back. The previous production app is sitting in the staging slot, already warm. This is seconds, and it is why you swap rather than deploy straight to production.
2. Redeploy the previous artefact from an earlier workflow run. Minutes.
3. Revert the commit and rebuild. A full pipeline run, with all the build risk.

If the deploy contained a destructive migration, none of these are clean. That is the real argument for expand and contract.

Reading a failed run

Open the run and find the first red step, not the last. Later failures are usually consequences: a deploy step fails because the artefact it wanted was never uploaded. Expand that first step and read from the top of its output — the first error is the cause, and the tail is often a generic non-zero exit from the shell.

Then check the step’s context: which commit, which artefact name, which environment, which slot. Two failures that look identical in the log (“resource not found”) are usually one typo in app-name and one missing role assignment. If the message is genuinely opaque, re-run with debug logging enabled to get the runner’s own diagnostics.

What to check in your own code

• Open .github/workflows/ and confirm there is exactly one place the artefact is built, and that deploy jobs download it rather than rebuilding.
• Grep for AZURE_CREDENTIALS and creds:. Their presence means a long-lived secret is stored; migrate to azure/login@v2 with client-id/tenant-id/subscription-id.
• Confirm every job that logs in to Azure has permissions: id-token: write.
• Check whether the production deploy job has an environment: with a required reviewer — or whether pushing to main ships straight to production with no gate.
• Check slot-name: is present on the deploy step. A workflow deploying with no slot is overwriting production in place.
• In the portal, list your slot’s app settings and confirm anything environment-specific (database URL, connection strings, API keys) is marked as a deployment slot setting.
• Confirm WEBSITE_SWAP_WARMUP_PING_PATH points at a path that fails while dependencies are unready.
• Find where migrations run. If the answer is “in the app’s startup code”, that is a race condition on every scale-out.
• Look at the last schema migration that shipped. Was it additive? If it dropped or renamed anything, you had a window where the live version was broken.
• Check for a concurrency group on the deploy workflow, so two merges in quick succession cannot deploy and migrate simultaneously.