Scheduling and orchestration: making jobs run when you think they do

An agent set up a nightly job for you: pull yesterday’s survey responses, clean them, write them to the database, refresh a dashboard. It worked when you tested it. Three weeks later someone notices Tuesday is missing, or that Monday’s numbers are doubled, and nothing anywhere said a word. Scheduled work fails in a particular way: quietly, in the gap between “the code is correct” and “the code ran once, at the right time, on the right data”.

This topic is about that gap. Almost all of it comes down to four questions: when does it fire, who fires it, what happens when two fire at once, and what happens when none do.

Cron syntax, and the fields that bite

A cron expression is five numbers describing a repeating calendar time. It is not an interval.

* * * * *
│ │ │ │ └── day of week   (0-6, Sunday = 0 in standard cron)
│ │ │ └──── month         (1-12)
│ │ └────── day of month  (1-31)
│ └──────── hour          (0-23)
└────────── minute        (0-59)

Two traps live here. The first is that */5 in the hour field does not mean “every five hours from now” — it means “at hours 0, 5, 10, 15, 20”, so the gap between the 20:00 run and the next one is four hours, not five. Cron matches clock times; it does not count elapsed time.

The second is that the dialects disagree. Azure Functions uses NCRONTAB, which adds a sixth field at the front for seconds. Cloudflare Workers uses five fields but numbers weekdays 1 = Sunday to 7 = Saturday, unlike standard cron. So the same-looking string means three different things:

0 30 9 * * 1-5     # Azure NCRONTAB: 09:30:00, Monday to Friday
30 9 * * 1-5       # standard cron:   09:30, Monday to Friday
30 9 * * 1-5       # Cloudflare:      09:30, Sunday to Thursday  ← not what you meant

For Cloudflare, use the three-letter names so the ambiguity disappears: 30 9 * * MON-FRI. If an agent hands you a cron string, the check is: count the fields, then name the platform, then read the weekday numbers out loud.

Time zones and daylight saving

Every platform here evaluates cron in UTC by default. Azure Functions, Container Apps jobs, Cloudflare cron triggers and GitHub Actions all do. That means a job you think of as “9am” drifts by an hour twice a year in the UK, which is exactly the sort of bug that looks like a data problem.

Azure Functions has a real fix: set the app setting WEBSITE_TIME_ZONE, and the trigger tracks the zone’s daylight-saving changes.

# Linux Premium/Dedicated plans use tz database names; Windows uses tzutil names.
az functionapp config appsettings set \
  --name iff-research-jobs --resource-group iff-prod \
  --settings WEBSITE_TIME_ZONE="Europe/London"   # now "0 0 9 * * *" really means 9am local

Microsoft’s docs carry a caveat worth knowing: WEBSITE_TIME_ZONE is not supported on Linux Consumption or Flex Consumption plans, where setting it can cause TLS oddities and break metrics. On those plans you keep UTC and accept the drift, or compute local time inside the job.

GitHub Actions now accepts an IANA timezone alongside the cron. During a spring-forward transition a schedule in a skipped hour advances to the next valid time (a 02:30 job runs at 03:00), and an autumn fall-back hour happens twice — so anything keyed to “the 1am run” should be idempotent.

on:
  schedule:
    - cron: "30 5 * * 1-5"
      timezone: "Europe/London"   # without this, 05:30 UTC — an hour off in summer

Container Apps jobs and Cloudflare cron triggers have no time zone setting at all. Write UTC and convert in your head.

Overlapping runs, and “only one at a time”

Suppose the job usually takes 4 minutes and runs every 10. One day the source API is slow and it takes 25. Does the 10-minute mark start a second copy? The answer depends entirely on the platform, and most agents never mention it.

When the platform does not guarantee it, take the lock yourself. A lock is a flag that one process holds while it works, so others see it and stand down. The cheapest correct one for a job that already talks to PostgreSQL is a session-level advisory lock: an arbitrary named lock the database hands out, released automatically when the connection closes — including when your container is killed mid-run.

from sqlalchemy import create_engine, text

engine = create_engine(DB_URL)
LOCK_KEY = 8817342  # any stable int; pick one per job and write it down

with engine.begin() as conn:
    got_it = conn.execute(
        text("SELECT pg_try_advisory_lock(:k)"), {"k": LOCK_KEY}
    ).scalar()
    if not got_it:
        print("another run holds the lock; exiting cleanly")   # exit 0, not an error
        raise SystemExit(0)
    ingest_survey_responses(conn)
    conn.execute(text("SELECT pg_advisory_unlock(:k)"), {"k": LOCK_KEY})

Two smells in agent-written versions of this. pg_advisory_lock (without try_) waits instead of declining, so a stuck run makes every later run pile up behind it. And a lock implemented as a row in a job_locks table, or a file on disk, does not get released when the process is killed — you end up with a job that is permanently “already running” until a human deletes the row.

Missed runs, and why “yesterday” is the wrong instruction

Scheduled triggers do not queue. If the function app was restarting, or the repository’s scheduled workflow was delayed, or the container could not start, that occurrence is gone. GitHub is explicit that the schedule event can be delayed at busy times — the top of every hour especially — and that queued jobs may be dropped under sufficient load. Azure Functions will not retry a failed timer invocation either; it waits for the next slot.

Azure does at least tell you when it noticed a slip. The timer object carries past_due, and the schedule status includes the last and next occurrence:

import azure.functions as func

app = func.FunctionApp()

@app.timer_trigger(schedule="0 0 2 * * *", arg_name="timer", run_on_startup=False)
def nightly_ingest(timer: func.TimerRequest) -> None:
    if timer.past_due:                      # an occurrence was missed; don't ignore this
        logging.warning("timer past due — backfilling from the watermark")
    ingest_since_watermark()

Leave run_on_startup as False. Microsoft warn against True in production: the function fires whenever the app wakes, restarts or scales, which on a consumption plan means unpredictable extra executions and unpredictable cost.

The durable fix is not retries — it is making the job state-driven. Instead of “process yesterday”, the job asks the database what it already has and processes everything after that. That single change makes a missed run self-healing and a double run harmless.

-- before: agents love this, and it silently loses a day whenever a run is skipped
SELECT * FROM raw_responses WHERE submitted_at::date = CURRENT_DATE - 1;

-- after: a watermark. Catches up after a missed run, re-runs are idempotent.
SELECT * FROM raw_responses
WHERE submitted_at > (SELECT last_seen_at FROM ingest_watermark WHERE job = 'survey')
ORDER BY submitted_at
LIMIT 5000;

Pair it with an idempotent write — one that can run twice with the same result — using the source system’s own id:

INSERT INTO responses (source_id, respondent_id, payload)
VALUES (:source_id, :respondent_id, :payload)
ON CONFLICT (source_id) DO UPDATE SET payload = EXCLUDED.payload;

Picking the runner

You already own five schedulers. Roughly:

• Azure Functions timer trigger — short, frequent jobs. Built-in singleton behaviour. Watch the plan’s execution timeout for long work.
• Container Apps job (Schedule) — long or heavy work in a container: document processing, a big export. You set --replica-timeout, --replica-retry-limit and --parallelism explicitly, which is honest about what happens on failure.
• Durable Functions — a multi-step workflow where step 3 depends on step 2 and you need the progress to survive a restart. An orchestrator function calls activity functions and its state is checkpointed, so a crash resumes rather than restarts. Reach for it when you catch yourself storing “which step are we on” in a database column.
• GitHub Actions schedule — fine for maintenance and reports, wrong for anything time-critical. It only runs from the default branch, is delayed under load, and in public repositories is automatically disabled after 60 days with no repository activity.
• Cloudflare cron trigger — a scheduled() handler for edge-shaped work: expiring KV sessions, refreshing a D1 news table. Trigger changes take up to 15 minutes to propagate, and the CPU limits of a Worker still apply.

az containerapp job create --name nightly-export --resource-group iff-prod \
  --environment iff-env --trigger-type Schedule \
  --cron-expression "0 2 * * *" \
  --replica-timeout 3600 --replica-retry-limit 1 --parallelism 1 \
  --replica-completion-count 1 --image iffacr.azurecr.io/export:2026-09-01 \
  --cpu 1.0 --memory 2.0Gi   # note: cron here is always UTC

Do you need Airflow?

Orchestration tools — Airflow, Prefect, Dagster — exist for dependency graphs: twenty tasks where several feed one, with backfills over historical dates, per-task retries and a UI showing which cell of the run-by-date grid is red. If you have that, they earn their keep.

If what you have is “fetch, transform, load, once a night”, they do not. You would be running a scheduler, a metadata database and a web server to replace one cron line. The usual right answer at your scale is a queue plus a worker: the scheduled trigger only enqueues work items, and a separate worker drains the queue. This gets you retries, visible backlog, and the ability to re-run one item without re-running the night — which is most of what Airflow was giving you.

# scheduled trigger: cheap, fast, just fans out work
for batch_id in unprocessed_batch_ids():
    queue_client.send_message(json.dumps({"batch_id": batch_id, "attempt": 0}))
# the worker processes one batch per message; a failure retries that message only

Alerting when a job silently stops

The failure that hurts is not the crash — a crash logs an exception. It is the job that stops being invoked: the trigger was deleted, the storage lease broke, the workflow was auto-disabled, the schedule string was edited. There is no error, because there is no run.

The fix is to alert on absence. Have the job record a heartbeat on success, then alert when the newest heartbeat is older than it should be. With Application Insights, that is a scheduled log alert on a query like this:

// Alert if the nightly ingest has not logged a success in the last 26 hours
customEvents
| where name == "ingest_succeeded"
| summarize last_success = max(timestamp)
| where last_success < ago(26h) or isnull(last_success)

Note the window is longer than the interval, so one late run does not page you. The same shape works anywhere: a job_runs table with a finished_at column, and one query asking whether the newest row is stale. Dead-man’s-switch services do this as a hosted ping, which is the same idea with someone else’s cron.

What to check in your own code

• Count the fields in every cron string, and name the dialect. Six fields means NCRONTAB seconds; on Cloudflare, read weekday numbers as 1 = Sunday.
• Grep for WEBSITE_TIME_ZONE. If it is absent, every Azure Functions schedule is UTC — decide whether that is intended.
• For each scheduled job, answer out loud: what happens if two run at once? If the platform does not prevent it, find the lock, and check it is pg_try_advisory_lock (or equivalent) rather than a table row.
• Grep for CURRENT_DATE - 1, date.today(), yesterday in job code. Each one is a job that loses data on a missed run.
• Check run_on_startup / runOnStartup is false in production.
• Confirm every write path has a unique key and an ON CONFLICT (or a MERGE) so a re-run is harmless.
• Find the alert that fires when the job does not run. If there isn’t one, that is the most valuable thing on this list.
• For scheduled GitHub Actions in a public repository, check the last run date — 60 days of inactivity disables it silently.