Scaling and cost

Your research API was fine with three users. Then a client ran a survey with forty thousand respondents, the dashboard got slow, and someone suggested “scaling it”. Meanwhile the Azure bill has drifted upwards and nobody can say which line is which. These two problems are the same problem viewed from different ends: what you are paying for is capacity, and most capacity decisions are invisible until they bite.

Up and out are different operations

Azure draws the line clearly. Scaling up means changing your App Service plan’s pricing tier to get more CPU, memory and disk on each machine. Scaling out means increasing the number of VM instances running your app. Basic, Standard and Premium plans scale out to 3, 10 and 30 instances respectively.

Scaling up is safe and boring: your code doesn’t notice. Scaling out changes the assumptions your code was written under, because one process becomes several, each with its own memory, its own connection pool, and its own copy of any background scheduler.

# Scale up: a bigger box (still one instance)
az appservice plan update -g iff-rg -n iff-plan --sku P1V3

# Scale out: more boxes
az appservice plan update -g iff-rg -n iff-plan --number-of-workers 4

Reach for up when a single request is slow or memory-hungry (a big document parse, a pandas transform). Reach for out when there are simply too many requests.

Autoscale rules that don’t oscillate

Autoscale watches a metric and adds or removes instances against rules you define. Three documented behaviours decide whether yours works.

First, always pair a scale-out rule with a scale-in rule on the same metric. With only one direction, the app walks to the ceiling and stays there. With two directions driven by different metrics, you can satisfy both at once — CPU under 45% while memory is over 90% — and the app oscillates.

Second, scale-out fires if any rule matches; scale-in requires all rules to match. One forgotten conservative scale-in rule can stop the app ever coming back down, which shows up on the bill rather than in the logs.

Third, leave a real margin between the thresholds. This is the difference between a working configuration and one that flaps:

// Before: two points of margin. Removing an instance pushes CPU back over 70%.
{ "scaleOut": "CpuPercentage > 70", "scaleIn": "CpuPercentage < 68" }

// After: enough margin that a scale-in doesn't immediately trigger a scale-out.
{ "scaleOut": "CpuPercentage > 70 for 10m", "scaleIn": "CpuPercentage < 40 for 10m",
  "minimum": 2, "maximum": 8, "default": 2 }

Flapping — repeatedly adding and removing instances — is logged. Look in the activity log for Flapping (autoscale spotted the risk and aborted the scale attempt) and FlappingOccurred (it scaled to a different count that breaks the loop). Those entries are the signal that your thresholds are too narrow.

Also set a sensible default instance count: when autoscale cannot read the metric, it scales out to the default if current capacity is below it.

What breaks when there are two of you

This is where agent-written code most often fails, because a single-instance app and a multi-instance app behave identically in development.

# Before: module-level cache. Works perfectly on one instance.
_survey_cache: dict[str, list[dict]] = {}

@app.get("/api/surveys/{survey_id}/responses")
async def responses(survey_id: str) -> list[dict]:
    if survey_id not in _survey_cache:          # each instance has its own copy
        _survey_cache[survey_id] = await load_responses(survey_id)
    return _survey_cache[survey_id]

At four instances you now have four independent caches with four independent staleness. “I cleared the cache and it’s still wrong” means you cleared one instance’s cache. The same defect appears in in-memory rate limiters (each instance counts a quarter of the traffic), in-process schedulers with BackgroundScheduler (every instance runs the nightly job, so it runs four times), and anything written to local disk.

# After: shared state lives in a shared store, keyed and expiring
@app.get("/api/surveys/{survey_id}/responses")
async def responses(survey_id: str, cache: Annotated[Redis, Depends(get_cache)]):
    key = f"survey:{survey_id}:responses"
    if cached := await cache.get(key):
        return json.loads(cached)
    rows = await load_responses(survey_id)
    await cache.set(key, json.dumps(rows), ex=300)   # one truth, all instances
    return rows

Connection pools fail the same way, and more expensively, because the pool is per process:

# Before: fine on one instance, 300 connections at ten instances
engine = create_async_engine(DATABASE_URL, pool_size=20, max_overflow=10)

# After: size against the database's limit divided by max instances,
# leaving headroom for migrations and admin sessions
engine = create_async_engine(
    DATABASE_URL,
    pool_size=5,            # 5 + 5 = 10 per instance; 10 instances = 100 max
    max_overflow=5,
    pool_pre_ping=True,     # drop connections the server has already closed
    pool_recycle=1800,
)

The symptom to recognise: connection refusals or long stalls that appear only under load, minutes after autoscale added instances.

Timeouts and health probes

Every outbound call needs an explicit timeout. Without one, a stalled LLM endpoint holds a worker forever, requests queue behind it, the health probe starts timing out, the instance gets pulled from the load balancer, and the remaining instances inherit the load and fail the same way. That is how one slow dependency becomes a full outage.

# Before: no timeout — a hang becomes an outage
async with httpx.AsyncClient() as client:
    r = await client.post(LLM_URL, json=payload)

# After: bounded, and a hang becomes a retryable error
timeout = httpx.Timeout(connect=5.0, read=30.0, write=10.0, pool=5.0)
async with httpx.AsyncClient(timeout=timeout) as client:
    r = await client.post(LLM_URL, json=payload)

Health check on App Service pings a path you choose on every instance once a minute. If an instance fails to return a status code in the 200–299 range ten times in a row (configurable down to two with WEBSITE_HEALTHCHECK_MAXPINGFAILURES), it is removed from the load balancer, and pinging continues; if it recovers, it returns. If it stays unhealthy for an hour, the instance is replaced — at most one per hour and three per day per plan.

Design the endpoint accordingly. It should touch the components the app genuinely cannot work without, return a 500-level code when they fail, respond well inside a minute, allow anonymous access, and not redirect — Health check doesn’t follow 302s, and a domain redirect returning 301 marks the worker unhealthy.

@app.get("/health")
async def health(db: Annotated[AsyncSession, Depends(get_db)]):
    try:
        await asyncio.wait_for(db.execute(text("SELECT 1")), timeout=3.0)  # cheap
    except Exception:
        raise HTTPException(status_code=503, detail="db unavailable")
    return {"status": "ok"}

Where the money actually goes

Five drivers dominate a platform like yours:

A budget is your early-warning system, and it is important to know what it is not. Budgets notify; they do not stop resources or cap consumption. Thresholds are a percentage of the budget amount, and you can set both actual alerts (spend already crossed the line) and forecasted alerts (the projection is heading past it). Cost data typically lands 8–24 hours after the fact and budgets are evaluated every 24 hours, so a runaway resource can burn for more than a day before the first email.

# A budget with a forecasted alert, so you hear about it before the money is gone
az consumption budget create-with-rg -g iff-rg --budget-name iff-monthly \
  --amount 500 --category Cost --time-grain Monthly \
  --time-period '{"start-date":"2026-10-01","end-date":"2027-09-30"}' \
  --notifications '{"Warn":{"enabled":"true","operator":"GreaterThanOrEqualTo","threshold":80.0,"contact-emails":["you@example.com"]}}'

Finding the thing quietly costing the most

Start in Cost Management: Cost analysis, daily view, grouped by Meter. That names the specific thing being billed rather than the service. Then filter to the Azure Monitor, Log Analytics, Insight and Analytics, and Application Insights services to isolate telemetry costs.

When telemetry is the culprit, the workspace can tell you which table:

Usage
| where TimeGenerated > ago(32d)
| where StartTime >= startofday(ago(31d)) and EndTime < startofday(now())
| where IsBillable == true
| summarize BillableDataGB = sum(Quantity) / 1000. by bin(StartTime, 1d), DataType
| render columnchart

And when the table is AppDependencies, this usually names the exact chatty call:

AppDependencies
| where TimeGenerated >= startofday(ago(30d))
| summarize Bytes = sum(_BilledSize), Calls = sum(ItemCount) by Target, OperationName
| top 20 by Bytes desc

Nine times out of ten the answer is one of: a log level left at DEBUG, a retry loop hammering a dependency, an autoscale minimum nobody lowered after a busy month, or storage that only ever grows.

What to check in your own code

• Grep the back end for module-level mutable state: a bare = {} or = [] at module scope that gets written to inside a handler.
• Grep for BackgroundScheduler, APScheduler or asyncio.create_task at startup. On N instances, those jobs run N times.
• Find every create_engine / create_async_engine and multiply pool_size + max_overflow by your autoscale maximum. Compare that to your database’s connection limit.
• Grep for httpx.AsyncClient( and requests.get( without a timeout. Each one is a potential hang.
• Check whether /health exists, what it actually checks, and how long it takes. Then confirm Health check is enabled in the portal and pointed at it.
• Open the autoscale setting: are the scale-out and scale-in metrics the same, is the margin wide, and is the minimum instance count one you’d be happy paying for at 3am?
• Search the activity log for Flapping.
• Confirm at least one budget exists with both an actual and a forecasted alert, and that the email goes somewhere a human reads.
• Run the Usage query above. If one table is more than half your ingestion, that is your next hour’s work.
• Check retention on your Log Analytics workspace and the access tier on your export buckets. Both are settings nobody ever revisits.