ADR 0031 — Multi-provider LLM failover and pipeline observability

Status: Accepted — temporary budget amendment pending routing evaluation Date: 2026-08-06 Audience: public Context phase: LLM resilience, observability, and temporary cost envelope Supersedes: ADR 0001 section 1 and ADR 0025 section 6 only for the temporary monthly amounts Amended: 2026-08-07 and 2026-08-08 — see Amendments Related: ADR 0011/0012 (postmortem + floor recalibration), ADR 0015 (engine-math gate), ADR 0029/0030 (answering engine)

Context

From 2026-07-31 to 2026-08-06, signal classification failed 200 of 200 items on every weekday run, and every GitHub Actions run reported success. Six days of complete data loss, unnoticed.

Three independent gaps combined:

1. The wipeout looked like a partial success. runSignalsStage classified any run with failures as "partial", and httpStatusForStage maps "partial" to HTTP 200 by explicit policy. Scoring zero out of two hundred was indistinguishable from a healthy run at the workflow level.
2. The error was discarded. lib/signals.ts caught each per-item failure, pushed it into an in-memory array, reported failed: signals.failed.length, and dropped the messages. The file contained no console.error at all. Even after the outage was identified, the provider's actual message could not be recovered from anything the system had persisted.
3. Nothing watched for absence. There was no run-history table, no health endpoint, and no alerting of any kind on any of the nine workflows. The only freshness surface was human-readable text on the public homepage.

Compounding all three: Anthropic was the sole provider for signals, chat, insights, weekly notes, NRR extraction and both knowledge adapters — seven files each constructing their own client, with retry logic in exactly one. lib/openrouter.ts was production-quality but wired only to social drafting. A single account-level failure took down every model-backed path at once.

The retry predicate that did exist (429 || 529 || >= 500) could not see the failure: Anthropic returns HTTP 400 for an exhausted credit balance, which is neither retryable nor, under that predicate, distinguishable from a malformed request. 200 items failed in about five seconds.

Decision

1. Provider ladders, not a single provider

Every structured LLM call goes through completeStructured in lib/llm/, which walks a per-task ladder. Rung 0 of every ladder is the model that task already used, so a healthy system is byte-identical; rung 1 is deepseek/deepseek-v4-flash via OpenRouter.

DeepSeek was chosen for the fallback because it is already priced in lib/cost.ts, already proven on the social path, and at $0.098/$0.196 per 1M tokens is roughly 30× cheaper than Sonnet — so a prolonged failover cannot breach the $4.17/mo ceiling from ADR 0001. It loses Anthropic prompt caching, which is a real cost difference and the reason the fallback rung is a cheap model rather than a peer-tier one.

Ops can pin any task with LLM_LADDER_<TASK>; an unparseable override is ignored with a warning rather than throwing, because a typo in an env var must not take down the pipeline it was meant to protect.

2. Error kinds drive retry and failover separately

Kind	Retry same provider	Advance to next
rate_limit (429)	yes	no — moving would stampede the fallback
overloaded (529), server (5xx), timeout	yes	yes
auth (401/403), credit (400 + billing body)	no	yes
invalid (400, other)	no	no — fails identically everywhere
unknown	no	yes

The credit vs invalid split is the crux. Both arrive as HTTP 400 and they want opposite handling, so classification reads the response body, not just the status.

3. Fallback-scored signals are attributable, never silently mixed

signals.provider is written alongside the existing signals.model_id, carried onto signal_outcomes.provider by runPostmortem, and recomputePillarFloors counts only primary-provider outcomes. ingestSignals reports fallback_scored, and the recalibration reports excluded_non_primary.

This matters because signals feed lib/postmortem precision, which feeds the monthly pillar_confidence_floors recalibration (ADR 0012). A different classifier model shifts the signal distribution, so postmortem precision must segment by model rather than pool verdicts from different models into one track record. Treating a DeepSeek-scored window as evidence about Sonnet's calibration would corrupt the floors.

lib/signals.ts is not on the ADR 0015 red-line list, so this change does not require a /strategy-gate PASS. The segmentation rule above is the condition under which that remains true: if a future change lets fallback-scored signals flow into floor recalibration unsegmented, that IS an engine-math change and needs the gate.

4. Fail loud at the point of judgement, not by changing the HTTP contract

httpStatusForStage keeps its "partial" → 200 policy and its regression test unchanged. Instead classifySignalsOutcome judges the outcome correctly: scoring nothing while failing everything is "error", as is failing a majority of attempted items. The 2026-08-05 run would have returned 502 and gone red.

The same reasoning applies to runAnalysisStage (all three sub-steps failing is now "error", a single failure remains a degradation) and to ingestNews returning zero items across every ticker.

5. Errors are persisted, grouped

pipeline_errors stores one row per distinct failure signature with an occurrences count. Grouping is what makes capture safe: the reason errors were previously reduced to a count is that writing 200 rows per bad run was worse than writing none. A total wipeout now writes one row reading 200 × HTTP 400: Your credit balance is too low.

6. A watchdog independent of the pipeline it watches

/api/cron/watchdog runs every two hours on its own workflow and alerts on absence — no signals in 48h, no daily run in 30h, stale prices, zero news. Staleness assertions are suppressed Sunday and Monday because daily-scan is weekdays-only and a weekend gap is not a fault.

Escalation, outermost first: /api/health returns 503 for an external monitor → the watchdog workflow reddens and GitHub emails the owner → the watchdog emails via Resend → workflows self-report failures to /api/cron/alert-ingest. Each layer covers the failure of the one inside it.

7. Alerting is deterministic and quiet by default

No LLM call anywhere in the alerting path — an alerting system that spends tokens stops working exactly when spend is the problem. Criticals email immediately on the transition into firing, then at most once per 24h while still firing, with an all-clear on recovery. Warnings ride the existing digest. alert_state.last_notified_at is set only after Resend accepts, mirroring owner_notification_deliveries, so a failed send retries rather than being silently consumed.

Consequences

Good. A provider outage degrades to a second provider instead of zeroing the pipeline. When something does break, the owner learns within ~2h with the verbatim provider error attached. cost_ledger.run_id finally carries data. One shared retry policy replaces one implementation and six absences.

Costs and risks.

• A fallback-scored window is not model-homogeneous. Mitigated by signals.provider + the segmentation rule, but it is a real caveat when reading precision over any window where llm.failover_active fired.
• Chat failover is partial by design: only before the first token, and the fallback is a single non-streamed completion with no tool loop. OpenRouter has no streaming transport in lib/openrouter.ts and a mid-answer provider switch has no clean semantics. Full streaming failover is deferred to its own spec.
• Alert fatigue is the main way this fails. The 24h cooldown, warning-to-digest routing, and weekend suppression are the countermeasures; if criticals start arriving routinely, the thresholds are wrong and should be revisited rather than the emails ignored.
• pipeline_runs and pipeline_errors grow unboundedly. Small at current volumes; revisit retention if they become material.

Alternatives considered

Alert only, keep one provider. Smaller, and would have caught this outage within hours. Rejected because it leaves the single point of failure intact — detection without recovery still means a dark pipeline until someone acts.

Change httpStatusForStage so "partial" pages. Rejected: the existing policy is deliberate and correct for genuine partial degradation. The bug was the classification, not the mapping.

Sentry or a hosted monitoring service. Rejected on the $50/yr total budget (ADR 0001) and because the failures that matter here are domain-level ("no signals in 48h"), which generic exception tracking would not have caught — nothing threw.

One row per failed item. Rejected: 200 rows per bad run is why error capture was skipped in the first place. Grouping is what makes it viable.

---

Amendments — 2026-08-07

A review of the implementation found ten defects, all confirmed. Four changed decisions recorded here; the rest were straightforward bugs.

A1. ledger is a distinct error kind

Originally a cost-ledger write failure escaped as a generic error, which the router classified unknown and therefore failed over — spending a second paid call while the first stayed unrecorded. That inverts the strict-cost contract it was meant to serve.

ledger is now its own kind, excluded from both RETRYABLE_KINDS and FAILOVER_KINDS. The provider answered and we paid; moving on cannot help.

A2. The ladder walk is bounded by a deadline

The Anthropic SDK defaults to a 600-second timeout and 2 internal retries. Wrapped in the ladder's own 4 retries that is up to 15 transport attempts, and one hung primary call could consume the entire 60s/300s function budget before the fallback rung was reached — a failover that never runs under the failure mode it exists for.

Clients are now constructed with maxRetries: 0 (the ladder owns retry policy) and an explicit per-call timeout, and the whole walk carries a deadlineMs budget (default 50s) checked before every attempt, every backoff sleep, and every rung advance.

A3. Ledger purpose is not the task name

Deriving cost_ledger.purpose from the task name silently moved x-reply spend out of the social pool — costPoolForPurpose routes on a social_ prefix, so x_reply billed to core while admission still checked the social ceiling — and broke cost-driver continuity for two other tasks.

LEDGER_PURPOSE in lib/llm/registry.ts now maps tasks to their historical purpose strings explicitly: x_reply → social_x_reply_shadow, hermes_operator_answer → hermes_ask_sorbet, nrr_extract → nrr_extraction.

A4. llm.failover_active is critical, and warnings reach the digest

It was a warning, but shouldNotify suppresses warnings and nothing read alert_state — so it reached nobody at all, and neither did stale.news or stage.partial.

Failover is now critical: the primary provider is down and the signal set is no longer model-homogeneous, which has engine consequences under §3. The 24h cooldown caps it at one email a day. Separately, the weekly cost pulse now renders every firing alert from alert_state, so warnings genuinely ride the digest as this ADR originally claimed.

Scoping corrections

• Only stages that actually invoke a provider (usesProvider: true) may resolve global llm.* alerts. Previously any successful stage could — a market-data run that makes no LLM call could mark a live outage recovered.
• workflow.failed.* alerts are point-in-time events with no stage that owns their resolution, so the watchdog expires them after 26h rather than leaving them firing and re-paging indefinitely.
• The all-clear email is sent before the row is marked resolved, and the row is resolved only if the send succeeded. The previous order lost the recovery notice permanently whenever Resend failed.
• Failure reporting moved into a composite action with required inputs. The copied-step version relied on inherited env that six workflows did not provide, and the check that "verified" it matched step-scoped env: — so it passed on reporters that would have posted empty credentials.

---

Amendments — 2026-08-08

The 2026-08-07 22:30 UTC daily-scan was the first live production run of this infrastructure under a real, extended Anthropic outage. It worked as designed — chat kept serving via the DeepSeek fallback, and two independent alerts fired and emailed — but it also surfaced a real mechanical bug and a real alert-resolution gap that a lab test couldn't have found.

B1. The signals stage is bounded by wall-clock, not just item count

Failover roughly doubles per-item latency (a failed primary attempt, then a fallback call). With SIGNAL_CONCURRENCY=6 and a backlog north of a thousand unscored items, the fixed 200-item cap could no longer reliably finish inside the route's 300s maxDuration. Vercel hard-killed the function mid-batch — a platform-level FUNCTION_INVOCATION_TIMEOUT that bypasses Node's catch/finally entirely, so withRunLog never closed its pipeline_runs row and pipeline_errors recorded nothing.

ingestSignals now takes an opts.deadlineAt (absolute epoch ms). The mapPool dispatch loop stops claiming new items once the deadline passes, letting in-flight workers finish naturally (each already bounded by completeStructured's own ~50s per-call deadline from A2). Unscored items are simply picked up by the next invocation. Both /api/cron/daily/signals and the combined /api/cron/daily route now pass a deadline measured from actual request entry, leaving margin inside their respective maxDuration budgets. This is a general-purpose fix — it would have prevented the same failure mode on a large legitimate news day, independent of any provider outage.

B2. A watchdog reaper closes runs a platform-level kill orphaned

A0's pipeline_runs ledger assumed every stage exits through its own try/catch/finally. A FUNCTION_INVOCATION_TIMEOUT doesn't — the row above sat in status='running', finished_at=null indefinitely, with no pipeline_errors row, and no stage.error.daily.signals alert (only the workflow-level GitHub Actions backstop caught this one, because it doesn't depend on the app process surviving).

reapStuckRuns (lib/observability/run-log.ts) runs on every watchdog pass: it force-closes pipeline_runs rows still running past a threshold (default 10 minutes — comfortably past the longest current maxDuration, 300s), guards against the row having genuinely finished in the interim with a conditional WHERE status='running' on the update, writes a synthetic pipeline_errors row (error_kind: "timeout") so there's a durable record even though the process never ran its own handler, and feeds each reaped run through evaluateRunAlerts so it pages exactly like an internally-caught error would have.

B3. llm.provider_account.* resolution didn't cover every path that could fix it

Confirmed live: a genuine Anthropic recovery — verified independently via cost_ledger and a real social_posts row — left llm.provider_account.anthropic and stage.error.chat.stream firing for hours, because only daily/signals (usesProvider: true) and the weekly route were wired to resolve them. The chat route only dispatches on failure (a clean success runs none of that code), and lib/social.ts's generation path — the one Hermes exercises — doesn't call the dispatcher in either direction. A correct recovery sat reported as ongoing until the next cron stage that happened to own that scope succeeded.

reconcileProviderAlerts (lib/alerts/notify.ts) runs on every watchdog pass: for each firing llm.provider_account.* or stage.error.chat.stream row, it checks cost_ledger for a same-provider row with nonzero usd_cost created after the alert's last_fired_at. cost_ledger is the right source of truth here — every provider call writes to it regardless of which code path triggered the call, so it doesn't require every call site to individually own alert resolution the way the scoping rule in the prior amendment assumed.

B4. The chat route stopped writing a spurious zero-cost ledger row

When the chat stream fails over at hop 0 (before any token arrives), usage stays all-zero. The finally block wrote a $0.000000, service='anthropic' cost_ledger row unconditionally, alongside the real DeepSeek row for the same turn — paired rows a few seconds apart in production. Beyond confusing the cost-driver report, this also undermined B3: a $0 non-charge is not evidence Anthropic answered anything. recordAnthropicCall is now only called when usage.input_tokens + usage.output_tokens > 0.

C. Temporary core and social budget expansion pending routing evaluation

Context and evidence

ADR 0001 set a $4.17 monthly core-operations ceiling when Sorbet's signal load was smaller. ADR 0025 later isolated social distribution in its own $1 monthly pool. The bounded signals stage can now examine up to 200 unscored items per run, with a paid classifier round-trip for each relevant item and additional latency when the provider ladder advances. Higher incoming signal volume therefore creates more token spend and longer catch-up runs. Social drafting and X-reply shadow generation also operate over a larger governed evidence surface than the original $1 pool anticipated.

On 2026-08-08, the owner supplied this Hermes chat response. It is preserved as received; the final failure sentence was truncated in the source:

Cronjob Response: sorbet-daily-growth-planner
(job_id: 9ae13e6b2d57)
-------------

⚠️ Cron 'sorbet-daily-growth-planner' failed: Skipped to prevent unintended spend: global inference config drifted since this job was created (model 'anthropic/claude-opus-4.6' -> 'openai/gpt-5-mini'), and this job is unpin...

To stop or manage this job, send me a new message (e.g. "stop reminder sorbet-daily-growth-planner").

This is evidence of a routing/configuration stop, not budget exhaustion. Hermes reported that it skipped rather than silently spending through a global model change. That fail-closed posture is correct. The response does not verify the job's current enabled state, complete pinning configuration, whether any tokens were consumed, the current global model after the message, or the truncated end of the failure sentence. Those remain unverified until the live Hermes job record is inspected.

The owner approved temporary headroom while Sorbet revisits the complete LLM-routing design and evaluates proposed models against the models selected at the evaluation freeze. The expansion is an operational bridge, not evidence that a higher-cost model is better, permission to inherit an unreviewed global model, or a substitute for quality, latency, reliability and cost evaluation.

Decision

1. Set the core SORBET_MONTHLY_CAP_USD default to $10/month.
2. Set the separate SORBET_SOCIAL_MONTHLY_CAP_USD default to $5/month.
3. Keep the pools independently metered and non-transferable. Purposes prefixed with social_ charge social; other purposes, including the private hermes_ask_sorbet operator path, charge core.
4. Retain each pool's 80% soft alert, 90% degrade/fallback, and 100% hard stop.
5. Keep the signals stage's 300-second platform duration, deadline-aware dispatch, 200-item query limit, bounded concurrency, retry policy, and 400-token per-item output limit. Budget headroom does not independently justify changing these controls.
6. Do not change ladder order, selected models, publishing autonomy, approval requirements, safety checks, credentials, allowlists, activation modes, or kill switches as part of the budget change.
7. Agent-driven Hermes cron jobs must pin an exact provider and model. A model mismatch must continue to skip and alert; nobody should unpin or bypass the guard merely because more monthly budget is available.
8. Treat both increases as temporary until the evaluation gate below produces an explicit owner decision to retain, revise, or roll them back.

Pool	Previous	Temporary cap	Soft alert	Degrade	Hard stop
Core operations	$4.17	$10.00	$8.00	$9.00	$10.00
Social distribution	$1.00	$5.00	$4.00	$4.50	$5.00
Total possible spend	$5.17	$15.00	—	—	—

Against the canonical $5,000 paper AUM, the temporary combined ceiling is $180/year, or 3.6% annualized. Core alone is 2.4%; a literal 2% annual core ceiling would be approximately $8.33/month. Public surfaces must not describe either $10/month core or $15/month combined as 2%.

Routing and comprehensive evaluation gate

Before the temporary limits become durable, produce a versioned evaluation record that:

1. freezes exact provider/model IDs, ladder configuration, prompts, tool schemas, fixtures, pricing assumptions, and run date;
2. evaluates Sorbet classification/generation and Hermes operator/planner workloads as separate roles rather than assuming one winner for all tasks;
3. compares every incumbent/job-pinned model and proposed replacement on the same representative corpus, including high-volume and failure cases;
4. measures evidence fidelity, unsupported claims, instruction resistance, structured/tool-call validity, policy compliance, owner preference, latency, token use, provider-reported cost, retry/failover behavior, and variance across repeated runs;
5. verifies that Hermes cron jobs remain pinned and global configuration drift produces a zero-inference skip plus an actionable alert;
6. archives raw outputs and a scored comparison without production writes; and
7. ends with an explicit owner decision for every role and both budget caps.

The existing npm run social:eval harness is smoke evidence only: six candidates over three synthetic rotations with deterministic validation and token/cost/latency capture. It does not yet cover representative operator planning, routing drift, retries/failover, adversarial evidence, repeated-run variance, or owner-scored quality. Until the comprehensive gate passes, new models remain challengers; a global configuration change is not promotion.

Consequences and follow-up

• The default maximum becomes $15/month across two independent pools.
• Explicit deployment variables retain their values until updated and redeployed; changing repository defaults alone is not production evidence.
• More budget provides token headroom but does not resolve provider errors, serverless timeouts, retry amplification, scheduling backlog, or model drift.
• Hermes job 9ae13e6b2d57 needs read-only inspection before any resume or replacement. This amendment does not stop, edit, unpin, or rerun it.
• Revisit this amendment when the evaluation gate completes, either pool reaches degrade for two consecutive periods, or routing drift recurs.

Detailed provenance and pending verification are retained in implementation-notes/adr0031-budget-routing-evidence.md.