{ "formatVersion": 1, "scope": "Curated public research extract; not the full raw run.", "canonical": "https://embiro.ai/research/container-apps-recovery", "fieldNote": 22, "slug": "container-apps-recovery", "title": "Container Apps: replica, streaming and queue recovery tests", "description": "A South Africa North experiment with replicated API and authentication services, private PostgreSQL, interrupted streams and durable jobs. Recovery results were promising, but the full production migration gate remained closed.", "category": "Infrastructure", "art": "africa-core", "thumbnail": { "src": "/research/thumbnails/container-apps-recovery.webp", "alt": "Two forest-green compute towers connected to a mint database, with one interrupted copper path and queued ivory blocks." }, "date": "2026-11-04", "published": "2026-11-04", "updated": "2026-11-04", "author": "Embiro Labs Research", "status": "Exploratory benchmark", "headline": "A pending job recovered after its worker failed. Work already dispatched was held for reconciliation.", "metric": "91.5 s", "metricLabel": "Recovery after a pre-dispatch worker crash", "method": [ "South Africa North, zone-redundant Azure Container Apps Consumption environment. Two API and two authentication replicas at 0.5 vCPU / 1 GiB each; two queue workers and one separate synthetic inference backend at 0.25 vCPU / 0.5 GiB each. Replica counts were fixed; this was not an autoscaling experiment.", "Real application implementations from frozen Git revisions, with Node.js 22.23.3 for the gateway and Node.js 24.22.2 for authentication. A benchmark-only wrapper supplied fault controls and an artificial upstream. No production image or routing was replaced.", "An empty marked database on the existing private PostgreSQL 16 D2 HA server. TLS verification remained enabled. The test role owned only the synthetic database; no customer rows or production credentials were copied.", "A B2s_v2 VM generated requests and hosted baseline containers. Three rounds of 20 requests per client at 1, 4, 8 and 16 concurrent clients: 1,740 requests per topology. Requests alternated between authenticated model metadata and usage reads. Loopback HTTP and ACA HTTPS are different timing boundaries, and the baseline shares CPU with the generator.", "A separate 3600.0-second soak targeted eight authenticated metadata requests/second with four clients, recording 28,798 responses or network outcomes. It overlapped controlled rollout, fault injection and diagnostic load bursts. Minute p95 values are retained separately rather than presented as one pooled percentile.", "One abrupt API process exit, one auth process exit, a worker exit after leasing but before dispatch, a worker exit after dispatch, and an exit during streaming. Provider restart counts corroborated the corrected fault hook. Twelve batch jobs tested idempotency, completion and one measured receipt per job.", "The artificial upstream emitted a chunk approximately every 250 ms and reported fixed synthetic usage. Short streams, cancellation and five-minute continuous output were measured. A candidate-only 360-second gateway deadline allowed the ingress boundary to be probed; production deadlines were unchanged.", "Warm functional results are reported separately from failed startup. The first soak aborted while replicas were recovering. An initial PID-1 self-kill hook did not terminate the process; those failure claims were invalidated and rerun with abrupt process exit. No discarded run is counted as a successful fault test.", "Public regional meter arithmetic put the fixed candidate ACA allocation at approximately $0.297/hour while fully active, plus approximately $0.124/hour for the temporary VM, amortised disk and IP. Registry builds, DNS, requests and traffic are additional. These are planning rates, not an invoice or model-serving unit cost." ], "sections": [ { "title": "Testing the application layer separately from inference", "paragraphs": [ "Our previous experiment moved the control-plane data to dedicated managed PostgreSQL with HA. This follow-up asked whether replicated application containers could preserve access, accounting and queued work when a process disappeared. Inference stayed separate, and every upstream request in this experiment was synthetic.", "The load steps measured lightweight control-plane requests. They do not describe how many people can generate speech, images or text simultaneously on the pilot GPUs. Those limits remain governed by the inference workers and their admission controls." ] }, { "title": "Recovery depends on the dispatch boundary", "paragraphs": [ "A leased job whose worker exited before dispatch completed on the surviving worker after 91.5 seconds. The approximately 90-second lease prevented two workers from executing it at once. The twelve-job batch completed with one measured receipt per job.", "When the worker exited after dispatch, the job became uncertain after 89.4 seconds. There was one recorded upstream dispatch, with no automatic replay. An interrupted connection cannot prove that a real GPU has stopped. The test retained the admission and accounting hold until the separate synthetic backend confirmed completion, then exercised the audited reconciliation path.", "The API continued serving the concurrent metadata requests during its process fault. Authentication retained its shared session data, but one of 30 immediate session lookups failed during the auth exit; subsequent lookups on the survivor succeeded. This is an observed transient failure, not a zero-downtime claim." ] }, { "title": "Continuous streaming and silent waits behave differently", "paragraphs": [ "The continuous diagnostic stream delivered output for five minutes and ended normally. In contrast, the request with no upstream output for 250 seconds returned HTTP 504 after approximately 240 seconds. These observations describe this configuration and run; they are not a promise of unlimited streaming duration.", "A deliberate exit of the serving API replica interrupted its active stream. The surviving replica could serve new work, but it could not reconstruct that existing connection. Client cancellation also retained the conservative capacity hold until backend completion was confirmed. Long GPU warm-up waits therefore remain a durable-job concern, with explicit status and cancellation handling." ] }, { "title": "Why production stayed on its current host", "paragraphs": [ "The proposed separate auth app returned HTTP 503 for console workspace loading. The current implementation intentionally permits its operator connection only over loopback. Removing that check would weaken an existing boundary. An isolated same-host check, with loopback preserved, passed sign-in, workspace loading, key creation and revocation.", "The next design step is an explicit private operator path. Co-locating API and auth containers within each replicated app, behind a proxy that excludes operator routes, is one option to benchmark. A separate private operator service is another. Either needs a complete security and cost review before changing production.", "Production migration also needs a matched latency baseline, safe handling of transient session lookup failures, real private inference transport and controller-failover tests, and a rehearsed rollback. Two web replicas alone do not remove a single transport-host dependency. Production routing, GPU cost controls and inference placement were left unchanged." ] } ], "limitations": [ "This is one exploratory synthetic run, not an availability SLA, regional disaster-recovery result, enterprise certification or model-quality benchmark.", "Only process failures were injected. A whole Azure availability-zone failure was not simulated, and the shared production PostgreSQL server was not failed over during this run.", "The baseline and candidate have different ingress paths. Their relative latency does not establish a production regression, speedup or a pass against a matched-baseline gate.", "The one-hour soak covers authenticated model metadata reads, not a full production traffic mix. The test does not establish concurrent inference-user capacity.", "The database also served production traffic, and baseline containers shared the load-generator VM. CPU contention, burst credits, background work and network timing can influence the measurements.", "A five-minute continuous stream passed only with the diagnostic candidate deadline extended. Production request deadlines were not changed.", "The initial replay check reached only the surviving auth replica. After both replicas were healthy, the repeat observed both replicas and rejected every replay.", "The image is a conceptual generated illustration, not a photograph or a deployment diagram." ], "tables": [ { "title": "Corrected recovery and security observations", "columns": [ "Check", "Observation", "Interpretation" ], "rows": [ [ "API process exit", "1,135 concurrent metadata requests; 0 errors", "Surviving replica served this sampled load" ], [ "Auth process exit", "1 failed lookup out of 30; subsequent lookups succeeded", "Session data survived; transient failure remains" ], [ "Leased worker exit", "91.5 seconds to completion", "Recovered before dispatch, without duplicate execution" ], [ "Running worker exit", "89.4 seconds to uncertain status", "One dispatch; no automatic replay" ], [ "Batch accounting", "12 jobs; one measured receipt each", "Ledger total matched retained receipts" ], [ "Revocation, nonce replay and zero budget", "Passed after both replicas were healthy", "Shared state and cost limits remained enforced" ], [ "Split-app console workspace", "HTTP 503", "Private operator path blocks production cutover" ], [ "Same-host console sanity", "Sign-in, state, key issue/revoke passed", "Existing loopback boundary works" ] ], "note": "Failed and invalidated initial runs are retained in the source archive. Uncertain status is not a successful inference completion." }, { "title": "Warm metadata-load diagnostic", "columns": [ "Clients", "Requests per topology", "VM loopback p95", "ACA HTTPS p95" ], "rows": [ [ "1", "60", "6.6 ms", "21.7 ms" ], [ "4", "240", "19.2 ms", "21.8 ms" ], [ "8", "480", "49.4 ms", "30.4 ms" ], [ "16", "960", "54.5 ms", "94.2 ms" ] ], "note": "Three rounds at each concurrency. All requests returned HTTP 200. Different timing boundaries and shared load-host resources prevent a production speed comparison." }, { "title": "Sustained traffic and long-stream boundary", "columns": [ "Scenario", "Result", "Boundary" ], "rows": [ [ "One-hour metadata soak", "28,798 requests; 0 non-200/network outcomes", "3600.0 seconds; four clients" ], [ "Continuous diagnostic stream", "300.3 seconds; HTTP 200; normal end marker", "Candidate deadline 360 seconds" ], [ "250-second silent upstream", "HTTP 504 after 240.0 seconds", "No successful streamed completion" ], [ "API exit during stream", "Connection interrupted after three received chunks", "No transparent stream continuation" ] ], "note": "Synthetic responses isolate transport behavior. Timing is client-observed; it does not measure model token generation speed." } ], "references": [ { "label": "Azure Container Apps ingress", "url": "https://learn.microsoft.com/en-us/azure/container-apps/ingress-overview" }, { "label": "Azure Container Apps health probes", "url": "https://learn.microsoft.com/en-us/azure/container-apps/health-probes" }, { "label": "Azure Container Apps networking", "url": "https://learn.microsoft.com/en-us/azure/container-apps/networking" }, { "label": "Our dedicated PostgreSQL HA experiment", "url": "https://embiro.ai/research/dedicated-postgres-ha" } ], "audio": [], "evidence": [ { "file": "results.json", "sha256": "b47471fa9ccfa1c9e41653feee7509095dbff1635b4425a6f18e395b2f326da5" }, { "file": "methodology.json", "sha256": "0b953f3c728c3d29382c28499662a2d18063844246da13849af1647f3a52211c" } ] }