Official · Infrastructure · advanced
Roll out configuration to a disconnected fleet
Safely converge 40,000 gateways while some sites are disconnected and old controllers can return.
The question
Design a tenant-aware configuration controller for a managed gateway fleet. Explain the journey from an approved change to an observed running version, including an interrupted rollout. Identify the responsibilities of management and request-serving components; separate visual planes only if that helps your explanation.
- 40,000 gateways across 800 tenants and four regions poll or receive changes. About 10% of sites can be disconnected for six hours.
- A gateway already serving traffic must keep serving its last known-good configuration while management is unavailable.
- Operators need canaries, rollback and a reliable answer to “which version is actually running?” Two controllers can temporarily believe they own a rollout.
What to cover
Diagram or describe approval, distribution, acknowledgement and serving paths, naming the state owner at each step.
Define desired versus observed state, version ordering, stale-leader fencing and reconnect behavior.
Estimate steady-state and reconnect-burst load with stated polling or push assumptions.
Walk through a faulty canary, a disconnected site and two active controllers; define measurable rollback/stop gates.
Compare two distribution approaches and state what you would monitor to detect a stuck rollout.
How your practice is reviewed
- Reconciliation and version safety (30 points): Desired/observed state, monotonic versions or equivalent safeguards, idempotency and reconnect behavior form a coherent protocol.
- Isolation and serving continuity (25 points): Tenant authentication/authorization and continued serving under management failure are addressed end to end.
- Quantified capacity (20 points): Calculations expose polling, reconnect bursts and retention assumptions; claimed convergence is supported rather than merely repeated.
- Rollout decisions and tradeoffs (25 points): Canary gates, rollback limits, ownership and an explicitly rejected alternative are operationally credible.
Discuss & learn
Ask the community, or tag @Coach for a contextual AI answer.