Abstract
Databricks’ serverless platform launches tens of thousands and thousands of VMs day by day, and every VM wants community configuration akin to allowed locations and personal endpoints, earlier than serving buyer workloads. With every node fetching config at startup and polling for updates all through its lifetime, this interprets to billions of community config requests per day. The outdated structure fetched this from a number of upstream providers synchronously, creating latency, and availability bottlenecks.We re-architected community config supply utilizing event-driven pipelines and snapshot pre-computation, decreasing RPC latency by 97.5% (5,000ms → 125ms), reaching 99.99% service availability.
Downside Assertion
Databricks’ serverless compute platform powers nearly all of our information and AI merchandise, akin to, SQL warehouses, notebooks, ML serving endpoints, and extra. The platform launches tens of thousands and thousands of VMs day by day throughout AWS, Azure, and GCP.
Earlier than any serverless workload can execute, the VM must know its community configuration: What storage locations can it entry? Are there non-public hyperlink endpoints via which it ought to route visitors? Is there current modifications in Unity Catalog that grants entry to new storage locations? Can we begin consuming new locations shared by way of Delta Sharing?
The problem is that community configuration isn’t saved in any single place. It have to be assembled from a number of upstream providers, every contributing a chunk of the total image.
The Outdated Structure
Within the authentic design, each time a serverless cluster began, our community configuration service would synchronously name all upstream providers, mixture their responses, compute the per-workspace community configuration, and return it to the serverless dataplane. This occurred on the important path of cluster creation.
Whereas the outdated structure was easy and labored effectively with small scale, this structure suffered from basic issues, mirrored within the following metrics we observe on our operational dashboard:
Latency: With a number of upstream providers on the important path, the RPC latency for serving community configuration was 5,000ms at p99. This impacted serverless cluster begin up latency.Server Success Fee: Every upstream service has its personal availability traits. With a number of providers in sequence, the compound availability drops rapidly, translating to elevated chance of severless cluster launch failures per yr.
As serverless utilization continued its speedy progress, the synchronous mannequin grew to become more and more unsustainable. Every synchronous name triggered costly operations throughout all workspaces, usually doing duplicated computation. This added load that grew proportionally with the variety of tenants and their configured sources.
Answer: Occasion Pushed Precomputation
We carried out a ground-up re-architecture of how Databricks delivers community configuration. It’s constructed on the core ideas:
Occasion-driven pipeline: As an alternative of constructing synchronous calls to all upstream providers, the brand new system subscribes to vary occasions by way of a message queue. When a buyer creates a brand new Unity Catalog connection or modifies a community coverage, the upstream service emits an occasion. The system processes it and updates the pre-computed configuration.Snapshot pre-computation: Community configurations are computed asynchronously within the background and saved in a pre-computed snapshot retailer. The serving path turns into a single, skinny storage fetch, utterly decoupled from the upstream providers.Static stability: Within the occasion of any upstream service outage, we are able to preserve a static config, offering static stability for the serverless clusters.
The structure cleanly separates two paths. The administration path runs asynchronously within the background: upstream providers emit change occasions to a message queue, which an occasion processor consumes to resolve which workspaces are affected and fan out per-workspace replace notifications. A neighborhood occasion supervisor then fetches the related particulars from upstream, recomputes the workspace’s community configuration, and shops the end in a pre-computed snapshot retailer. A periodic reconciler additionally re-syncs all workspaces within the background, guaranteeing eventual consistency even when occasions are missed. The serving path, in contrast, is important and quick: when a serverless cluster begins up and desires community configuration, the community configuration service serves it immediately from the snapshot retailer with a single storage learn, requiring no upstream service calls and meaningfully decreasing load on upstream providers.
Key Design Choices
Upstream providers push change occasions to the message queue. The system processes these occasions within the background. A low-frequency reconciler periodically re-syncs all workspaces as a security internet, offering the reliability of synchronous framework with the effectivity of push.Community configurations are computed and saved regionally inside every service partition, co-located with the workspaces they serve. This distributes computation, reduces blast radius throughout incidents, and eliminates cross-partition dependencies on the serving path.Occasions carry solely workspace and useful resource identifiers. This retains occasions light-weight, makes them idempotent (they are often replayed in any order), and avoids transferring delicate buyer information via the messaging pipeline.
How Occasions Circulation
When a buyer creates a brand new Unity Catalog connection, Unity Catalog emits a change occasion to the message queue. The occasion processor then receives the occasion, determines which workspaces are hooked up to the affected metastore, and followers out a per-workspace replace notification. In every workspace’s partition, the occasion supervisor receives this notification, fetches the up to date connection particulars, recomputes the workspace’s community configuration, and shops it with a brand new model mark. From that time on, when a serverless cluster requests the community config, it’s served immediately from the snapshot retailer with no upstream calls wanted.
Affect
After rolling out the brand new structure, the outcomes had been transformative throughout all operational metrics:
Past the topline metrics:
Upstream name quantity diminished by 86%. The system solely calls upstream providers when an occasion signifies a change, not on each request.We noticed significant enchancment within the freshness of the networking configuration.Legacy synchronous framework absolutely deprecated.
Conclusion
This undertaking taught us a number of classes about working community infrastructure at cloud scale:
Pre-computation decouples important paths. By shifting costly aggregation to the background, the serving path turns into trivially easy and quick. That is the one most impactful architectural determination. It turned a multi-service dependency chain right into a single storage learn.
Occasion-driven structure trades consistency for scalability and reconciliation offers the security internet. Occasion-based push handles the widespread case effectively, whereas a periodic reconciler catches something that falls via the cracks.
Design for extensibility from day one. The modular, stage-based structure means including help for a brand new upstream information supply requires solely a brand new stage implementation with zero modifications to the core pipeline. As Databricks’ product floor expands, the community configuration system scales with it.
Immediately, this technique serves billions of community config requests per day throughout Databricks’ international serverless fleet, with ~125ms latency and 99.99% availability. As serverless compute continues its speedy progress, the event-driven structure ensures that community configuration supply scales proper alongside it.
We’re at all times in search of engineers who take pleasure in tackling distributed techniques challenges at international scale. If issues like these excite you, we might love to listen to from you, please take a look at open roles at databricks.com/careers!
