Massive information and analytics organizations usually battle to make entry to information and insights actually self-serve. The business tried to unravel this downside, fairly unsuccessfully, for many years, however now AI is giving us a reputable approach to do exactly that.
At GitHub scale, offering devoted analytics assist to dozens of product groups is difficult, and subsequently many groups are left to unravel this downside on their very own. Although there’s a variety of helpful product telemetry that product and engineering groups can use to make choices, determining which information mannequin, which grain, which filter, after which write the question and validate the outcome has all the time been tough with out the assist of an information analyst.
Enter Qubot, our inner GitHub Copilot-powered analytics agent. Qubot permits any Hubber (that’s what we name GitHub staff) to ask questions on any information mannequin in GitHub’s information warehouse in plain language and get a solution inside seconds.
Qubot just isn’t a reporting software or a dashboard alternative. As a substitute, it’s meant for exploratory questions like “Which cohort of customers has the very best retention on this function?” or “What product contributed to maneuver this metric probably the most final week?” Qubot has zero value upkeep and helps groups ramp up rapidly on datasets they might be unfamiliar with.
On this weblog put up, we’ll go over how we constructed Qubot, the way it’s modified, and what we realized.
How Qubot works
The structure has three most important elements: consumer interface, context layer, and question engine.
Person interface
Qubot is accessible by way of Slack, VS Code, and the Copilot CLI. The Slack interface doesn’t require any configuration, and it’s the popular collaboration software of Hubbers. When somebody posts a query within the Qubot Slack channel, a Qubot occasion is spawned as a Copilot Cloud Agent working on github.com. The reply is supplied immediately in Slack, permitting the consumer to share the outcome with others, but additionally iterate within the thread to evolve or refine the query. All the outcomes are additionally saved as a markdown report in a pull request that the consumer can reference to positive tune the question or use it in a dashboard.
Qubot can be out there in VS Code and the Copilot CLI, for customers that need an expertise extra built-in with their workflows. Qubot might be put in with one command as a plugin, and it turns into out there in any agent session in VS Code or Copilot CLI alongside every other customized brokers, expertise, and instruments configured by the consumer.
Context layer
Our information warehouse comprises information at completely different phases of curation: uncooked occasions (bronze), conformed information and dimensions (silver), and curated datasets designed for particular enterprise use circumstances (gold). The context layer is in-built a federated method, with information that’s tailor-made to the kind of information.
For bronze information, we have now telemetry context contributed by product groups, with schema data and metadata.
For silver information, we have now examples of queries, utilization steering, necessary filters and so on, maintained by the information and analytics workforce.
For gold information, we have now enterprise guidelines and metric definitions, contributed by groups proudly owning these datasets.
We additionally leverage our ETL pipelines to systematically enrich the context layer with extra alerts and derived metadata. The context is loaded at runtime by way of the GitHub MCP Server, fetching it from the context layer.
Context agent
The context layer is consistently enriched with new information continued throughout a number of repositories. At GitHub, we primarily use markdown for documentation, so we don’t must interface with a number of completely different instruments.
We’ve streamlined federated context contribution by way of a context agent. Groups can contribute by way of a standardized template or by referencing a repository containing related context. The agent then ingests, organizes, and normalizes this data right into a structured format that has confirmed efficient for Qubot based mostly on our evaluations.
Analysis framework
Each change to the context layer or agent configuration will get evaluated earlier than it ships. When somebody desires to counterpoint the context layer with new information, they will open a pull request. The brand new context goes by way of an offline eval framework that measures accuracy of the response, latency to find the proper reply, and catches regressions earlier than they attain customers.
The benchmarking framework for evaluating Qubot throughout structured check circumstances has three elements:
Take a look at circumstances: A curated dataset of prompts with identified appropriate solutions, ground-truth SQL, and metadata (area, issue).
Automated run orchestration: A script that automates launching every check case as an agent process with the GitHub CLI gh agent-task create, runs a number of parallel trials, polls for completion, and saves detailed JSON outcomes.
Stats aggregation: A reporting script that reads the saved outcomes and computes per-test-case metrics: completion charge, accuracy, and period (avg/min/max).
The tip-to-end circulation is: outline check circumstances → run Qubot N occasions per case → accumulate outcomes → combination stats → evaluate configurations.
Question engine
Qubot connects to each Kusto and Trino, the 2 question engines that energy most of GitHub’s analytics workloads, by way of a MCP server. We developed a customized implementation of the Trino MCP server, whereas for Kusto we deployed a neighborhood model of the Cloth RTI MCP Server. Kusto is quick and well-suited to exploratory questions over current occasion information. Trino handles complicated joins and deeper historic evaluation.
Fairly than forcing customers to know which to make use of, Qubot defaults to Kusto and switches to Trino mechanically when the query requires it.
What modified, and what we realized
Qubot has been extensively adopted at GitHub, with lots of of enthusiastic customers working 1000’s of queries. The variety of questions that Hubbers ask within the information and analytics Slack channels has lowered dramatically, as a result of now they will discover the information with higher autonomy and attain out just for sophisticated questions. It additionally permits Hubbers that by no means dared to dip into the information warehouse to entry the information they should drive their resolution making. That is without doubt one of the causes for providing a number of interfaces like Slack, Copilot CLI, and VS Code; Hubbers are very technical, however we wished to supply an possibility with no barrier to entry and nil configuration.
We rapidly found that the context layer is essential to enriching the reasoning capabilities of Copilot and to create an professional analytics agent. In our experiments we discovered that structured and properly curated context not solely makes Qubot extra correct, but additionally 3 times sooner at returning the proper reply. This has profound implications on the analytics engineering self-discipline, as a result of it makes any such artifact a first-class citizen in how information is modeled, somewhat than an afterthought.
Qubot has been a uncommon instance of profitable hub-and-spoke execution. It removes pressure from the information and analytics workforce, as product groups personal the telemetry for his or her surfaces and enterprise groups personal the definition of their gold information. Qubot acted as a gravitational power to centralize all this distributed information right into a single software that may profit all GitHub, offering incentives to accomplice groups to contribute to Qubot, as an alternative of making a number of instruments restricted to their very own domains.
Acknowledgements
Qubot engineering workforce: Weijie Tan, Tobias Tschuemperlin, Vamsi Anamaneni
Particular thanks: Yaswanth Anantharaju
Written by
