In\u00a0March\u00a0and\u00a0April\u00a0we shared updates on GitHub\u2019s availability and infrastructure investments. As that work continues and we method some main milestones, we needed to start out sharing extra common updates in our month-to-month availability reviews.\u00a0\u00a0So\u00a0earlier than we dive into incidents from Might,\u00a0right here\u2019s\u00a0how\u00a0we\u2019re\u00a0monitoring with\u00a0our ongoing work to make GitHub extra dependable.\u00a0\u00a0<\/p>\n
Our progress in making GitHub extra resilient<\/p>\n
The\u00a0quick model:\u00a0GitHub\u2019s site visitors is rising quickly, pushed largely by AI-assisted and agentic improvement workflows, and\u00a0we\u2019ve\u00a0been reworking our infrastructure to maintain up with it. Meaning shifting to Azure for elastic capability,\u00a0breaking our monolith aside into remoted providers, and\u00a0eliminating\u00a0the shared failure factors which have pushed previous incidents.\u00a0<\/p>\n
Right here\u2019s\u00a0the place we stand.\u00a0We\u2019re\u00a0now serving\u00a040% of monolith site visitors from Azure\u00a0(up from 8% in February), with Git site visitors at 30% and repository replication at\u00a099%. We\u2019ve greater than doubled our efficient capability in 4 months. On the similar time,\u00a0we\u2019re\u00a0finishing the isolation of our major database cluster: splitting customers, authentication, and authorization into impartial domains in order that an issue in a single can now not cascade throughout the platform. Our\u00a0new customers\u00a0service is absolutely minimize over, and dealing with double the site visitors at\u00a0considerably decrease\u00a0database value. Stateless authentication tokens are additionally rolling out,\u00a0eliminating\u00a0per-request database lookups that\u00a0amplified\u00a0strain throughout site visitors spikes.\u00a0<\/p>\n
We’re making\u00a0structural adjustments that completely take away failure modes.\u00a0We\u00a0acknowledge that we\u00a0have\u00a0work\u00a0to do, however\u00a0we\u2019re\u00a0dedicated to getting it performed and making GitHub dependable when and the place you want it. The precept guiding\u00a0our\u00a0determination is easy:\u00a0availability, then capability, then options.\u00a0<\/p>\n
Thanks to your partnership as we hold constructing GitHub\u2019s reliability and resilience.\u00a0<\/p>\n
In Might, we skilled 9 incidents that resulted in degraded efficiency throughout GitHub providers.<\/p>\n
Might 04 15:45 UTC (lasting 55 minutes)<\/p>\n
On Might 4, 2026, between 15:34 and 16:40 UTC, github.com skilled a service disruption that produced elevated latency and an elevated price of request failures throughout a broad set of customer-facing providers. Whole buyer influence lasted roughly one hour and 6 minutes.\u00a0<\/p>\n
Essentially the most considerably\u00a0impacted\u00a0service was pull requests, which was\u00a0statused\u00a0Purple at some stage in peak influence. Points, actions, webhooks, and Git operations skilled elevated latency and intermittent errors.\u00a0Quite a few\u00a0dependent providers\u2014together with\u00a0Codespaces, Pages, Packages, OAuth and GitHub Apps, Market, and Copilot\u2014additionally noticed various levels of degraded efficiency because of shared information dependencies. At peak, roughly 1.3% of requests returned a 5xx response, averaging round 0.46% throughout the period of the incident.\u00a0<\/p>\n
The disruption was triggered by a routine on-line schema migration working towards a big,\u00a0heavily-accessed\u00a0database desk. The migration had been progressing with out difficulty for a number of hours, however as site visitors\u00a0ramped up\u00a0towards the weekly peak, the mixed load from the migration and regular manufacturing site visitors saturated database connection capability. This produced question competition on a major database and cascading timeouts throughout providers that rely upon it.\u00a0<\/p>\n
The incident was detected inside roughly three minutes of the primary indicators of influence by way of a mix of automated monitoring and on-call statement. As soon as the contributing migration was\u00a0recognized, it was paused, and dependent providers recovered shortly thereafter. Time to mitigation was roughly 33 minutes, and full decision adopted roughly half-hour later\u00a0<\/p>\n
As\u00a0follow-up, we’re implementing a number of enhancements to scale back the probability and blast radius of an identical occasion. Migrations towards massive, high-traffic tables will likely be extra tightly aligned with low-traffic home windows and can use dynamic throttling that adapts to dwell cluster load. We’re including automated circuit breakers that may pause in-flight migrations when latency or connection utilization on the underlying database crosses secure thresholds, and we’re extending our monitoring in order that migration-induced strain (i.e., write price, lock time, and connection saturation) triggers alerts earlier than buyer influence happens. In parallel, we’re reviewing connection-pool capability to make sure satisfactory headroom is\u00a0maintained\u00a0whereas migrations are working.\u00a0\u00a0<\/p>\n
Might 05 13:37 UTC (lasting 3 hours and 49 minutes)<\/p>\n
Might 06 07:19 UTC (lasting 2 hours and 25 minutes)<\/p>\n
On Might 5 and Might 6, GitHub Actions was degraded throughout two associated incidents affecting hosted runners. The 2 occasions have been linked: remediation work carried out after the Might 5 incident launched the configuration\u00a0difficulty that triggered the Might 6 incident.\u00a0<\/p>\n
On Might 5, 2026, from 13:22 to 17:05 UTC, GitHub Actions hosted runners within the East US area have been degraded.\u00a0Roughly\u00a013.5% of jobs requesting a normal runner failed and ~16% of requested bigger runners with personal networking pinned to East US failed or have been delayed by greater than 5 minutes. Copilot code evaluate requests have been additionally\u00a0impacted. Roughly 8,500 code evaluate requests timed out throughout this window. Affected customers noticed an error touch upon their pull requests and have been capable of retry by rerequesting a evaluate. Most runner requests have been picked up by different areas mechanically, however a\u00a0portion\u00a0of requests nonetheless routing to East US have been\u00a0impacted.\u00a0<\/p>\n
This was triggered by a scale-up operation for hosted runner VMs within the East US area. This can be a common operation, however the VM create load hit an inner price restrict when VM creates pull photos from storage. Present backoff logic was not triggered\u00a0due to\u00a0the response code\u00a0returned\u00a0on this case. The speed limiting and VM creation failures have been mitigated by decreasing load to permit for restoration and permitting queued work to be processed. By 15:34 UTC, queued and failed job assignments have been largely mitigated, with lower than 0.5% of runner assignments\u00a0impacted\u00a0between 15:34 and full restoration at 17:05.\u00a0<\/p>\n
On Might 6, 2026, from 06:45 to 09:15 UTC, GitHub Actions Normal Ubuntu hosted runners have been\u00a0once more\u00a0degraded, and roughly 17.1% of jobs requesting a normal runner failed. The difficulty was brought on by surprising configuration information launched throughout remediation work for the day gone by\u2019s incident, which blocked new allocations as each day load ramped up. We eliminated the problematic information at 08:51 UTC, permitting allocations to renew and runner swimming pools to cut back up and get better.\u00a0\u00a0<\/p>\n
We’re bettering our system\u2019s throttling habits when limits happen, bettering our controls to\u00a0extra rapidly mitigate related conditions\u00a0sooner or later, and reviewing all limits end-to-end for related operations.\u00a0As well as, we’re updating the filter logic for this allocation information to be resilient to irregular information shapes and bettering monitoring to alert when allocations are blocked, permitting the workforce to reply earlier than buyer influence begins.\u00a0<\/p>\n
Might 06 11:21 UTC (lasting 38 minutes)<\/p>\n
On Might 6, 2026,\u00a0between 11:02 and 11:13 UTC, customers have been unable to start out or view Copilot cloud agent or distant periods. Throughout this time,\u00a0all\u00a0requests to the session API returned errors, stopping customers from creating new periods or viewing current ones. The difficulty was brought on by a configuration change to the service\u2019s community routing that inadvertently eliminated the ingress path for the service. The workforce reverted the change at 11:13 UTC which restored service. The incident remained open till 11:59 UTC whereas the workforce verified full restoration. We’re taking steps to enhance our deployment validation course of to stop related configuration adjustments from impacting manufacturing site visitors sooner or later.\u00a0<\/p>\n
Might 06 15:25 UTC (lasting 3 hours and 39 minutes)<\/p>\n
On Might 6, 2026,\u00a0between 15:12 and 19:02 UTC,\u00a0creation\u00a0of latest pull request evaluate threads on\u00a0github.com failed. This included new line feedback and file feedback on pull requests. Present\u00a0pull requests and beforehand created feedback have been unaffected.\u00a0<\/p>\n
This incident was brought on by a 32-bit integer key reaching its most worth in a\u00a0Vitess\u00a0lookup desk used throughout\u00a0pull request\u00a0thread creation. The first desk\u00a0was\u00a0migrated to a 64-bit integer key,\u00a0however the\u00a0Vitess\u00a0lookup desk remained 32-bit. As soon as the values within the major desk handed the obtainable 32-bit ID\u00a0house\u00a0makes an attempt\u00a0to create new evaluate threads started failing, leading to\u00a0a\u00a0close to 100% failure price for brand new thread creation requests. We mitigated the difficulty by updating the impacted lookup desk definitions throughout all shards to make use of 64-bit integer column varieties, growing the obtainable ID\u00a0vary,\u00a0and restoring regular operation. Service was absolutely restored as soon as the schema adjustments competed globally.\u00a0<\/p>\n
To assist forestall related incidents, we’re increasing current monitoring of database columns to incorporate\u00a0Vitess\u00a0lookup tables\u00a0and\u00a0enabling earlier detection of any tables approaching a column dimension restrict.\u00a0\u00a0\u00a0\u00a0<\/p>\n
Might 07 05:02 UTC (lasting 1 hour and 54 minutes)<\/p>\n
On Might 7, 2026, between 04:12 and 06:13 UTC,\u00a0Copilot cloud\u00a0agent\u00a0and Copilot code evaluate agent periods for pull requests have been delayed or\u00a0did not\u00a0begin.\u00a0\u00a0\u00a0<\/p>\n
Through the influence\u00a0window, new\u00a0Copilot coding agent periods triggered by pull requests weren’t created, whereas\u00a0evaluate agent periods dropped by ~50% from the baseline.\u00a0\u00a0\u00a0<\/p>\n
The difficulty was brought on by follow-up restoration work from a separate\u00a0pull\u00a0request\u00a0incident.\u00a0As a part of that restoration, we ran a big database migration, which precipitated replication delays on a number of duplicate hosts.\u00a0<\/p>\n
Though these replicas weren’t serving person site visitors, our safeguards accurately handled the elevated replication lag as a sign to decelerate writes to the affected database cluster.\u00a0Because of this, some pull request background processing was briefly delayed.\u00a0That processing\u00a0is answerable for\u00a0sending the inner occasions that Copilot brokers use to start work, so affected brokers didn’t begin till the database replicas caught up.\u00a0<\/p>\n
The system recovered as soon as replication lag returned to regular and pull request processing resumed.\u00a0We’re making crucial background processing extra resilient to replication\u00a0lag\u00a0so it degrades gracefully and retains important occasions flowing below pressure, decreasing the prospect of comparable secondary influence throughout future restoration work.\u00a0<\/p>\n
Might 15 08:13 UTC (lasting 35 minutes)<\/p>\n
On Might 15, 2026, from 07:43 to 08:48 UTC, GitHub Actions skilled a degradation that precipitated workflow runs to fail or\u00a0expertise\u00a0delayed begins for a subset of consumers. The incident was triggered by a deliberate failover of supporting infrastructure utilized by GitHub Actions. Throughout that operation, an automatic service discovery replace didn’t propagate accurately, which precipitated site visitors to be routed incorrectly and elevated request timeouts in a core dependency for workflow orchestration.\u00a0<\/p>\n
At peak influence, 42% of Actions runs failed. Downstream providers that rely upon\u00a0Actions\u00a0workflow execution have been additionally\u00a0impacted, together with GitHub Pages and Copilot cloud providers. At 08:12 UTC, responders manually corrected the service discovery routing difficulty. Timeout and failure charges recovered shortly after, and we continued\u00a0monitoring\u00a0till full stabilization was confirmed throughout all affected providers. The incident was marked resolved at 08:48 UTC.\u00a0<\/p>\n
To\u00a0forestall recurrence, we’re taking a number of steps. First, we’re implementing failover guardrails that\u00a0validate\u00a0service discovery state earlier than finishing failover operations. Second, we’re\u00a0strengthening verification\u00a0checks. Lastly, we’re bettering dependency resilience to scale back timeout cascades throughout infrastructure occasions.\u00a0<\/p>\n
Might 26 10:57 UTC (lasting 2 hours and 21 minutes)<\/p>\n
On Might 26, 2026, between 10:40 and 12:56 UTC, GitHub Actions jobs have been degraded. From 10:40 to 12:16 UTC, all newly queued runs\u00a0did not\u00a0begin. From 12:16 to 12:56 UTC, runs that required downloading actions for his or her workflows continued to fail. GitHub Pages, Copilot\u00a0code\u00a0evaluate, Copilot coding agent,\u00a0Octoshift, and GitHub Enterprise Importer have been additionally\u00a0impacted\u00a0because of their dependency on\u00a0GitHub\u00a0Actions.\u00a0<\/p>\n
This was brought on by our automated account evaluate system incorrectly suspending the service account utilized by GitHub Actions to authenticate workflow runs and obtain actions.\u00a0<\/p>\n
We mitigated by restoring the account at 12:16 UTC, marking it\u00a0exempt\u00a0from additional automated evaluate at 12:20 UTC, and redeploying a associated service at 12:48 UTC to flush cached account state. Full restoration was confirmed at 12:56 UTC.\u00a0<\/p>\n
Throughout this incident, a small variety of\u00a0points,\u00a0pull requests,\u00a0feedback, and\u00a0discussions have been marked as hidden when the service account was disabled. No information was misplaced. All content material hidden due to this incident has been\u00a0restored\u00a0and full search index restoration\u00a0was accomplished.\u00a0<\/p>\n
To stop a recurrence, we’ve added an allowlist of all service accounts that can’t be suspended by automated techniques, and guaranteeing these protections are enforced persistently throughout all account administration tooling. We’re additionally bettering diagnostic tooling for accounts and decreasing cache propagation delays to shorten time to mitigate related incidents sooner or later.\u00a0<\/p>\n
Might 28 19:01 UTC (lasting 1 hour and 40 minutes)<\/p>\n
On Might 28, 2026, between 18:27 and 20:41 UTC, the GitHub Copilot service was degraded because of a problem with the Responses API of an upstream supplier affecting the GPT-5.2, GPT-5.3-Codex, GPT-5.4, and GPT-5.5 fashions. Requests routed to those fashions by way of the Responses API returned elevated error charges, which additionally affected\u00a0Copilot coding agent and Copilot code evaluate. No different fashions have been\u00a0impacted.\u00a0<\/p>\n
We mitigated the incident by shifting site visitors away from the affected fashions whereas the upstream supplier deployed a repair.\u00a0<\/p>\n
GitHub is working to enhance automated failover for the affected fashions and strengthen monitoring to stop related incidents sooner or later.\u00a0<\/p>\n
Observe our standing web page for real-time updates on standing adjustments and post-incident recaps. To be taught extra about what we\u2019re engaged on, try the engineering part on the GitHub Weblog.<\/p>\n