The Wayback Machine - https://web.archive.org/web/20240927022820/https://www.githubstatus.com/
GitHub header
All Systems Operational
Git Operations ? Operational
Webhooks ? Operational
Visit www.githubstatus.com for more information Operational
API Requests ? Operational
Issues ? Operational
Pull Requests ? Operational
Actions ? Operational
Packages ? Operational
Pages ? Operational
Codespaces ? Operational
Copilot Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Sep 27, 2024

No incidents reported today.

Sep 26, 2024
Degraded performance for some Copilot users
Resolved - This incident has been resolved.
Sep 26, 05:08 UTC
Update - Monitors continue to see improvements. We are declaring full recovery.
Sep 26, 05:08 UTC
Update - Copilot is operating normally.
Sep 26, 05:03 UTC
Update - We've applied a mitigation to fix the issues and are seeing improvements in telemetry. We are monitoring for full recovery.
Sep 26, 03:51 UTC
Update - We believe we have identified the root cause of the issue and are monitoring to ensure the problem does not recur.
Sep 26, 02:34 UTC
Update - We are continuing to investigate the root cause of the latency previously observed to ensure there is no reoccurrence, and better stability going forward.
Sep 26, 01:46 UTC
Update - We are continuing to investigate the root cause of the latency previously observed to ensure there is no reoccurrence, and better stability going forward.
Sep 26, 01:03 UTC
Update - Copilot users should no longer see request failures. We are still investigating the root cause of the issue to ensure that the experience will remain uninterrupted.
Sep 26, 00:29 UTC
Update - We are seeing recovery for requests to Copilot API in affected regions, and are continuing to investigate to ensure the experience remains stable.
Sep 25, 23:55 UTC
Update - We have noticed a degradation in performance of Copilot API in some regions. This may result in latency or failed responses to requests to Copilot. We are investigating mitigation options.
Sep 25, 23:40 UTC
Investigating - We are investigating reports of degraded performance for Copilot
Sep 25, 23:39 UTC
Sep 25, 2024
Incident with Actions Runs
Resolved - This incident has been resolved.
Sep 25, 19:19 UTC
Update - We're seeing issues related to Actions runs failing to download actions at the start of a job. We're investigating the cause and working on mitigations for customers impacted by this issue.
Sep 25, 19:14 UTC
Investigating - We are investigating reports of degraded performance for Actions and Pages
Sep 25, 19:11 UTC
Incident with Git Operations
Resolved - On September 25, 2024 from 14:31 UTC to 15:06 UTC the Git Operations service experienced a degradation, leading to 1,381,993 failed git operations. The overall error rate during this period was 4.2%, with a peak error rate of 12.5%.

The root cause was traced to a bug in a build script for a component that runs on the file servers that host git repository data. The build script incurred an error that did not cause the overall build process to fail, resulting in a faulty set of artifacts being deployed to production.

To mitigate the impact, we rolled back the affecting deployment.

To prevent further occurrences of this cause in the future, we will be addressing the underlying cause of the ignored build failure and improving metrics and alerting for the resulting production failure scenarios.
Sep 25, 16:03 UTC
Update - We are investigating reports of issues with both Actions and Packages, related to a brief period of time where specific Git Operations were failing. We will continue to keep users updated on progress towards mitigation.
Sep 25, 15:34 UTC
Investigating - We are investigating reports of degraded performance for Git Operations
Sep 25, 15:25 UTC
Sep 24, 2024
Incident with Codespaces start and creation
Resolved - This incident has been resolved.
Sep 24, 21:04 UTC
Update - Codespaces is operating normally.
Sep 24, 21:04 UTC
Update - We have successfully mitigated the issue affecting create and resume requests for Codespaces. Early signs of recovery are being observed in the impacted region.
Sep 24, 21:01 UTC
Update - Codespaces is experiencing degraded performance. We are continuing to investigate.
Sep 24, 21:00 UTC
Update - We are investigating issues with Codespaces in the US East geographic area. Some users may not be able to create or start their Codespaces at this time. We will update you on mitigation progress.
Sep 24, 20:56 UTC
Investigating - We are investigating reports of degraded availability for Codespaces
Sep 24, 20:54 UTC
Sep 23, 2024

No incidents reported.

Sep 22, 2024

No incidents reported.

Sep 21, 2024

No incidents reported.

Sep 20, 2024

No incidents reported.

Sep 19, 2024

No incidents reported.

Sep 18, 2024

No incidents reported.

Sep 17, 2024

No incidents reported.

Sep 16, 2024
Incident with Pages and Actions
Resolved - On September 16, 2024, between 21:11 UTC and 22:20 UTC, Actions and Pages services were degraded. Customers who deploy Pages from a source branch experienced delayed runs. Approximately 1,100 runs were delayed long enough to get marked as abandoned. The runs that weren't abandoned completed successfully after we recovered from the incident. Actions jobs experienced average delays of 23 minutes, with some jobs experiencing delays as high as 45 minutes. During the course of the incident, 17% of runs were delayed by more than 5 minutes. At peak, as many as 80% of runs experienced delays exceeding 5 minutes. The root cause was a misconfiguration in the service that manages runner connections, which caused CPU throttling and led to a performance degradation in that service.

We mitigated the incident by diverting runner connections away from the misconfigured nodes. We are working to improve our internal monitoring and alerting to reduce our time to detection and mitigation of issues like this one in the future.
Sep 16, 22:08 UTC
Update - Actions is experiencing degraded performance. We are continuing to investigate.
Sep 16, 21:55 UTC
Update - The team is investigating issues with some Actions jobs being queued for a long time and a percentage of jobs failing. A mitigation has been applied and jobs are starting to recover.
Sep 16, 21:53 UTC
Update - Pages is operating normally.
Sep 16, 21:52 UTC
Update - Actions is experiencing degraded availability. We are continuing to investigate.
Sep 16, 21:37 UTC
Investigating - We are investigating reports of degraded performance for Actions and Pages
Sep 16, 21:31 UTC
Disruption with Git SSH
Resolved - On September 16, 2024, between 13:24 UTC and 14:28 UTC, the Git Operations service experienced a degradation, leading to intermittent SSH connection drops. The overall SSH error rate during this period was 0.0005%, with a peak error rate of 0.3%.

The root cause was traced to a regression in the service reload mechanism, which resulted in SSH hosts dropping connections on an hourly basis. As SSH hosts were rebooted for routine security updates, the issue progressively affected more hosts.

To mitigate the impact, we removed the affected hosts from production traffic. The SSH regression has since been identified and resolved, with all SSH hosts fully restored. Additionally, we have implemented new monitoring to alert us of any SSH connection refusals moving forward.
Sep 16, 14:28 UTC
Update - We are no longer seeing dropped Git SSH connections and believe we have mitigated the incident. We are continuing to monitor and investigate to prevent reoccurrence.
Sep 16, 14:27 UTC
Update - We have taken suspected hosts out of rotation and have not seen any impact in the last 20 minutes. We are continuing to monitor to ensure the problem is resolved and are investigating the cause.
Sep 16, 14:11 UTC
Update - We are seeing up to 2% of Git SSH connections failing.

We have taken suspected problematic hosts out of rotation and are monitoring for recovery and continuing to investigate.
Sep 16, 13:38 UTC
Update - We are investigating failed connections for Git SSH. Customers may be experiencing failed SSH connections both in CI and interactively. Retrying the connection may be successful. Git HTTP connections appear to be unaffected.
Sep 16, 13:30 UTC
Investigating - We are currently investigating this issue.
Sep 16, 13:29 UTC
Sep 15, 2024

No incidents reported.

Sep 14, 2024
Incident with Pull Requests
Resolved - This incident has been resolved.
Sep 14, 22:43 UTC
Update - Pull Requests is operating normally.
Sep 14, 22:43 UTC
Update - we believe we have mitigated and are confirming recovery.
Sep 14, 22:41 UTC
Investigating - We are investigating reports of degraded performance for Pull Requests
Sep 14, 22:10 UTC
Sep 13, 2024
Processing delays to some Issues, Pull Requests and Webhooks
Resolved - On Sep 13, 2024, between 05:03 UTC and 07:13 UTC, the Webhooks and Actions services were degraded resulting in some customers experiencing delayed processing of Webhooks and Actions Runs. 0.5% of Webhook deliveries were delayed more than 2 minutes during the incident. 15% of Actions Runs started between 05:03 and 05:24 UTC saw run start delays or failures. At 05:24 UTC, we implemented a mitigation to shift traffic to healthy infrastructure and new Actions Runs resumed normal operations. During the rest of the incident window, Actions runs started before 05:24 UTC continued to see delays publishing logs or job results. No Actions runs or Webhook deliveries were lost, only delayed.

We mitigated the incident by immediately shifting traffic to a healthy cluster while investigating. The incident was caused by an erroneous configuration change on our eventing platform. A permanent fix was deployed at 06:22 UTC after which services began to recover and burn down their backed up queues, with full recovery by 07:13 UTC.

We are working to reduce our time to detection and develop test automation to prevent issues like this one in the future.
Sep 13, 07:13 UTC
Update - We are seeing improvements in telemetry and are monitoring the delivery of delayed Webhooks and Actions job statuses.
Sep 13, 06:49 UTC
Update - We've applied a mitigation to fix the issues being experienced in some cases with delays to webhook deliveries, and the delayed reporting of the outcome of some running Actions jobs. We are monitoring for full recovery.
Sep 13, 06:23 UTC
Update - Actions is experiencing degraded performance. We are continuing to investigate.
Sep 13, 05:59 UTC
Investigating - We are investigating reports of degraded performance for Issues, Pull Requests and Webhooks
Sep 13, 05:42 UTC
← Incident History Powered by Atlassian Statuspage