packagecloud.io Status Page

Nov 26, 2025 · resolved Nov 26

# Summary At 09:52AM UTC, we received the first customer report indicating HTTP 500 errors during Debian package uploads. Additional reports followed throughout the day, confirming that multiple package types and customers were affected. After an internal investigation, the issue was declared an incident at 18:19PM UTC. The underlying cause was found to be insufficient disk space on a subset of ECS tasks responsible for handling file uploads. These tasks had reached their disk capacity, preventing them from completing upload operations. Because the tasks were running at maximum disk usage, they also could not be automatically cycled out by ECS, allowing the problem to persist. During the investigation we observed an increase in device maximum capacity errors, and inspection of the ECS cluster revealed tasks of varying ages. This correlated with the accumulation of temporary files generated during package extraction and processing. Although temporary file cleanup mechanisms exist, they did not adequately prevent disk saturation. As a short-term remediation, a deploy was triggered to refresh all ECS tasks, which replaced the existing tasks with new ones. Following this deployment, upload functionality stabilized and no further customer impacting errors were observed. The issue was considered mitigated at 20:19PM UTC. Subsequent analysis confirmed that temporary file usage was a significant contributor to disk exhaustion. Additional improvements were implemented to ensure temporary files are consistently closed and cleaned up, regardless of whether errors occur during processing. ## Changes we’re making: To reduce the likelihood of similar incidents in the future, we are implementing the following long-term improvements: * Strengthening disk usage observability: Enhanced monitoring now provides realtime insight into ECS task disk consumption, enabling earlier detection of abnormal usage. * Improving temporary file lifecycle management: Further audit and cleanup of temporary file handling across the codebase is underway to minimize unnecessary disk usage. * Refining ECS task lifecycle practices: We are reviewing our deployment and recycling strategy to ensure tasks do not remain active long enough to accumulate excessive temporary data. * Enhancing health checks: We are evaluating load balancer health check behavior to ensure that resource-degraded tasks \(including those with insufficient disk\) are identified and replaced more promptly.

Oct 24, 2025 · resolved Oct 24

## Service Impact On October 24th, 2025 from approximately 01:53AM UTC until 04:24AM UTC our package download service experienced an outage affecting customer access. This was confirmed to have impacted Debian and RubyGems. ## Incident Summary At 01:13AM UTC we updated the S3 Bucket Policy for an S3 bucket containing customer packages via Terraform as part of a larger re-architecting project. We received customer reports about Debian package download issues from 01:53AM UTC and started doing isolated, per-customer investigations. At 03:40AM UTC we concluded that the issues were not isolated and declared the incident. More members from our team were called to help investigate the problem, where we found that service for pulling RubyGems had also been impacted. Around 04:00 UTC, the new bucket policy deployed at 01:13AM UTC was identified as a main factor to the access failure and was removed. At 04:19AM UTC our team identified that updating the IAM policy via Terraform did not have the expected result of merging with the pre-existing policy, but rather inadvertently overwrote the pre-existing access policies, causing the loss of access permissions necessary for package downloads. Afterwards it became apparent to our team making these changes that this particular behavior is a [known issue](https://github.com/hashicorp/terraform-provider-aws/issues/6334) with the Terraform AWS provider, which contributed to the difficulty of detection and prevention, thereby increasing the likelihood of the incident occurring for those unaware. After redeploying the pre-existing S3 bucket IAM policy, we confirmed service functionality for Debian package downloads was restored at 04:24 AM UTC. The team subsequently verified that pulling RubyGems and other supported package types were also functioning as expected, and the incident was closed at 04:47 AM UTC. ## Changes we're making We are revising our monitoring approaches to identify and address service disruptions proactively, while also improving our communication processes to ensure timely and accurate customer updates.

Jan 31, 2024 · resolved Jan 31

The website continues to be available, and we're confident the root cause has been addressed.

Jan 31, 2024 · resolved Jan 31

This incident has been resolved.

Dec 21, 2023 · resolved Dec 21

The incident has been resolved.