Clockwork Recruiting Status Page Status Page

Mar 22, 2023 · resolved Mar 22

Incident Details: Background job slowdown, hence delay in notification, indexing, reports and exports processing. Root Cause Analysis: There was a data export request by a firm to export all their people data (around 200K). For every such request we create a background job, with the list of people ids to be exported. Furthermore, due to the large size of the job (200k * 40 =~ 8MB), this also caused a slow down in the fetching of other records from the background jobs table (Database sort buffer getting filled up). This resulted in other job workers, i.e., ones for indexing, notifications etc. to slow down as well. Immediate Resolution: - Existing problematic export jobs were stashed, extra workers spawned to clean up the backlog. - A limit of 50K records was added for exporting of data. - Excel export and csv job separated into two so that things can be simplified. - Checks added to prevent creation of unnecessary jobs - Index added on background jobs table for faster access within a queue, to limit the impact of such huge jobs on other background processing.

Mar 3, 2023 · resolved Mar 3

Maintenance Patched

Oct 20, 2022 · resolved Oct 20

_On the morning of October 20 Clockwork views and application feature settings reverted to a non setting state for approximately 2 hours. This is the first time we’ve had any feature flag reversion. We immediately discovered the issue and remediated it as quickly as possible. We understand this was a significant inconvenience that we take very seriously. Below we have shared our postmortem and our actions to prevent this from occurring in the future._ * **Incident Details** All the feature flags stored in Redis were wiped away causing users to see old Clockwork views and hence users were not able to find certain functionalities. * **Investigation Summary** The team reviewed the code and the Redis Server configuration. Identified that the same Redis instance is shared between Demo and Production environments. The team worked on restoring the data from backup Redis servers. * **Data Exposure Summary** No, data was exposed. UI settings data stored in Redis was deleted, but has been recovered from the backup. * **Remediation Summary**  Review Feature Flagging System, use separate Redis namespace for Demo.  Upgrade to Redis 6.x and use of better security features and ACL, current version is 5.0. Setup Redis cluster with multi-AZ replication.

Jul 25, 2022 · resolved Jul 25

We pushed out a big release over the weekend. This incident has been resolved.

Jun 19, 2022 · resolved Jun 19

Planned update of available servers to core Clockwork application. Application downtime limited to 15 minutes during migration.