Lucidworks Platform Status Page

May 28, 2026 · resolved May 28

The service disruption affecting certain Lucidworks AI hosted models has been fully resolved, and all previously affected services are operating normally. End-user functionality is fully restored, with all dependent services operating as expected. The disruption originated during routine Kubernetes maintenance involving node upgrades, during which a cloud provider capacity shortfall prevented the instances hosting these models from re-launching. Stability was restored after successfully launching new nodes and pulling the required images. We will share a postmortem report containing the full root cause analysis within three business days.

May 12, 2026 · resolved May 13

## Summary On May 12, 2026, at 21:19 UTC, the Lucidworks SaaS Platform experienced an access disruption. Users encountered errors when attempting to use the Lucidworks Platform UI \([platform.lucidworks.com](http://platform.lucidworks.com)\). Other SaaS Platform services, including all search and Lucidworks AI functionality, were not affected. Lucidworks Engineering identified the issue at 21:25 UTC and began remediation efforts. The platform was fully restored and operational at 23:41 UTC. ## Root Cause The incident was caused by a database schema change that was included alongside an application code update. The Lucidworks Platform runs multiple instances of each application simultaneously, and new code is rolled out incrementally. New application instances are brought online while existing instances continue serving traffic. In this case, the database schema was updated before all application instances had received the new code, causing existing instances to query a column that no longer existed.The schema change removed a database column before all running instances of the application had been updated to use the new schema. During the rollout window, existing application instances attempted to query the removed column, resulting in errors that prevented users from accessing the platform. Lucidworks Engineering deployed a fix that restored the affected database column and repopulated the necessary data from the most recently-taken database backup. The incident was verified as resolved at 23:41 UTC. ## Lucidworks Actions Lucidworks will take the following actions as a result of this incident: * Enforce phased rollout procedures for database schema changes, ensuring destructive modifications are deployed separately from application code updates to maintain backward compatibility through the rollout window * Update automated code review tooling to detect and flag destructive database operations, adding an additional layer of defense during the review process * Evaluate automated incident creation from critical alerts to reduce the time between detection and formal incident response The Lucidworks engineering team is committed to ensuring this type of incident does not recur. These enhancements will strengthen the platform's resilience and reliability for all customers. ## Recommended Client Actions Lucidworks recommends that clients subscribe to Lucidworks status updates to receive real-time notifications about Lucidworks SaaS Platform incidents. To enable this feature, click **Subscribe to Updates** at [status.lucidworks.com](http://status.lucidworks.com).

Apr 24, 2026 · resolved Apr 24

## Summary On April 24, 2026, at 16:05 UTC, the Lucidworks SaaS Platform experienced an issue that made the entire platform temporarily unavailable and all requests returned IO errors. Lucidworks Engineering was made aware of the issue and reverted the change on April 24, 2026, at 16:08 UTC. The reversion took a few minutes to propagate through the system and successful responses were restored by April 24, 2026, at 16:13 UTC. ## Root Cause The incident was caused by a configuration change to the SaaS Platform gateway components that was deployed to production at 16:05 UTC on April 24, 2026. This change introduced an error in the configuration template that caused it to render syntactically invalid output in the production environment. When the deployment system processed this invalid configuration, it interpreted the malformed output as an indication that the gateway components \(routing and load-balancing infrastructure\) were no longer needed and removed these components from the platform. This caused all incoming requests to fail with IO errors as there were no gateway instances available to handle traffic. Lucidworks engineers were actively monitoring this deployment as it rolled out, and immediately noticed the problem that the deployment introduced. The configuration change was quickly reverted at 16:08 UTC. The deployment infrastructure then correctly reinstated the gateway components, which became fully operational by 16:13 UTC and restored normal platform operation. ## Lucidworks Actions Lucidworks has taken the following actions as a result of this incident: * Added pre-deployment validation to verify that configuration templates render syntactically valid output before deployment, and that changes of this nature are deployed to a development environment before deploying the changes to production * Modified the deployment workflow to require additional verification steps before pull requests can be merged ## Recommended Client Actions Lucidworks recommends that clients subscribe to Lucidworks status updates to receive real-time notifications about Lucidworks SaaS Platform incidents. To enable this feature, click **Subscribe to Updates** at [status.lucidworks.com](http://status.lucidworks.com).

Mar 6, 2026 · resolved Mar 6

## Summary On March 6, 2026, at 05:05 UTC, the Lucidworks SaaS Platform experienced a user interface disruption. Users were consistently redirected to a 404 error page when attempting to access [platform.lucidworks.com](http://platform.lucidworks.com). Other SaaS Platform services, including all search and LWAI functionality, were not affected. Lucidworks Engineering became aware of the issue at 05:40 UTC and the inaccessibility of the user interface was addressed by 05:45 UTC by reverting the most recently-deployed UI change. At 06:13 UTC, Lucidworks Engineering confirmed that access to the user interface had been fully restored and the platform was operational in its entirety. ## Root Cause The incident was caused by a configuration change to one of our currently in-development UI services that inadvertently affected routing across the entire platform. This change was carried out at 04:57 UTC. Following the code rollout, Lucidworks Engineering became aware that the SaaS Platform user interface was unavailable at 05:40 UTC during routine verification of this deployment. The deployment configuration setting for that UI service was left blank, which caused the service to claim routing priority for all platform traffic instead of only its intended path. The resulting global path misconfiguration caused the platform to become unavailable and redirected users to a 404 error page. Reverting this change restored the SaaS Platform to its previous configuration. The incident was verified as resolved at 06:13 UTC. ## Lucidworks Actions Lucidworks will take the following actions as a result of this incident: * Establish clear guidelines for infrastructure-sensitive changes * Include more explicit warnings to configuration files for critical settings and update internal documentation to highlight high-risk configuration areas Additionally, Lucidworks will take the following actions to further enhance our ability to detect, withstand, and respond to similar incidents in the future: * Enhance automated pre-deployment testing by integrating validation tools that run before any deployment to ensure errors are visible to engineering teams earlier in the process * Enhance real-time monitoring for routing anomalies, adding alerts for unexpected traffic patterns, and creating dashboards to provide visibility into platform routing health The Lucidworks engineering team is committed to ensuring this type of incident does not recur. Lucidworks is implementing multiple layers of prevention, detection, and response improvements. These enhancements will strengthen our platform's resilience and reliability for all customers. ## Recommended Client Actions Lucidworks recommends that clients subscribe to Lucidworks status updates to receive real-time notifications about Lucidworks SaaS Platform incidents. To enable this feature, click **Subscribe to Updates** on [status.lucidworks.com](http://status.lucidworks.com).

Dec 3, 2025 · resolved Dec 3

## Summary On December 3, 2025, at 13:30 UTC, the Lucidworks SaaS Platform experienced widespread service disruptions affecting multiple clients. This issue affected Lucidworks AI functionality \(including Neural Hybrid Search\), Commerce Studio and Analytics Studio accessibility, and Connected Search. API requests received HTTP `401` and `403` errors with `invalid or expired token` messages, and the [platform.lucidworks.com](http://platform.lucidworks.com) UI became inaccessible. During this time, Lucidworks infrastructure also experienced a high volume of malicious traffic from external sources. Lucidworks Engineering resolved the issue by 22:03 UTC on December 3, 2025, restoring service for all affected products. A similar issue occurred on December 5, 2025, beginning at 16:17 UTC. Following further coordination with our third-party identity provider \(explained in detail below\), this repeat incident was resolved by 19:00 UTC that day. ## Root Cause The incident was caused by a combination of three interconnected issues that occurred simultaneously on December 3, 2025. Okta, Lucidworks' identity provider for Platform authentication, [experienced a service outage](https://status.okta.com/#incident/a9CWR00000012EI2AY), and in response implemented aggressive rate-limiting measures. During this period, Lucidworks Platform IP addresses were blocked by Okta for an extended duration, preventing the Lucidworks authentication system from retrieving JSON Web Key Sets \(JWKS\) from Okta's servers. This resulted in HTTP `401` `invalid/expired` token errors for authentication requests. Concurrent with the Okta outage, Lucidworks Platform infrastructure experienced a distributed denial of service \(DDoS\) attack in the form of a high rate of malicious requests from multiple geographic sources. The attack traffic significantly exceeded Okta's rate limit thresholds and contributed to Okta's decision to block the Lucidworks Platform IP addresses. Additionally, for some Lucidworks Search customers, [a configuration flag \(failOnError\)](https://doc.lucidworks.com/docs/5.9/fusion/reference/config-ref/pipeline-stages/query-stages/lwai-vectorize-query#fail-on-error) in the Core Package vectorization stage was inadvertently left enabled in the Production environment following earlier debugging activities. When the Okta authentication issue prevented access to Lucidworks AI services, queries using Neural Hybrid Search failed completely rather than falling back to lexical-only search, amplifying the impact for clients using Neural Hybrid Search functionality. Lucidworks determined the root causes through analysis of Okta service status, review of authentication logs showing JWKS retrieval failures, traffic analysis identifying the DDoS attack pattern, coordination with Okta support, and code review of affected query pipeline configurations. Changing the Lucidworks Platform IP address immediately restored authentication functionality, confirming that Okta's IP blocking was the primary cause of the authentication failures. Two days later, on December 5, 2025, the Lucidworks Platform again began to experience `401` invalid/expired token errors for authentication requests. We again had to coordinate with Okta support to update their IP address allowlist in order to ensure our traffic was not blocked or unnecessarily rate-limited. ## Lucidworks Actions Lucidworks has taken the following actions as a result of this incident: * Implemented enhanced rate limiting and DDoS protection and firewall rules to defend against future attacks * Modified affected client query pipelines to disable the `failOnError` flag and enable automatic lexical-only fallback when vectorization services are unavailable * Changed the Lucidworks Platform IP address to restore authentication services and coordinated with Okta to maintain proper IP allowlisting * Escalated with Okta support to resolve the IP blocking issue and ensure continuity of authentication services Additionally, we intend to take the following actions to further enhance our ability to detect, withstand, and respond to similar incidents in the future: * Ensure debug configuration flags are removed from all Production query pipelines to prevent similar failures during future service disruptions * Implement caching mechanisms for JWKS to provide fallback authentication capabilities during Okta service disruptions * Enhance our firewall’s adaptive protection capabilities to more quickly detect and mitigate DDoS attacks * Implement improved synthetic monitoring with automated incident detection to reduce time to detection for similar issues * Establish a process to proactively notify Okta when Lucidworks Platform IP addresses change to maintain current allowlist configurations ## Recommended Client Actions Lucidworks recommends that all clients using Neural Hybrid Search upgrade to Fusion 5.9.15 or later as soon as possible. This version includes enhanced failsafe fallback mechanisms in the Neural Hybrid Query stage that automatically switch to lexical-only queries when Lucidworks AI services are unreachable, improving overall system resilience during service disruptions. Lucidworks also recommends that clients subscribe to Lucidworks status updates to receive real-time notifications about Lucidworks SaaS Platform incidents. To enable this feature, click **Subscribe to Updates** on [status.lucidworks.com](http://status.lucidworks.com).