Freshpaint - Processing for Some Events is Delayed – Incident details

Processing for Some Events is Delayed

ResolvedDegraded performance3 hours 58 minutes

Affected

Destinations

Degraded performance from 4:14 PM to 6:20 PM, Operational from 6:20 PM to 8:12 PM

Management Console

Degraded performance from 4:14 PM to 8:12 PM

  • ResolvedUTC

    We have verified that delivery of events to destinations is working normally without delays. The number of events which show traces in Event Verification in the Freshpaint UI may remain reduced for some customers as a mitigation for handling the current load.

  • UpdateUTC

    We have scaled our ingest capabilities and destination delivery is now working without delays. The Event Verification feature remains degraded due to the load on tracing services, and some delivered events may fail to appear in event verification in the web UI.

  • IdentifiedUTC

    Delivery of some events is delayed by up to 40 minutes due to degraded performance in event processing caused by increased load. We have increased the scale of our event processing architecture to handle increased load and are actively monitoring performance. The Event Verification section of the management UI is also experiencing some issues as a result of the same processing delays.