Event write processing is delayed
Incident Report for Keen
Postmortem

We experienced an unexpected burst of event traffic this morning from approximately 9:00-11:30 am PDT. Normally we operate with enough capacity to handle these types of spikes, but in this case we started to fall behind. We have identified two different factors that seem to have contributed to the apparent reduction in peak throughput. One is related to a de-duping mechanism (a process which ensures we don't save the same event twice), and the other is related to proper even distribution of incoming events in our queue layer. We are working to investigate both of these and hopefully ensure a higher ceiling for future bursts. We are also in contact with the customers involved in the traffic burst to try and better coordinate on future usage.

Posted Oct 25, 2016 - 14:09 PDT

Resolved
Event writes are caught up and processing normally again. Thanks for your patience!
Posted Oct 25, 2016 - 13:53 PDT
Identified
We've identified the cause of the backup and are making progress on clearing the backlog.
Posted Oct 25, 2016 - 11:48 PDT
Investigating
We are experiencing a delay in event writes being processed. No data has been lost. We're working to catch things back up as quickly as possible. Thanks for your patience.
Posted Oct 25, 2016 - 10:27 PDT
This incident affected: Stream API.