This morning one of our write queue machines timed out when contacting our coordination system and our write-path paused. This created a backlog that needed to be worked down after a manual restart of the write path. They delay was short because the duration of the incident was small.
We're actively working on upgrades to our write path's code to be more tolerant of coordination failure and expect to deploy fixes to that in the next 30 days.
Posted Feb 04, 2015 - 09:03 PST
Resolved
We have completed working through the small backlog of delayed events and all events are available for querying.
Posted Feb 04, 2015 - 09:01 PST
Update
We have begun working through the backlog of events, we will resolve this incident when the backlog is complete.
Posted Feb 04, 2015 - 08:58 PST
Identified
We currently have a small delay in writing events and making them available for querying. The backlog will be properly handled soon and events will be available.