Earlier this week we made a code change to more accurately measure the size of incoming events. This code change measured the number of bytes rather than the number of characters, more accurately enforcing our limits of event size.

A mistake was made in the patch that set the single-event size limit to the same value as the limit for a batch of events. This mistake allowed a single event to enter our write path that was too large for some of our internal code paths. The manifestation of the problem was very similar to failures we've seen before due to other symptoms, so we were slow in diagnosing the problem. After examining the write queue we realized the event was too large and quickly found the aforementioned error.

A config change was deployed to our API to correct the error and the bad event was skipped. In addition, the following remediation items will be instituted:

• Our runbooks will be updated to include instructions for on-call to examine items in the queue and to skip bad events
• Our write path code will be modified to drop (with errors) any event over a certain size.

We apologize for the inconvenience. Indexing and making your data available for query quickly is important and thank you for your patience.