The maintenance finished normally over night and all our systems are functioning normally now.
Posted Dec 03, 2013 - 09:01 PST
Update
The event indexer is now up-to date. The maintenance is still running so we're going to keep monitoring this closely. We will update this if anything changes and when the maintenance finishes.
Posted Dec 02, 2013 - 22:35 PST
Update
The indexer is taking longer than expected due to a much higher than normal load caused by a one-off maintenance job that's currently running. We're working on steps to mitigate the effects of this maintenance.
Posted Dec 02, 2013 - 20:08 PST
Monitoring
The lag seems to have been caused by a slow database node. We've restarted the node and the indexer is now catching up. We estimate our systems will be back to normal in a few hours. We will update this if something changes or if this work finishes.
We're looking into what happened with the database node, but have no definitive answers there yet.
Posted Dec 02, 2013 - 18:08 PST
Investigating
The indexer for our new architecture is behind right now. Incoming event data is safe but may not be available for queries immediately.