We're back up and running. Our internal load balancer (HAProxy) got stuck in an ornery state, and it took us a while to realize it was the load balancer instead of the actual services causing the errors. Sorry for the intermittent failures!
Posted Mar 31, 2015 - 16:07 PDT
Monitoring
We identified the root cause in our infrastructure management setup that caused the 503 errors in one of our data centers. We are monitoring, but believe the problem is resolved.
Posted Mar 31, 2015 - 15:57 PDT
Update
The team is at work continuing to resolve this issue!
Posted Mar 31, 2015 - 15:47 PDT
Investigating
We're still seeing 503 errors for some customers and are digging in.
Posted Mar 31, 2015 - 15:01 PDT
Monitoring
We've deployed the internal query service and things look back to normal. Sorry for the inconvenience!
Posted Mar 31, 2015 - 14:51 PDT
Investigating
Some users are seeing query error messages saying the query service is overloaded