It's been a bit of a tough day here at Unbounce, and we know it's been a tough day for some our customers too. So first off I'd like to start by apologizing for the issues we've had today.
It turns out that the root cause of our page server issues was another internal Unbounce service. This internal service was accidentally generating an enormous amount of traffic to our servers in our US East region. The amount and type of requests being sent were not something our page servers are optimized for, which caused them to become overloaded trying to handle these requests.
Once we were able to identify the specific systems we blocked their access and service in the region immediately returned to normal.
In the short term we will be leaving that badly behaving service blocked until the team can fix it. Additionally, we plan to come up with new safe guards on our page servers to prevent similar events like this from affecting page server performance.
If you have any additional questions please reach out to our amazing Customer Support team, or you can email me directly.
Sincerely,
Mike Thorpe
Manager of Infrastructure Operations
Posted Mar 11, 2016 - 15:27 PST
Update
We've added more capacity in the US East region and the US West region. We've added the US East region back into production.
No failures have been reported since 12:50 PDT.
Posted Mar 11, 2016 - 13:26 PST
Update
We've removed the US East region again, as we're seeing increased failure rates again.
Posted Mar 11, 2016 - 12:33 PST
Monitoring
Starting around 10:45am this morning we started seeing intermittent failures to talk to our page server instances on the east coast of North America (Those familiar with AWS, we're talking about us-east-1). Some customers might have seen error pages (503 server errors) if they were located in that region.
After some initial investigation, at 11:03 we decided to remove the entire set of servers in that region from our page servers group. Due to the way our request load balancing works, customers in the us east region would be sent servers in other (potentially further away) regions. This would likely cause slower than normal page load times.
At 11:23 the Eastern region page server instance problem stabilized (we're still unclear why) and we decided to add the region back into service.
Currently we are monitoring everything closely and we are still investigating the root cause. We will hopefully have more to report soon.