Sorry for the late notice, but we will be using our weekly scheduled maintenance window tonight beginning at 10:00pm PST. We should only be going into read-only mode, and will do our best to keep it short. Please check back here for updates.
22:05 PST – read-only mode begins
00:00 PST – all clear! Tonight we did some minor database maintenance, thanks for your patience!
A critical part of our infrastructure failed this morning and its backup is having issues as well. Our engineers are looking into it and will restore service as quickly as possible. Sorry for the inconvenience.
07:26 PDT – we’ve identified the problem and were able to fix it. now to bring everything else back online, sorry for the delay
07:45 – we’re in read-only, verifying everything is OK before we put everything back
07:50 – sorry, spoke too soon. a few components still need attention before even read-only. we’re working as fast as we can, sorry for the continued delay
10:09 – all clear. sorry about that. we’ve identified several areas of improvement to avoid this in the future.
We had a brief outage starting at 09:25 Pacific, and ending at 09:48, after a database component became bogged down. Unfortunately this came at a shift change, so we were a little slow in addressing it. Sorry for the inconvenience.
Our engineers are investigating an issue, more when info here once we get it. Sorry for the inconvenience.
12:55 PDT – looks like we’re back, sorry again
This evening at approximately 11:30pm Pacific, we need to take the site into read-only mode for an emergency update to our edge routers. It should only take approximately 15 minutes as we implement the necessary setting changes. We will do our best to keep as much functionality available during the maintenance, however it is possible a full outage may occur beyond read-only.
We are confident that this should help prevent the routing instability we saw earlier this morning. Sorry for the inconvenience, please check back here for updates after 10pm.
21:09 PDT – pushing back to 11:30pm tonight
23:57 – still have not been able to start due to a problem with one of the related routers that we can fix in the background. Hope to start shortly after this is complete. Sorry for the continued delay
03:05 – we’ve completed the troubleshooting and will be going into read-only shortly to start/complete the fix.
03:24 – reload complete, staying in read-only for a few more moments to make sure we’re good to go. routers are all happy!
03:38 – back out of read-only. thanks for your patience and sorry for the inconvenience
