![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
single point of failure
Apr. 9th, 2009 09:35 pmHow odd to wake up and find no internet. How much odder to see the cell phone indicating no service.
With that as context, somewhat less odd to get downtown, pop open the laptop right next to the ISP, and find that their own webserver was unavailable from their own central office.
So then, not at all odd to get to the office and find that the internet outage covered three counties.
That's when it got surreal. Nobody dares risk losing observing time on the biggest telescope in the world. The remote observing room is equipped with ISDN lines to serve as backup in case the internet goes away. We enabled the dialer -- and got nothing. Land line long distance service was gone too.
During the day the MTA started reporting the failed mail from the cron jobs. It was an odd pattern, for not everything had totally gone at once. E-mail to domains which still had the IP address in the DNS cache had been finding its way for a few hours after the initial incident, but as time went by more and more domains expired. The mail bounces paint a picture like a vigil watching an elderly relative going through progressive organ failure.
Then to leave the office and get to Trader Joe's and find that retailers could not perform credit or debit transactions. I wrote a check for groceries.
By the time I got home the girls informed me that the DSL had come back a few hours earlier.
That seemed hopeful, except that during the outage various bright folks had redirected the web traffic to another site so that moms and dads wouldn't panic about their kids. The substitute web page is still there, and traceroute indicates CENIC has been told that subnet is now coming from Los Angeles. But our web server? No, that one did not rate being rerouted, and it also did not rate being left alone.
Most of the internet is back, except for us. We don't exist.
With that as context, somewhat less odd to get downtown, pop open the laptop right next to the ISP, and find that their own webserver was unavailable from their own central office.
So then, not at all odd to get to the office and find that the internet outage covered three counties.
That's when it got surreal. Nobody dares risk losing observing time on the biggest telescope in the world. The remote observing room is equipped with ISDN lines to serve as backup in case the internet goes away. We enabled the dialer -- and got nothing. Land line long distance service was gone too.
During the day the MTA started reporting the failed mail from the cron jobs. It was an odd pattern, for not everything had totally gone at once. E-mail to domains which still had the IP address in the DNS cache had been finding its way for a few hours after the initial incident, but as time went by more and more domains expired. The mail bounces paint a picture like a vigil watching an elderly relative going through progressive organ failure.
Then to leave the office and get to Trader Joe's and find that retailers could not perform credit or debit transactions. I wrote a check for groceries.
By the time I got home the girls informed me that the DSL had come back a few hours earlier.
That seemed hopeful, except that during the outage various bright folks had redirected the web traffic to another site so that moms and dads wouldn't panic about their kids. The substitute web page is still there, and traceroute indicates CENIC has been told that subnet is now coming from Los Angeles. But our web server? No, that one did not rate being rerouted, and it also did not rate being left alone.
Most of the internet is back, except for us. We don't exist.
