So, by now I think everyone in the USA and the rest of the world has heard about the epic failure of a website healthcare.gov was at launch. To be fair I do admit that I understand the complexity of it as it has to tie into several government databases, all of which use different software and formatting. From what I heard, they had to have the website reach into more than 100 of these government databases all over the United States. Furthermore, in true government fashion none of these databases are located in the same data center.
With all these databases distributed all over the place we can only imagine the special kind of hell the website programmers had to go through to get it all working. However, just getting the site working and integrated into all these distributed government databases was easy compared to making it perform well. Google makes it look easy, you go to their page and type some search criteria and with magical speed, results appear. This is not difficult to do when you have all your data located in the same building and then replicate it to many other similar buildings around the country to distribute the traffic load.
By having healthcare.gov separated from the databases that it needed to operate created major performance problems for the site. I like to call this “Call Latency” this is the amount of time required to make a data request to a remote database and have it reply back. When working with a high traffic website keeping the call latency below 20 – 30ms is critical to the transaction performance of your website. A simple rule of thumb is, “The lower the better”. You want to keep your call latency as low as possible, thus allowing your website to process more transaction a second, thereby enabling a smother user experience. You don’t want the users of your website to have to wait because it is taking up to 20 – 30 seconds to process a single transaction, as was the case of the Healthcare.gov website.
Distributed data centers are a fact of life for high traffic websites, traffic load is sheared between these data centers and if one of the data centers goes down the others pick up the load. The healthcare.gov site was only located in one data center, and due to Murphy’s Law that data center had a hardware failure that brought down the website for several hours. Let this be a lesson to all you developers out there, don’t put all your eggs in one basket. Make sure your host site uses distributed data centers and has dynamic resource scaling to handle any spikes in traffic.
Anyway, I have found this whole thing rather funny in a sad sort of way. There are lessons to be taken from this and in the future when someone asks you to build a mission critical website for them it would be wise to remember the disaster that was the Healthcare.gov rollout.
Bibliography
Gallagher, S. (2013, 11 01). Just six people got insurance through HealthCare.gov on day one. Retrieved from ars technica: http://arstechnica.com/information-technology/2013/11/just-six-people-got-insurance-through-healthcare-gov-on-day-one/
Gallagher, S. (2013, 10 29). The seven deadly sins of HealthCare.gov. Retrieved from ars technica: http://arstechnica.com/information-technology/2013/10/the-seven-deadly-sins-of-healthcare-gov/
Kennedy, K. (2013, 10 25). Zients: HealthCare.gov problems to be fixed in a month. Retrieved from USA TODAY: http://www.usatoday.com/story/news/nation/2013/10/25/zients-update-healthcaregov-website/3187867/
Kliff, S. (2013, 10 22). These two paragraphs say everything about HealthCare.Gov’s problems. Retrieved from WashingtonPost: http://www.washingtonpost.com/blogs/wonkblog/wp/2013/10/22/these-two-paragraphs-say-everything-about-healthcare-govs-problems/
