WBSC: scalable highly-available web architecture

How we turned a WordPress website into a Cloud distributed application able to handle millions of HTTP requests per hour

WBSC

WBSC is the World Baseball Softball Confederation, the world's leading authority for Baseball and Softball, affiliated with the International Olympic Committee. WBSC has 208 associated national federations in 141 different countries.

The challenge

WBSC's website is the main source of information for most portals dedicated to the sport of Baseball, for international news agencies and specialist publications for all news related to international competitions and tournaments. For this reason, traffic on WBSC's website can spike rapidly, especially during tournaments or events.

It isn't rare for the number or users on the website to increase thousand-folds when posts are shared on social media, or when commercials about an international tournament or competition airs anywhere in the world.

WBSC's website is translated in five languages including Japanese, Korean and Chinese and its audience is really global. Apart from publishing news daily, the website also hosts important information like international rulings and regulations which are used on a daily basis all across the world; most importantly, the website hold all of the statistics relative to international teams and players, scorings and game data updated in real time during competitions. For this reason, the website isn't just a reference for fans or staff affiliated with its many associated national federations; it is also one of the main tools of the job for a plethora of editors, publishers and translators.

We built all of the Cloud infrastructure upon which WBSC's website resides; the site was made from the ground up by the design & digital studio VENTIE30, our partner in matters related to web and communication.

The project's key requirements were numerous and challenging; specifically, the website has to be secure, fast and always online with all of its functionality intact both for visitors and staff. Also, this website has a number of developers working on it constantly, and they need to be able to continuously deploy new functionalities and bug fixes in production without compromising the website's availability in a fast, secure and efficient manner. All operations must happen with no downtime.

Our solution

The first step was to transform the website in a distributed application, both horizontally and vertically, on different instances organized in different layers. At its core, WBSC's website is powered by WordPress, so its fundamental components are the web application, the database and the file system; the website also depends on a collection of web services and applications, like Play By Play, which updates and displays a game's scores and statistics in real time along with a rendition of the playing field.

Database

To ensure business continuity for database-related operations we chose an as a service solution with a main writer node and a variable number of synchronous read replicas; this way the loss of a single node still allows database readings. When the main writer node is not answering the disaster recovery system promotes one of the read replicas to the role of main writer, thus restoring full functionality. All backups are automated and replicated across multiple different geographical regions.

Because of the high number of reads that the database has to handle, especially when traffic scales rapidly, we engineered the infrastructure to include a caching layer: the output of the most intensive and frequent database queries is saved on multiple memcached nodes, which serve the data directly from RAM to the web application to maximize retrival speed.

File system

All files upon which the application can perform operations, like multimedia files uploaded directly from the website or configuration files, need to be shared among all of the cluster's web servers. For this reason we implemented a synchronously replicated high-availability NAS; the file system is backed-up regularly too, and backups are replicated on different data centers in different regions.

Web server

The web servers which host the application are reachable via a highly-available load balancer and they scale automatically, increasing or decreasing their number according to metrics which are continuously updated according to our analysis of website traffic data and instance resource consumption, so that the application can get better at scaling and adapt more rapidly to handle more traffic and reduce infrastructure costs. Unresponsive instances are automatically removed from the cluster so that they don't receive anymore traffic.

Static content

All files which are server directly to clients, like CSS style sheets, JavaScript scripts, images and video, are hosted on an object storage to maximize file availability and durability, and to distribute them efficiently across multiple servers to guarantee fast delivery performance regardless of the number of requests. At the object-storage level we handle rules for browser caching and for CDN edge-location caching. We also use object storage to save logs and aggregated metrics reports.

Content Delivery Network and Web Application Firewall

We implemented a distributed Content Delivery Network for WBSC's website across more than 50 data centers all over the world; this way the loading time for the website is uniform across the globe. The CDN pulls from the main infrastructure and caches each web page with ad-hoc invalidation times for each resource; the most frequently updated resources dynamicall invalidate themselves when an update is made from the source. The CDN layer implements a dynamic web application firewall with auto-updating exclusion rules to protect the website from the most common security threats.

From a single project to a continuous collaboration

Our work with WBSC challenged our team with interesting, complex problems for which the client needed specific, state of the art technologies, delivered fast; for a client like WBSC, time to market is of paramount importance. Our collaboration with VENTIE30 and with WBSC's personnel was smooth and it proved to be a growing opportunity for every actor involved in the project. The client's full satisfaction gave us the chance to start a continuous collaboration, giving us the job to ensure and monitor uptime, especially during tournaments, to optimize architecture and give consultancy for development-related matters.

The future

The natural evolution of web technologies and the client's growing needs allow us to plan for continuous development and improvement; specifically, we plan to migrate all of the website to HTTPS, implement HTTP2 protocol, configure end-to-end IPv6 and progressively tansition more and more components to a microservices architecture by developing web services exposing REST APIs; specifically, we would like to employ technologies like Lambda and API Gateway from Amazon Web Services, which we already use in production with other projects, using Node.js and Python. From a data standpoint, we are migrating from a SQL-only paradigm to a polyglot persistency model based on object storage, SQL and NoSQL using solutions like Amazon's DynamoDB.