- 100-200k visits/month
- 4+ million characters
- 100,000,000+ character fights logged
Raidbots is a series of tubes for WoW raiders that I've built over the last couple of years. It initially started as a way to collect data about player performance to get at real-world metrics. DPSBot aggregates hundreds of thousands of boss fight combat logs to see a data-backed comparison of classes and specs. EpeenBot highlights a specific player's parses against her own class/spec to see where she lies in the greater scheme of things. CompareBot is a utility to dig into multiple boss fight parses in order to see small variations of ability rotations or gear disparities to understand how their own performance can be improved.
I've spent a lot of time and effort optimizing the page response time on Raidbots. Raw data is available from World of Logs once a day which I download and process as a batch. The processing denormalizes a lot of data to make database lookups as fast as possible. Varnish is used to cache pages until the next batch is processed. Raidbots uses CloudFlare as a CDN to both get assets closer to the visitors as well as for minification of CSS, JS, and HTML.
Raidbots is hosted on its own dedicated non-virtual hardware both to minimize contention and to manage its sizable database (the Mongo database sits at about 100GB right now).