TSAR (the TimeSeries AggregatoR): How Twitter Counts 50+ Billion Daily Events in Real Time Using Open Source Technologies

No ratings

Presented at cloudopen 2014 by

Anirudh Todi

Twitter's 250+ million users generate over 50 billion tweet views per day. Aggregating these events in real time - in a robust enough way to incorporate into our products - presents a massive scaling challenge. In this talk I'll introduce TSAR (the TimeSeries AggregatoR), a robust, flexible, and scalable service for real-time event aggregation designed to solve this problem and a range of similar ones. I'll discuss how we built Tsar from the ground up, almost entirely on open-source technologies (storm, summingbird, kafka, aurora, and others), and describe some of the challenges we faced in scaling it to process tens of billions of events per day.