Chaos Engineering: Running Effective Gamedays to Build System Resilience

No ratings

Presented at Securi-Tay 2025 by

Tai Huynh

This 30-minute talk dives into the practical execution of chaos engineering through 'gamedays', structured events where teams intentionally simulate failures in their systems to identify and address weaknesses. Contrary to its playful name, a gameday is a focused, collaborative, and strategic event. We'll explore key aspects of planning and executing gamedays: from choosing the right applications and environments, to defining steady states and executing targeted and controlled chaos experiments. Attendees will learn how Datadog plans gamedays and selects gameday participants based on our system environment, agrees on steady-state definitions, and gradually scales from small latency injections to full-scale events. The session also covers how we use observability tooling for live insights and incident management, highlighting the importance of automation, transparency, and a holistic view for effective execution. By embracing chaos engineering through gamedays, teams can proactively identify system gaps, build resilience, and foster valuable cross-team learning.