This talk will discuss the key lessons learned in incident response at Google over the past 10 years. In that time we have evolved from some bad ideas and some hastily written bash scripts, to a globally distributed, cross functional organization, ready to face whatever comes our way. We will discuss the things that have made the team successful, but more importantly, the lessons we learned the hard way in the hope that others won't have to.