Beyond Repair: Proactive Maintenance Work at Scale

No ratings

Presented at SREcon 2016 by

FBAR has enabled Facebook's production engineering teams to automate break/fix responses to many of the single-host events, such as hardware failures or application daemon crashes, that occur on a daily basis, leaving engineers to focus on larger, more complex and interesting problems. To provide more opportunities and time for engineers to take on more meaningful work, the team developed automated responses for the less frequent, but larger-scale proactive work necessary to maintain a healthy infrastructure. This work takes on different shapes, including top-of-rack switch replacements, disruptive BIOS/firmware updates, or work on power supply (backup or primary).