Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Privacy

No ratings

Presented at Privacy Enhancing Technologies Symposium 2024 by

Joshua Kroll

Abstract: Man-made catastrophes have been studied for many decades because of the high impact severity of these events: Chernobyl, Three Mile Island, Fukushima-Daiichi, Bhopal, the Vajont Dam Collapse, Challenger, the Financial Crash of 2008. A common misconception is that these kinds of events are rare ``freak'' accidents and result from the inherently unforeseeable interactions in complex systems. A closer examination of these disasters reveals that the risks and hazards were well-known beforehand but not acted upon due to social structural, political and economic factors. As computing and data-driven technologies pervade more of society, and are used to manage consequential outcomes, understanding their capabilities, limitations, and attendant risks in context requires analysis of full sociotechnical systems. Sociotechnical analysis of risks in highly complex systems provides clear lessons for the design and evaluation of computing systems, transcending a technical focus on reliable or “responsibly designed” components to understand risks at a systemic level. We outline several ways in which the unlearned lessons of previous catastrophes can inform engineering practice beyond the traditional focus on narrow technical failure, considering especially the social and organizational dynamics that shape decision-making: improving risk perception, communication, and analysis at the organizational level; traceability of requirements and responsibilities; and holistic approaches to responsibility and safety.