Unprivileged Containers: Shaving Yaks To Get the Toothpaste Back In the Tube
No ratings
Presented at 44CON 2024 by
Yelp has been using docker for development and deployment for about a decade, but our development process has always relied heavily on shared development infrastructure which creates a nasty security problem in exchange for faster development: privilege escalation. Docker (and containerizers inspired by docker) have an architecture that’s built around allowing developers to iterate quickly and have a similar deployment environment to the one they develop in, but that architecture is based entirely on making privileged operations easy by running dockerd and containerd as root, and fundamentally assumes a single-tenant system as evidenced by the lack of authn/z controls or audit logging. The writing was always on the wall for our growing security team – one day we would have to try to get the toothpaste back in the tube, and in March 2024 we finally switched the toggle to make our containerized development unprivileged at no extra infrastructural cost… Other than the year and a half of blood, sweat and tears to try to thread the needle of securing development without affecting developer velocity, which was only possible through technical know-how, perseverance and timing. This talk will go into how we did it and the tradeoffs we had to make along the way. Various haircuts in the interminable yak shave included: Setting up unprivileged user namespaces to allow users to continue to act as root with no additional permissions Deduplicating base images now that everyone needs a copy with brand new magical mount features to overcome permission problems Building a solution to tell us which base image layers we can clean up without breaking images and containers Utilizing podman as a drop-in replacement for docker (and then repeatedly backporting patches to its API to actually make it behave like one) Leveraging in-house development patterns to pre-instantiate privileged networking stacks on user log-in using systemd Building an authz system for Docker that users can’t just turn off Adapting over a decade’s worth of accumulated antipatterns, assumptions and vestigial code in disparate codebases Technical details aside, however, also threading its way through this story is the narrative that “best practices” are not replacement for deep technical and business understanding, timing can be everything and that integrating with your engineering culture is the only way to get the toothpaste back into the tube or, more critically, to not have to.