Perth, Western Australia - 6th to 10th January 2014

<-- Back to schedule

Live upgrading many thousands of servers from an ancient RedHat 7.1 to a 10 year newer Debian

This talk will look at how we upgraded our ancient linux distribution on all the Google production servers to a more modern one based on debian stripped down and built from source.
We do live upgrades on running machines using an rsync-like rollout mechanism, and we slowly switched distributions on the fly without rebooting or downtime unless required by separately upgraded kernel, but doing a 10 year jump of linux userspace without breaking users was no piece of cake.
The talk will explain how something so crazy made sense, was even possible, executed, and worked out, as well as recommendations for maintaining linux big servers and dealing with upstream upgrades.


Marc has been using linux since 0.99pl15f (slackware 1.1.2, 1994), both as a sysadmin and userland contributor. He has worked for various tech companies in the Silicon Valley, including Network Appliance, SGI, VA Linux,, and now Google both a server sysadmin and software engineer. He has done hacking in various areas like mail with exim, mailman, SpamAssassin, as well as maintained linux distributions at multiple companies, including 2 at google.

For fun, when he's not trying to beat his mythtv into submission, or hacking misterhouse so that a motion sensor and video camera can trigger a blender to scare the cat off the kitchen counter, he goes snowboarding, mountain biking, racing cars, and flying RC or full size planes.