92-Disaster Recovery Lessons I Hoped Id Never Have to Learn - Bdale Garbee

From LCA2014 Delegate Wiki
Revision as of 13:48, 18 January 2014 by Mitch Davis (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

http://mirror.linux.org.au/linux.conf.au/2014/Thursday/92-Disaster_Recovery_Lessons_I_Hoped_Id_Never_Have_to_Learn_-_Bdale_Garbee.mp4

Incomplete transcript

Thank you, so you know, um, how you end up giving talks like this one, there are a couple of things that happened this past year that caused me to end up giving this particular talk. Those of you who read the talk description will understand the sort of principle reason, we'll get into more detail about that shortly, but the other reason that I'm here giving this talk is that I get this wonderful communication from Linux Australia and from the programme committee and so forth right around the time the CFP was closing, saying "Gee Bdale, we haven't seen any talk submissions from you this year". And I had one of those "<sigh> I just can't think about that right now" sorts of reactions, and when I said that, you know, I think trying to be nice me they said, "well under the circumstances would you like an extension on the paper submission deadline, and I said, "that's really very nice of you, I just can't imagine sort of having the time to come up with something that was talk-worthy to you know, would sort of justify that kind of treatment, and one of the folks replied and said, "Ahh come on now Bdale, you're going through an experience right now that you're bound to learn some interesting lessons from that, you know, we'd like to hear about, if you'd be willing to do it". And so, for better or for worse I said, "Ok, I guess I could try to that", and so here I am. Um, what I wanna do today, you know, look, I've been involved in the IT world for a long time, and for ten or twelve years of that career, I actually managed a group that maintained all the computing and networking infrastructure for a non-trivial slice of old HP, our R&D and manufacturing organizations. And so I spent way more than my share of time, you know, helping to write and review disaster recovery plans for you know, a major corporation. Um, and not surprisingly, some of that rubbed off on me. So, at least a few of the lessons that I learned through that process have percolated into the management of my personal infrastructure, and the things that sort of mattered the most to me, but um, for better or worse, nothing always goes exactly the way you expect, and there are probably a couple of things I learned in the process of recovering from the events of last June that, you know, maybe you'll learn something from too.

It it sort of traditional though, that when I give talks at places like LCA, that there's at least one or two rocket photos, and I would admit that there are a couple of non-gratuitous rocket photos in this presentation. It's sort of part of the story of what happened after the fire. So, without a whole lot more ado, let me ask a couple of questions. Um, so first of all, how many by show of hands, how many of you back up your computers. Liars! More serious question. Of those of you who back up your computers, how many of you do something to get your essential data off site? Right...

You know, I would have been one of the people who raised my hand on that too, um, it's a little embarrassing when I finally got to my bank to the safe deposit box and realised that the last snapshot of financial data I put there was <sheepish>twelve years old</sheepish>. You get busy you know? So, um, the most important question is, if someone called you right now, and said, "Gosh, sorry, your house is gone", would that be completely devastating to you? I mean, is that, is, someone asked me earlier today or yesterday, "Gee Bdale, if that happened to me, I don't know if I go on, how do you do it?". I mean would it be that kind of devastating, or would it just be a really bad day?

Think about it, ok? Because this sort of colors, you know, it certainly colored some of my thinking since all of this happened, and I think it's you it's, some point all of these things boil down to, you know, some element of risk management, how much do you think you could deal with, and what would you want to have preserved one way or another if you found yourself in a circumstance like this, so you didn't feel like you couldn't go on. So what exactly happened? Well, on the afternoon of the 11th of June, my daughter and I were sitting in our living room in our house in Black Forest, Colorado, um, which some of you in the room have visited before, ah, reading email and catching up on things. My wife and son are off at the ice skating rink, 'cause he's [indistinct] had a practice session, and of a sudden we smelled a little whiff of smoke, you know the kind of thing you might smell if someone had a barbecue going in the neighborhood. That seemed a little strange, didn't think much about it. Then, I don't know, somewhere between fifteen and thirty seconds later, there was a big whiff of smoke, that's like, that's not a grill, we both sort of looked at each other, jumped up, ran outside, looked at, you know, this is what we saw. Those are not clouds. That's a freakishly large column of smoke, that's certainly less than a kilometer away from the house. And there began what ended up being, sort of, a really crazy adrenaline-pumped afternoon. Um, it was interesting because in the first minute or so that we sort of stood there transfixed looking at it, we didn't hear much, and it didn't seem like things were particularly moving in our direction, but I realised very quickly that you know, wow maybe one of the neighbor's houses is burning or something. I went running down the driveway to see if I could see a little more about what was going, maybe you know, lend a hand to help or something, and ah, as I got to the end of my fairly long driveway, ah, there was a second or third fire department arriving, responding to the call, and one of the fire guys immediately looked at me and said, "do you live down there?", and I said "yes", and he said, "well if there's anything you care about down there, you should get it out right now". And that's exactly how much warning I had. We didn't receive evacuation orders, we were too close to the point of origin. By the time they started doing reverse 911 phone calls and all of that, they'd already killed all the power to our neighborhood, and everybody was either not there 'cause it was the work day and they were nowhere near home, or like us, they were frantically trying to cope.

I don't actually know how much time we spent putting things into a vehicle. Um, somewhere, you know, at least 20 minutes, maybe as much as an hour, I just don't know. The reason I don't know is that my whole sense of time went nuts. There's something that, running on that much adrenaline, every time we looked down the driveway and could see smoke and flames right sort of down at the end of the driveway, and, hadn't yet found the second cat, and knew that couldn't leave without her, um, you know the stress level's pretty high. We did eventually however, find the second cat, got her in her carrier, threw her in the back of the Suburban, and went to take off, and as we turned the corner at the bottom of the hill on our driveway and went to go up, this is what we saw. That photo you can't see it but as soon as you got past what you could see there, there's a wall of firefighters and flames and stuff had jumped the road and all of that. Actually right there, is active flame, and all of that stuff underneath those trees is either on fire, or just burned. That's the side of my driveway. So by the time we went to leave, that's how close the fire was to our house. So what ended up happening was because there's only one road out, and the fire had already, due to high winds, been pushed across the road, and we weren't able to drive out, the fire guys after scratching their heads a little bit said, "Ok, park your vehicles here on your neighbor's property, and we will walk you out through the area that's already burned". And so my daughter and I sort of following directions, took the least we thought we had to carry, which is one cat carrier each, cell phone, keys and wallets, and that was pretty much it. I was literally wearing a pair of [indistinct] and pair of shorts that I'm wearing today. I was not wearing this T-shirt, I was wearing a drag racing T-shirt, and that's what I had on when I walked out.

We got dropped off after a substantial amount of walking, and then being, ah, sort of handed off between a couple of different fire groups that were going out to refill their vehicles and replenish and come back and fight some more. And an intersection just about two miles away from the house, and that's what it looked like, looking back in the direction of our house. In case there's any doubt, my house is any doubt, my house is just to the left of that freakishly large pile of smoke is. Even at this point, it was a little surreal, it's like, well, "that's bad, but it doesn't look that horrible", and there's a gazillion firefighters, and there's, you know, a steady stream of water trucks coming and going and, ah, you know, they'll get this under control.

By 9:30 that night, this is what the view from the other side of Colorado Springs looked like of Black Forest, and you know, it didn't get a whole lot better for a little bit. So just to give you a few statistics. The fire started around 1pm local time on Tuesday the 11th of June. By the time the fire had burned itself out, or gotten under control, something like 94,000 acres, which I'm told is 380 square kilometers had been evacuated, that's about 13,000 homes, about 38,000 people. This fire did start in a semi-rural residential, you know, part of this particular forested area. There's something around 500 firefighters involved, some numbers, I think the most they ever had fighting at one time was 457 or something like that, but I don't actually know exactly. They declared the fires fully contained on the 20th of June. To the best of my understanding, our house actually caught fire early in the morning of Thursday the 13th. Which is kind of interesting, it wasn't the day the fire started, even though it started so close. That's because it started just south of us, and the wind was blowing hard east. Later the winds changed, and it kinda got blown back. Both of those times, the firefighters managed to keep the fire away from our house. Unfortunately at about four o'clock in the morning on the 13th, there's a flare-up just south-west of us, when the winds went to about 40mp/h, at you know, four in the morning when there's no air support for dropping water or anything. Apparently my house ended up under an ember plume that was something like a quarter of a mile long, and it was embers in the middle of the night that ignited the house. There was a scanner log entry from about 4:30-5 that morning where they said, you know, darn, another house is burning, and I'm pretty sure that was my house that, I don't know absolutely know for sure. By the time it was all done, I had plenty of company. There's something, the numbers seem to vary, you know, in terms of exactly which ones you believe. I've heard everything from a low of 486 to a high of 511 homes completely destroyed. There are a bunch more that were damaged. Unfortunately two people died, and that's sad both because you know, they were just normal people trying to get their stuff loaded and get out, and from what I understand it seems like the fire jumped over them, surrounded them, and they were probably asphyxiated, I don't really know for sure. But the other negative consequence of this is that all investigation of origin is therefore criminal investigation, until such time as they can prove it was not intentionally set. It has to be treated as if this were a you know, a murder/manslaughter whatever investigation.

So we were finally let back into our property on the 21st of June, and that's what it looked like.