PAPod 535 - Reimagining Safety: Lessons from the Resilience Master

WEBVTT 00:00:00.017 --> 00:00:03.137 Fighting zombies and the zombies don't die they 00:00:03.137 --> 00:00:06.377 come back over and over again yeah and 00:00:06.377 --> 00:00:12.237 the zombies are out in force now and none of these things for example all the 00:00:12.237 --> 00:00:20.257 insane statements relative to the washington crash none of them are in a sense 00:00:20.257 --> 00:00:25.437 they're all old they're all things we've seen and fought and dealt, had to deal with before, 00:00:25.697 --> 00:00:29.997 but there's kind of a new intensity, a new, 00:00:30.657 --> 00:00:34.277 blatantness, a new, a new kind of misdirect. 00:00:34.677 --> 00:00:38.077 But again, it's as always, safety is controversial. 00:00:38.337 --> 00:00:46.537 Safety is never not controversial, even though we all want to pretend it is. Oh, my God.

00:00:44.240 --> 00:00:55.920 Music. 00:00:55.057 --> 00:00:57.657 Hey, everybody, and welcome to the Pre-Accident Investigation Podcast. 00:00:58.017 --> 00:01:03.837 I am your host, Todd Conklin, and it is so good to be with you. 00:01:04.197 --> 00:01:07.537 What a week.

Oh, my goodness. What a couple of weeks. 00:01:08.137 --> 00:01:14.177 You know the James Reason case because this podcast is the origin of the James 00:01:14.177 --> 00:01:17.717 Reason podcast, so you know how that happened. 00:01:18.177 --> 00:01:22.597 The conversation didn't stop when we talked about Reason.

In fact, it went on. 00:01:22.837 --> 00:01:27.937 And the great thing about this conversation is if you don't take a note in this 00:01:27.937 --> 00:01:32.017 conversation, I'm not sure I guess what would make you take a note. 00:01:32.477 --> 00:01:37.117 Because the podcast you're about to jump into is a sweet podcast. 00:01:39.057 --> 00:01:42.177 Oh, it has got lots and lots of content.

00:01:42.837 --> 00:01:47.337 And it's an interesting conversation, and I'm so excited you're here to be a 00:01:47.337 --> 00:01:50.397 part of it. I think you're going to enjoy it immensely. 00:01:50.817 --> 00:01:57.597 We're going to talk to David Woods. Now, David Woods is really the origin of 00:01:57.597 --> 00:02:02.937 something that's in The Ohio State University called the Cognitive Systems Engineering Laboratory.

00:02:03.377 --> 00:02:08.017 And we've talked before about this, and it's something to think about. 00:02:08.137 --> 00:02:12.857 Now, David put a series of videos together, and it's the easiest way for me 00:02:12.857 --> 00:02:19.677 to tell you how to find the videos, is just go to YouTube and look up David Wood's Resilience. 00:02:19.997 --> 00:02:24.257 And there's a series of 12 videos, and I think even more are coming. 00:02:24.697 --> 00:02:29.237 They're all pretty short, six, eight minutes, maybe nine minutes tops.

00:02:30.117 --> 00:02:33.557 Completely worth watching. You will love them. 00:02:33.717 --> 00:02:37.977 And I would start in order, but I would definitely watch it because you're going 00:02:37.977 --> 00:02:44.457 to hear kind of where the contemporary academic thinking is now from the master 00:02:44.457 --> 00:02:46.137 of contemporary academic thinking. 00:02:46.297 --> 00:02:52.997 There's just no question in my mind that they get no better or smarter or wiser 00:02:52.997 --> 00:02:59.677 or more just profoundly futuristic than David Woods.

00:03:00.390 --> 00:03:06.290 He's, Professor Woods is remarkable. And he's remarkable in the way he sees the world. 00:03:06.450 --> 00:03:10.310 And he's just consistently seen these things for a long time. 00:03:10.430 --> 00:03:15.870 And now he's at a point in his academic career where, and this is the best time 00:03:15.870 --> 00:03:20.110 ever, where now things are starting to kind of come to a sharp point.

00:03:20.570 --> 00:03:24.510 And he's got some pretty strong ideas. And he's going to talk a bunch, 00:03:24.650 --> 00:03:28.130 well, in the entire podcast today is about the fundamentals. 00:03:28.630 --> 00:03:31.890 Now, you know I'm a big believer in first principles. We've talked about it 00:03:31.890 --> 00:03:35.450 a ton of times on the podcast.

If you don't have fundamentals, 00:03:35.450 --> 00:03:40.190 you can't have the next step, and the next step is applied use of these ideas. 00:03:40.390 --> 00:03:45.210 They're based upon these fundamentals. They support the theory and support the 00:03:45.210 --> 00:03:47.850 ideas, and much of the work we do, you and I do, 00:03:47.990 --> 00:03:55.430 is based upon these very fundamentals, and that's what those videos are brilliant in describing. 00:03:56.210 --> 00:04:00.990 Please, if you get a chance, and I urge you to make a chance, 00:04:01.190 --> 00:04:03.390 watch them because they're really worthwhile.

00:04:03.690 --> 00:04:08.530 Let's jump into this podcast because the great thing about when you talk to 00:04:08.530 --> 00:04:14.230 David Woods is you just need to get him started and then kind of sit back and 00:04:14.230 --> 00:04:16.370 take notes, which is exactly what I do. 00:04:16.510 --> 00:04:19.050 You'll hear that as we progress through, But I think you're going to find this 00:04:19.050 --> 00:04:20.650 podcast super interesting. 00:04:20.890 --> 00:04:26.010 So without any more ado, remember David Woods, Cognitive Systems Engineering Laboratory. 00:04:26.510 --> 00:04:32.170 And just go to YouTube and look up David Woods Resilience and you'll see these 00:04:32.170 --> 00:04:33.350 videos.

Here's the conversation. 00:04:33.530 --> 00:04:36.730 It's just the three of us, you, me, and Dave Woods. 00:04:40.730 --> 00:04:47.210 So, Todd, why don't we talk about compromise? All right.

Let's compromise and talk about compromise. 00:04:48.070 --> 00:04:54.130 Well, what does compromise mean in safety? What does compromise mean in systems engineering? 00:04:55.130 --> 00:04:58.870 So I want to tell a couple stories.

00:04:59.510 --> 00:05:05.710 One story is really simple. What do I tell people as they finish up their PhD 00:05:05.710 --> 00:05:11.850 or graduate work and go off into the world to fight battles over safety and 00:05:11.850 --> 00:05:15.790 design and the future and automation? 00:05:15.790 --> 00:05:19.790 And I go to them Hey look if you, 00:05:20.645 --> 00:05:24.665 If you never compromise, you're not doing your job. 00:05:25.125 --> 00:05:29.305 If you always compromise, you are not doing your job.

00:05:31.685 --> 00:05:39.545 The need to compromise is continuous navigation in order to push people further 00:05:39.545 --> 00:05:40.905 than their comfort zone. 00:05:41.305 --> 00:05:46.925 Because they want to retreat and retrench in what's familiar and comfortable when it comes to safety. 00:05:46.925 --> 00:05:51.565 And we see this in the reaction to failures I wrote about with my colleagues 00:05:51.565 --> 00:05:54.305 in the original Behind Human Error in 1994, 00:05:54.305 --> 00:05:58.425 which all emerged as 00:05:58.425 --> 00:06:03.285 soon as I hit the ground in the real world from grad school and walked into 00:06:03.285 --> 00:06:08.965 nuclear control rooms in the aftermath of accidents and started looking at accidents 00:06:08.965 --> 00:06:15.005 and redesigning control rooms to help operators handle accidents in the making. 00:06:15.005 --> 00:06:18.465 The pressures to compromise 00:06:18.465 --> 00:06:21.705 are we're there i mean i tell 00:06:21.705 --> 00:06:25.385 the story of walking in as you 00:06:25.385 --> 00:06:28.125 know i was still 27 years old i hadn't 00:06:28.125 --> 00:06:31.445 even made it to 28 and i am 00:06:31.445 --> 00:06:39.965 supposed to run a study looking at new post three mile island accident upgrades 00:06:39.965 --> 00:06:45.005 to the control room and we had been designing an innovative one based on the 00:06:45.005 --> 00:06:50.985 octagon pattern display to show you a dynamic picture of where are you moving?

00:06:51.085 --> 00:06:56.385 Are you moving away from the normal configuration for the operating context? 00:06:56.645 --> 00:06:59.525 How are you moving away? You got diagnostic cues. 00:06:59.665 --> 00:07:03.725 You had a big picture overview, an overview of health safety, 00:07:03.825 --> 00:07:06.925 what was deteriorating, what was recovering, what was stabilizing.

00:07:07.245 --> 00:07:10.965 And there were other approaches which were compromising. 00:07:11.105 --> 00:07:15.305 Instead of taking this as an opportunity for innovation. They were taking it 00:07:15.305 --> 00:07:20.305 as an opportunity, how to change it as little as possible. It was really those operators.

00:07:20.685 --> 00:07:23.425 It was really that utility, right? 00:07:23.605 --> 00:07:30.085 Second-rate organization, second-rate people, right? All of these things we've heard all along. 00:07:30.765 --> 00:07:37.545 And so the sponsor says, and I'm sitting in the back of the room and the sponsor 00:07:37.545 --> 00:07:39.685 goes, I know I'm the one who's going to execute, right?

00:07:39.885 --> 00:07:46.125 And the sponsor goes, oh, it's simple. We're going to compare different difficult situations. 00:07:46.525 --> 00:07:48.805 We'll look in this high-fidelity simulator. 00:07:49.385 --> 00:07:55.705 We can track errors and response time, and then we can do a statistical analysis, 00:07:55.725 --> 00:07:58.645 and we'll know if any of these new backfits really help or not.

00:07:59.301 --> 00:08:02.221 And I'm going, wait a minute. In the back of the room, I'm thinking, 00:08:02.361 --> 00:08:05.661 there's no way this will work. They're solving dynamic problems. 00:08:06.961 --> 00:08:10.441 There's thousands of variables.

There's variations. By the way, 00:08:10.561 --> 00:08:14.801 when you run one of these things, depending on an action that could be correct, 00:08:14.961 --> 00:08:17.961 but exactly when it's taken, it's continuous change. 00:08:18.141 --> 00:08:21.061 Something can get worse. Something else may not be as hard.

00:08:21.281 --> 00:08:26.201 They may innovate and recognize something and act early, diffusing the situation 00:08:26.201 --> 00:08:31.641 before things get too far. In other cases, they may just be going slow and things 00:08:31.641 --> 00:08:34.901 continue to get bigger and bigger and more difficult to handle. 00:08:35.681 --> 00:08:40.041 They can misstep. They can missee.

The information can be misleading. 00:08:40.301 --> 00:08:44.101 We designed misleading information cues into the scenarios. 00:08:44.641 --> 00:08:47.601 And I'm sitting there going, there's no way this is going to work. 00:08:47.981 --> 00:08:53.901 And I spoke up.

I started to say, you know, it violates every assumption of 00:08:53.901 --> 00:08:58.201 these kinds of statistical analyses. And what was the comment back? 00:08:58.321 --> 00:09:02.861 Oh, no, no, no, no, no, no, no, no, no, no, no. Just go collect these numbers 00:09:02.861 --> 00:09:06.741 and run the analysis of variance, and that's it.

00:09:07.141 --> 00:09:12.081 And I'm going back on, this is a fiasco. I have to do underground research. 00:09:12.521 --> 00:09:17.081 I had to compromise and not compromise, because I knew down the road they'd 00:09:17.081 --> 00:09:21.041 be unhappy when I came back with null results showing absolutely nothing. 00:09:21.661 --> 00:09:25.581 It would have been boring and expensive with no information value.

00:09:25.701 --> 00:09:27.921 I had to create a different kind of study. 00:09:28.101 --> 00:09:33.581 I had to go back and invent process tracing for the modern systems and world 00:09:33.581 --> 00:09:38.981 from the history of protocol analysis in the early 20th century and problem solving. 00:09:39.221 --> 00:09:45.861 That was like chess, not how do you recognize unexpected behavior in a dynamic, 00:09:46.321 --> 00:09:49.001 complex, highly interdependent system. 00:09:50.401 --> 00:09:55.961 It's a little example.

Now, think about human factors. Think about safety. 00:09:56.581 --> 00:09:59.341 What are we asked to do all the time? Compromise.

00:10:00.130 --> 00:10:03.650 What does everybody say? Well, you got to get your foot in the door. 00:10:04.110 --> 00:10:09.150 What do they say? You've got to reach out.

You've got to touch them where they 00:10:09.150 --> 00:10:13.430 live in terms of their expectations and mental models about safety. 00:10:13.690 --> 00:10:17.630 And then you can start to move them. And then you can start to connect. 00:10:18.070 --> 00:10:21.410 You build it up.

You get your foot in the door. 00:10:21.850 --> 00:10:31.090 So what did I start with as a joke in my 1999 presidential address for human factors. 00:10:31.710 --> 00:10:35.790 I said, you know, I've seen this all the time. I hear this all the time.

00:10:36.030 --> 00:10:40.530 You know, when they crack the door open, get your foot in there and then you 00:10:40.530 --> 00:10:43.370 can expand it and get inside and really help. 00:10:43.670 --> 00:10:45.710 Right. I mean, this is common. 00:10:46.130 --> 00:10:48.130 And I said, you know what really happens?

00:10:48.830 --> 00:10:53.910 It says they slammed the door shut. And then later when problems arise or accidents 00:10:53.910 --> 00:10:58.990 happen that have human factors, human systems, human automation problems, what do they do? 00:10:59.170 --> 00:11:02.890 They pick up your foot and wave it around and say, hey, I have human factors. 00:11:03.070 --> 00:11:04.350 It's right here.

They lift their foot. 00:11:06.170 --> 00:11:11.690 So what does that mean for today and in today's context? 00:11:12.170 --> 00:11:14.390 Well, I have good news. 00:11:15.010 --> 00:11:22.510 The good news is what we started really, it's now 25 years since I first called 00:11:22.510 --> 00:11:29.090 for resilience engineering based on NASA accidents and investigations that occurred in 99, 00:11:29.610 --> 00:11:33.410 20 years since we had the first meeting on resilience engineering, 00:11:33.630 --> 00:11:38.110 is we've actually made progress on fundamentals.

00:11:38.910 --> 00:11:43.770 And, well, fundamentals, what's the reaction I get to fundamentals? 00:11:44.690 --> 00:11:50.690 Fundamentals, you know, they go, but those fundamentals, they don't show anybody 00:11:50.690 --> 00:11:53.570 what to do practically tomorrow in their organization. 00:11:53.850 --> 00:11:55.370 I'm looking at them like, what do you mean? 00:11:55.710 --> 00:12:02.870 I go, you know, nobody says, hey, you know, we've been building airframes, 00:12:03.050 --> 00:12:06.510 airplanes for decades.

We know how to build them. 00:12:06.990 --> 00:12:10.650 Hey, this AI system, look at this wing. 00:12:11.270 --> 00:12:15.490 It's beautiful. And, you know, it's going to be great.

00:12:15.610 --> 00:12:21.290 This AI program generated a new image of an aircraft wing. Let's go build it. 00:12:22.610 --> 00:12:25.790 Wait a minute. Don't you have to take it to a wind?

You have to put it together. 00:12:25.990 --> 00:12:29.910 You have to run some tests and simulation and the wind tunnel. 00:12:30.110 --> 00:12:33.390 It's a radically different wing. Oh, no.

00:12:34.390 --> 00:12:37.770 Aerodynamics is a solved problem. We don't need to do those tests. 00:12:37.890 --> 00:12:40.650 We don't need to do those analyses. You should be laughed out the door.

00:12:41.110 --> 00:12:44.350 Actually, today, I wonder whether you would still be laughed out the door. 00:12:44.610 --> 00:12:48.730 You should be laughed out the door, right? That's not engineering. 00:12:48.730 --> 00:12:50.250 That's not systems engineering.

00:12:51.821 --> 00:12:57.681 Yet that's what people who work on complex systems and safety are asked to do every day. 00:12:58.081 --> 00:13:02.561 You know, it feels good this way. You know, this is consistent with what we 00:13:02.561 --> 00:13:04.001 learned in an MBA program. 00:13:04.381 --> 00:13:09.481 You know, it would cost.

Hey, that's not going to happen to us. 00:13:11.021 --> 00:13:15.881 Compromise. Compromise. Push it, you know, just make a small change.

00:13:16.581 --> 00:13:23.601 And you see this in what happened in the twin Boeing 737 MAX accidents with 00:13:23.601 --> 00:13:25.401 the MCAS automated system. 00:13:25.981 --> 00:13:30.641 And, you know, and then all these rationalizations and discounting mechanisms 00:13:30.641 --> 00:13:33.801 are out there in a case where, what did we do? 00:13:33.921 --> 00:13:37.121 We didn't listen to the fundamentals. We took safety for granted.

00:13:37.121 --> 00:13:41.261 It. We don't have to go back and test. We don't have to check. 00:13:41.441 --> 00:13:46.161 We don't have to make sure certain things in an integrated human automation 00:13:46.161 --> 00:13:49.021 system work.

Do we understand what they are? 00:13:49.261 --> 00:13:55.161 Yeah, but compromise because otherwise it might raise some regulatory risk. 00:13:55.381 --> 00:13:57.181 It might delay the schedule. 00:13:57.581 --> 00:13:59.741 They killed 246 people.

00:14:00.652 --> 00:14:04.992 And then the other irony in this is it's cost them $30 billion and counting. 00:14:05.212 --> 00:14:07.312 I don't know where they are, $50 billion now? 00:14:07.832 --> 00:14:12.572 You know, the idea there's an economic case for safety has been blown up by Boeing. 00:14:13.172 --> 00:14:16.792 They said, I can save money now, so let's take shortcuts.

00:14:17.812 --> 00:14:22.472 Engineering shortcuts, safety shortcuts. And when it blows up, 00:14:22.572 --> 00:14:23.972 but they didn't think it would happen to them. 00:14:24.072 --> 00:14:28.092 It hadn't happened to them because they'd been taking these shortcuts for a 00:14:28.092 --> 00:14:32.072 long time. and their cumulative toll was building.

00:14:32.632 --> 00:14:38.472 So where do we stand? The good news is we have foundations. What's that like? 00:14:38.692 --> 00:14:44.072 It's like physics, aerodynamics, only it's for adaptive systems in this universe.

00:14:44.632 --> 00:14:50.792 Adaptive systems at a human system scale, that means technology is part of human systems. 00:14:51.032 --> 00:14:56.032 And we have fundamentals, we have laws, we have theorems, right? 00:14:56.132 --> 00:14:59.032 People always would say, where's your numbers don't you 00:14:59.032 --> 00:15:02.412 have numbers you know every time that 00:15:02.412 --> 00:15:05.572 happened to me when i started about alarms 1982 00:15:05.572 --> 00:15:08.912 30 years old and go 00:15:08.912 --> 00:15:14.832 okay let me show you the math for how how why alarms are misdesigned and how 00:15:14.832 --> 00:15:19.332 to do it better and you know what they said i didn't mean that math i don't 00:15:19.332 --> 00:15:24.152 understand that math and and and it gives a different answer than i expected 00:15:24.152 --> 00:15:26.592 and and if i to follow that answer, 00:15:26.832 --> 00:15:29.772 it'll cost me money. I'll have to do something different.

00:15:30.372 --> 00:15:32.132 Different. Oh my God, different. 00:15:33.032 --> 00:15:38.392 And so I said, wait a minute. You asked for the math.

Here it is. 00:15:38.752 --> 00:15:44.032 Well, now guess what? We've got some math. We've got some advanced theorems.

They're pretty weird. 00:15:44.312 --> 00:15:49.872 What do they show? Well, the world is inherently messy. The world is inherently nonlinear.

00:15:50.232 --> 00:15:53.672 All these compromises, is let's pretend it's linear. 00:15:54.532 --> 00:15:59.352 Let's pretend these are mostly independent parts. We can add in a few interactions. 00:16:00.806 --> 00:16:06.606 And again, this is an old story.

This went back to the Lewis report in 1978, 00:16:06.606 --> 00:16:12.846 a year before the Three Mile Island accident happened, when they were asked to check on the accuracy, 00:16:13.786 --> 00:16:20.926 of probabilistic risk assessments on the safety of nuclear power plants. And what did they say? 00:16:21.466 --> 00:16:26.486 Basically, there's too many interactions and interdependencies, right? 00:16:26.626 --> 00:16:30.646 And it's hard to take into account the adaptive behavior of people.

00:16:31.186 --> 00:16:37.466 And so the only thing we can be sure of is that the analyses are an order of 00:16:37.466 --> 00:16:40.026 magnitude more uncertain than they pretend to be. 00:16:40.626 --> 00:16:44.866 An order of magnitude more uncertain, right? What were they highlighting? 00:16:45.106 --> 00:16:48.126 The complexity that's evident in our world everywhere.

00:16:48.646 --> 00:16:54.186 So the fact that we have laws, and yes, some of these you may say are provisional. 00:16:54.186 --> 00:16:56.986 Theorems can be overturned by new work. 00:16:57.506 --> 00:17:03.406 Laws can be qualified. Laws may be modified as we gain more evidence.

00:17:03.646 --> 00:17:08.826 But that's what the foundations are. That's our physics. That's our aerodynamics. 00:17:10.126 --> 00:17:14.726 And so I run through this and I go, and some of these are actually very simple.

00:17:15.746 --> 00:17:19.206 Let's take one. And we'll take one from social science first, 00:17:19.206 --> 00:17:24.126 because one of the differences about resilience as a systems engineering approach 00:17:24.126 --> 00:17:29.806 is that we take key ideas from all kinds of things that are non-traditional 00:17:29.806 --> 00:17:31.526 and don't normally talk to each other. 00:17:31.826 --> 00:17:38.386 Because that's what about adaptive systems in the human slash biological world. 00:17:38.706 --> 00:17:44.006 So let's take one, reciprocity.

So Eleanor Ostrom gets the Nobel Prize in 2009 00:17:44.006 --> 00:17:51.006 for studies and work synthesizing the role of reciprocity in systems and human 00:17:51.006 --> 00:17:53.566 systems, human system scales, 00:17:53.766 --> 00:17:57.426 and how these things work to make systems that are adaptive. 00:17:58.510 --> 00:18:03.370 And that they can persevere over challenges, over longer timescales. 00:18:03.830 --> 00:18:07.690 And there's different ways to think about reciprocity. Well, 00:18:07.770 --> 00:18:13.870 it turns out in the fundamentals that a particular way to think about reciprocity is fundamental.

00:18:14.310 --> 00:18:19.830 So if you want to build a safe system, you need to build reciprocity between 00:18:19.830 --> 00:18:23.050 the different units, roles, and layers in your organization. 00:18:23.050 --> 00:18:28.590 Now, we see a lot of lip service to that, and we have always, right? 00:18:29.050 --> 00:18:33.870 But do you really build reciprocity? What does it really take?

00:18:34.130 --> 00:18:37.390 Now, we can describe that very simply. You don't have to know the theorems. 00:18:37.490 --> 00:18:41.190 You don't have to get into complex nonlinearities. 00:18:41.410 --> 00:18:47.910 You have to simply say and look at your system and say, where is reciprocity demonstrated?

00:18:47.990 --> 00:18:54.330 Where does reciprocity break down? how does management actually tell everyone 00:18:54.330 --> 00:18:57.090 that there is reciprocity going on? 00:18:57.210 --> 00:19:03.490 Or is it only ad hoc between horizontally between a few local roles because they're what? 00:19:03.670 --> 00:19:08.010 Committed to make the system work, fill the gaps, close the holes.

00:19:08.550 --> 00:19:11.330 Be the ad hoc source of resilient performance. 00:19:11.490 --> 00:19:18.290 So they find ways to reciprocate under stress in order to make the system work, 00:19:18.290 --> 00:19:27.750 generally in an environment that downplays or restricts or works against forms of reciprocity. 00:19:28.670 --> 00:19:33.550 And we can see this in successful stories of organizations that are highly adaptive 00:19:33.550 --> 00:19:36.290 to a dynamic world with changing risks. 00:19:36.630 --> 00:19:43.050 And so we can see these things play out.

So it's very practical, But it's different. 00:19:43.510 --> 00:19:48.390 And that's when the compromise, the bad sense of compromise creeps in. 00:19:48.750 --> 00:19:54.430 No, we can't go that far. We can't do that.

That doesn't fit with standard management practices. 00:19:54.850 --> 00:20:00.810 And what do they end up with? A slow organization that doesn't recognize things 00:20:00.810 --> 00:20:04.410 fast, that can't adapt quickly to changing circumstances. 00:20:04.410 --> 00:20:10.430 That's stuck in stale approaches in a world that is increasingly telling us 00:20:10.430 --> 00:20:13.850 about the turbulence and change that's ongoing.

00:20:14.190 --> 00:20:20.170 The thing we know for sure is that new kinds of challenges and shocks will occur. 00:20:20.530 --> 00:20:23.810 We don't know which ones, but we know they'll occur. Now, we know something 00:20:23.810 --> 00:20:26.310 about what makes such events shocking. 00:20:26.790 --> 00:20:28.450 That's back in the fundamentals.

00:20:29.050 --> 00:20:31.910 We know a bunch of stuff about how 00:20:31.910 --> 00:20:35.150 to be adaptive that's what's in the fundamentals do you 00:20:35.150 --> 00:20:38.910 want to put it into practice or do you want to say no no no no we're just we're 00:20:38.910 --> 00:20:42.770 just repairing a little bit over here that that's out of whack and a little 00:20:42.770 --> 00:20:48.350 bit over there and this discounting and narrowing that we wrote about in behind 00:20:48.350 --> 00:20:53.910 human error 40 years ago it's a 40 30 years ago 30 years ago 30 years ago. 00:20:55.378 --> 00:21:02.698 Plays out. So today's version of this isn't different than older versions. It's just that it's now.

00:21:03.198 --> 00:21:08.038 But the difference is we do have tools and techniques to deal with it that come 00:21:08.038 --> 00:21:09.238 out of those foundations. 00:21:09.518 --> 00:21:13.618 We have techniques to build reciprocity. We have tools that could help you tell 00:21:13.618 --> 00:21:16.278 whether you have reciprocity. Let's take another one.

00:21:17.138 --> 00:21:20.018 Saturation. This comes from engineering. It's in the textbooks. 00:21:20.238 --> 00:21:21.878 It's in the textbooks, for God's sake.

00:21:22.178 --> 00:21:32.298 And it gets ignored. But the textbooks try to hide and compromise away the implications that any agent, 00:21:32.598 --> 00:21:38.098 any system that's trying to control a dynamic, influence a dynamic world can 00:21:38.098 --> 00:21:39.698 saturate its responses. 00:21:40.038 --> 00:21:44.558 What does that mean? Hey, you got finite resources.

The world keeps changing. 00:21:44.938 --> 00:21:48.998 Others are adapting around you. So whatever scale you're at, 00:21:49.058 --> 00:21:54.538 not just a control system, at every scale, this idea of saturation matters. 00:21:54.818 --> 00:21:59.658 What was competent can still be surprised.

00:21:59.958 --> 00:22:04.978 What was increasingly competent over before will get surprised because those 00:22:04.978 --> 00:22:07.118 limits will matter eventually. 00:22:07.118 --> 00:22:11.758 And the world will continue to throw events that surprise, right, 00:22:11.978 --> 00:22:19.118 the model you use to build competence or the changing model you use to enhance a form of competence. 00:22:19.798 --> 00:22:23.518 And there's a bunch of assumptions about the way we think about the world that 00:22:23.518 --> 00:22:25.398 turn out to be wrong and the fundamentals. 00:22:25.698 --> 00:22:31.918 So what's going on in the real world is the need to revise and reframe.

00:22:33.223 --> 00:22:37.063 And so we can talk about that. It's in the fundamentals. Can people do it? 00:22:37.203 --> 00:22:40.803 Yes.

Is it hard to do? Yes. Do machines do it? No.

00:22:41.043 --> 00:22:43.723 Does anyone try to make machines reframe? No. 00:22:44.343 --> 00:22:48.663 So let's go back to saturation. It turns out it's universal.

00:22:49.143 --> 00:22:53.603 It turns out in adaptive systems, they adapt before they reach saturation. 00:22:53.823 --> 00:22:57.363 Why? Because if you hit saturation, your effectiveness at responding to whatever 00:22:57.363 --> 00:22:59.983 is going on just went to zero or near zero. 00:23:00.303 --> 00:23:02.763 What does that mean?

That's a brittle collapse. 00:23:03.223 --> 00:23:07.863 Brittle collapse has big penalties. It's a step change in penalties. 00:23:08.203 --> 00:23:10.143 Could be an economic penalty.

00:23:10.503 --> 00:23:15.963 Could be a weakening of how your system works that will mean the next time it's 00:23:15.963 --> 00:23:17.363 challenged, it won't work as well. 00:23:18.303 --> 00:23:24.303 And it has costs right now. So biology wants to avoid brittle collapse. 00:23:24.463 --> 00:23:29.123 This is why I can summarize a lot of the resilience engineering as we build 00:23:29.123 --> 00:23:32.463 competent but brittle systems because we ignore these fundamentals.

00:23:32.683 --> 00:23:36.763 We want to use new tools and techniques that we're developing in resilience 00:23:36.763 --> 00:23:40.823 engineering to build competent and extensible systems. 00:23:41.103 --> 00:23:45.963 Yes, should you do work to build up new forms of competency, 00:23:46.263 --> 00:23:49.463 take advantage of new technical capability? Sure. 00:23:50.043 --> 00:23:55.663 It doesn't change the larger equation about how adaptive systems work, 00:23:55.783 --> 00:24:00.703 about how adaptive systems are challenged, about how adaptive systems can malfunction.

00:24:01.063 --> 00:24:06.043 Those fundamentals hold, just like aerodynamics holds, just like physics holds. 00:24:07.083 --> 00:24:12.383 So everyone needs to understand saturation, approaching saturation, 00:24:12.523 --> 00:24:13.983 what are the forms of adaptation. 00:24:15.264 --> 00:24:19.664 That can go on. There's some basic classes.

There's opportunities for research 00:24:19.664 --> 00:24:24.844 to develop and understand other forms of adaptation that go on approaching saturation 00:24:24.844 --> 00:24:29.204 so that systems can be extensible as well as competent. 00:24:29.544 --> 00:24:31.804 Again, it's pragmatic. 00:24:32.384 --> 00:24:35.384 There's things to do, but it's different. 00:24:35.784 --> 00:24:40.504 Now, there's one other challenge for us, and that is it's not as mature.

00:24:40.984 --> 00:24:48.464 Why do people think about aerodynamics as or control engineering as easy and 00:24:48.464 --> 00:24:53.504 resilience engineering is, oh, we can't do that, is because the others are mature. 00:24:53.724 --> 00:24:58.924 So they've created very efficient tools, tooling and tools. 00:24:59.204 --> 00:25:01.824 You don't have to go back to first principles. 00:25:02.324 --> 00:25:10.904 You don't have to do custom craft work to translate laws into engineering practice, right?

00:25:11.064 --> 00:25:16.284 We have engineering practices. We have software that supports those engineering practices. 00:25:16.764 --> 00:25:21.924 But what do they tell us? They go, you have to fit the old engineering practices.

You have to compromise. 00:25:22.364 --> 00:25:29.764 And you go, but those practices don't address the complexity penalties that arise as growth goes on. 00:25:30.064 --> 00:25:39.464 New roles, new scales, new interdependencies, a more tangled network of roles 00:25:39.464 --> 00:25:41.864 and interdependencies across layers. 00:25:42.504 --> 00:25:48.384 Tangled, hard to see, hard to notice where the connections are going to matter 00:25:48.384 --> 00:25:50.684 as the world continues to change.

00:25:51.444 --> 00:25:56.584 So we're not as mature? Yes. Are we different? 00:25:57.064 --> 00:26:04.104 Yes.

Does that mean we're not actionable? Not in the least. Are you willing to change? 00:26:04.644 --> 00:26:11.484 Or are you going to stay stuck in old ways when the world is shouting at you?

00:26:12.064 --> 00:26:16.984 It's different. Your old assumptions don't hold anymore. 00:26:17.344 --> 00:26:22.844 You need to revise, reframe how you do things if you want to be adaptive to 00:26:22.844 --> 00:26:26.884 be viable in the world we live in now. 00:26:27.164 --> 00:26:31.964 You need to be able to synchronize across roles and layers, for example, 00:26:32.124 --> 00:26:33.464 by building reciprocity.

00:26:33.804 --> 00:26:38.984 You need to be able to be highly responsive when disturbances arise. 00:26:40.244 --> 00:26:45.244 And that means you have to understand how to adapt ahead of reaching saturation, 00:26:45.524 --> 00:26:51.104 whether that's about a mass casualty event in a hospital system or whether that 00:26:51.104 --> 00