From the Bunker: Test Czar

This is the first in a series of posts “from the bunker,” interviews with the people behind the scenes at every hot fire and cold flow that takes place at XCOR…

Bryan Campen: Where did the name “Test Czar” originate?

Jeff Greason: The reason we came to call this role the Test Czar—it’s cold in the winter in Mojave and I had a Russian Navy version of a Ushanka, you know the full fur hats you’ve probably seen.

czarvalant

Not Jeff Greason: XCOR Engineer Mike Valant performs test czar duties at a recent hot fire, complete with Russian Ushanka

I had picked one up in Moscow not long after the movie The Hunt for Red October came out in theaters. Anyway it was cold out, and I was wearing that hat and the guys started calling me czar, and it stuck.

12-09-12_90sec-coldflow-8648

Jeff Greason acts as test czar during a recent cold flow, without the standard Ushanka

BC: So what does a test czar do?

JG: If you’re familiar with the Apollo missions, there was always a flight director.

And during that time, in the sixties, there were different people in front of computer screens, each responsible for some piece of the system.

800px-Mission_Operations_Control_Room_at_the_conclusion_of_Apollo_11-1

Mission Control during the Apollo 11 mission

And there were all these people with individual roles and responsibilities that made-up mission control.

NASA also found it very helpful to have somebody who was only worried about the big picture, and not about the specific details of any one of those systems.

When doing this job, you have to be aware of two polar opposite traps that are very easy to get into when doing test operations.

From time to time, anomalies crop up and you have to make an informed decision on whether you can live with the anomaly on that day—or not. If you always stop everything, any time there is anything unusual, you’d never get anything done.

But the other disease that’s easy to fall into is go fever. You’ve been working toward this test a long time, you get up early that morning, you have momentum built-up, people are doing their steps and some little problem crops up and it’s very, very easy to say “well you know, don’t worry about that, we’ll deal with that later. Let’s just keep going.”

The guy reading the checklist reads a step, people say it’s done and he reads another step, done. He wants to read the next step. But what if it’s not done?

We have to stop.

So the czar looks at the big picture, watches the momentum, the ebb and flow of the day. He watch for people who are not listening, for the danger signs, and for who is going too fast. But you also watch the other extreme: Not only are you making sure the team doesn’t go too fast, but the czar will also step in sometimes and say “OK, I understand this is an interesting problem. My judgment is that that is not worth stopping [the test] for, so let’s make sure we capture that problem for later analysis.”

Or, maybe that stop is indicating for us something we need to check on again later in the day, so we’ll add a step to the checklist later on to check the health of that system, to make sure we don’t forget.

You have to keep your eye on all these moving parts. And if it gets too stressful that’s usually a sign that it’s time to slow down.

We also have a Safety Officer, or “Safety”, whose only job is to stop things, in some sense. [Note: we’ll cover the Safety Officer role in a later post]

Safety is there looking for problems.

Everybody else is looking for how to get it done.

And the czar is there to look at the big picture, to get the test done safely or with the appropriate amount of risk. I have to decide if the probability that we are going to get the data we want is the worth the risk that we are going to be taking. It’s something we think about ahead of time, and also something we think about the day of the test.

In one sense, on a perfect test day with no anomalies, I don’t have anything to do. All I do is watch what other people are doing, make sure that I am happy with how that’s all going.

But you’re there poised at every second, not letting your eye fall off of anything, to be ready to step in and say “Wait a minute, stop, we need to pay closer attention to that. That’s a potential danger sign that we just missed.”

My job is to be ready to stop the test, but also be ready to say “we’re ready to go.”

It’s the rare test day that goes by without there being some hard call to make about what we want to live with—or not.