This is the second in a series of posts “from the bunker,” interviews with the people behind the scenes at every hot fire and cold flow that takes place at XCOR. Today we talk about the role of Checklist with XCOR engineer Brandon Litt.
The team confers during a test day. Left to right: Doug Jones (Chief Test Engineer), Jeremy Voigt (Control), Randall Clague (Safety Officer), Mike Valant (Senior Engineer), Jeff Greason (Test Czar and CEO), Brandon Litt (Checklist), Derek Nye (A&P) and Geoff Licciardello (Test Engineer)
Bryan Campen: So what does Checklist do?
Brandon Litt: The checklist reader gives commands to everybody on the test crew. And no one does anything on the test stand, on the control box, with the DAQ, unless the checklist specifically tells them to do it.
It has been through so many iterations that it is now the best way to get the test done successfully and safely.
The speed at which I read the checklist really sets the tempo for the day. If I’m reading it pretty fast, people might be on edge. We might miss things or skip lines if it were too fast. So it’s nice to go slow and deliberately and make sure everything is done the right way.
But if I’m going too slow people might get a little bit sluggish and lose focus, so I need to keep things going at a steady pace throughout the day. I also have to make sure that the commands I give are called back to me by the rest of the team.
BC:Have you ever stressed anyone out by going too fast?
BL: Oh yeah, when I started I’d get comments –“slow that up” or “take a pause here”– I’d push the radio button a little too early and maybe cut someone off while they were trying to say something.
So I tell Red Team to open a valve, wait for them to call back that they have opened that valve successfully, and then move on to the next step.
It requires a lot of mental focus to ensure that I am very precise.
I don’t just give commands. I wait for feedback.
BC:How many items are on the checklist?
BL: That’s a really good question. For a standard engine day we start with a rollout checklist, it includes all the things we bring with us when we get to the test site. We go through a stand setup checklist, which maybe has fifty items. Then we go through a control checklist which pretty much brings up the electrical system on the stand, and test every valve on the whole stand. It’s probably another fifty items. Then we go on to the pressurization steps for the engine run. The final checklist is only a few steps, it’s mostly verifications before we hit go.
BC:What are the best and most challenging parts of the job?
BL: It’s disappointing to go out there all day, read this checklist methodically, slowly, everything happens right, but for some reason we just have to call it a day, shut down and go home.
Counterpoint to that, it’s equally as amazing when do I do all those things, we push the button and that engine just works.
All of the positions on the crew don’t just do their respective jobs. All of the prep work in the hangar, and rollout, and for setting up the stand once you get out there, it’s a big group effort and everybody really has to chip in to get that thing ready to go.
So the checklist acts as that final verification. When everything is set up, if I am [for instance] trusting new engineers to do certain things during the setup, I’m going to verify those steps later in the checklist.
And rest assured I’m not just going to assume they were done correctly.
This is the first in a series of posts “from the bunker,” interviews with the people behind the scenes at every hot fire and cold flow that takes place at XCOR…
Bryan Campen: Where did the name “Test Czar” originate?
Jeff Greason: The reason we came to call this role the Test Czar—it’s cold in the winter in Mojave and I had a Russian Navy version of a Ushanka, you know the full fur hats you’ve probably seen.
Not Jeff Greason: XCOR Engineer Mike Valant performs test czar duties at a recent hot fire, complete with Russian Ushanka
I had picked one up in Moscow not long after the movie The Hunt for Red October came out in theaters. Anyway it was cold out, and I was wearing that hat and the guys started calling me czar, and it stuck.
Jeff Greason acts as test czar during a recent cold flow, without the standard Ushanka
BC: So what does a test czar do?
JG: If you’re familiar with the Apollo missions, there was always a flight director.
And during that time, in the sixties, there were different people in front of computer screens, each responsible for some piece of the system.
Mission Control during the Apollo 11 mission
And there were all these people with individual roles and responsibilities that made-up mission control.
NASA also found it very helpful to have somebody who was only worried about the big picture, and not about the specific details of any one of those systems.
When doing this job, you have to be aware of two polar opposite traps that are very easy to get into when doing test operations.
From time to time, anomalies crop up and you have to make an informed decision on whether you can live with the anomaly on that day—or not. If you always stop everything, any time there is anything unusual, you’d never get anything done.
But the other disease that’s easy to fall into is go fever. You’ve been working toward this test a long time, you get up early that morning, you have momentum built-up, people are doing their steps and some little problem crops up and it’s very, very easy to say “well you know, don’t worry about that, we’ll deal with that later. Let’s just keep going.”
The guy reading the checklist reads a step, people say it’s done and he reads another step, done. He wants to read the next step. But what if it’s not done?
We have to stop.
So the czar looks at the big picture, watches the momentum, the ebb and flow of the day. He watch for people who are not listening, for the danger signs, and for who is going too fast. But you also watch the other extreme: Not only are you making sure the team doesn’t go too fast, but the czar will also step in sometimes and say “OK, I understand this is an interesting problem. My judgment is that that is not worth stopping [the test] for, so let’s make sure we capture that problem for later analysis.”
Or, maybe that stop is indicating for us something we need to check on again later in the day, so we’ll add a step to the checklist later on to check the health of that system, to make sure we don’t forget.
You have to keep your eye on all these moving parts. And if it gets too stressful that’s usually a sign that it’s time to slow down.
We also have a Safety Officer, or “Safety”, whose only job is to stop things, in some sense. [Note: we’ll cover the Safety Officer role in a later post]
Safety is there looking for problems.
Everybody else is looking for how to get it done.
And the czar is there to look at the big picture, to get the test done safely or with the appropriate amount of risk. I have to decide if the probability that we are going to get the data we want is the worth the risk that we are going to be taking. It’s something we think about ahead of time, and also something we think about the day of the test.
In one sense, on a perfect test day with no anomalies, I don’t have anything to do. All I do is watch what other people are doing, make sure that I am happy with how that’s all going.
But you’re there poised at every second, not letting your eye fall off of anything, to be ready to step in and say “Wait a minute, stop, we need to pay closer attention to that. That’s a potential danger sign that we just missed.”
My job is to be ready to stop the test, but also be ready to say “we’re ready to go.”
It’s the rare test day that goes by without there being some hard call to make about what we want to live with—or not.
This weekend saw the departure of a dear friend and longtime member of the XCOR family. Mouser, age 11, passed away in her sleep early Saturday morning.
It is very important for a spacecraft manufacturer to be free of pests, especially mice. They work their way into rocket plumbing and aircraft, and chew up wiring among other nuisances.
So early on, XCOR knew it needed a cat who would be experienced with catching rodents.
Not a cat that had been raised in a shelter, or anywhere away from nature (though we love shelter cats).
Certainly not a house cat.
Mouser was born some time in 2002. We adopted her soon after. She was, from the first, hell on mice.
In the entire time we had Mouser, we never saw a single rodent (after she rid us of the initial batch in Hangar 61, firmly encamped prior to our arrival).
She also caught the occasional bird and had showdowns with several squirrels who on rare occasion wandered into the shop.
Mouser provided comic relief on long days and nights of testing and project deadlines, and was attuned to the rhythms and sounds of the shop. She trained herself to run for cover whenever a rocket test was about to occur, and did not like the noise of the teacart engines that we test in the hangar.
For example, Mouser gradually learned to associate steps being read on Doug Jones’s checklist with impending noise. Whenever Doug arrived at the moment in the checklist where he yells ‘HEARING PROTECTION EVERYONE!”–on cue, Mouser scurried through the office cat door and under Aleta’s desk, making up for her lack of hearing protection.
Mouser also had a bit of a following outside of XCOR. Once she was featured by our friends at “The Fruit Guys” who keep us stocked with snacks, and she was adored by many of our investors, clients, visiting government officials, Mojave Spaceport personnel and XCOR family members.
Like all cats, Mouser liked sleep, food and laying in sunlight. On sunny mornings, we opened the hangar doors and Mouser would wander outside–but never too far–to get her dose of sunshine. Occasionally someone would set a chair out in the hangar, in the streaming sunlight for her to nap on, or she would just chill on the floor.
One particular chair she liked so much, we put a piece of tape on the back and dubbed it the “Mouser hair chair” because it was so covered with fur. She liked being spun around in swivel chairs as well. She knew the sound of her own name and would give you a sharp meow if alert, or a groggy mrrowr if half asleep. Greeting her was also frequently an occasion for her to ask for a snack, by means of following you around if you passed near the food bowl. And to keep people from accidentally over-feeding her, Mouser had her own employee timesheet with a log for who had fed her for breakfast, mid-day snack and dinner.
Mouser will be remembered as a fixture in the early days of XCOR, with us every step as we grew from ten people to over fifty. She saw the construction and flights of two rocket powered aircraft, countless engines, pumps and other significant company achievements.
More importantly, she played her part in making those flights possible.
Every visitor who saw Mouser loved her and loved the idea of a cat in a rocket hangar.
Rest in peace, Mouser. And wherever you rest there is, we know, a chair with a ray of sunshine ready just for you.
After running through LOX fill, setup, and pressurization checklists, the cold flow commences with a beautiful stream of liquid oxygen flowing from the engine. Flow rates and temperatures are recorded and checked both in real time and later during a data review. Functionality of the entire system is evaluated. If necessary, more flows are conducted, or the system is prepared for a hot fire test in the coming days.
Tune in this weekend and we will update you on where to find XCOR on the road (and online), and an update on how we will answer more of your questions in the coming weeks.
As always, ping us on Twitter or Facebook with your comments, or leave a comment right here.
Geoff Licciardello acts as control, Jeremy Voigt reads checklist, and Lee Draper and Dale Amon perform Data Acquisition. More on the specifics of each role can be found here.
Instead of a control bunker, a table is set with the same control box and data acquisition setup we would have for a live hotfire. Since cold flows are close by the hangar, crews larger than those at a hot fire can participate in discussions between flows.
XCOR CEO Jeff Greason keeps a keen eye on the test for any sign of a problem.
A number of XCOR staff discuss the results of a cold flow and decide what to do next.
Tomorrow, experience the cold flow!
Questions or comments? Hit us up on Twitter or Facebook, or in the comments section below.
Jeremy Voigt (Control) and Ray Fitting (Red Team) are shown maneuvering the test stand into position with a hitch dolly.
The bottle trailer is a test stand we use to supply our valves and tanks will pressurization gas. The trailer holds 10 gaseous helium tanks and 8 gaseous nitrogen tanks. It also carries a standalone gaseous nitrogen sphere that bypasses the normal test stand plumbing for use in and around the hangar.
Ray Fitting opens one of the valves on a bottle trailer tank.
The idea behind the bottle trailer is similar to that of XCOR’s other test stands. We needed a mobile platform that could be versatile and make our pressurization gasses available wherever our testing might be conducted.
After LOX fill, the stand is moved into position in front of the hangar doors and the bottle trailer is rolled up next to it. Plumbing lines are connected between the two test stands and we are ready to get underway
Tomorrow: join us as we set-up for a cold flow.
Questions or comments? Connect with us on Twitter or Facebook, or connect with us below.
We ended last week with a hot fire in action. This week we’re going to chill, and show you a “day in the life” of a cold flow.
Your questions and comments matter to us, so comment or ask us anything on Twitter, Facebook or right here on the blog.
So “what is a cold flow”, you ask? A cold flow is a test where fuel or liquid oxygen is moved through the plumbing system as it would be during a real engine test without lighting the igniter. The point is to serve as a final check of plumbing and valves before a hot fire. This allows us to find and fix minor leaks and valve reliability before heading all the way out to the test site.
The day of a cold flow begins much like the day of a hot fire. First, the stand is rolled out, checked and then moved to the large stationary LOX dewar (tank) for fill.
In the top photo, Geoff Licciardello reads the LOX fill checklist while senior engineer Mike Valant handles the fill hose and senior technician Mike Laughlin observes. At bottom, Derek Nye, Geo and Jeremy Voigt make further preparations in advance of a cold flow.
If the cold flow will be a test of the fuel system, kerosene will be loaded from storage barrels using an electric pump.
When flowing liquid oxygen during a cold-flow test, the LOX is pumped out into the atmosphere (into the air) where it evaporates instantly and becomes gaseous oxygen. When flowing fuel during a cold-flow test (never done at the same time as LOX), the fuel is pumped into a closed fuel container to be re-used.
As you may have guessed by now, a cold flow has the added advantage of happening right outside the XCOR hangar. In this series we’ll showcase a liquid oxygen cold-flow happening just outside the hangar doors.
After the crew has run through the stand setup, controls, and pressurization checklists (and a little lunch) they settle into the day’s series of hotfire objectives. Sometimes it can be a duration goal, a certain number of successful tests in a row with a new part, a certain pressure setting, fuel / LOX ratio change, or any combination of test objectives to further the test program.
The countdown for a hot fire is really the rundown of a detailed checklist that has been customized to test objectives. A majority of the checklist is nearly identical to many other tests, but there may be slight differences depending on the objective. There are built-in holds for tasks such as chilling down the pumps or waiting for the system to pressurize.
Regardless, as the checklist nears the bottom, the tension in the bunker starts to rise. Is it all correct? Did we set it up right? Are there any leaks? Will the new design work properly? All of those questions and more run through the team.
Then it’s time … “Engine run in three, two, armed! …”
Check out this video to experience what it’s like to prep and execute a hot fire:
This image (the thumbnail shot in the video above) was taken remotely by XCOR photographer and video engineer Mike Massee. It shows a brilliant plume overpowering the bright noon desert sky.
When a test series is concluded, the stand is disconnected, the bunker test equipment broken down and everything is packed for the return. Our convoy makes its way back to the XCOR hangar, and on a good test day where everything goes smoothly, the crew typically returns between 3 and 4:30 pm. On later days it’s between 9 and 10pm.
After each test, an “after-action” meeting is held while all the information is still fresh in everyone’s heads. Each person present at the test is encouraged to bring up any issue, no matter how minor. There are no “dumb comments”, and all comments major and minor are captured on a “squawk list”. Suggested test stand or procedural changes are assigned to be completed before the next test. Data review is scheduled for the next day after everyone has had a chance to process and review the data on their own. And based on the data review, goals are set for the next test series.
Thanks for joining us for the “day in the life of a hot fire” series. Let us know what you think of it by connecting with us on Twitter or Facebook, or by emailing me at bryan [at] xcor [dot] com.