This is the first in a series of posts “from the bunker,” interviews with the people behind the scenes at every hot fire and cold flow that takes place at XCOR…
Bryan Campen: Where did the name “Test Czar” originate?
Jeff Greason: The reason we came to call this role the Test Czar—it’s cold in the winter in Mojave and I had a Russian Navy version of a Ushanka, you know the full fur hats you’ve probably seen.
Not Jeff Greason: XCOR Engineer Mike Valant performs test czar duties at a recent hot fire, complete with Russian Ushanka
I had picked one up in Moscow not long after the movie The Hunt for Red October came out in theaters. Anyway it was cold out, and I was wearing that hat and the guys started calling me czar, and it stuck.
Jeff Greason acts as test czar during a recent cold flow, without the standard Ushanka
BC: So what does a test czar do?
JG: If you’re familiar with the Apollo missions, there was always a flight director.
And during that time, in the sixties, there were different people in front of computer screens, each responsible for some piece of the system.
Mission Control during the Apollo 11 mission
And there were all these people with individual roles and responsibilities that made-up mission control.
NASA also found it very helpful to have somebody who was only worried about the big picture, and not about the specific details of any one of those systems.
When doing this job, you have to be aware of two polar opposite traps that are very easy to get into when doing test operations.
From time to time, anomalies crop up and you have to make an informed decision on whether you can live with the anomaly on that day—or not. If you always stop everything, any time there is anything unusual, you’d never get anything done.
But the other disease that’s easy to fall into is go fever. You’ve been working toward this test a long time, you get up early that morning, you have momentum built-up, people are doing their steps and some little problem crops up and it’s very, very easy to say “well you know, don’t worry about that, we’ll deal with that later. Let’s just keep going.”
The guy reading the checklist reads a step, people say it’s done and he reads another step, done. He wants to read the next step. But what if it’s not done?
We have to stop.
So the czar looks at the big picture, watches the momentum, the ebb and flow of the day. He watch for people who are not listening, for the danger signs, and for who is going too fast. But you also watch the other extreme: Not only are you making sure the team doesn’t go too fast, but the czar will also step in sometimes and say “OK, I understand this is an interesting problem. My judgment is that that is not worth stopping [the test] for, so let’s make sure we capture that problem for later analysis.”
Or, maybe that stop is indicating for us something we need to check on again later in the day, so we’ll add a step to the checklist later on to check the health of that system, to make sure we don’t forget.
You have to keep your eye on all these moving parts. And if it gets too stressful that’s usually a sign that it’s time to slow down.
We also have a Safety Officer, or “Safety”, whose only job is to stop things, in some sense. [Note: we’ll cover the Safety Officer role in a later post]
Safety is there looking for problems.
Everybody else is looking for how to get it done.
And the czar is there to look at the big picture, to get the test done safely or with the appropriate amount of risk. I have to decide if the probability that we are going to get the data we want is the worth the risk that we are going to be taking. It’s something we think about ahead of time, and also something we think about the day of the test.
In one sense, on a perfect test day with no anomalies, I don’t have anything to do. All I do is watch what other people are doing, make sure that I am happy with how that’s all going.
But you’re there poised at every second, not letting your eye fall off of anything, to be ready to step in and say “Wait a minute, stop, we need to pay closer attention to that. That’s a potential danger sign that we just missed.”
My job is to be ready to stop the test, but also be ready to say “we’re ready to go.”
It’s the rare test day that goes by without there being some hard call to make about what we want to live with—or not.
This weekend saw the departure of a dear friend and longtime member of the XCOR family. Mouser, age 11, passed away in her sleep early Saturday morning.
It is very important for a spacecraft manufacturer to be free of pests, especially mice. They work their way into rocket plumbing and aircraft, and chew up wiring among other nuisances.
So early on, XCOR knew it needed a cat who would be experienced with catching rodents.
Not a cat that had been raised in a shelter, or anywhere away from nature (though we love shelter cats).
Certainly not a house cat.
Mouser was born some time in 2002. We adopted her soon after. She was, from the first, hell on mice.
In the entire time we had Mouser, we never saw a single rodent (after she rid us of the initial batch in Hangar 61, firmly encamped prior to our arrival).
She also caught the occasional bird and had showdowns with several squirrels who on rare occasion wandered into the shop.
Mouser provided comic relief on long days and nights of testing and project deadlines, and was attuned to the rhythms and sounds of the shop. She trained herself to run for cover whenever a rocket test was about to occur, and did not like the noise of the teacart engines that we test in the hangar.
For example, Mouser gradually learned to associate steps being read on Doug Jones’s checklist with impending noise. Whenever Doug arrived at the moment in the checklist where he yells ‘HEARING PROTECTION EVERYONE!”–on cue, Mouser scurried through the office cat door and under Aleta’s desk, making up for her lack of hearing protection.
Mouser also had a bit of a following outside of XCOR. Once she was featured by our friends at “The Fruit Guys” who keep us stocked with snacks, and she was adored by many of our investors, clients, visiting government officials, Mojave Spaceport personnel and XCOR family members.
Like all cats, Mouser liked sleep, food and laying in sunlight. On sunny mornings, we opened the hangar doors and Mouser would wander outside–but never too far–to get her dose of sunshine. Occasionally someone would set a chair out in the hangar, in the streaming sunlight for her to nap on, or she would just chill on the floor.
One particular chair she liked so much, we put a piece of tape on the back and dubbed it the “Mouser hair chair” because it was so covered with fur. She liked being spun around in swivel chairs as well. She knew the sound of her own name and would give you a sharp meow if alert, or a groggy mrrowr if half asleep. Greeting her was also frequently an occasion for her to ask for a snack, by means of following you around if you passed near the food bowl. And to keep people from accidentally over-feeding her, Mouser had her own employee timesheet with a log for who had fed her for breakfast, mid-day snack and dinner.
Mouser will be remembered as a fixture in the early days of XCOR, with us every step as we grew from ten people to over fifty. She saw the construction and flights of two rocket powered aircraft, countless engines, pumps and other significant company achievements.
More importantly, she played her part in making those flights possible.
Every visitor who saw Mouser loved her and loved the idea of a cat in a rocket hangar.
Rest in peace, Mouser. And wherever you rest there is, we know, a chair with a ray of sunshine ready just for you.
After running through LOX fill, setup, and pressurization checklists, the cold flow commences with a beautiful stream of liquid oxygen flowing from the engine. Flow rates and temperatures are recorded and checked both in real time and later during a data review. Functionality of the entire system is evaluated. If necessary, more flows are conducted, or the system is prepared for a hot fire test in the coming days.
Tune in this weekend and we will update you on where to find XCOR on the road (and online), and an update on how we will answer more of your questions in the coming weeks.
As always, ping us on Twitter or Facebook with your comments, or leave a comment right here.
Geoff Licciardello acts as control, Jeremy Voigt reads checklist, and Lee Draper and Dale Amon perform Data Acquisition. More on the specifics of each role can be found here.
Instead of a control bunker, a table is set with the same control box and data acquisition setup we would have for a live hotfire. Since cold flows are close by the hangar, crews larger than those at a hot fire can participate in discussions between flows.
XCOR CEO Jeff Greason keeps a keen eye on the test for any sign of a problem.
A number of XCOR staff discuss the results of a cold flow and decide what to do next.
Tomorrow, experience the cold flow!
Questions or comments? Hit us up on Twitter or Facebook, or in the comments section below.
Jeremy Voigt (Control) and Ray Fitting (Red Team) are shown maneuvering the test stand into position with a hitch dolly.
The bottle trailer is a test stand we use to supply our valves and tanks will pressurization gas. The trailer holds 10 gaseous helium tanks and 8 gaseous nitrogen tanks. It also carries a standalone gaseous nitrogen sphere that bypasses the normal test stand plumbing for use in and around the hangar.
Ray Fitting opens one of the valves on a bottle trailer tank.
The idea behind the bottle trailer is similar to that of XCOR’s other test stands. We needed a mobile platform that could be versatile and make our pressurization gasses available wherever our testing might be conducted.
After LOX fill, the stand is moved into position in front of the hangar doors and the bottle trailer is rolled up next to it. Plumbing lines are connected between the two test stands and we are ready to get underway
Tomorrow: join us as we set-up for a cold flow.
Questions or comments? Connect with us on Twitter or Facebook, or connect with us below.
We ended last week with a hot fire in action. This week we’re going to chill, and show you a “day in the life” of a cold flow.
Your questions and comments matter to us, so comment or ask us anything on Twitter, Facebook or right here on the blog.
So “what is a cold flow”, you ask? A cold flow is a test where fuel or liquid oxygen is moved through the plumbing system as it would be during a real engine test without lighting the igniter. The point is to serve as a final check of plumbing and valves before a hot fire. This allows us to find and fix minor leaks and valve reliability before heading all the way out to the test site.
The day of a cold flow begins much like the day of a hot fire. First, the stand is rolled out, checked and then moved to the large stationary LOX dewar (tank) for fill.
In the top photo, Geoff Licciardello reads the LOX fill checklist while senior engineer Mike Valant handles the fill hose and senior technician Mike Laughlin observes. At bottom, Derek Nye, Geo and Jeremy Voigt make further preparations in advance of a cold flow.
If the cold flow will be a test of the fuel system, kerosene will be loaded from storage barrels using an electric pump.
When flowing liquid oxygen during a cold-flow test, the LOX is pumped out into the atmosphere (into the air) where it evaporates instantly and becomes gaseous oxygen. When flowing fuel during a cold-flow test (never done at the same time as LOX), the fuel is pumped into a closed fuel container to be re-used.
As you may have guessed by now, a cold flow has the added advantage of happening right outside the XCOR hangar. In this series we’ll showcase a liquid oxygen cold-flow happening just outside the hangar doors.
After the crew has run through the stand setup, controls, and pressurization checklists (and a little lunch) they settle into the day’s series of hotfire objectives. Sometimes it can be a duration goal, a certain number of successful tests in a row with a new part, a certain pressure setting, fuel / LOX ratio change, or any combination of test objectives to further the test program.
The countdown for a hot fire is really the rundown of a detailed checklist that has been customized to test objectives. A majority of the checklist is nearly identical to many other tests, but there may be slight differences depending on the objective. There are built-in holds for tasks such as chilling down the pumps or waiting for the system to pressurize.
Regardless, as the checklist nears the bottom, the tension in the bunker starts to rise. Is it all correct? Did we set it up right? Are there any leaks? Will the new design work properly? All of those questions and more run through the team.
Then it’s time … “Engine run in three, two, armed! …”
Check out this video to experience what it’s like to prep and execute a hot fire:
This image (the thumbnail shot in the video above) was taken remotely by XCOR photographer and video engineer Mike Massee. It shows a brilliant plume overpowering the bright noon desert sky.
When a test series is concluded, the stand is disconnected, the bunker test equipment broken down and everything is packed for the return. Our convoy makes its way back to the XCOR hangar, and on a good test day where everything goes smoothly, the crew typically returns between 3 and 4:30 pm. On later days it’s between 9 and 10pm.
After each test, an “after-action” meeting is held while all the information is still fresh in everyone’s heads. Each person present at the test is encouraged to bring up any issue, no matter how minor. There are no “dumb comments”, and all comments major and minor are captured on a “squawk list”. Suggested test stand or procedural changes are assigned to be completed before the next test. Data review is scheduled for the next day after everyone has had a chance to process and review the data on their own. And based on the data review, goals are set for the next test series.
Thanks for joining us for the “day in the life of a hot fire” series. Let us know what you think of it by connecting with us on Twitter or Facebook, or by emailing me at bryan [at] xcor [dot] com.
So far this week, we have discussed the build-up to a hot fire.
Today, we’ll talk about the roles we play in the process.
In the bunker, a wall of video monitors gives the test crew excellent situational awareness. Cameras are placed to accommodate the specific test and points of interest. If needed, high speed and infrared cameras gather additional data. And on the radio, crew members are called-out by their positions.
Crew positions, roles and responsibilities for each hot fire include the following:
Control:Runs the control box and actually commands the rocket engine to fire.
DAQ:Runs the Data AcquisitionUnitwhich records data from instruments on the stand. They are also responsible for displaying the data for instant review by the test crew after the test.
Red Team: This team directly interacts with the test stand while everyone else is inside the bunker. They return to the bunker during pressurization and during the actual hotfire.
Test Czar: This individual is in charge of the test and has the final say on go / no-go conditions, and sets the test objectives and parameters. Acts as a safety net for the rest of the team, considering all possible risks before giving the green light.
Spotter: Acts as an additional safety check and spends most of the day seated on top of the bunker to keep an eye out for anyone approaching the test site. The spotter makes sure that the area down range of the rocket engine remains clear, and is only called-in just before the actual test.
Video: In charge of setting up and maintaining all the video feeds from the test site. The objective is to make sure to record all of the video during press, chill and hotfire.
Checklist:Acts as the pace setter for the test. By reading the checklist methodically and accurately they set the tempo for the test day. The checklist contains all of the commands for all of the other positions except the Test Czar.
Safety: The primary safety officer in charge of ensuring all protocols are followed for the whole test. Safety may stop a test at anytime for any reason if they feel the crew or the uninvolved public are at risk.
On Tuesday we told you about preparing to roll out for a hot fire. Wednesday was the journey to the test stand.
Today, what happens when we arrive …
Once we are at the test site, there is a flurry of activity. We attach the truss and fuselage to the test stand, a concrete slab rated for up to 80,000 lbf of thrust. It is a monster of a concrete pad, and contains various types of mounting brackets that can be used for a range of engine test stands.
Next to the test stand is our bunker. It is an old WWII ammunition storage bunker built by the US Navy Seabees for the Marine Corps, right around 1942. It is capable of withstanding the explosive energy of 435,000 pounds of TNT.
Shot atop the XCOR bunker, this photo shows the test stand in the foreground, and a bunker similar to ours in the background.
The bunker has very thick concrete walls and is covered by yards of earth. As a result it is cool in the summer and warm in the winter, with six inch-thick steel doors. It would make one heck of a man cave!
The interior of the XCOR bunker
We purposefully do not trick it out, because our philosophy is to have one culture at XCOR, not to have different mini-cultures grow up inside the company such as “the guys in the test bunker” or “those test stand fabricators” or “the composite team”. Everyone is one big team, and we all enjoy the simplicity of a day in the bunker!
As all of our test equipment is portable, very little remains out at the test bunker. When we arrive, the test stand is bolted to the test pad, cables are unreeled from the pad to the bunker, cameras are set up, and computer gear is electrically connected. Once everything is in position, gas and fluid cables for pressurization and valve control are attached and the “control room” inside the bunker goes live. The whole team is ready to ”get on checklist”.
XCOR senior shop mechanic Mike Laughlin checks a high-pressure hose fitting in preparation for a hotfire test.
Tomorrow we will show you the roles and responsibilities for the test, and you can decide what role you want to play!