Geoff Licciardello hauls the control box out to the XCOR bunker.
Today we are breaking things up, and providing you the first part of a two-part interview…
This is the fourth in a series of posts “from the bunker,” interviews with the people behind the scenes at every hot fire and cold flow that takes place at XCOR. Today we continue our discussion about the role of Control, this time with engineer Geoff Licciardello. Geo previously worked as Control on both the Lynx truss (5K18) and LOX pump tests, and currently works on all hydrogen program testing at XCOR.
Geoff Licciardello: A control box is the box that will remotely control the valves and every system on the test stand.
BC: When do you start your work as Control, a day ahead of tests?
GL: It depends on what we’ve been doing. If this is the first time we bring a stand online, usually we’ll have at least one day, sometimes more days, of shakedown testing before we run out to the bunker to do a real test. We’ll run through all the systems, do some simulated tests, maybe some cold flows to address any initial issues from new hardware before a new test.
But if it’s a mature system, sometimes it’s just “get everything ready to go,” then go out to the bunker for a test.
BC: What’s a shakedown?
GL: A shakedown is basically putting a stand through its paces and finding any issues with the stand. Basically you’re just putting the stand through all of its paces so that you can find any issues that come up and address them before it’s actually time to run the engine. It can be disruptive to a test if you have a simple error that could have been spotted earlier, one of those things that you can’t always fix in the field. Being able to find those problems ahead of time and then resolve them really saves a test day a lot of trouble.
BC: What’s the difference between a shakedown and a click test?
GL: So a click test is when we have the control box hooked-up to the stand and electrical power is up, but nothing else is enabled. We don’t have fluid in the tanks, we don’t have pressure on the stand. We are essentially just checking each valve making sure that all our wiring is correct and that the correct valve on the stand will click long before we test.
So it’s a way of detecting electrical or wiring issues.
A shakedown is much more thorough—in a shakedown we run through things as close to doing a real test as possible, so that we can find every issue that may crop up in the field.
BC: How does Control integrate with the rest of the team?
GL: It’s really important for Control, Checklist and Red Team to have very good communication. During a test day, those are the three roles that are doing a lot of the work.
Checklist is making sure we go through the steps in the proper order.
Then Control and Red Team are the ones who are actually turning valves, running systems.
If there isn’t good communication, that leads to problems. So it is really important that everyone is on the same page.
Control is a lot of responsibility. I am the one right there with all the switches when it’s go-time. If something happens I have to be ready to react and I have to know the system very well because I am the one actually commanding things. Control needs a good understanding of the system so that they know what actions will cause an issue or not.
More from Geo soon…
If you have questions or comments, you can ask them right here, or connect with us on Twitter or Facebook.
This is the third in a series of posts “from the bunker,” interviews with the people behind the scenes at every hot fire and cold flow that takes place at XCOR. Today we talk about the role of Control with XCOR engineer Jeremy Voigt (and soon with engineer Geoff Licciardello, who also works in this role at XCOR for another program).
Jeremy Voigt (Control, center). To his left is Brandon Litt (Checklist), and to his right is Jeff Greason (Test Czar).
Bryan Campen: So you started off as an intern? How did you end up in the role of Control?
Jeremy Voigt: For hot fires, interns never get to do things like Control. When I was an intern I started off as Spotter.
Then I graduated to video. Now all of a sudden instead of being outside, I’m inside sitting next to the head table. I can hear everything going on, all of the radio calls. At the same time I started reading checklists for another test stand, and was Red Team for that stand as well.
When I came back as a full time engineer I was given the responsibility of the Truss Test Stand. I started out on Checklist and worked my way up to Control Box.
So now I run Control Box and plan the tests for the Truss Test Stand.
BC: What does that mean?
JV: I’m in charge of the test stand. That means that when there are modifications that need to be done, I will either do them myself or delegate them to people to do, and then check over their work.
Then when it comes time to test, I talk with senior engineers–Doug [Jones], Jeff [Greason] and Dan [DeLong]–and we figure out the objectives of the test and what we need to do to meet them.
Then I will go to the crew and make sure everybody knows what’s going on and make sure they are ready for the test. I’ll send out emails on when we’re going to have the test, when we will have after action meetings and data reviews—all those things.
So I’m kind of the point of contact for the rest of the test crew.
On Control Box, I start days before the actual test. I will do this whenever we’ve broken into the stand. I’ll do a click test where I actuate every single valve. And I don’t do that on the day of the test, I do it beforehand, so that we know when we go out there that we won’t have any problems.
BC: What’s a click test?
JV: A click test is where I go through and actuate all of the valves and make sure the electronics wiring from the control box to the stand to each of the valves works. That they all move when they are supposed to and that they don’t move when they are not supposed to.
BC: Ok, here’s something I’ve heard you mention offhand that I want to ask you about specifically. Is it true during a hot fire that time actually slows down for you?
JV: Yeah. I’m counting in in my head, I have my hand on the stop button and I’m watching the video screens…
…and I think to myself “Oooh, that must have been about a minute! I wonder how the count’s going?”
And then you hear Checklist next to you scream out “Fifteen seconds!”…
On the left, Brandon Litt (Checklist) shouts the count with Jeremy (Control) to his right.
…and you can barely hear him and he’s shouting at the top of his lungs.
And then I say to myself “Oh man, I’ve got a long way to go.”
Every second feels like an hour of just watching, of making sure everything is going right. It’s an unbelievable feeling.
At larger companies, someone my age wouldn’t ever get near that experience. The thing I like about XCOR is I get that opportunity. It’s a lot of fun, getting to fire the rocket engine.
To work on this and then get to be able to test it, that’s unique to XCOR, and I never take it for granted.
We received a lot of photo inquiries the past few weeks, and have begun to catalog our content, by subject and week, over at Flickr. So this weekend we have a simple update: if you’re interested in the photos from this week’s stories, head over there!
Our dear friend Mouser passed away last Saturday morning. We’ve made a photo set for her “best of” shots here:
We began our series of interviews “from the bunker” on Tuesday, and will continue those next week. The shots so far, right here:
This is the second in a series of posts “from the bunker,” interviews with the people behind the scenes at every hot fire and cold flow that takes place at XCOR. Today we talk about the role of Checklist with XCOR engineer Brandon Litt.
The team confers during a test day. Left to right: Doug Jones (Chief Test Engineer), Jeremy Voigt (Control), Randall Clague (Safety Officer), Mike Valant (Senior Engineer), Jeff Greason (Test Czar and CEO), Brandon Litt (Checklist), Derek Nye (A&P) and Geoff Licciardello (Test Engineer)
Bryan Campen: So what does Checklist do?
Brandon Litt: The checklist reader gives commands to everybody on the test crew. And no one does anything on the test stand, on the control box, with the DAQ, unless the checklist specifically tells them to do it.
It has been through so many iterations that it is now the best way to get the test done successfully and safely.
The speed at which I read the checklist really sets the tempo for the day. If I’m reading it pretty fast, people might be on edge. We might miss things or skip lines if it were too fast. So it’s nice to go slow and deliberately and make sure everything is done the right way.
But if I’m going too slow people might get a little bit sluggish and lose focus, so I need to keep things going at a steady pace throughout the day. I also have to make sure that the commands I give are called back to me by the rest of the team.
BC:Have you ever stressed anyone out by going too fast?
BL: Oh yeah, when I started I’d get comments –“slow that up” or “take a pause here”– I’d push the radio button a little too early and maybe cut someone off while they were trying to say something.
So I tell Red Team to open a valve, wait for them to call back that they have opened that valve successfully, and then move on to the next step.
It requires a lot of mental focus to ensure that I am very precise.
I don’t just give commands. I wait for feedback.
BC:How many items are on the checklist?
BL: That’s a really good question. For a standard engine day we start with a rollout checklist, it includes all the things we bring with us when we get to the test site. We go through a stand setup checklist, which maybe has fifty items. Then we go through a control checklist which pretty much brings up the electrical system on the stand, and test every valve on the whole stand. It’s probably another fifty items. Then we go on to the pressurization steps for the engine run. The final checklist is only a few steps, it’s mostly verifications before we hit go.
BC:What are the best and most challenging parts of the job?
BL: It’s disappointing to go out there all day, read this checklist methodically, slowly, everything happens right, but for some reason we just have to call it a day, shut down and go home.
Counterpoint to that, it’s equally as amazing when do I do all those things, we push the button and that engine just works.
All of the positions on the crew don’t just do their respective jobs. All of the prep work in the hangar, and rollout, and for setting up the stand once you get out there, it’s a big group effort and everybody really has to chip in to get that thing ready to go.
So the checklist acts as that final verification. When everything is set up, if I am [for instance] trusting new engineers to do certain things during the setup, I’m going to verify those steps later in the checklist.
And rest assured I’m not just going to assume they were done correctly.
This is the first in a series of posts “from the bunker,” interviews with the people behind the scenes at every hot fire and cold flow that takes place at XCOR…
Bryan Campen: Where did the name “Test Czar” originate?
Jeff Greason: The reason we came to call this role the Test Czar—it’s cold in the winter in Mojave and I had a Russian Navy version of a Ushanka, you know the full fur hats you’ve probably seen.
Not Jeff Greason: XCOR Engineer Mike Valant performs test czar duties at a recent hot fire, complete with Russian Ushanka
I had picked one up in Moscow not long after the movie The Hunt for Red October came out in theaters. Anyway it was cold out, and I was wearing that hat and the guys started calling me czar, and it stuck.
Jeff Greason acts as test czar during a recent cold flow, without the standard Ushanka
BC: So what does a test czar do?
JG: If you’re familiar with the Apollo missions, there was always a flight director.
And during that time, in the sixties, there were different people in front of computer screens, each responsible for some piece of the system.
Mission Control during the Apollo 11 mission
And there were all these people with individual roles and responsibilities that made-up mission control.
NASA also found it very helpful to have somebody who was only worried about the big picture, and not about the specific details of any one of those systems.
When doing this job, you have to be aware of two polar opposite traps that are very easy to get into when doing test operations.
From time to time, anomalies crop up and you have to make an informed decision on whether you can live with the anomaly on that day—or not. If you always stop everything, any time there is anything unusual, you’d never get anything done.
But the other disease that’s easy to fall into is go fever. You’ve been working toward this test a long time, you get up early that morning, you have momentum built-up, people are doing their steps and some little problem crops up and it’s very, very easy to say “well you know, don’t worry about that, we’ll deal with that later. Let’s just keep going.”
The guy reading the checklist reads a step, people say it’s done and he reads another step, done. He wants to read the next step. But what if it’s not done?
We have to stop.
So the czar looks at the big picture, watches the momentum, the ebb and flow of the day. He watch for people who are not listening, for the danger signs, and for who is going too fast. But you also watch the other extreme: Not only are you making sure the team doesn’t go too fast, but the czar will also step in sometimes and say “OK, I understand this is an interesting problem. My judgment is that that is not worth stopping [the test] for, so let’s make sure we capture that problem for later analysis.”
Or, maybe that stop is indicating for us something we need to check on again later in the day, so we’ll add a step to the checklist later on to check the health of that system, to make sure we don’t forget.
You have to keep your eye on all these moving parts. And if it gets too stressful that’s usually a sign that it’s time to slow down.
We also have a Safety Officer, or “Safety”, whose only job is to stop things, in some sense. [Note: we’ll cover the Safety Officer role in a later post]
Safety is there looking for problems.
Everybody else is looking for how to get it done.
And the czar is there to look at the big picture, to get the test done safely or with the appropriate amount of risk. I have to decide if the probability that we are going to get the data we want is the worth the risk that we are going to be taking. It’s something we think about ahead of time, and also something we think about the day of the test.
In one sense, on a perfect test day with no anomalies, I don’t have anything to do. All I do is watch what other people are doing, make sure that I am happy with how that’s all going.
But you’re there poised at every second, not letting your eye fall off of anything, to be ready to step in and say “Wait a minute, stop, we need to pay closer attention to that. That’s a potential danger sign that we just missed.”
My job is to be ready to stop the test, but also be ready to say “we’re ready to go.”
It’s the rare test day that goes by without there being some hard call to make about what we want to live with—or not.
This weekend saw the departure of a dear friend and longtime member of the XCOR family. Mouser, age 11, passed away in her sleep early Saturday morning.
It is very important for a spacecraft manufacturer to be free of pests, especially mice. They work their way into rocket plumbing and aircraft, and chew up wiring among other nuisances.
So early on, XCOR knew it needed a cat who would be experienced with catching rodents.
Not a cat that had been raised in a shelter, or anywhere away from nature (though we love shelter cats).
Certainly not a house cat.
Mouser was born some time in 2002. We adopted her soon after. She was, from the first, hell on mice.
In the entire time we had Mouser, we never saw a single rodent (after she rid us of the initial batch in Hangar 61, firmly encamped prior to our arrival).
She also caught the occasional bird and had showdowns with several squirrels who on rare occasion wandered into the shop.
Mouser provided comic relief on long days and nights of testing and project deadlines, and was attuned to the rhythms and sounds of the shop. She trained herself to run for cover whenever a rocket test was about to occur, and did not like the noise of the teacart engines that we test in the hangar.
For example, Mouser gradually learned to associate steps being read on Doug Jones’s checklist with impending noise. Whenever Doug arrived at the moment in the checklist where he yells ‘HEARING PROTECTION EVERYONE!”–on cue, Mouser scurried through the office cat door and under Aleta’s desk, making up for her lack of hearing protection.
Mouser also had a bit of a following outside of XCOR. Once she was featured by our friends at “The Fruit Guys” who keep us stocked with snacks, and she was adored by many of our investors, clients, visiting government officials, Mojave Spaceport personnel and XCOR family members.
Like all cats, Mouser liked sleep, food and laying in sunlight. On sunny mornings, we opened the hangar doors and Mouser would wander outside–but never too far–to get her dose of sunshine. Occasionally someone would set a chair out in the hangar, in the streaming sunlight for her to nap on, or she would just chill on the floor.
One particular chair she liked so much, we put a piece of tape on the back and dubbed it the “Mouser hair chair” because it was so covered with fur. She liked being spun around in swivel chairs as well. She knew the sound of her own name and would give you a sharp meow if alert, or a groggy mrrowr if half asleep. Greeting her was also frequently an occasion for her to ask for a snack, by means of following you around if you passed near the food bowl. And to keep people from accidentally over-feeding her, Mouser had her own employee timesheet with a log for who had fed her for breakfast, mid-day snack and dinner.
Mouser will be remembered as a fixture in the early days of XCOR, with us every step as we grew from ten people to over fifty. She saw the construction and flights of two rocket powered aircraft, countless engines, pumps and other significant company achievements.
More importantly, she played her part in making those flights possible.
Every visitor who saw Mouser loved her and loved the idea of a cat in a rocket hangar.
Rest in peace, Mouser. And wherever you rest there is, we know, a chair with a ray of sunshine ready just for you.
After running through LOX fill, setup, and pressurization checklists, the cold flow commences with a beautiful stream of liquid oxygen flowing from the engine. Flow rates and temperatures are recorded and checked both in real time and later during a data review. Functionality of the entire system is evaluated. If necessary, more flows are conducted, or the system is prepared for a hot fire test in the coming days.
Tune in this weekend and we will update you on where to find XCOR on the road (and online), and an update on how we will answer more of your questions in the coming weeks.
As always, ping us on Twitter or Facebook with your comments, or leave a comment right here.
Geoff Licciardello acts as control, Jeremy Voigt reads checklist, and Lee Draper and Dale Amon perform Data Acquisition. More on the specifics of each role can be found here.
Instead of a control bunker, a table is set with the same control box and data acquisition setup we would have for a live hotfire. Since cold flows are close by the hangar, crews larger than those at a hot fire can participate in discussions between flows.
XCOR CEO Jeff Greason keeps a keen eye on the test for any sign of a problem.
A number of XCOR staff discuss the results of a cold flow and decide what to do next.
Tomorrow, experience the cold flow!
Questions or comments? Hit us up on Twitter or Facebook, or in the comments section below.