Today we conclude an interview with XCOR’s Mike Massee, the guy behind the videos captured at every hot fire.
If you have questions or comments, you can ask them right here, or connect with us on Twitter or Facebook.
Bryan Campen: Jeff [Greason] often talks about being able to read shock diamonds with the naked eye – the brain is faster than any computer in being able to read a rocket engine during a hot fire – so being in a bunker, it’s very important to have video. What set of selections do you have in terms of equipment or set-up to make it better, to make it an even faster read?
Mike Massee: Early on when we started, when I started in 2001 the entire world of broadcast video was still in standard definition. And my big goal was to move into high definition as soon as it became affordable at a reasonable rate to do so, because there’s so much more situational awareness with that clarity. So the first chance I had, I started switching out things with HD.
That made a huge difference, having big monitors with lots of pixels. Instead of a little fuzzy dot that was a valve, you could see the valve, you could see which way it was actuated, and resolution is, probably the most important thing is having more resolution. As the world starts to get into 4K, which is quad HD basically, almost four thousand pixels across on a video screen, that will bring an even more window-like clarity to the world. I like to call the video wall we have in the bunker the world’s most expensive window. We are looking right through the wall at what’s on the other side of it – except that you can zoom and look into different spots.
I’m reminded of the video screen in Back To The Future II that they have in their future home. They have this screen that they pull down in front of the window that gives them a view of whatever they want to look at. That’s how I’ve always kind of seen it, it is just a big window.
BC: Have you ever destroyed any equipment? Be honest.
MM: I’ve put a lot of equipment in the line of fire. Certainly very, very close to operating rocket engines. But since there has never been a hard start in the history of the company, I have not lost any cameras. The worst camera that got damaged – and it still worked after that – was in a really hellacious sand storm. So the worst damage came from Mojave weather and not from the rocket engines or the propulsion systems, interestingly enough. So that’s–I’ve really been very fortunate that I’ve put some very expensive equipment on the line and we’ve not had to replace any of it. Our engines, since they’re safer and reliable, I feel comfortable putting equipment out there, and I haven’t really had any problems with that. The main thing is they have to, if they’re close to the engine, they have to survive a very harsh vibratory environment because there’s a lot of vibration radiation that comes off of them. So there are some special considerations there. But, other than that, I’ve had very good luck here. That’s just a testament to the company’s propulsion technology as a whole.
Today we kick-off an interview with XCOR’s Mike Massee, the guy behind the videos captured at every hot fire.
If you have questions or comments, you can ask them right here, or connect with us on Twitter or Facebook.
Bryan Campen:So what do you do at XCOR and specifically on test days?
Mike Massee: I run media imaging and graphics at XCOR. That includes test documentation, video and the onboard video systems for the Lynx.
On the day of a hot fire, I set up and record multiple channels of video at the test site. This system both gathers visual documentation for the test as well as provides situational awareness in real time for the crew. I also take still photographs and remote controlled still photographic imagery of rocket engine tests.
BC: You’ve been doing this a while, right? Your particular focus in terms of documentation with film here is on rocket firings and everything surrounding it, probably more than almost anybody in this space.
Methane Blast – This photo of XCOR’s 5M15 LOX-Methane engine is one of the most popular that Mike has taken. It has been featured in countless magazines and websites and several books. This engine was developed as a lunar ascent module prototype together with ATK and NASA. This photo has been nicknamed “The Death Ray” shot by XCOR personnel.
MM: In terms of sheer quantity we tend to do a lot of tests here because of the reusability of our engines and the frequency at which we test. Because there are non-toxic propellants, the set up and handling requirements are such are that we have a very rapid turnaround cycle, so I’ve photographed a lot of different engine types over the years and a lot of different tests on each one of those engine types.
I’ve come to specialize in rocket photography and in shock diamond photography, things like that. Some of it is technical photography and some artistic, beautiful stuff.
BC: What was the learning curve like?
MM: The curve is learning about the dynamic range or brightness of the rocket engine plumes, and how to set the camera to take advantage of that artistically and forensically. The different rocket engine fuels all produce plumes of varying brightness and color. For example, kerosene is much, much, much, much brighter than anything else we test with. And so, you can play around with that a lot by intentionally underexposing the photo to highlight the plume or in the case of, say, methane and alcohol and hydrogen rocket engine plumes, they’re more visibly dimmer.
BC: So when is the best time to shoot, or how do you work around your circumstances?
MM: It’s much more difficult to get a clear photograph of the plumes in broad daylight. So you have to be kind of creative, shoot at different times of day or highlight the engines and compose them and frame them in different ways so that the plume actually shows up in the photo. There are also issues with high frequency vibrations which affects what shutter speed you use.
The 3E17 RCS engine prototype used N2O and Ethane, storable, non-cryogenic propellants.
BC: I have two questions then. One, how do you set up usually? And what’s the crew looking for in a shot?
MM: For any given test day there’s typically a default set-up which includes an overview of the stand, and the stand is framed closely and the entire stand in one view, and then there’s a wide overview of the area for situational awareness. Kind of like a security camera. And beyond that, there’s a number of specialty cameras where they can say hey, I need a shot of the pumps on the other side of the stand because we can’t see them in the other shots, or I want a close up of the engine plume, or I want a picture of this particular valve – and then we move a camera out there and there’s power and transmission of that back to the bunker.
So there is a basic setup for most engine tests that’s the same, and then there are any number of specialty cameras that people would like to place in different areas, depending on what we’re looking at. If cameras are available and I want to do it, I place them for artistic shots to use for promotional purposes.
An early run of the Lynx “5K18″ main engine on XCOR’s original “5K” stand (designed to handle engines up to 5,000 lbs of thrust) this photo is taken in bright noon daylight, but underexposed to highlight the luminosity of the plume.
BC: What cameras do you use to shoot, what’s the setup?
MM: A lot of the regular cameras are off the shelf inexpensive HD cameras, but there are a lot of converter boxes and switchers and quad splitters and things that allow us to monitor and record the cameras remotely in real time and routers to put any image on any screen. It’s a little television studio in a box, and that’s something I’ve built up over time. I have a background as a broadcast engineer.
Some of the specialty cameras we have are infrared and high speed. We use infrared cameras to detect hot spots, leaks, and things like that. And high speed cameras for special tests that have some type of phenomena that’s so high frequency that it cannot be caught in a regular frame rate. So, where regular video is 30 or 60 frames per second, the high speed camera can go up to about 10,000 frames per second or more if you sacrifice resolution for frame rate.
Next week: Find out if Mike has ever destroyed equipment during a hot fire.
Geoff Licciardello (Control, right) with engineer Lee Draper (DAQ, left) and engineer Jeremy Voigt (Checklist, center) during a recent cold flow test at the XCOR hangar.
This is the fifth in a series of posts “from the bunker,” interviews with the people behind the scenes at every hot fire and cold flow that takes place at XCOR.
Today we complete an interview which began last week with XCOR’s Geoff Licciardello (in the role of Control). Geo previously worked as Control on both the Lynx truss (5K18) and LOX pump tests, and currently works on all hydrogen program testing at XCOR.
We start with a brief answer to a question of how time changes for the crew when an engine is fired.
Bryan Campen: Do you have this experience with time that Jeremy mentioned as well, that it slows down during a hot fire?
Geoff Licciardello: Yes. I call it the hot fire time warp. I might be set up to do a two second engine run, and I flip that switch and the engine starts running, and it feels like a long time for those two seconds.
But then I watch the video afterwards and go “oh, that was really fast!” It felt like ten seconds in front of the box, but everything seems to slow down a little bit when you’re actually there in person firing the engine. And I don’t know how to explain it, but everything seems to move in slow motion almost.
BC: When you began at XCOR, what was your first role on the test crew?
GL: My first job on the test crew was as an intern, where I ran DAQ for a couple tests. When I came back full time they threw me over to running the checklist. Checklist was where I spent most of my time my first couple of years at XCOR, at first with the 5K18 stand, and then the LOX pump test stand.
As the hydrogen program came online I transitioned to being Control for all hydrogen work, because I was experienced running pumps on the other stand. That transitioned to being Control for hot fires.
That’s my progression so far. We try to do cross-training as well so different people are trained in different roles. Then if for some reason I can’t come in one day, someone else could take over for me. And you need that understanding before you start flipping valves, or become a test czar for a test.
BC: So that’s your progression. I’m curious how Control integrates with what roles during a test day, starting with Test Czar…
GL: During the actual test day there’s not much interaction between Test Czar and Control. Usually it’s between Checklist and Control. On the day, Czar usually has the job of watching and waiting with his hand over the shoulder of Control. If there is an issue, he’ll clap you really hard on the shoulder to shut things down so that you can deal with things if there’s an anomaly.
But it’s Checklist and Control that have the tight interactions.
Every test we run through the checklist. It’s really important that we do things in the right order, that we set everything at the right pressures and go through it all in a very orderly fashion.
When a command is called out, I repeat back that command to let Checklist know I’ve done it. There are a lot of steps on the list that need to happen in the right sequence.
Having everyone understand where they are at and where they want to be keeps things very smooth for test day.
Chief Engineer Dan DeLong (Test Czar, left) in the XCOR bunker with engineer Mark Street (Control, right).
BC: One other question, about the Test Czar with the hand on the shoulder of Control. Can you explain that for readers?
GL: Sure. The tradition we’ve had at XCOR is, during an engine run, it’s a tense moment where things could happen quickly. Everyone is looking at the monitors and switches. Control has a lot do and is very focused on doing the job, but the Test Czar doesn’t have to worry about flipping switches. He is watching the engine and the stand, and if things need to stop quickly he is the best person to see that and to know that we should stop.
BC: Why not just shout?
GL: Engines are loud. It’s better than a verbal command you can’t see or feel.
BC: Anything else that I haven’t asked that you’d like to cover?
GL: Control Box is exciting. You feel a really strong connection with the hardware. And you are tangibly connected to everything that happens.
For anyone who wants to experience, the cool part is that you are the one right there with all the buttons, running the engine, flipping the switches. You are very involved with the test. It’s a lot of responsibility but at the same time, when you get a successful test off there is a high level of satisfaction.
If you’ve made it this far… congrats, and stay tuned for some pretty big news tomorrow!
Geoff Licciardello hauls the control box out to the XCOR bunker.
Today we are breaking things up, and providing you the first part of a two-part interview…
This is the fourth in a series of posts “from the bunker,” interviews with the people behind the scenes at every hot fire and cold flow that takes place at XCOR. Today we continue our discussion about the role of Control, this time with engineer Geoff Licciardello. Geo previously worked as Control on both the Lynx truss (5K18) and LOX pump tests, and currently works on all hydrogen program testing at XCOR.
Geoff Licciardello: A control box is the box that will remotely control the valves and every system on the test stand.
BC: When do you start your work as Control, a day ahead of tests?
GL: It depends on what we’ve been doing. If this is the first time we bring a stand online, usually we’ll have at least one day, sometimes more days, of shakedown testing before we run out to the bunker to do a real test. We’ll run through all the systems, do some simulated tests, maybe some cold flows to address any initial issues from new hardware before a new test.
But if it’s a mature system, sometimes it’s just “get everything ready to go,” then go out to the bunker for a test.
BC: What’s a shakedown?
GL: A shakedown is basically putting a stand through its paces and finding any issues with the stand. Basically you’re just putting the stand through all of its paces so that you can find any issues that come up and address them before it’s actually time to run the engine. It can be disruptive to a test if you have a simple error that could have been spotted earlier, one of those things that you can’t always fix in the field. Being able to find those problems ahead of time and then resolve them really saves a test day a lot of trouble.
BC: What’s the difference between a shakedown and a click test?
GL: So a click test is when we have the control box hooked-up to the stand and electrical power is up, but nothing else is enabled. We don’t have fluid in the tanks, we don’t have pressure on the stand. We are essentially just checking each valve making sure that all our wiring is correct and that the correct valve on the stand will click long before we test.
So it’s a way of detecting electrical or wiring issues.
A shakedown is much more thorough—in a shakedown we run through things as close to doing a real test as possible, so that we can find every issue that may crop up in the field.
BC: How does Control integrate with the rest of the team?
GL: It’s really important for Control, Checklist and Red Team to have very good communication. During a test day, those are the three roles that are doing a lot of the work.
Checklist is making sure we go through the steps in the proper order.
Then Control and Red Team are the ones who are actually turning valves, running systems.
If there isn’t good communication, that leads to problems. So it is really important that everyone is on the same page.
Control is a lot of responsibility. I am the one right there with all the switches when it’s go-time. If something happens I have to be ready to react and I have to know the system very well because I am the one actually commanding things. Control needs a good understanding of the system so that they know what actions will cause an issue or not.
More from Geo soon…
If you have questions or comments, you can ask them right here, or connect with us on Twitter or Facebook.
This is the third in a series of posts “from the bunker,” interviews with the people behind the scenes at every hot fire and cold flow that takes place at XCOR. Today we talk about the role of Control with XCOR engineer Jeremy Voigt (and soon with engineer Geoff Licciardello, who also works in this role at XCOR for another program).
Jeremy Voigt (Control, center). To his left is Brandon Litt (Checklist), and to his right is Jeff Greason (Test Czar).
Bryan Campen: So you started off as an intern? How did you end up in the role of Control?
Jeremy Voigt: For hot fires, interns never get to do things like Control. When I was an intern I started off as Spotter.
Then I graduated to video. Now all of a sudden instead of being outside, I’m inside sitting next to the head table. I can hear everything going on, all of the radio calls. At the same time I started reading checklists for another test stand, and was Red Team for that stand as well.
When I came back as a full time engineer I was given the responsibility of the Truss Test Stand. I started out on Checklist and worked my way up to Control Box.
So now I run Control Box and plan the tests for the Truss Test Stand.
BC: What does that mean?
JV: I’m in charge of the test stand. That means that when there are modifications that need to be done, I will either do them myself or delegate them to people to do, and then check over their work.
Then when it comes time to test, I talk with senior engineers–Doug [Jones], Jeff [Greason] and Dan [DeLong]–and we figure out the objectives of the test and what we need to do to meet them.
Then I will go to the crew and make sure everybody knows what’s going on and make sure they are ready for the test. I’ll send out emails on when we’re going to have the test, when we will have after action meetings and data reviews—all those things.
So I’m kind of the point of contact for the rest of the test crew.
On Control Box, I start days before the actual test. I will do this whenever we’ve broken into the stand. I’ll do a click test where I actuate every single valve. And I don’t do that on the day of the test, I do it beforehand, so that we know when we go out there that we won’t have any problems.
BC: What’s a click test?
JV: A click test is where I go through and actuate all of the valves and make sure the electronics wiring from the control box to the stand to each of the valves works. That they all move when they are supposed to and that they don’t move when they are not supposed to.
BC: Ok, here’s something I’ve heard you mention offhand that I want to ask you about specifically. Is it true during a hot fire that time actually slows down for you?
JV: Yeah. I’m counting in in my head, I have my hand on the stop button and I’m watching the video screens…
…and I think to myself “Oooh, that must have been about a minute! I wonder how the count’s going?”
And then you hear Checklist next to you scream out “Fifteen seconds!”…
On the left, Brandon Litt (Checklist) shouts the count with Jeremy (Control) to his right.
…and you can barely hear him and he’s shouting at the top of his lungs.
And then I say to myself “Oh man, I’ve got a long way to go.”
Every second feels like an hour of just watching, of making sure everything is going right. It’s an unbelievable feeling.
At larger companies, someone my age wouldn’t ever get near that experience. The thing I like about XCOR is I get that opportunity. It’s a lot of fun, getting to fire the rocket engine.
To work on this and then get to be able to test it, that’s unique to XCOR, and I never take it for granted.
We received a lot of photo inquiries the past few weeks, and have begun to catalog our content, by subject and week, over at Flickr. So this weekend we have a simple update: if you’re interested in the photos from this week’s stories, head over there!
Our dear friend Mouser passed away last Saturday morning. We’ve made a photo set for her “best of” shots here:
We began our series of interviews “from the bunker” on Tuesday, and will continue those next week. The shots so far, right here:
This is the second in a series of posts “from the bunker,” interviews with the people behind the scenes at every hot fire and cold flow that takes place at XCOR. Today we talk about the role of Checklist with XCOR engineer Brandon Litt.
The team confers during a test day. Left to right: Doug Jones (Chief Test Engineer), Jeremy Voigt (Control), Randall Clague (Safety Officer), Mike Valant (Senior Engineer), Jeff Greason (Test Czar and CEO), Brandon Litt (Checklist), Derek Nye (A&P) and Geoff Licciardello (Test Engineer)
Bryan Campen: So what does Checklist do?
Brandon Litt: The checklist reader gives commands to everybody on the test crew. And no one does anything on the test stand, on the control box, with the DAQ, unless the checklist specifically tells them to do it.
It has been through so many iterations that it is now the best way to get the test done successfully and safely.
The speed at which I read the checklist really sets the tempo for the day. If I’m reading it pretty fast, people might be on edge. We might miss things or skip lines if it were too fast. So it’s nice to go slow and deliberately and make sure everything is done the right way.
But if I’m going too slow people might get a little bit sluggish and lose focus, so I need to keep things going at a steady pace throughout the day. I also have to make sure that the commands I give are called back to me by the rest of the team.
BC:Have you ever stressed anyone out by going too fast?
BL: Oh yeah, when I started I’d get comments –“slow that up” or “take a pause here”– I’d push the radio button a little too early and maybe cut someone off while they were trying to say something.
So I tell Red Team to open a valve, wait for them to call back that they have opened that valve successfully, and then move on to the next step.
It requires a lot of mental focus to ensure that I am very precise.
I don’t just give commands. I wait for feedback.
BC:How many items are on the checklist?
BL: That’s a really good question. For a standard engine day we start with a rollout checklist, it includes all the things we bring with us when we get to the test site. We go through a stand setup checklist, which maybe has fifty items. Then we go through a control checklist which pretty much brings up the electrical system on the stand, and test every valve on the whole stand. It’s probably another fifty items. Then we go on to the pressurization steps for the engine run. The final checklist is only a few steps, it’s mostly verifications before we hit go.
BC:What are the best and most challenging parts of the job?
BL: It’s disappointing to go out there all day, read this checklist methodically, slowly, everything happens right, but for some reason we just have to call it a day, shut down and go home.
Counterpoint to that, it’s equally as amazing when do I do all those things, we push the button and that engine just works.
All of the positions on the crew don’t just do their respective jobs. All of the prep work in the hangar, and rollout, and for setting up the stand once you get out there, it’s a big group effort and everybody really has to chip in to get that thing ready to go.
So the checklist acts as that final verification. When everything is set up, if I am [for instance] trusting new engineers to do certain things during the setup, I’m going to verify those steps later in the checklist.
And rest assured I’m not just going to assume they were done correctly.
This is the first in a series of posts “from the bunker,” interviews with the people behind the scenes at every hot fire and cold flow that takes place at XCOR…
Bryan Campen: Where did the name “Test Czar” originate?
Jeff Greason: The reason we came to call this role the Test Czar—it’s cold in the winter in Mojave and I had a Russian Navy version of a Ushanka, you know the full fur hats you’ve probably seen.
Not Jeff Greason: XCOR Engineer Mike Valant performs test czar duties at a recent hot fire, complete with Russian Ushanka
I had picked one up in Moscow not long after the movie The Hunt for Red October came out in theaters. Anyway it was cold out, and I was wearing that hat and the guys started calling me czar, and it stuck.
Jeff Greason acts as test czar during a recent cold flow, without the standard Ushanka
BC: So what does a test czar do?
JG: If you’re familiar with the Apollo missions, there was always a flight director.
And during that time, in the sixties, there were different people in front of computer screens, each responsible for some piece of the system.
Mission Control during the Apollo 11 mission
And there were all these people with individual roles and responsibilities that made-up mission control.
NASA also found it very helpful to have somebody who was only worried about the big picture, and not about the specific details of any one of those systems.
When doing this job, you have to be aware of two polar opposite traps that are very easy to get into when doing test operations.
From time to time, anomalies crop up and you have to make an informed decision on whether you can live with the anomaly on that day—or not. If you always stop everything, any time there is anything unusual, you’d never get anything done.
But the other disease that’s easy to fall into is go fever. You’ve been working toward this test a long time, you get up early that morning, you have momentum built-up, people are doing their steps and some little problem crops up and it’s very, very easy to say “well you know, don’t worry about that, we’ll deal with that later. Let’s just keep going.”
The guy reading the checklist reads a step, people say it’s done and he reads another step, done. He wants to read the next step. But what if it’s not done?
We have to stop.
So the czar looks at the big picture, watches the momentum, the ebb and flow of the day. He watch for people who are not listening, for the danger signs, and for who is going too fast. But you also watch the other extreme: Not only are you making sure the team doesn’t go too fast, but the czar will also step in sometimes and say “OK, I understand this is an interesting problem. My judgment is that that is not worth stopping [the test] for, so let’s make sure we capture that problem for later analysis.”
Or, maybe that stop is indicating for us something we need to check on again later in the day, so we’ll add a step to the checklist later on to check the health of that system, to make sure we don’t forget.
You have to keep your eye on all these moving parts. And if it gets too stressful that’s usually a sign that it’s time to slow down.
We also have a Safety Officer, or “Safety”, whose only job is to stop things, in some sense. [Note: we’ll cover the Safety Officer role in a later post]
Safety is there looking for problems.
Everybody else is looking for how to get it done.
And the czar is there to look at the big picture, to get the test done safely or with the appropriate amount of risk. I have to decide if the probability that we are going to get the data we want is the worth the risk that we are going to be taking. It’s something we think about ahead of time, and also something we think about the day of the test.
In one sense, on a perfect test day with no anomalies, I don’t have anything to do. All I do is watch what other people are doing, make sure that I am happy with how that’s all going.
But you’re there poised at every second, not letting your eye fall off of anything, to be ready to step in and say “Wait a minute, stop, we need to pay closer attention to that. That’s a potential danger sign that we just missed.”
My job is to be ready to stop the test, but also be ready to say “we’re ready to go.”
It’s the rare test day that goes by without there being some hard call to make about what we want to live with—or not.