Test and evaluation are crucial parts of the systems engineering process. But, as one defence expert says, it’s usually one of the first things to go when the pressure is on. “It’s important to realise this and stick to principles, make the right call and make sure our problems don’t become the warfighter’s.”
Test and evaluation (T&E) needs to be an early and constant consideration in defence acquisition.
“If programs have not discovered and corrected design problems or key failure modes [before production], those problems will almost certainly become the warfighter’s problems,” is how one expert put it.
At the same time, and despite the difficulty of pushing the limits of technology, a program manager frequently works with negative incentives in a ‘break even or lose’ situation. In this scenario, T&E can come under strong pressure when juggling cost, schedule and performance.
William Keegan, a retired Army Colonel and 26-year US Army veteran, is president of the International Test & Evaluation Association (ITEA). As such, he’s pretty familiar with the obstacles to excellence in acquisition.
“What is preventing us from unlocking the future? I assume that future here means successful acquisition programs for defence purposes, or successful procurement and acquisition in engineering in general, and what are the obstacles to that?” he said, adding that any program will have natural habits, bureaucracies and competing ambitions to overcome.
“However, there are a couple of questions I have, and one is: Do we have a suitable set of best practices in our business that help us unlock the future? And then there’s a second aspect, which is even if we have best practices, what are we doing to actually follow these?”
Under the pump
Institutional issues like missing schedules and the political consequences of this can compromise best practices, as can cost overruns. It’s not uncommon to open a newspaper and see an article about a big-ticket program going over budget or being delayed. And delays can be so distasteful that programs might just move to the next milestone, absorb a failure, and hope to fix it before production ramps up.
Another example of an organisational temptation is to sacrifice testing to save money and put this money into something else, such as fixes. This comes with inherent risks, namely the potential inability to sufficiently test a system before a user receives it.
“It’s almost always the performance that suffers, and then we wait to fix it later, and it’s not easy to avoid, because the pressure can be tremendous,” he said.
What is necessary is for testers to work with systems engineers from the beginning, and for requirements to be well-defined, testable, and measurable. And when the pressure is on, leadership must maintain discipline in managing the program.
“Leadership needs to rise above the fray and actually back up a program management decision that makes sense,” Keegan said.
We as testers must be flexible and agile as well, in order to get the optimal quality and amount of test data given the available resources.
“As an operational tester, for me, it’s about getting the capability to the war fighter as quickly as possible, it’s not just about getting the first plane, or the first ship, or the first rifle or whatever, the first tank, to the soldier. Because that one vehicle is not going to be what the Force depends upon, it’s going to depend upon the fleet of those vehicles, ships or planes.
“And the cost of testing is very low when it comes to lifecycle cost, especially operational testing, but it looks like a lot of money at the end because most of the big testing comes at the end, especially government testing.”
Learn from the past
Following a military career of command in the field in the 82nd Airborne Division, he concluded his military service in the Office of the Secretary of Defense, Director of Operational Test & Evaluation, where he was involved with the testing of all major defense weapon systems. Keegan more recently held senior defense T&E related roles in industry as Vice President, Senior Vice President, and Chief Operating Officer.
He moved to Adelaide in June last year and is Director of T&E for the SEA 1000 Program, the $50 billion for Australia’s fleet of submarines to replace its Collins Class fleet. Though he oversees T&E for the biggest defence procurement program in Australian history, Keegan is appearing at SETE 2018 as President of ITEA, the premier non-profit organisation that advances T&E education and best practices.
He finishes by suggesting that a sense of history is needed for program teams and their leaders. Historically, most program costs are underestimated. Programs also are never issue-free. It’s important to realise this and stick to principles, make the right call, and make sure our problems don’t become the warfighter’s.
“It’s best to kind of get a good sense of … the history of how programs that are difficult and high-tech have been managed and understand what is in the realm of the possible; do your best to meet your cost, schedule and performance objectives, but when a tough call needs to be made, that in fact we’re driven by our principles,” he advises.
“We’re driven by our principles, not only by our pressures, and we have to balance them, not just give in to the pressures and abandon our principles.”
This is the best long-term approach to avoid significant cost growth and greater schedule slippage late in the acquisition process.
William Keegan is a keynote speaker at SETE 2018 and will be speaking on the theme Unlocking the Future through Systems Engineering. The Systems Engineering Test and Evaluation (SETE) Conference 2018 will be held in Sydney from 30 April to 2 May. To learn more and to register, click here.