It can be a minefield selecting the right validation technique your concept in question. In order to develop a concept iteratively, different validation techniques can be used. Typically, the needs of the project reside on two main questions:
- What maturity level is your concept and what does that mean for the considerations you have to make at that stage?
- What information or evidence do you need to obtain during the validation exercise?
As far back as MAEVA there was a desire to identify a specific order in which validation techniques would be used as you progressed along the concept development lifecycle. What the E-OCVM changed was to introduce a more considered approach, assessing which technique was fit for purpose. Co-authored by Think, it stated that various inputs would come together in a validation strategy and only then would you determine the most appropriate validation technique. Whilst the E-OCVM admitted there were “likely suspects” at each maturity level linked to type of evidence required, appetite for cost and risk, it was not a fixed rule. This approach was diluted to some extent during the SESAR Development phase where – in a need to encourage some level of standardisation and equivalence across projects – a more direct link between techniques and maturity was again introduced. This led to thinking things like “Real Time Simulations are a V3 validation technique”.
Naturally, as authors of the original guidance we have pushed back against the direct linking and prefer to think more strategically. We don’t recommend a firm link between maturity and technique. Instead we encourage you to consider all the inputs. In the diagram below we illustrate what other factors might influence your choice of validation technique. Obviously, other aspects come into the mix as well such as time available, budget and expertise, and even within our mix below there might be reasons why some techniques would shift position. But that’s the fun of the Validation Strategy!
Perhaps one of the most influential factors behind choice of validation technique is the type of evidence that you require from the validation exercise. What are your stakeholder’s expectations from the validation exercise? Have you identified the intended benefits and drawbacks of your concept? ICAO have a series of 11 Key Performance Areas (KPAs) plus Human Performance to categorise the performance of your concept. Which ones are relevant? Every KPA can be assessed to some degree in almost all validation techniques but some validation techniques are better than others.
As with maturity, there are nuances and subtleties which make a predetermination unwise. As above, some are more suited than others but they are never exclusive. Take capacity assessments for example. Analytical techniques such as modelling, FTS or data analysis are often well suited to measuring capacity with their ability to quickly simulate a variety of large scale traffic volumes and scenarios and spot subtle changes. During human in the loop validation techniques such as RTS, airspace and airport capacity is often fixed into a 60 minute traffic samples so it may give the false impression that the solution has limited impact on capacity. Nevertheless, capacity can be measured indirectly in RTS by assessing any changes in controller workload and linking a human workload capacity to traffic capacity. It is not always as statistically accurate but it gives an extra level of analysis.
Our rough guide is shown in the diagram below but we have often encountered scenarios where the Validation Strategy means we take a different path.
Overall, you can see that it’s not easy to pick the most appropriate validation technique and there are different factors that have to go into the decision making. That’s why we place so much emphasis on validation as a critical consideration of any programme which will help save money in the long run.
If you still have questions regarding validation techniques or If you would like more information on the suitability of each technique relative to the KPA please email us for a more detailed printable guide which we will be happy to send. Alternatively, feel free to contact us and we will be happy to help.
Author: Sarah McLarty, ATM Consultant
Recent Comments