TOP 10 TIPS FOR PLANNING A REAL TIME SIMULATION

RTS usually have a long lead time for preparatory activities.  Have you got a working version of the Concept of Operations so the validation team know what they are working with before you begin preparatory activities?  Also, highlight any dependencies to the project manager to ensure all members of the team are working coherently.  Are you dependent on any software build drops prior to conducting system testing?  Ensuring you have planned multiple test days for data, system and any prototyping sessions prior to your RTS will de-risk the RTS significantly.  This will also help to identify any resourcing demands for future activities.

It is important to gather stakeholder’s expectations for the concept as well as the validation exercise early on in the planning stages as they are a key input to objectives creation and help to decipher what kind of data needs to be gathered and how.  Stakeholders also may have input into the type of operational environment they would like the concept simulated in.  Remember to keep them engaged throughout the whole RTS and why not invite them to the actual RTS so they remain in the loop and the opportunity to provide feedback?

It is key to set realistic and achievable objectives in your RTS.  Consider a range of inputs such as the current and target maturity of your concept, Stakeholder expectations including validation expectations, concept requirements, the validation strategy, any benefit mechanisms and associated key performance areas.  Make your objectives unbiased e.g. to assess the impact of the concept on workload then use success criteria to help you decide if you have met your validation objective or not e.g. controller workload reduces compared to the reference scenario with the introduction of the concept.

How are you going to satisfy success criteria?  What metrics do you need to consider?  Do you need to gather data from the validation platform/prototype?  If so, make sure you’ve tested this during data and system testing prior to the RTS.  If you are going to use questionnaires, think about how much time you have to administer these and how much detail you require.  If you are issuing questionnaires, perhaps consider Post Run and Post Participation questionnaires with different aims/levels of detail. During the runs it is important for observers to note down any observations to use as information to be discussed in debriefs and for the validation report which will be written at the end. Ensure the observations are as detailed as possible, as if the runs have been recorded the observations will be useful to refer to when watching the re-plays. 

How many independent variables (things that are changed/controlled) and dependent variables (the variable being tested) do you have to consider? What is going to vary throughout the validation exercise e.g. participants, rotation, traffic sample, operational environment, concept variant?  Picking a validation exercise design to suit the concept maturity is important.  E.g. if you are requiring high fidelity quantitative results, a matched design (one major variable to test e.g. reference and solution scenario) may be preferred.  If you are prototyping and want to try out different combinations of variables or concept variants then a randomised design may be better.

The last thing you want is controllers getting distracted by unrealistic traffic presentation, density or traffic mix/callsigns.  Even unrealistic wind profiles or inaccurate aircraft performance or maps can throw them off.  Ensure you set the appropriate traffic sample density to test your concept.  If your controllers are new to the concept or are in the first few runs of the RTS, it might be worth preparing some lower density traffic samples as training samples so they that get used to working within a simulation environment and understand how the concept/system works prior to pumping up the traffic levels!

Planning a RTS is not a one-man (or woman) band.  You will need regular coordination with different members of the project team from project controllers, to Safety Experts, to Human Performance Experts and engineers depending on the needs of your project.  Ensure you work together towards deliverables and include them in review cycles as it is likely that your work is interconnected.  Remember you are working towards the same goal of developing a successful and mature concept.

The experimental design will influence how you construct your timetable for the RTS.  This will dictate how many measured runs you need to perform.  However, there are considerations too such as training time, length of runs so you can keep participants engaged, the length of breaks, time required for questionnaires and allowance for any contingency runs/time.  Also consider, participant attendance.  Have you got the same participants throughout the whole RTS or do they vary? This will influence when you do certain runs, such as runs with lighter traffic to ease the participants in or when matched runs are conducted.

Ensure participants are thoroughly briefed in preparation for the RTS.  This includes any theory and practical hand’s on experience.  Ensure participants know what the intended method of operations is and any procedures they need to be aware of.  It can be useful for them to attend some testing events so they are familiar with the concept/system.  However, you do not want them overfamiliar with the traffic samples as this might lead to overexposure and boredom throughout the RTS.

It is important to be fully as prepared as you can be when planning an RTS but typically you will have to think on your feet if things don’t go to plan.  Identifying risks and mitigating against them is useful to help identify possible things that might go wrong, e.g. participant absence, software/hardware bugs, training taking longer than anticipated or even customer demands midway through RTS activities.  Go into the simulation with a rough plan and prepare for problem solving and firefighting throughout the activity!