N22A-T005 TITLE: Spatial Disorientation Assessment and Evaluation Tool

OUSD (R&E) MODERNIZATION PRIORITY: General Warfighting Requirements (GWR)

TECHNOLOGY AREA(S): Air Platforms;Human Systems

OBJECTIVE: Develop and validate a survey-based assessment tool aimed at measuring perceptions regarding the experience and severity of a spatial disorientation-related illusion, as well as to evaluate the effectiveness of knowledge/skill acquisition and attitudinal changes from spatial disorientation training protocols.

DESCRIPTION: Spatial disorientation (SD) is one of the most cited accident-causing factors in aviation and accounts for 33% of all aviation accidents [Ref 2]. This trend has increased over time due to the rise in licensed pilots and hours flown; however, little research has been done to address the measurement of knowledge, skills, and attitudes required to combat an SD incident. Rather, the majority of current and prior literature focuses on improving the technology utilized to improve SD training. While technological updates to modern SD training simulations have been shown to improve SD-related outcomes [Refs 3�6] (i.e., subjective identification of SD illusions, successful simulation, and elicitation of illusions), the lack of observational and survey scales to assess the true effect that SD training methods have on aviators is concerning. Specifically, no current or prior literature attempts to analyze and present the specific knowledge, skills, and abilities (KSA) that their study's training conditions were meant to target. This systematic lack of KSA identification during training assessment is concerning as they remain the most predictive and valid metrics of competencies that relate to an individual�s abilities to perform a task [Ref 1].

Recent advances in the SD training domain have sought to mitigate this challenge by producing a set of training competencies that are believed to be associated with SD training outcomes. A recent Training Systems Requirements Analysis focused on advanced spatial disorientation was developed via a subject matter expert review of prior SD training and competency literature. Various current and prior SD training programs also informed this analysis in order to ensure that the information taught in future SD training programs, to both indoctrination and refresher aviators, will improve their knowledge of SD, their skills in employing tactics against it, and their attitudes towards utilizing training and safety procedures for SD. However, while previous analyses provide the most comprehensive list of competencies for SD training to date, the competencies and methods of measuring said competencies have not undergone documented validation. Psychometric validation is a statistically quantitative process concerned with determining if the metrics utilized to measure latent constructs (i.e., illusion identification ability) are measuring latent constructs reliably and consistently. Without the validation of questions and behavioral observations to underpin analysis results, it is unclear whether the protocols will truly target key SD avoidance, mitigation, and countermeasure competencies required by aviators. Further, it is possible that without appropriate psychometric validation, future efforts will have opposing effects on SD training by missing key components of the required KSA.

Developing a validated SD assessment and evaluation tool provides an opportunity to formulate a data-driven method to both measure SD mitigation and countermeasure knowledge and behavior, while also providing a differential measurement to assess training effectiveness resulting in validated training methods. A software-based assessment tool would assist trainers in not only developing more effective training protocols and procedures, but also personalizing SD training feedback to student aviators. The final decision support tool product will enable a standardized, reliable, and valid measurement of real-time training SD episode mitigation and reaction knowledge and skills. The hardware and software must meet the system DoD accreditation and certification requirements to support processing approvals for use through the policy cited in Department of Defense Instruction (DoDI) 8510.01, Risk Management Framework (RMF) for DoD Information Technology (IT) [Refs 7, 8], and comply with appropriate DoDI 8500.01, Cybersecurity [Refs 7, 8, 9]. Finally, research into the effectiveness of the instructional strategies and technologies developed based on these concepts is necessary to determine feasibility prior to transition.

PHASE I: Develop a psychometrically-based validation protocol to assess relevant SD competencies (e.g., application of procedures, communication, safety of flight management, automated and/or manual aircraft control, leadership, crew resource management, problem solving, decision making, situation awareness, workload management). Design the framework of the software-based tool to ensure a high level of end-user use reliability and usability. Develop the user-interaction architecture of the software tool for user input, output, and modification of the validated survey. Deploy psychometric validity testing. The Phase I effort will include prototype plans to be developed under Phase II.

PHASE II: Develop a validated questionnaire and observation tool of SD mitigation and countermeasure KSA from validity testing. Incorporate the initial questionnaire and observation tool into the software-based application for prototype demonstration and testing. Deploy confirmatory testing of validated questionnaire and observation tool. Finalize the questionnaire and observational assessment tool.

PHASE III DUAL USE APPLICATIONS: Obtain management framework certification for an authority to operate to successfully transition to a NAVAIR program office. Based on Phase II results, finalize and refine the methodology (questionnaire/observation tool) and software developed to meet training requirements for a wider variety of SD events/scenarios or platforms to support transition and commercialization of the product. Investigate the potential of expanding the software-based application to validate additional relevant training environments to extend transition applicability.

The validation framework and evaluation software has applicability to commercial industries including commercial airlines and corporate training. Demonstration of a methodologically sound software technology to validate training system needs has broader DoD and commercial applicability.

REFERENCES:

1. Bloom, B. S. (1956). "Taxonomy of educational objectives. Handbook 1: Cognitive domain." Addison-Wesley Longman Ltd; 2nd edition. https://www.amazon.com/Taxonomy-Educational-Objectives-Handbook-Cognitive/dp/0582280109/ref=sr_1_2?crid=194EH99NIT4AZ&dchild=1&keywords=taxonomy+of+educational+objectives&qid=1616611649&s=books&sprefix=Taxonomy+of%2Caps%2C438&sr=1-2.
2. Gibb, R., Ercoline, B., & Scharff, L. (2011). "Spatial disorientation: decades of pilot fatalities." Aviation, Space, and Environmental Medicine, 82(7), 717�724. https://doi.org/10.3357/ASEM.3048.2011.
3. Kallus, K. W., & Tropper, K. (2004). "Evaluation of a spatial disorientation simulator training for jet pilots." International Journal of Applied Aviation Studies, 4(1), 4556. https://www.academy.jccbi.gov/ama-800/Spring_2004.pdf#page=45.
4. Tropper, K., Kallus, W., & Boucsein, W. (2009). "Psychophysiological evaluation of an antidisorientation training for visual flight rules pilots in a moving base simulator." The International Journal of Aviation Psychology, 19(3), 270�286. https://www.worldcat.org/title/psychophysiological-evaluation-of-an-antidisorientation-training-for-visual-flight-rules-pilots-in-a-moving-base-simulator/oclc/770679956&referer=brief_results.
5. Kallus, W., Tropper, K., & Boucsein, W. (2011). "The importance of motion cues in spatial disorientation training for VFR-pilots." The International Journal of Aviation Psychology, 21(2), 135�152. https://www.worldcat.org/title/the-importance-of-motion-cues-in-spatial-disorientation-training-for-vfr-pilots/oclc/710990109&referer=brief_results.
6. Stroud, K. J., Harm, D. L., & Klaus, D. M. (2005). "Preflight virtual reality training as a countermeasure for space motion sickness and disorientation." Aviation, Space, and Environmental Medicine, 76(4), 352-356. https://www.ingentaconnect.com/content/asma/asem/2005/00000076/00000004/art00006
7. Department of Defense. (2014). Risk Management Framework (RMF) for DoD Information Technology (IT). Washington D.C.: Executive Services Directorate. https://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodi/851001_2014.pdf.
8. BAI Information Security Consulting & Training. (2020). BAI: Information Security RMF Resource Center. Retrieved from Risk Management Framework. https://rmf.org/.
9. Department of Defense Instruction 8510.01. https://www.acqnotes.com/wp-content/uploads/2016/08/DoDI-8510.01-Risk-Management-Framework-for-DoD-Information-Technology-%E2%80%93-24-May-2016.pdf.
10. FREQUENTLY ASKED QUESTIONS https://navysbir.com/n22_a/N22A-T005_FAQs.pdf

KEYWORDS: Spatial disorientation; training; validated training methods; decision support tool; psychometric validation; training competencies