N241-011 TITLE: Generative Artificial Intelligence for Scenario Generation and Communications Analysis

OUSD (R&E) CRITICAL TECHNOLOGY AREA(S): Trusted AI and Autonomy

The technology within this topic is restricted under the International Traffic in Arms Regulation (ITAR), 22 CFR Parts 120-130, which controls the export and import of defense-related material and services, including export of sensitive technical data, or the Export Administration Regulation (EAR), 15 CFR Parts 730-774, which controls dual use items. Offerors must disclose any proposed use of foreign nationals (FNs), their country(ies) of origin, the type of visa or work permit possessed, and the statement of work (SOW) tasks intended for accomplishment by the FN(s) in accordance with the Announcement. Offerors are advised foreign nationals proposed to perform on this topic may be restricted due to the technical data under US Export Control Laws.

OBJECTIVE: Develop the capability to rapidly generate high threat density scenarios with tactically representative red (adversary) threats that adapt in real time. Additionally, this effort will develop the capability to conduct automatic analysis of blue (friendly) communications to understand speed and accuracy of information exchange.

DESCRIPTION: As the carrier airwing of the future prepares for the high-end fight, the training paradigm will shift to almost exclusively Live, Virtual, Constructive (LVC) environments due to expanded range capabilities of the peer threat competitors and Operations Security (OPSEC) considerations. As a result, warfighters are able to train as they fight with higher fidelity scenarios that more accurately represent red kill chains. This high-fidelity, data rich environment provides unique opportunities for instructional strategies to better support end-to-end training and improve readiness. Specifically, LVC environments increase the amount of—and access to—data that can support improved scenario generation, performance assessment, and debrief when utilized appropriately. However, LVC training is not without its challenges. These challenges include resource requirements to develop these high-fidelity scenarios as they can be cumbersome and labor intensive. Moreover, scenarios that do not contain significant variations may lose utility very quickly as operators can begin to anticipate scenario outcomes after a few exposures. Consequently, a need exists for rapid generation of real-time, adaptive, high-fidelity scenarios.

Additional challenges lie in the assessment of performance. The carrier airwing of the future will rely on integrated tactics that require a level of coordination and information exchange across platforms that have not been required in past tactics. The complexity of coordination associated with integrated tactics necessitates a significant amount of voice communications across the different platforms to provide Situational Awareness (SA) and elicit decision-making. While communication is critical to cross platform coordination and overall tactical execution, it remains one of the most challenging training objectives to meet during Air Defense events.

As such, this effort seeks to alleviate identified challenges with scenario generation and performance assessment through the investigation of generative artificial intelligence (AI) (e.g., DALL-E, ChatGPT) or other forms of AI to support scenario generation and communications assessment. This SBIR effort shall focus on utilizing AI to learn from pilot-in-the-loop red threat behavior to rapidly generate constructive threat presentations that adapt to trainee behavior in a tactically feasible manner. Additionally, AI shall be applied to further the state-of-the-science in communications analysis [Ref 6]. Specifically, AI shall support analysis of blue recorded communications and provide an initial assessment in terms of accuracy of the words said (relative to ground truth) and speed at which they are said. This analysis will include digesting communication recordings, assessing quality of communications-based accuracy and speed, and then providing these results via automated debrief.

These capabilities will improve the quality of training and readiness via end-to-end training enhancements. First, high-fidelity Air Defense scenarios that can be rapidly generated and are adaptive will yield greater training utility and provide cost avoidance associated with scenario development manpower and human-in-loop threat support manpower. Next, development of a communications analysis and debrief capability will improve SA, and decision making will benefit the Fleet by decreasing instructor workload, reducing human error and manpower time requirements, and automatically provide instructors with information on communication protocol adherence and timeliness to improve SA and increase debriefing capabilities.

This effort will specifically look at Air Defense training scenarios within LVC environments to increase speed at which high-fidelity, adaptive scenarios can be generated and assessed to enhance operator performance. This capability will be developed with the intention of a transition path to the Next Generation Threat System (NGTS).

Work produced in Phase II may become classified. Note: The prospective contractor(s) must be U.S. owned and operated with no foreign influence as defined by 32 U.S.C. § 2004.20 et seq., National Industrial Security Program Executive Agent and Operating Manual, unless acceptable mitigating procedures can and have been implemented and approved by the Defense Counterintelligence and Security Agency (DCSA) formerly Defense Security Service (DSS). The selected contractor must be able to acquire and maintain a secret level facility and Personnel Security Clearances. This will allow contractor personnel to perform on advanced phases of this project as set forth by DCSA and NAVAIR in order to gain access to classified information pertaining to the national defense of the United States and its allies; this will be an inherent requirement. The selected company will be required to safeguard classified material during the advanced phases of this contract IAW the National Industrial Security Program Operating Manual (NISPOM), which can be found at Title 32, Part 2004.20 of the Code of Federal Regulations. Reference: National Industrial Security Program Executive Agent and Operating Manual (NISP), 32 U.S.C. § 2004.20 et seq. (1993). https://www.ecfr.gov/current/title-32/subtitle-B/chapter-XX/part-2004

PHASE I: Research and develop an integration plan for development of a proof of concept, standalone, capability to rapidly generate high-threat density scenarios with tactically representative red threats that adapt in real time. This will include investigating unclassified sample data to determine appropriate AI models for future development. Additionally, Phase I will focus on identifying the most appropriate AI model or models to support automatic analysis of blue communications based on accuracy and speed results. An unclassified sample dataset will be provided to help support this investigation and to understand speed and accuracy of information exchange. Both objectives will use generative or other forms of AI. Performance assessment should focus on communications but may also include tactical assessments. Noise filtering shall be investigated to support communication analysis as the noise content in the operational environment for Air Defense is significant.

Demonstrate the feasibility of application into the larger, integrated training system. The plan shall detail integration into NGTS to allow for transition into an operational LVC environment. Additionally, the plan shall include a Subject Matter Expert (SME) evaluation of capabilities and methods for conducting an Analysis of Alternatives to identify best practice method moving forward for training delivery.

Provide prototype plans to be developed under Phase II.

PHASE II: Research, develop, design, and deliver a proof-of-scenario generation and communication assessment capabilities for Air Defense training scenarios through execution of the integration plan developed in Phase I. During Phase II, the sample data provided will be more tactically and operationally relevant and classified at the SECRET level. Developers can expect the scenarios to be more tactically complex, have larger amount of communication, and communications will include significant background noise. Noise will include, but is not limited to, background noise (engines, alerts, etc.), static, and the like. Integration with NGTS will enhance the capability with scenarios and performance data already resident in NGTS. Design and develop the tool to include visualizations, usability documentation, and technology evaluation. Demonstration of the tool, along with documentation of usability of the training software is critical. Risk Management Framework guidelines should be considered and adhered to during the development to support information assurance compliance.

Work in Phase II may become classified. Please see note in the Description paragraph.

PHASE III DUAL USE APPLICATIONS: Introduce additional data from NGTS as well as other Live-and-Virtual entities within the scenario. Scenario generation shall be enhanced to include external (live and/or virtual) entities. The AI implementation should account for any differences or effects external entities may have on the AI model. The voice communication assessment capabilities shall be flexible as to be deployed in varying training configurations. Training locations may differ in their setup of radios and networked communications, which will require easy and configurable settings and controls. Integration testing and demonstration of capabilities will be conducted in a distributed simulation via Distributed Interactive Simulation (DIS) protocol at the SECRET level. Software shall be integrated with NGTS to facilitate transition into operational LVC environment. Documentation and any supporting materials shall be delivered to NGTS team for maintenance and future enhancements.

The AI voice assessment model can be leveraged in the private sector as a speech-to-text model in environments with high noise or when non-standard English speech is in use, such as the brevity communications made during a tactical aviation scenario. Most AI speech models are trained with common English phrases. The data and voice communications from the tactical aviation domain will provide more robust speech-to-text analysis for the private sector in areas such as air traffic control or brevity communications training.

REFERENCES:

1. Baidoo-Anu, D., & Owusu Ansah, L. (2023). Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4337484
2. Park, K., Mott, B. W., Min, W., Boyer, K. E., Wiebe, E. N., & Lester, J. C. (2019, August). Generating educational game levels with multistep deep convolutional generative adversarial networks. In 2019 IEEE Conference on Games (CoG) (pp. 1-8). IEEE. https://doi.org/10.1109/CIG.2019.8848085
3. Svenmarck, P., Luotsinen, L., Nilsson, M., & Schubert, J. (2018, May). Possibilities and challenges for artificial intelligence in military applications. In Proceedings of the NATO Big Data and Artificial Intelligence for Military Decision Making Specialists’ Meeting (pp.1-16). https://www.sto.nato.int/publications/STO%20Meeting%20Proceedings/STO-MP-IST-160/MP-IST-160-S1-5.pdf
4. Wang, K., Gou, C., Duan, Y., Lin, Y., Zheng, X., & Wang, F.-Y. (2017). Generative adversarial networks: Introduction and outlook. IEEE/CAA Journal of Automatica Sinica, 4(4), 588–598. https://ieeexplore.ieee.org/iel7/6570654/8039012/08039016.pdf
5. Weisz, J. D., Muller, M., Houde, S., Richards, J., Ross, S. I., Martinez, F., Agarwal, M., & Talamadupula, K. (2021, April). Perfection not ? Human-AI partnerships in code translation. In 26th International Conference on Intelligent User Interfaces (pp. 402-412). https://doi.org/10.1145/3397481.3450656
6. Leini, Z., & Xiaolei, S. (2021, February). Study on speech recognition method of artificial intelligence deep learning. In Journal of Physics: Conference Series (Vol. 1754, No. 1, p. 012183). IOP Publishing. https://doi.org/10.1088/1742-6596/1754/1/012183

KEYWORDS: Artificial Intelligence (AI); Scenario Generation; Communications Assessment; Voice Analysis; Live, Virtual, Constructive; Automated Debrief

---