N211-078 TITLE: Operator Analytics and Training Integration through Artificial Intelligence and Machine Learning
RT&L FOCUS AREA(S): Machine Learning/AI
TECHNOLOGY AREA(S): Human Systems
OBJECTIVE: Develop an analytical toolset that mines sonar operator performance data via the tactical system along with training data gathered from the Moodle Learning Management System (LMS).
DESCRIPTION: The AN/SQQ-89 A (V)15 system collects a large amount of data from which it is possible to infer sonar operator performance data. The interactions that the operators have with the system can provide valuable insights into the fleet�s training readiness. Applying Big Data mining techniques, methodologies, and analysis to the sonar system of the SQQ-89 would enable more relevant and real-time decision-making. The Chief of Naval Operation (CNO) has stated that there is a need to "Establish data-driven decisions as a foundation for achieving readiness in our warfighting enterprises. Lead Type Commands (TYCOMs), supported by Systems Commands (SYSCOMs), Budget Submitting Offices, and higher echelons will develop and maintain authoritative and accessible data for decision-quality information." By coupling artificial intelligence (AI) that mines the SQQ-89�s operator performance data with learning analytics mined via machine learning applied to the Moodle LMS, decision-makers would possess the data necessary to make more informed decisions.
Additionally, the CNO desires to "Focus Navy efforts for fielding artificial intelligence/machine learning (AI/ML) algorithms on areas that most enhance warfighting, training, and corporate decisions." The current state of learning analytics used in the Moodle LMS include simple metrics such as activity completion and quiz/exam scores but lack the power to contribute to the understanding of student learning. An AI/ML toolset would provide the power necessary to measure an operator�s abilities and determine areas for improvement quickly and accurately over traditional assessments. Currently no tools exist that can provide this capability. Obtaining this toolset would benefit the Navy by decreasing operational costs. Training should use study time more efficiently than the chosen control conditions so participants did not waste time studying items they already knew and were able to concentrate on items that require more training maximizing the operator�s time and increasing training efficiency.
The Navy seeks an AI/ML toolset that captures data generated by operators interacting with the SQQ-89 system as well as the Moodle LMS hosted on the tactical system. The AI/ML toolset must also be able to format that data using "strategies to transform data into appropriate forms, to include smoothing, attribute construction, aggregation, normalization, and discretization (Susnea, pg. 74)." Such formatting would show patterns and provide additional insights into an operator�s performance and behavior while using the SQQ-89.
In addition to formatting the data, the toolset must be able to perform descriptive modeling, which is a mathematical process that describes real-world events and the relationships between factors responsible for them. The toolset must also perform data analysis through predictive modeling, which is a process that uses data mining and probability to forecast outcomes. The toolset must also visualize the data in an easily digestible format to ease a decision-maker�s ability to make better and more informed decisions quickly and with high confidence in the data.
Initial testing of the AI/ML toolset may be demonstrated at the contractor facility but a more robust evaluation of a fully developed toolset will be conducted using representative data gathered from a fleet test event, at a developer site such as the Lockheed Martin Anti-Submarine Warfare Laboratory in Manassas, VA, or from an appropriate Navy training facility such as Fleet Anti-Submarine Warfare Training Center San Diego, CA. (FASW-TC). In order to properly evaluate the toolset, the test should include data from a team of sonar operators and their interactions with the SQQ-89 tactical system. Ideally, the interactions would include real-world or synthetic scenarios that span the detect-to-engage timeline. In addition to the data derived from the tactical system, a fully populated Moodle LMS comprised of training data from each of the participants of the sonar team would be preferred as the AI/ML toolset will need to uncover findings and correlations between the two sets of data.
Metrics used to assess the AI/ML toolset will refer to data quality, which can be defined as the degree to which a set of characteristics of data fulfills requirements. Examples of characteristics include: completeness, validity, accuracy, consistency, availability, and timeliness. The toolset must be able to apply big data analysis to the information present in the Moodle LMS as well as the learning record store (LRS) present in the learning architecture on the SQQ-89.
Work produced in Phase II may become classified. Note: The prospective contractor(s) must be U.S. Owned and Operated with no Foreign Influence as defined by DOD 5220.22-M, National Industrial Security Program Operating Manual, unless acceptable mitigating procedures can and have been implemented and approved by the Defense Counterintelligence Security Agency (DCSA). The selected contractor and/or subcontractor must be able to acquire and maintain a secret level facility and Personnel Security Clearances, in order to perform on advanced phases of this contract as set forth by DCSA and NAVSEA in order to gain access to classified information pertaining to the national defense of the United States and its allies; this will be an inherent requirement. The selected company will be required to safeguard classified material IAW DoD 5220.22-M during the advance phases of this contract.
PHASE I: Develop a concept for an AI/ML toolset that mines data from the AN/SQQ-89A (V) 15 sonar system and Moodle LMS. Demonstrate the feasibly of the concept meets the parameters listed in the Description through modeling and analysis. The Phase I Option, if exercised, will include the initial design specifications and capabilities description to build a prototype solution in Phase II.
PHASE II: Develop and deliver a prototype of the AI/ML toolset and supporting architecture. Demonstrate at a Government- or company-provided facility that the prototype meets all parameters detailed in the Description. ASW personnel will conduct independent testing in the Fleet.
It is probable that the work under this effort will be classified under Phase II (see Description section for details).
PHASE III DUAL USE APPLICATIONS: Support the Navy in transitioning the technology to Navy use through system integration and qualification testing for the toolset prototype developed in Phase II. The AI/ML toolset prototype will be delivered to support a single transition event. Assist with the integration of the prototype into a future Advanced Capability Build of the AN/SQQ-89A (V) 15 Surface Ship Undersea Warfare Combat System.
The AI/ML toolset can be adapted to other technical fields including radio-frequency engineering and medical diagnostic tools. Big data and learning analytics are a relatively new field, but an architecture that allows adapting to different learning and training domains would be useful to the wider education and business community.
REFERENCES:
KEYWORDS: Sonar Operator Training; High Velocity Learning; Big Data Analytics; Artificial Intelligence; Machine Learning; Moodle LMS