N24B-T028 TITLE: Real-Time Detection of Operator Workload as Input to Scalable Autonomy During Dynamic Mission Operations

OUSD (R&E) CRITICAL TECHNOLOGY AREA(S): Advanced Computing and Software; Human-Machine Interfaces; Trusted AI and Autonomy

OBJECTIVE: Develop a method for real-time assessment of operator workload during dynamic flight operations to use as input for scalable autonomy in human-autonomy teams.

DESCRIPTION: Changes in the competitive capabilities of our adversaries has brought us to a new playing field in which we need to quickly develop and successfully leverage and integrate new technologies in support of our warfighters and the mission to maintain naval superiority. New developments in Machine Learning and Artificial Intelligence (ML/AI) provide opportunities for the integration and development of new autonomous and automated systems, ranging from advanced automated sensors, decision aids and mission systems to fully autonomous platforms that will work alongside the warfighter as a teammate rather than a tool. Successful integration of autonomy in warfighting systems will depend not only on their reliability and predictability, but their ability to work effectively with the human operator.

Effective human-autonomy teaming in naval operations will only be achieved if the human operator and autonomy system or agent are reactive to—and collaborative with—each other. A reduction in workload due to automation does not always result in superior operator and system performance; if task load is manageable, then offloading of tasks can result in underload and a loss of situational awareness [Ref 3]. Furthermore, automation does not always result in reduced workload. The paradox of automation is that monitoring the autonomous system, in addition to other mission responsibilities, can inadvertently increase workload. This increase in workload is thought to be due to the taxing nature of passive monitoring [Ref 4], which ultimately can result in complacency.

One proposed strategy to enhance human-autonomy teaming effectiveness is to build autonomous systems that adapt to the needs of the human operator in real time via dynamic workload thresholds based upon performance, psychophysiological activity, and/or other relevant metrics. The goal of such a strategy is to maintain situational awareness while moderating workload by increasing or decreasing levels of autonomy (i.e., number and types of tasks that are offloaded, type of decision aids provided, level of transparency, level and types of automated/autonomy functions, etc., [Ref 2]) based upon indicators of operator workload states [Refs 4 and 5]. Ideally, high operator workload would be addressed by increasing levels of automation or autonomous features, offloading/modifying tasks, and enhancing operator performance. Likewise, lower operator workload states would require minimal autonomy in order to ensure that the human remains in the loop to maintain engagement and situational awareness.

The current state of autonomous functions of a system is either: (a) to be active at all times, (b) be manually turned on/off by the user, or (c) be manually selected by the user from various predetermined levels of autonomy. Thus, innovation is still needed to develop adaptive automation in real time, so that autonomy can be scaled to match the current operator need in order to ensure mission success. For this, we first need to:

Identify valid, consistent, and resilient metrics/tools to estimate various dimensions of operator workload (i.e., cognitive, temporal, physical, etc.) in real time, and develop a model to combine these into a single workload estimation measurement.

• Multiple metrics/methods/tools are expected to be combined for a more rounded and accurate estimation of workload and could include physiological and/or psychophysiological measures and metrics, as well as operator performance metrics, amongst others.

• The resulting tools and methods need to be unobtrusive to operator performance and comfort.

• The tools and methods need to be able to be resilient and function in naval aviation operational environments, to include in-aircraft use.

• Develop a model for operationally defining workload thresholds (i.e., overloaded or underloaded), which will require changes in system automation level or autonomous behaviors.

• Propose tasks and task allocation strategies between operator and autonomy/automation that would result in increased and/or decreased levels of autonomy/automation as needed, and would be based on the real-time workload indicators.

A solution that addresses the above-mentioned needs would provide a first step in supporting future human-autonomy teams that are inherent in the ever-growing manned-unmanned missions.

Note: NAVAIR will provide Phase I performers with the appropriate guidance required for human research protocols so that they have the information to use while preparing their Phase II Initial Proposal. Institutional Review Board (IRB) determination as well as processing, submission, and review of all paperwork required for human subject use can be a lengthy process. As such, no human research will be allowed until Phase II and work will not be authorized until approval has been obtained, typically as an option to be exercised during Phase II.

PHASE I: Identify the metrics, methods, and tools that will be used for the real-time assessment of operator workload. These should be validated in a simulation environment that dynamically induces varying levels of operator workload (e.g., overload or underload). The Phase I effort will include prototype plans to be developed under Phase II. Note: Please refer to the statement included in the Description above regarding human research protocol for Phase II.

PHASE II: Develop, demonstrate, and validate an unobtrusive and affordable stand-alone kit for the dynamic assessment of operator workload, and its use and effectiveness as input for scalable automation/autonomy. An ideal kit would measure operator workload in an unobtrusive manner, so as not to interfere with operator task load, and would be viable for use in various naval aviation environments to include in-aircraft use. It will also include the development of an algorithm to operationally define overload and underload, as well as optimal workload. In addition, strategies should be proposed for manipulating the levels of automation in response to workload. Note: Please refer to the statement included in the Description above regarding human research protocol for Phase II.

PHASE III DUAL USE APPLICATIONS: Final testing would involve validation of the technology in a naval aviation relevant use case that involves dynamic automation level modifications based on the workload assessment and demonstration that the intervention results in the intended changes in operator workload and enhanced system performance.

The real-time assessment of workload as input to scaling automation levels or autonomy behavior, could be used in any application that involves the interaction of a human operator with an automated system for extended periods and in dynamic environments. Some of these could be autonomous-car or transit vehicle operation, search and rescue mission systems, reconnaissance and surveillance mission systems, and monitoring systems and applications (e.g., scientific, medical, and nuclear).

REFERENCES:

1. Hooey, B. L.; Kaber, D. B.; Adams, J. A.; Fong, T. W. and Gore, B. F. "The underpinnings of workload in unmanned vehicle systems." IEEE Transactions on Human-Machine Systems, 48(5), 2017, pp. 452-467. https://doi.org/10.1109/THMS.2017.2759758
2. Parasuraman, R.; Sheridan, T. B. and Wickens, C. D. "A model for types and levels of human interaction with automation." IEEE Transactions on systems, man, and cybernetics-Part A: Systems and Humans, 30(3), 2000, pp. 286-297. https://doi.org/10.1109/3468.844354
3. Young, M. S. and Stanton, N. A. "Attention and automation: New perspectives on mental underload and performance." Theoretical issues in ergonomics science, 3(2), 2002, pp. 178-194. https://doi.org/10.1080/14639220210123789
4. Parasuraman, R.; Bahri, T.; Deaton, J. E.; Morrison, J. G. and Barnes, M. "Theory and design of adaptive automation in aviation systems." Naval Air Warfare Center, Warminster, PA, Tech. Rep. NAWCADWAR-92, 17 July 1992, pp. 033-60. https://apps.dtic.mil/sti/pdfs/ADA254595.pdf
5. Kaber, D. B.; Riley, J. M.; Tan, K. W. and Endsley, M. R. "On the design of adaptive automation for complex systems." International Journal of Cognitive Ergonomics, 5(1), 2001, pp. 37-57. https://doi.org/10.1207/S15327566IJCE0501_3

KEYWORDS: Human-autonomy teaming; adaptive automation; operator workload; real-time monitoring; neuroergonomics; psychophysiology

TPOC-1: Ada Fraticelli

Phone: (301) 342-4845

 

TPOC-2: Brittany Neilson 

Phone: (240) 587-1320

---