TECHNOLOGY AREA(S): Electronics

ACQUISITION PROGRAM: IWS 5.0: Undersea Warfare Systems

OBJECTIVE: Develop machine learning (ML) approaches using artificial intelligence (AI) to create realistic synthetic video sequences suitable for use in training simulators for periscope operators and as training data for other ML exploitation tools to enable rapid approaches to fielding this capability.

DESCRIPTION: Currently tools are freely available on the internet that allow individuals to create incredibly realistic and believable audio and video clips for speeches and discussions that never happened. These tools use a variety of ML tools and limited exemplars of training data such as actual speeches and videos of a person.

Tools like these are being used to create more complex, realistic synthetic scenes using training data to develop new models and approaches that do not require a three dimensional (3-D) model of the environment. Complex, physics-based models are often used in current simulations. This requires a fundamental understanding of the entire phenomenon in question and requires extreme computational power. The mantra for the armed services has always been �Train like we fight, fight like we train.� The Navy utilizes many simulators to train and conduct experiments, but these often utilize low-resolution representations that limit the effectiveness of the simulation. It is imperative that training systems and simulators be as realistic as possible, enabling experiences like what may be experienced while deployed. The Navy is looking for technology to create realistic synthetic video sequences suitable for use in training simulators. The goal is to increase the fidelity of the simulated sensor imagery used within the Submarine Multi-Mission Team Trainer (SMMTT).

Providing realistic synthetic data will improve operator responses, reduce operator uncertainty under stress, and improve decision-making. ML synthesis tools can enable development of realistic synthetic video and imagery for use with simulations. ML approaches are being leveraged for image and video processing applications, but a limiting factor is the availability of training data. High-quality synthesis approaches that utilize ML can also provide an alternate means to creating the large volumes of training data that are needed to �teach� a deep learning algorithm. However, current approaches to video scene synthesis focus on frame interpolation and static scene creation.

Scene-generation tools are available in industry. However, existing tools are not sufficient to develop dynamic periscope scene content covering 360 degrees and at least 60 frames per second (fps) across the world�s range of weather and lighting conditions. Innovation is required to support real-time generation of synthetic dynamic scenes that represent phenomena associated with weather, the surface of the ocean in different lighting and sea states, any viewable terrain or infrastructure when near land, attributes of shipping, and combat effects, such as explosions. Possible approaches include using generative adversarial models, deep predictive coding models, and image-to-image translation. The Navy needs both high fidelity data and scene content for training simulations, and large volumes of synthetic data to train ML algorithms that will improve target detection, classification and tracking systems. Metrics for the work will include computational performance, image similarity metrics, and user assessments.

PHASE I: Develop a concept for creating realistic synthetic video sequences suitable for use in training simulators. Demonstrate the feasibility of the concept to meet all the requirements as stated in the Description. Establish feasibility through modeling and analysis of the design.

The Phase I Option, if exercised, will include the initial design specifications and capabilities description to build a prototype solution in Phase II.

PHASE II: Develop, and deliver for testing, a prototype of the realistic synthetic video sequences suitable for use in training simulators. Testing will include benchmarking computational performance, image similarity metrics compared to actual periscope video scenes (which will be provided), and user assessments. Validate the prototype through application of the approach for use in a simulation environment. Provide a detailed test plan to demonstrate that the simulation achieves the metrics defined in the Description. Develop a Phase III plan.

PHASE III DUAL USE APPLICATIONS: Support the Navy in transitioning the software suite to Navy use in current Navy training systems or simulators to provide dynamic scene content. Work with the training working group for IWS 5.0 to increase the fidelity of the sensor imagery used within the SMMTT.

Modelling dynamic textures has been an ongoing topic of investigation for applications for film and video production. The technology developed under this topic could provide an improved approach to creating dynamic scene content for this industry and other DoD programs. Complex, physics-based models are often used in current simulations. This requires a fundamental understanding of the entire phenomenon in question and requires extreme computational power.

The innovation sought would reduce reliance on physics and processing capacity. This new approach could be used for frame prediction and interpolation across frames to construct new video sequences from limited data or to enhance video compression methodologies for all industries producing video imagery or needing to store large quantities of video imagery (e.g., law enforcement, border protection, news and broadcast entities).

REFERENCES:

1. Ilisescu, Corneliu et al. �Responsive Action-based Video Synthesis.� Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems,� May 06-11, 2017, pp. 6569-6580. https://arxiv.org/pdf/1705.07273.pdf

2. Wang, Ting-Chun et al. �Video to Video Synthesis.� NIPS Proceedings, 2018. https://arxiv.org/abs/1808.06601

3. You, Xinge et al. �Kernel Learning for Dynamic Texture Synthesis.� IEEE Transactions on Image Processing, Vol.. 25, No. 10, OCTOBER 2016. Kernel-Learning-for-Dynamic-Texture-Synthesis.pdf

4. Dosovitskiy, Alexey and Brox, T. �Generating images with perceptual similarity metrics based on deep networks.� NIPS Proceedings, 2016. https://papers.nips.cc/paper/6158-generating-images-with-perceptual-similarity-metrics-based-on-deep-networks

KEYWORDS: Machine learning; Video Synthesis; Generative Adversarial Models; Dynamic Scene Synthesis; Data Simulation; Training Simulators; ML