N252-103 TITLE: Virtual Sea-Shield -- Immersive Content Creation with Generative AI for Automated C5ISR Situational Awareness

OUSD (R&E) CRITICAL TECHNOLOGY AREA(S): Human-Machine Interfaces;Integrated Sensing and Cyber;Trusted AI and Autonomy

The technology within this topic is restricted under the International Traffic in Arms Regulation (ITAR), 22 CFR Parts 120-130, which controls the export and import of defense-related material and services, including export of sensitive technical data, or the Export Administration Regulation (EAR), 15 CFR Parts 730-774, which controls dual use items. Offerors must disclose any proposed use of foreign nationals (FNs), their country(ies) of origin, the type of visa or work permit possessed, and the statement of work (SOW) tasks intended for accomplishment by the FN(s) in accordance with the Announcement. Offerors are advised foreign nationals proposed to perform on this topic may be restricted due to the technical data under US Export Control Laws.

OBJECTIVE: Develop an Artificial Intelligence (AI)-driven 4D content generation environment that efficiently processes all-domain Command, Control, Communication, Computers, Cyber, Intelligence, Surveillance, and Reconnaissance (C5ISR) information into high-fidelity immersive observables for collaborative situational awareness across the desired layers of naval operations.

DESCRIPTION: Virtual Sea-Shield (VSS) is an all-domain multilayered protective environment that utilizes AI-assisted automatic situational awareness to deter threats (i.e., jamming, unmanned vehicle attacks, missiles, etc.) from targeting naval assets. VSS is a decentralized 4D (time and space) digital content generation environment for building a self-healing adaptive multilayered defensive and offensive naval sea shield across various naval operations. VSS utilizes generative AI for automated content creation by processing all-source intelligence, operational objectives, and engagement plans. In autonomous operations, VSS can track, penetrate, and trigger automatic strikes on designated hostile assets, personnel, logistics nodes, ammunition depots, and supply lines – blunting adversaries’ advances and disrupting its ability to maneuver, regroup, and launch attacks. Above all, VSS is a critical access infrastructure for a novel Autonomous Sea-Basing to support the deployment of forces, equipment, supplies, and warfighting capabilities.

Today, a multidisciplinary team of decision-makers, analysts, and warfighters conduct extensive wargaming exercises with a considerable investment in time and resources as they sift through vast amounts of data to look for indication and warning (I&W) signs and supporting evidence to anticipate and counter adversarial trends, strategies, and tactics that may have unfavorable impacts on the U.S. National Security interests. This human-centric approach to attain proper visibility into future events is fraught with the risk of costly errors, human biases, and omissions of valuable insights that may influence the assessment of evidence, skew statistical analysis, or color the recognition of cause and effect.

This SBIR topic envisions a systematic approach to integrate AI-based immersive technologies into the C5ISR environment with improved accuracy and agility over the current human-centric paradigm. VSS leverages state-of-the-art immersive technologies such as augmented reality (AR), virtual reality (VR), spatial reality (SR), and mixed reality (MR) as extended reality (XR) to transport humans from observation experience to immersion experience as they stimulate human attention to focus on I&W signs of emerging situations. The immersive technologies engage end-users to enrich collaborative teaming through natural language, dynamic visualization, and sound as they navigate spatial domains (i.e., sea, air, land, and space) and cyber. Generative AI-driven MR overlays provide a true-to-life presence and awareness that go beyond simply displaying pre-programmed digital elements. A successful VSS will generate realistic and adaptive content overlays that openly respond to human interactions with the contents and induce collaboration – during which generative AI tools will learn adversarial tactics and anticipate the evolution of engagement scenarios, and their consequences based on human decision-making effects. By doing so, VSS brings about the synergy between humans and AI. It enables warfighters to home in on deterrence options, experiment with novel human-to-machine (i.e., unmanned aerial vehicle (UAV), unmanned underwater vehicle (UUV), space assets) or machine-to-machine teaming engagement opportunities, conduct joint operations rehearsal, or perform training exercises.

The VSS performance test and demonstration applications for the proof-of-concept (Phase I) and prototyping (Phase II) may include operational scenarios that capture a series of hostile joint military and commercial mobilization and exercise activities to control contested waters such as amphibious landing and sea-lane blockade. With generative AI, the immersive VSS must automatically extract knowledge from multimodal datasets to demonstrate creative teaming of the blue forces required to penetrate an anti-access area-denial environment, where and how, with sustainable persistence while inside and outside a contested space. The quantitative metrics must include timely decision-making, and precise command and control for engagement strategies and tactics supported by the layered deployment of forces, equipment, and supplies that signify adaptive sea basing. The Phase I proof-of-concept will be unclassified. AI-assisted creative teaming and collaborations will be limited to unclassified multimodal contents (e.g., text, images, video, audio, spatiotemporal 4D models, or other data types) over commercial 5G networks.

The VSS end-to-end design must consider distributed deployment, fault-tolerant connectivity, optimal configuration throughputs, and scalability.

Note 1: Phase I will be UNCLASSIFIED and classified data is not required.

Note 2: Contractors must provide appropriate dataset release authorization for use in their case studies, tests, and demonstrations, and certify that there are no legal or privacy issues, limitations, or restrictions with using the proposed data for this SBIR project.

Note 3: Work produced in Phase II may become classified. However, the proposal for Phase II will be UNCLASSIFIED. The prospective contractor(s) must be U.S. owned and operated with no foreign influence as defined by 32 U.S.C. § 2004.20 et seq., National Industrial Security Program Executive Agent and Operating Manual, unless acceptable mitigating procedures can and have been implemented and approved by the Defense Counterintelligence and Security Agency (DCSA) formerly Defense Security Service (DSS). The selected contractor must be able to acquire and maintain a secret level facility and Personnel Security Clearances. This will allow contractor personnel to perform on advanced phases of this project as set forth by DCSA and ONR in order to gain access to classified information pertaining to the national defense of the United States and its allies; this will be an inherent requirement. The selected company will be required to safeguard classified material during the advanced phases of this contract IAW the National Industrial Security Program Operating Manual (NISPOM), which can be found at Title 32, Part 2004.20 of the Code of Federal Regulations.

Note 4: If the selected Phase II contractor does not have the required certification for classified work, ONR or the related DON Program Office will work with the contractor to facilitate certification of related personnel and facility.

PHASE I: Develop a concept for generative AI models to generate VSS context-relevant content representing naval operational domains (i.e., sea surface, undersea, air, space, land, and cyber). Leverage generative mesh methods to produce high-resolution 3D textures and shades for the virtual world including text-to-3D and multimodal 2D imagery to 3D content. Apply the retrieval-augmented generation method to improve the accuracy and reliability of generative AI models with facts from external sources. Develop domain-relevant common large language models (LLMs) for defining, packaging, assembling, and editing 3D data for digital content creation applications and visualization. Identify training tools to speed up LLMs training and reasoning about unforeseen events (i.e., unknown unknowns). The VSS end-to-end design must consider distributed deployment, fault-tolerant connectivity, optimal configuration throughputs, and scalability. Demonstrate VSS can perform semantic extraction for automated analysis of all-source intelligence, live multidomain intelligence, surveillance, and reconnaissance (ISR) signals, and can learn I&W signs. Demonstrate the VSS performance facilitates novel virtual collaborative teaming configurations (e.g., human-machine and machine-machine teams) to achieve synchronized operational planning and execution at key points in the contested littoral with immersive decision trees that compute engagement plans, options, and risks associated with the ups and downs of operational encounters.

The feasibility study, test, and demonstration must utilize a combination of OSINT datasets, synthetic datasets from DoN, MCML, unclassified AIS maritime traffic, unclassified commercial satellite imagery, or similar. AI-based data mining includes entity extraction (people, places, and objects), relationships, and transactions -- rigorously searched, recognized, differentiated, and organized into groupings of related facts and relations. Accuracy metrics to ingest and classify multimodal data: structured data mining and interpretation - accuracy of 95% over 98% captured content; unstructured data mining and interpretation – accuracy of 90% over 95% captured content. The software validation, verification, and performance consistency must analyze the VSS sensitivity (i.e., true-positive rate), specificity (i.e., true-negative rate), precision (i.e., positive predictive value), miss rate (i.e., false negative rate), false discovery rate, and false omission rate.

Deliverables: end-to-end initial prototype technology. Conduct appropriate T&E of the conceptual system. Prepare a plan for Phase II. Deliver a final report.

PHASE II: Develop a prototype of the immersive VSS from the candidate technologies in Phase I. Test and demonstrate VSS scenarios with representative operational data sources. Demonstrate the immersive human-machine interaction with the scenarios. Validate the VSS execution timelines to meet the Ops-Floor end-to-end operational requirements. Assess prototype’s performance against the metrics detailed in Phase I. Conduct an "end-user satisfaction" survey, on a scale of 0 to 5: a) VSS situation understanding that includes events that go dark, disguised activities and maneuvers, dormant or not radiating targets; b) alignment with formal I&W signals; c) alignment with prioritized deterrence and engagement options; d) timeliness for responsive cross-domain decision-making and collaboration; and e) immersive virtual training. Develop a plan for Phase III with a transition path to a program of record (PoR).

Deliver prototype software, systems interface requirements for mobile and stationary devices, design documentation, source code, user manual, and final report.

It is probable that the work under this effort will be classified under Phase II (see Description section for details).

PHASE III DUAL USE APPLICATIONS: Advance these VSS capabilities to TRL-7 and integrate the technology into the Maritime Tactical Command and Control PoR or ISR processing platforms at the Marine Corps Information Operations Center.

Once validated conceptually and technically, demonstrate dual-use applications of this technology in the video gaming industry.

REFERENCES:

1. Bokhovkin, Alexey; Tulsiani, Shubham and Dai, Angela. "Mesh2Tex: Generating Mesh Textures from Image Queries." Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 8918-8928. https://arxiv.org/abs/2304.05868#
2. Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P. and Ommer, B. "High-Resolution Image Synthesis with Latent Diffusion Models." IEEE/CVF Conference on Computer Vision and Pattern Recognition, April 2022. https://arxiv.org/abs/2112.10752
3. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, Colin Raffel, Extracting training data from large language models." 30th USENIX Security Symposium (USENIX Security 21), 2021, pp. 2633-2650. https://arxiv.org/abs/2012.07805
4. Park, Taesung; Liu, Ming-Yu; Wang, Ting-Chun and Zhu, Jun-Yan. "Semantic image synthesis with spatially-adaptive normalization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2019. https://arxiv.org/abs/1903.07291
5. Zhou, Bolei; Lapedriza, Agata; Khosla, Aditya; Oliva, Aude and Torralba, Antonio. "Places: A 10 million image database for scene recognition." IEEE Transactions on Pattern Analysis and Machine Intelligence, 40, 2018, pp. 1452-1464. https://ieeexplore.ieee.org/document/7968387

KEYWORDS: Immersive; Artificial Intelligence; Generative AI; Generative Mesh; Virtual Reality; Spatial Reality; Augmented Reality; Mixed Reality; Large Language Models

TPOC 1: Allen Moshfegh
[email protected]

TPOC 2: Behzad Kamgar-Parsi
[email protected]

---