Rowan researchers are creating virtual and augmented reality for the U.S. Army

In training scenarios and live combat, members of the military must make crucial decisions based on limited information while exposed to the elements. To help them prepare for these conditions—specifically, troops housed in gunner turrets on armored and tactical vehicles—an interdisciplinary research team at Rowan University is developing a virtual reality (VR) and artificial intelligence system designed to keep military personnel safe, during both combat and training. 

“The most important asset that the nation has is the soldier,” said Dr. Shreekanth Mandayam, a professor and graduate coordinator in the Department of Electrical and Computer Engineering in the Henry M. Rowan College of Engineering and principal investigator on the project. “One of the places where the soldier is most exposed to danger is inside a turret. This project aims to figure out ways to improve the survivability of the person who is most vulnerable to harm.”

The $2.4 million project is funded by the U.S. Army Combat Capabilities Development Command – Armaments Center at Picatinny Arsenal, N.J. 

The scope of the project is fourfold: To create a realistic and accurate virtual reality environment, develop an artificial intelligence system to help military personnel make decisions, build a communications system between the vehicle operator and the gunner, and create a user-friendly interface for each of these systems. 

First, Mandayam and his team of engineering students and faculty at the Rowan Virtual Reality Center will create a VR simulation where the user will assume the point of view of the gunner in the turret. Researchers will speak with Army officials to learn the full breadth of experiences troops are likely to encounter. 

Meanwhile, a team of biomedical art students and faculty, led by Amanda Almon, an associate professor and program coordinator of the Biomedical Art and Visualization Bachelor of Fine Arts in the Ric Edelman College of Communication & Creative Arts, will create storyboards, illustrations and 3D models of the visual environment in the VR simulation. To accurately represent operations, Almon and her team will collaborate with experts and reference terrain maps, videos and photos of the landscape and architecture, historical weather conditions and battlefield environments.

Since the art and VR teams work closely together to fabricate an entire virtual world, each field of experts must learn the vernacular of the other discipline in order to collaborate, Almon said.

“One of the challenges is interdisciplinary communication,” Almon said. “If we can bridge those gaps, therein lies the excitement of the grant. Then we can really do amazing things because we're using all of our collective talents and skills.”

Within the virtual reality scenario, users will be faced with a bevy of threats, from rocky terrain to enemy action. An artificial intelligence team, led by Dr. Nidhal Bouaynaya, associate dean for research and graduate studies and professor in the Department of Electrical and Computer Engineering in the Henry M. Rowan College of Engineering, will build an AI system that will identify the known and unknown threats in the scenario and further assess whether or not to engage. 

“It can actually predict what is going to happen or predict if the situation is safe or not safe,” Bouaynaya said. “That will enhance a lot of the operations and also make them safer for our military, especially when it comes to threats in engagement scenarios. You have an AI system that is sensing everything around you, integrating and processing all the information from the different sensors and making decisions in real time.”

The next facet of the project is to create a communication system that effectively relays this threat assessment information to not only the gunner in the turret but to others who need it, such as generals and tactical vehicle operators. 

“What is the best information to give first, and in what order do you present information so a person in a very stressful situation can understand and be able to act on it?” Mandayam said.

Dr. Polly Tremoulet, associate professor of psychology in the College of Science & Mathematics, will use psychological and engineering concepts to determine the most effective means of presenting information to users—from threat assessment to providing a 360-degree view of the landscape. Through interviews and observations of training exercises where experienced turret gunners interact with actual or simulated vehicle interiors, Tremoulet and her team will determine what information to display and how (such as on a headset screen or a screen in the turret).

“We don't want to overwhelm them. We don't want to present too much information,” Tremoulet said. “We’re trying to strike that balance between if there’s something important, how do we get their attention with this visual display but not blindside them because they stop monitoring something that's also important.” 

The first two years of the project will include creating the virtual scenario, the AI and communications systems and graphical user interface. Then, the system will be tested in the field. Taken together, the integrative work will result in a virtual and augmented reality scenario for training using cutting-edge technology. 

“It’s a complex project that brings together experts from different colleges and disciplines together to solve a very critical and important problem to protect our soldiers,” Mandayam said.