What is the concept behind, and driving force of, your research?

Dr Joel Martin: Firefighters have high rates of musculoskeletal injuries (MSKIs), which creates a major burden on fire departments and, in turn, communities. Many departments are experiencing staffing issues, which are in some part attributable to MSKIs. Many injuries occur during training exercises, which are inherently dangerous due to the demands of the occupation. While many of the tasks have a physical component that can’t be replaced with augmented reality, the cognitive components (e.g. reaction time, decision-making, visual scanning, etc.) can be trained with augmented reality (AR) and other types of VR simulations. Essentially, the aim of the research is to make firefighting training safer and more effective.

Dr Craig Yu: I am a computer science professor at George Mason University (GMU), where I direct the Design Computing and Extended Reality (DCXR) Lab. I have been working on generative algorithms for virtual environment synthesis for more than 15 years. From wheelchair VR training, earthquake safety training, driving training, to healthcare training and fire evacuation training, my team has been devising generative algorithms for synthesizing all sorts of VR training scenarios.

In recent years, there have been tremendous advances in VR/AR devices and artificial intelligence (AI) technologies such as computer vision and large language models (LLMs), which promise to bring VR/AR training to the next level of effectiveness, scalability, and success. As a researcher, I observe many unique opportunities to integrate the power of AI into VR/AR training methods, to bring novel, powerful capabilities such as training personalization/adaptiveness, infinite training scenario synthesis, automated analysis of trainee performance, AI-driven coaches, and automatic generation of feedback and improvement advice. It is the excitement of such opportunities that has driven my research.

On the entrepreneurial side, I also wear the hat of being the CEO of a VR tech startup, Great Victory Legends, incubated at the GMU’s Virginia Serious Game Institute. Our startup specializes in devising AI-driven VR training tools and solutions, and transferring some of the research from our lab to the industry. This is the dawn of AI-driven VR/AR training. Seeing the possibilities to transfer my research to make a direct impact in different training domains (e.g. healthcare, defense, education) also motivates me to pursue research in this direction, as it is truly exciting and rewarding to see how people could collaborate with AI to improve their skills.

Using augmented reality in a training scenario removes the risk of physical injury, but how adept is the software at translating the real-life risks of aerial firefighting scenarios?

Dr Yu: How adept the software is at translating the real-life risks of aerial firefighting scenarios depends on the level of realism we can attain in the synthesized VR/AR scenarios. Although I said that I devise AI algorithms to automatically generate VR/AR training scenarios, in creating such algorithms, we also need useful inputs from domain experts (e.g. firefighting experts) to provide expert inputs to refine our generative AI algorithms – for example, telling the algorithms what the goals are, constraints, important factors to consider in the training scenarios. Such information would allow our AI algorithm to generate many VR/AR scenarios that are ‘random’ yet reasonable, akin to situations that first responders would face in the real world.

Is using AR for training now within reach of even smaller operators, given that the price point of the tech needed, like headsets, is changing?

Dr Yu: That depends. The popularity of consumer-grade VR/AR headsets (e.g. Meta Quest) that only cost a few hundred per piece definitely make VR/AR training much more economic, accessible, and scalable. They are fine for a lot of basic training that doesn’t require high-end graphics rendering, sophisticated simulation, and detailed human pose and eye tracking. However, if you want high-end VR/AR training which incorporates the advanced factors like I just said, you probably need more high-end VR/AR headsets, and the computation would probably be done on a high-end computer with a powerful graphics card (think about generating highly realistic fire simulation responsive to the trainee’s action). In addition, we will need additional, sometimes even bespoke, controllers and haptic feedback devices, to enhance the training immersiveness.

If you want high-end VR/AR training which incorporates the advanced factors, you probably need more high-end VR/AR headsets, and the computation would probably be done on a high-end computer with a powerful graphics card

Can the AR program you’re currently developing be adapted to different levels of ability and training experience?

Dr Martin: Definitely: firefighters must prepare for numerous different scenarios, which have overlap with other military branches and occupations. Some refer to firefighters as ‘all-hazard’ responders as they literally must be prepared to respond to any type of emergency. Locally in Fairfax County, there are areas that are urban with high rises, but other parts of the county are densely wooded, with waterfalls, rivers, etc. Also, firefighters must perform different types of training based on their occupational roles. For instance, some are responsible for entering the structure, identifying the cause of the fire, and extinguishing; others are driving the fire engine. Then the captains and chiefs must coordinate the tasks of all the firefighters on the fireground and communicate with those who may be driving to the emergency.

Dr Yu: Definitely. Achieving VR/AR training adaptiveness through AI is a key objective of our research project. We already have some preliminary results on creating such algorithms for other training domains. For firefighting training, our AI algorithm should be able to consider the trainees’ skills, background, performance, weaknesses, etc., to adapt the AR training scenario to guide the trainee to improve effectively. Such an approach is particularly feasible in AR training, because we can closely track the trainee’s behaviors and reactions with respect to different virtual simulated events (e.g. explosion), so gaining insights about the trainee’s overall performance is quite possible. With that, our AI can then adapt the training scenario to be effective in reinforcing the trainee’s skills and in overcoming the trainee’s weaknesses.

Achieving VR/AR training adaptiveness through AI is a key objective of our research project. We already have some preliminary results on creating such algorithms for other training domains

How customizable is the training program in other ways – from nighttime firefighting to daytime, for instance, and from high to low altitudes?

Dr Yu: As the VR/AR training is a simulation after all, everything within the simulation is customizable. So you can easily switch between nighttime and daytime, generate different weather conditions, change the current status of the plane, etc. We could even ask our AI to adapt training scenarios on the fly. Say you have already proven yourself in putting out fire during the daytime, as the AI could see from your performance data, how about putting out fire on a windy night? Our AI should be powerful enough to generate combinations of training scenario parameters to make sure that a trainee is experienced and proficient in dealing with different situations.

What stage is your research at currently, and what do you need from aerial firefighting organizations to get it to the next level?

Dr Yu: As mentioned, we are experienced in devising personalized VR/AR training scenarios for different domains, so many of the insights we obtained previously could be leveraged to build AR firefighting scenarios as well. My team, consisting of a postdoc researcher, PhD students, and undergraduate research assistants, is investigating how we could incorporate the latest AI models (e.g. large language models) to power AR firefighting training. We are investigating such training in VR first, as things are in complete control in VR and can be tested more easily, but then we will move on to extend the approach to work in AR, which would be more complex as our simulations would then need to consider the trainee’s locomotion and object interaction in real spaces. We aim to generate such AR training scenarios using the new Fuse building at GMU as our first test bed.

We aim to generate such AR training scenarios using the new Fuse building at GMU as our first test bed

Of course, our team will also consult our firefighter collaborators to understand their requirements and to devise realistic, useful training scenarios, even as a proof of concept. Similarly, if we create aerial firefighting VR/AR training scenarios, the first thing we should do is to chat closely with the domain experts on their current training practices and requirements, such that we can translate such considerations into prompts and formulations that our generative AI algorithm would understand, after which it can generate useful scenarios automatically.

What are your hopes for the future use of AI and AR in firefighting scenarios?

Dr Martin: Expanding on initial efforts to simulate a wider range of firefighting scenarios, and from input obtained from firefighters create more authentic simulations that reflect the complexities of the emergencies that are responded to. Obtaining user feedback and buy-in from stakeholders will be key.

Dr Yu: There are many training procedures that we could potentially simulate using AI-driven AR training. First, we want to demonstrate the effectiveness of AR training on a few use cases, and then we need to think about how to scale up in terms of the variety of training tasks that we could mimic. Eventually, the power of AI should allow us to implement training approaches and pursue training analysis that may not yet be possible, especially when it comes to personalization and training adaptiveness. As a VR training researcher, it is my mission and my passion to innovate how we can fully unlock the potential of VR/AR training with the power of AI. Eventually, we may enter an era where humans can quickly become highly skilled in different tasks and problem-solving situations with the help of AI.