Gaming Students Develop Augmented-Reality Flight Training Tool

A team of Simulation Science, Games and Animation (SSGA) students got to work this summer developing augmented-reality (AR) holograms that will eventually play a part in training the next generation of aviators at Embry-Riddle Aeronautical University.

The project was funded by an Undergraduate Research Institute (URI) grant, empowering Prescott Campus students to create virtual CRJ-700 models to be used in College of Aviation classrooms, using HoloLens 2 goggles.

“The holograms they have made will be a wonderful tool to start using AR for teaching, practicing of skills and, ultimately, testing,” said Dr. Michelle Hight, assistant professor of Aeronautical Science and Fixed Wing program chair.

The holograms developed by the SSGA team work like this: A student wearing the headset can step through “slides” inside interactive virtual training modules, giving them access to view schematics and information about the CRJ-700 aircraft in 3D. Another type of hologram model in development could also let them look around the flight deck, manipulating the buttons, switches and controls as they would in real life — the HoloLens 2 can even “see” their hands.

Some of the other holograms under development by the SSGA team depict the landing gear operation and the external fuel panel.

“We encouraged the team to consider the reasons for using augmented reality rather than virtual reality on the project,” associate professor and SSGA Program Chair Derek Fisher explained. “They determined that while virtual reality might provide a more immersive alternate experience, AR provides opportunities to focus on interacting with the computer-generated jet while collaborating in the familiar environment of the classroom, with real-life colleagues.”

The holograms created by the SSGA team for the AR environment have informational labels and can be rotated and resized by the user, giving students the opportunity to do a unique sort of virtual hands-on study, becoming familiar with where controls and important parts of the aircraft are located. Specifically, the module the SSGA students completed has a detailed schematic of the exhaust and radar danger zones. When it’s time for quizzing, the parts’ labels can be removed, and students will be asked to identify either location of parts or schematics to test their knowledge.

“This is the first step down a very good road for us as a university, potentially toward an entire class in AR,” Hight added. “Even if students had to attend class from home, this brings the jet to life. Every student who has a HoloLens 2 could join in 3D, standing or sitting in a virtual classroom, interacting with hologram learning tools rather than a flat screen. This is a timely pedagogical development for a world evolving and responding to a global pandemic.”

Virtual Tools, Real Training

SSGA students learn to design and develop interactive and immersive experiences on a daily basis, according to Fisher, which made them the obvious choice to be added to the hologram-development team.

Through a pair of HoloLens 2 augmented-reality goggles, students can see and interact with holograms of aircraft and their parts.

“The goal on this particular project was to leverage the emerging technology of the HoloLens 2 and connect the technology and information back to the human experience,” Fisher said.

Over the summer, a team of five students worked on the AR prototype over a five-week period. Next, the team will improve the software so that multiple users can interact with the same jet simulation simultaneously. After testing the new prototype, the group will seek funding to expand the program to a full-class experience.

Some advanced forms of the software have even already been used in flight deck procedures training, allowing students to participate in virtual tutorials. The ongoing development of the technology, though, is providing students with valuable learning opportunities. Students like Evan LaBate, a senior SSGA major, gained hands-on experience building simulations he envisions will change the course of flight training on campus.

“Participating on this project has helped me decide that I want to do virtual or augmented reality simulations, either for military or police, as my career,” said the Chandler, Arizona, native. “I believe this technology can be used to help the military simulate live firefights, or even help with Air Force, in giving pilots updated heads-up displays.”

Fellow SSGA senior Josh Snow, from Phoenix, Arizona, agreed.

“This project was a massive learning experience,” he said. “Jobs in the field of interactive technology are plentiful but demanding, often working on large projects in small teams. The skills I have picked up from this research opportunity have better prepared me for all facets of working on and or managing a team in my career.”

Snow’s career plans have taken many shapes over the year, he added. Growing up, he dreamt of being a racecar driver, mechanic and paramedic, but once he hit high school, something clicked.

“I was exposed to simulation and visualization as the 3D design lead of my robotics team, and I fell in love with it,” he said. “Taking a complex mechanical device and bringing it into the virtual world, I could play with variables that I couldn’t dream of touching in reality and, furthermore, communicate concepts, ideas and products in a way that would help people to better understand them.”

Today, his dream job is simply to “bring ideas to life,” through video games, architecture, virtual reality or any other medium that requires a mix of both creativity and technical knowledge. The SSGA program has given him that opportunity.

“We are on the leading edge of this technology at Embry-Riddle — both for the pilot students, who are the end users, and the Games & Sims students, the developers,” Hight added. “This is really the future of pilot training. Imagine what this could do for distance learning.”

Posted In: Aviation | Computers and Technology | Engineering