top of page

Mixed Reality: CyberSecurity VR

Explorations, Texas A&M University Undergraduate Research Journal
Volume 11, Fall 2019 
Fig1.png
Figure 1. “Cyber Security VR” Mixed Reality Demonstration.

            Though Virtual Reality (VR) has been technically possible for decades, it hasn’t seen wide scale commercial viability until recently. Currently, we are in a period of rapid innovation and growth in VR technology and application. VR has made its way into video games, classrooms, and even the workplace, because its use is intuitive. Essentially, virtual reality uses the brain's natural perception of depth to transport you to another world. You use the same motions and visuals that you have had since birth to look and interact with virtual environments. This makes it so that people who have never touched a controller can interact in VR with ease. However, there is a great disconnect between reality and the simulated environment. Without using the equipment first hand, it’s difficult to convey the core experience of interaction in VR. If VR is to enter and improve massive industries, its producers need to showcase and sell their content in the most visually appealing and marketable way. Our team is exploring a method of filming which is already changing the way we look at VR.

           Our research team has set out to address the problem of visual inaccessibility in VR demonstration. The challenge was to display how a person acts and interacts in the virtual world in a naturalistic, aesthetic, and practical way. To achieve this, we built a “mixed reality” system, which essentially composites a live actor into a virtual environment. To do this, we film an actor in VR performing in the desired application. We use a surround green screen studio to distinguish outlines of the actor from all angles. Then, we attach a positional tracker to the real camera, and create a virtual camera to mirror the position, rotation, and field of view. This creates a “virtual cameraman”, which moves 1 to 1 in real and virtual space. The actor is then composited into the shot in the position they would be occupying. This creates a final image which is similar to a live action camera shot of an actor, yet the environment is completely virtual (Figure 1).

Results: Virtual Filmmaking

         For our three guiding criteria, we had to consider a wide array of functions. In order to make the footage seem naturalistic, objects would have to interact with the actor as they would in the real world. Therefore, we would have to include variable opacity in our composition. If an object is in between the camera and the actor, it must block the subject from view in the same way it would naturally. If the object has translucency, then it should only partially cover the subject. We achieved this through the four panel compositing method, developed by Kert Gartner and the Owlchemy Lab team for the production of VR game trailers.

4 panel-01.png
Figure 2. Four output displays captured at once. From top left to bottom right: Foreground, Foreground Alpha Matte, Background, reference POV angle.

            In this method, we split up the virtual camera into 4 views which will act as different layers in our final composition (Figure 2). We then stack the layers, along with the green screen footage in the correct order, to produce the final shot. The background layer displays everything that the virtual camera sees; this will capture anything behind the subject. Then, the actor is chroma keyed into the frame so that only their form is visible. Then, we use an alpha matte (a black and white template) to extract only the objects between the camera and the subject. In places where the alpha matte is white, something is obstructing the camera’s view. In that position on screen, we render the image in front of the actor with the foreground layer. Any places where the alpha matte is a gradient between white and black, are either translucent objects or objective markers for navigation and direction in the program. All of this combines to give a final image which places a person seamlessly into a digital space (Figure 3).

Fig3.png

            Now that we have our actor in the scene, we must consider the aesthetics of the shot. This will ensure the greatest production value and “readability” of our demonstrations. Fortunately, much of the cinematography is similar to live action production. In the production of our video on Texas A&M University’s “Cyber Security VR” application, developed by Dr. Hwaryoung Seo’s research team, we approached the shoot as if we were filming on location. In the software, you tour a high security server farm, and perform maintenance on a faulty server rack. It is important that this training can be done in VR, because actual server farms are highly secure and losses can be costly. In order to film the demonstration of the software, we decided to do multiple run throughs from varying angles. This would give us coverage to get the best shot for any given action. However, there were some challenges, as in many instances our camera would spawn behind a wall or prop, and the action would be blocked. In the preproduction of our next demonstration for “Muscle Action VR”, an educational software which simulates how your muscles move in real time, we decided to attach the physical camera to a steady cam and film the demonstration at any angle that the camera operator moves. This would add dynamic movement to the shot, while also allowing us to adjust angles to obtain more presentable footage

           Lastly, it was important that our footage be practical and applicable. A large portion of VR demonstration occurs at live events. For this reason, we wanted to be able to display actors in virtual environments in real time. With this method, VR demos would become more communal, allowing passersby to immediately understand the application before trying it themselves. In order to do this, we used LIV VR compositing software to composite each video layer in real time and output to a display. While this has proven to work in the lab, we will showcase both the live broadcast method and the final version of Muscle Action VR at this year’s VIZA GOGO student showcase in May.  

Figure 3. Composition Layering Chart
Conclusion: Applications for the Applications

           It is not likely that we will completely forgo two-dimensional displays as our main form of information transfer any time soon. In order for VR to thrive, it has to be available to audiences whose primary interface is through screens. By developing methods which can artistically capture VR on a plane, we can make it more accessible to the general public. In entertainment, we can advertise products and games by showing real people using the software, and having fun. In education we can step into virtually any environment, and experience learning topics firsthand. Entire classes of third graders can explore life science in a unique way by visiting jungles, oceans, and highlands while staying in the safety of their classrooms. Furthermore, doctors and nurses in training can experience surgery without the threat of human mortality. In standard VR, these would all be singular experiences, but with Mixed Reality, entire groups can interface with the programs. This is just the beginning of how VR can expand into more practical roles. With the advent of Mixed Reality, we can view VR content in a more human way.

Acknowledgements

Texas A&M University: Department of Visualization

Dr. Hwaryoung Seo

Bill Jenks

​

Graduate Student Collaborators:

Eunsun “Sunny” Chu

Anantha Natarajan

Austin Payne

Creative Anatomy Research Team

​

Kert Gartner

Owlchemy Labs Development Team

LIV Development Team

​

Programs used:

LIV Compositor

OBS Virtual Camera

Black Magic Media Express

Adobe After Effects

Texas A&M University “Cyber Security VR”

Texas A&M University “Muscle Action VR”

Owlchemy Labs “Job Simulator”

References

  1. How Virtual Reality Will Change How We Learn and How We Teach                                                         Babich, Nick. “How Virtual Reality Will Change How We Learn and How We Teach.” Adobe Blog, Adobe, 5 Jan. 2018, theblog.adobe.com/virtual-reality-will-change-learn-teach/.                                                                                                                                                   

  2. Making High Quality Mixed Reality VR Trailers and Videos                                                                        Gartner, Kert. “Making High Quality Mixed Reality VR Trailers and Videos.” KERT GARTNER, Owlchemy Games, 5 Apr. 2017, www.kertgartner.com/making-mixed-reality-vr-trailers-and-videos/.                                                                                                                                          

  3. Soft Interaction: Texas A&M University Research                                                                                         Seo, Jinsil Hwaryoung. “Soft Interaction Lab.” Soft Interaction Lab, Texas A&M University, softinteraction.com/.   
                                                             

  4. LIV Software                                                                                                                                                    LIV. “Main Page.” LIV Wiki, Techstars Company, 24 Jan. 2019, wiki.liv.tv/index.php/Main_Page.

Introduction: The Expansion of VR
Methods: Mixed Reality in Action
bottom of page