The Great Learning Mystery: Virtual Reality, Augmented Reality, and Mixed Reality
Warning: Long, comprehensive blarticle ahead!
Turmoil abounds in applying virtual reality (VR), augmented reality (AR), and mixed reality (MR) in learning and development efforts in some corporations. Some industries throw money at these efforts and fail to produce the desired results; they deny the potential benefits and effectiveness of newer technologies in learning applications.
This is intended as a hybrid article and blog post, so the tone is far more casual than it would be in an academic article. I am attempting to write in such a manner that the lay person could understand the concepts of VR/AR/MR and virtual worlds(VW). A friend referred to it as a blarticle.
NOTE: While I am using Wikipedia as a resource for references, I have reviewed the references within the Wikipedia articles, and they are sound references.
Definitions, concepts, assumptions, and clarifications
“Virtual reality (VR) is an interactive computer-generated experience taking place within a simulated environment, that incorporates mainly auditory and visual, but also other types of sensory feedback like haptic. This immersive environment can be similar to the real world or it can be fantastical, creating an experience that is not possible in ordinary physical reality.”(1)
Wikipedia includes information about augmented reality (AR) as follows: Augmented reality systems may also be considered a form of VR that layers virtual information over a live camera feed into a headset or through a smartphone or tablet device giving the user the ability to view three-dimensional images.(1,2)
VR is further distinguished from simulated reality – where reality could be simulated to be indistinguishable from true reality. We will focus only on the VR and AR definitions for the purposes of this article, not including simulated reality.
Review the definition of mixed reality (MR) or hybrid reality. This is “is the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real time. Mixed reality takes place not only in the physical world or the virtual world, but is a mix of reality and virtual reality, encompassing both augmented reality and augmented virtuality via immersive technology.”(3)
In this article, the focus for discussion will be the petrochemical industry. This industry is chosen only to represent a common industry that could greatly benefit (and in some cases, has already benefited) from the innovative applications of VR, AR, and MR.
Let me take just a moment to clarify what I mean by innovative. I don’t necessarily mean cutting edge technology. I personally LOVE technology. I will jump in head first and try new things. And there are risks associated with using the very latest and greatest technology for your business. The first being expense – new technology is more expensive. And the second risk is that new technology is risky because it often doesn’t work optimally when released. This is a real risk, and when it comes to business, I rarely (never) recommend using a new technology out of the gate. I will often wait until the third or fourth release to dive in.
The reference to innovative applications is referring to tried and true technology that is being applied in an innovative manner. In the field of learning and education, there are countless opportunities to apply existing technology in ways that can enhance the learning, while the technology may be well-established or even considered old. Keep in mind, old is a relative term with regard to learning technology.
Problem: Every day, we see headlines for impressive applications of VR. Typically, we are seeing the biggest applications in medical science, the military, space exploration, and university research in scientific areas (typically paid for with large grants). In other words, we are witnessing applications in areas where these organizations can spend astronomical amounts of money for VR development and implementation. This tends to discourage more common industries, companies, and people from delving into any applications of VR or AR. It can make matters more discouraging when specialized vendors who niche-holders in the industry only express interest and respond to RFIs (requests for information) with only the most cutting-edge solutions and the astronomical price tags. Additionally, most vendors that respond in these areas have their expertise in the technical area(s), as opposed to their baseline expertise being in the area of learning and development / education (as a discipline).
When I dove into virtual world learning applications as a specialty, my focus was on finding ways to bring virtual world learning into areas of corporate training in such a way that it was exponentially impactful, relatively affordable, sustainable, and practical. Please be aware that I was doing this in 2007 – which was a tall order at the time. Practicality, affordability, and sustainability were not the most common three words on the tips of the tongues of the major vendors. Much of the time, another vendor had proposed a similar project and given such a high price tag that when my boutique firm came along and offered a project at a fraction of the cost, the client simply could not believe it could be done for that amount.
In the effort to be leading edge, supercool, and to explore the possibilities of VR and AR; many companies tried to develop internal resources. I had clients contacting me about a year ago, because they had selected training professionals from their internal resources and had decided to develop them in the areas of VR and AR. The clients were having difficulty completing assets and projects. In fact, they were truly having issues distinguishing between the development of assets and the learning project itself.
Within the world of digital learning development of VR, AR, and MR – you are going to have projects within projects. The learning project is a project and each asset that is developed may represent a project.
So, what is an asset? The very condensed definition of an asset is any sort of object you can view and / or with which you can interact in your project. It’s your content within your game, project, setting, etc. That may not be the broadest generalization of an asset within VR/AR, however, it is the relevant one for the purposes of this article. Virtual assets often refer to virtual currency or anything that is perceived of value; in this case, we are using asset to specifically refer to content within the virtual environment.
Assets may include avatars, buildings, furniture, equipment, or interactions that are object-based but not visible; you may interact with a script within an object, but may never see the object itself. Anything that can be viewed in a virtual setting is an asset. Anything with which you can interact is an asset. And all those things must be purchased, designed, and/or developed.
Companies new to asset development can easily get bogged down in the old chicken/egg question: What comes first: Asset development or learning development?
Almost every time, the request for support is coming to me based on asset development. For example, the client has decided a virtual pump or compressor is needed. The vision is that this asset will be created – this virtual pump or compressor, and it will be able to be taken apart and rebuilt in completion. After that, this pump or compressor will be used to teach maintenance personnel how to care for a pump or compressor. However, this approach is backwards. Learning needs to define the purpose and functionality of the asset.
The other dilemma here is that someone from the internal team is selected to create this asset. That person must be developed to perform these tasks. The organization may pay for the selected person to take some asset development courses, and turn him/her loose. This is quite unrealistic and unfair to the person in question.
Solution: Within a learning project, it is preferable to design the learning first. And here is why:
The asset should optimally be designed to fit the project. Complex generic assets are often a waste of time and resources. Learning projects begin with solid instructional design (ID). The ID process will determine the specific requirements and functionality for assets.
If one develops a virtual pump that can be completely disassembled and reassembled without considering the ID process, one may leave off a critical part or function of that pump. It is highly probably to miss some aspect or detail of that virtual pump, and more importantly, the creation of this complex asset will likely be cost-prohibitive in terms of dollars and man hours.
The most critical issue: if you are in a refinery or plant, what is most impactful in terms of learning to work on a pump? Would it be more advantageous to work on a virtual pump or to work on a pump that is out of service? As an educator in this field and someone who used to work on rotating equipment in a petrochemical environment, I can tell you that actually taking a real piece of equipment apart and putting it back together is difficult to replace in terms of real value. And it’s practical – a refinery or plant will likely always have equipment that is out of service on which to practice.
The key to this issue is to start with the learning outcomes and objectives, NOT the asset. The necessary assets and their requirements should emerge from the learning outcomes and objectives, as opposed to the other way around.
Let’s go back to our example of the pump. I have VR developers argue that there are many issues that you cannot easily teach on a pump that is out of service. For instance, how do you show someone what can go wrong if a gasket is tightened improperly?
Let’s do this according to the learning outcomes and objectives. My outcome might be that the learner will understand the possible negative consequences of failing to tighten gaskets using a particular method. My objectives could be as follows:
- The learner will use the X method to tighten the gasket in such a way that the equipment is free of leaks and functions properly.
- The learner will observe negative consequences when methods other than X method are used to tighten the gasket.
My learner will review the proper method. I can do this easily with a pump that is out of service, and there is no need to program haptics. However, I cannot demonstrate leakage easily or safely if the proper method fails to be performed optimally. I can, however, show videos or animated renderings of what happens if other methods are used or if the desired method is performed incorrectly. It may not be as exciting as experiencing a leakage or failure in VR, and I will assert that it will get the job done.
This is where AR or MR might be the most affordable and sustainable way to support this learning in real time. I can easily outfit my pumps with a barcode sticker. When a mechanic comes to work on the pump, she/he can use a tablet or phone to read the barcode; instructions for performing the proper methods come up, with videos and renderings. There are also videos and rendering for negative consequences. This is an example of using AR instead of VR; it is much more affordable and sustainable than true VR. It is a practical application of AR. Asset development is within the realm of possibility by internal personnel. It is likely that the application of those applications in AR (the bar codes, interactions within a rendering) will require the use of a company that specializes in that area.
Part of the reason I am a proponent for this approach is because most examples we are encountering in refineries and plants are based on existing scaffolding in terms of learning. It is likely that the pump maintenance person is already a skilled mechanic. He may not be specifically skilled in that type of pump, but he has all the basic and maybe even the higher level skills of understanding what he needs to do regarding general care and maintenance of rotating equipment. In my own experience, I learned to work on my car in college. I could change my oil, grease my wheels, even rebuild a carburetor. With the right equipment, I replaced my front end suspension and rebuilt an engine – including renting a honer and honing the cylinders of the engine while sitting on the floor of my sister’s garage. This made working on extrusion equipment a piece of cake when I was doing polymers research for a major chemical company; it was far easier than working on a car engine. At the end of the day, rotating equipment is rotating equipment – the general principles apply across the board. There is a great deal of positive transfer of learning between a car engine and an extruder.
It is important to examine the existing scaffolding when developing learning. And it is critical to avoid getting caught up in, “Let’s do something because it is cool.” You can do creative application without be superfluous or breaking the bank.
What does it really take to develop good assets?
Quite simply, asset development is the development of the content used in VR/AR/MR. One of the major issues in the field of applications is the lack of understanding of all the skill sets currently required in asset development. Currently, there is no encompassing authoring tool for asset development. Rather, assets are developed using many tools and skills.
In speaking to some colleagues in the 3D animation and asset development field, I have created a list of currently needed skills for comprehensive asset development. This is a high level list, and it is very likely not complete.
- Graphic design
- Video editing
- UV Mapping
- Game design
- Haptics – equipment/coding
Within each discipline and expertise, there are multiple tools and platforms that can be used. Being a great animator, photographer, and videographer doesn’t mean you can create great VR, AR, or MR. Those assets must be edited, rendered, sometimes edited again at different points, and scripted to perform within a platform. And there are a variety of platforms.
I would assert that it is completely unrealistic to take people, have them take a couple of Photoshop classes, maybe a video editing class, and believe they can develop viable assets. Even if they could develop some decent assets in a short time, would they be able to design the learning to make that asset perform optimally? My experience with clients contacting me to discuss these matters leads me to believe this isn’t working well.
Companies are likely much better off sending internal trainers/instructional designers to learn more about the creation, management, and embedding of assets from a project management and program management perspective. The internal master trainer or instructional designer will then focus back on the learning outcomes and objectives, ensuring that asset development occurs within a framework of practical and optimum utilization.
The Trend of Tools for Asset Development
Companies need to be aware of the trends for asset development. Currently, trends for tools are towards making life easier for specialized developers within the discipline, not creating cross-disciplinary tools. Graphic art tools get more powerful for specialized users; they don’t get easier for non-graphic artists to use. Rendering tools get easier for 3D specialists to use; they don’t make it easy to learn to be a proficient 3D specialist.
My prime example of this is being in a CMO (Chief Marketing Officer) meeting / education day a few years ago. A VP from marketing for Sony was doing a presentation regarding VR asset development. Sony and Google are two of the most ardent developers of consumer technology for VR and AR utilization. They both see the possibilities of VR and AR, and they recognize the opportunity to create growing demand.
The VP talked about all the cool and exciting consumer tools coming out. He enthusiastically shared Sony’s commitment to driving the cost of these tools down; and they have been doing this quite successfully. He then shared about a camera framework being released that would allow for simultaneously photography and videography from multiple perspectives. This framework was mere months from being released and would be a gamechanger for asset development and splicing.
I then asked him about the move to create tools that would do the same for asset scripting and embedding. He made it clear; neither Sony nor any company of which he was aware had any intention or desire to develop authoring per se for VR/AR/MR development. It simply wasn’t on the radar, nor do these companies see it as practical. I, for one, concur. While I desperately wish this would happen, it simply isn’t practical at this point in time.
What does this mean for the future? Gartner’s Hype Cycle
As an educator who sees and has participated in amazing possibilities for VR/AR/MR, I have a great fear: That these applications will die a horrible death before ever really coming to fruition. Something similar happened with the virtual world and Second Life, and while we have some diehard cultists who continue to swear that MR/VW (virtual world) development is far from dead, I believe mainstream development and application has at the very least stagnated and, at worst, been set back decades in terms of development. The inability of a company to affordably produce the capability to enter and participate in a virtual world with a mainstream browser is a prime example of how this industry has been set back in a major way. Linden Labs showed us what was possible, and then promptly set their sights to become the 3D answer to social media, prompting a mass exodus of major players. While Unity and Unreal continue to provide environments for asset management and application, neither of those companies are really promoting the social use that made Second Life the exploding opportunity that once existed. New players on the market simply lack the cash flow to keep things progressing at a reasonable pace, with period setbacks and functionality glitches with development upgrades.
Virtual worlds are an example of a product that seemed to have great answers to learning and has diminished in popularity. There are still us mainstreamers that still dream of possibilities, and there are few solutions. One that I believe shows ongoing promise is Sinespace . Information about their enterprise learning cases is located here. They have a well done game engine that is template-based and quite powerful as part of their offerings. They are quite affordable as well.
Many corporations won’t risk doing any VW development of any kind, as they continue to distrust sustainability after their failed investments in other worlds, such as Second Life. .
VW development is a great example of Gartner’s Hype Cycle. In Wikipedia, Gartner’s Hype Cycle is described as a “branded graphical presentation developed and used by the American research, advisory and information technology firm Gartner, for representing the maturity, adoption and social application of specific technologies. The hype cycle provides a graphical and conceptual presentation of the maturity of emerging technologies through five phases.”(4)
Table 1 – The Five Phases of Gartner’s Hype Cycle (4)
|1||Technology Trigger||A potential technology breakthrough kicks things off. Early proof-of-concept stories and media interest trigger significant publicity. Often no usable products exist and commercial viability is unproven.|
|2||Peak of Inflated Expectations||Early publicity produces a number of success stories—often accompanied by scores of failures. Some companies take action; most don’t.|
|3||Trough of Disillusionment||Interest wanes as experiments and implementations fail to deliver. Producers of the technology shake out or fail. Investment continues only if the surviving providers improve their products to the satisfaction of early adopters.|
|4||Slope of Enlightenment||More instances of how the technology can benefit the enterprise start to crystallize and become more widely understood. Second- and third-generation products appear from technology providers. More enterprises fund pilots; conservative companies remain cautious.|
|5||Plateau of Productivity||Mainstream adoption starts to take off. Criteria for assessing provider viability are more clearly defined. The technology’s broad market applicability and relevance are clearly paying off. If the technology has more than a niche market then it will continue to grow.|
Some opinions would place VR/AR on the very slight upside of the Trough of Disillusionment and starting to move up the Slope of Enlightenment. I agree in this case – the placement would be accurate. My fear is this is a precarious place for VR/AR to reside in terms of mainstream learning applications. Without efforts to educate potential clients and users in the corporate and mainstream learning arenas, we could see the stagnation and eventual death of AR/VR that we are seeing among the virtual world authoring applications.
Tips for Development
As companies and mainstream educational institutions desire to create and utilize AR/VR/MR, consider these things:
- Focus on your instructional design FIRST and foremost. Assets and requirements will emerge from solid learning outcomes and objectives. Let the learning professionals get the learning design done; then let the technology experts work within that framework.
- Your instructional designers and similar disciplines are your first line of production – consider developing these people as project and program managers for AR/VR development, not necessarily investing in them to become some sort of author or scripter. Invest in their instructional design capabilities to extend to include conceptual technology applications. Otherwise, you will only have part of a larger picture.
- Review the disciplines needed for asset creation. Inventory the skillsets you have in your organization that may be easily adapted. Reach out to consultants and specialists in the areas you are lacking. Consider that developing someone in a discipline takes years, not days.
- Find consultants that integrate learning and technology at a conceptual level. These consultants will bring your skilled folks along, teaching them about the technology and integration, asset development, and what they can contribute.
- Understand that your core business will likely never be asset development and management. Are you planning on becoming a VR/AR development company? No? Then use care investing your money in developing internal personnel for this purpose. Find skilled and budget-minded professionals to support and help you learn.
There is an old curse attributed to many different cultures: May you live in interesting times.
And, wow, do we EVER live in interesting times!
While VR/AR/MR continue to be somewhat elusive and mysterious, keep moving forward. Consider the possibilities and work with experts that support YOUR vision and YOUR needs. Consider that learning and educational expertise is as important as the technology that is to be implemented.
NOTE: I use Wikipedia for nearly all of these references in this particular instance. I have reviewed the Wikipedia information and references, and they are accurate. Please feel free to review the Wikipedia reference lists for more information.