Our Training Committee held a very successful webinar on “The Impact and ROI of VR/AR in Training” on 16th January 2019.
You can watch the Webinar recording here
You can also watch the solution demos here
It proved a popular event, with over 300 people registering to watch four enlightening presentations on very different Virtual Reality and Augmented Reality training applications. During the lively question-and-answer session it became clear that the number of great questions was larger than the time available to answer them all! This post allows the presenters to respond the questions that couldn’t be answered during the webinar.
Presenter 1 : David Trainor, Sentireal
Presentation Theme
David presented the “AppUccino” software, which delivers AR experiences for special-needs trainees to learn how to be a barista and work successfully in a commercial coffee shop. AppUccino directs trainees through the process of making americano, espresso, latte and cappuccino drinks. It generates AR overlays on the physical coffee-making equipment, including the industrial coffee machine itself, the separate coffee bean grinder and the separate water heater. A short video of part of the AppUccino AR experience can be seen here. Trainee performance measurements are taken as they prepare the drink using AppUccino and these measurements are uploaded to a cloud application, which analyzes them and presents training progress information to authorized training managers. An initial pilot with AppUccino indicated that trainees retained twice the amount of information and developed their skills 50% faster using AppUccino.
Webinar Questions for David Trainor
You said it took ~9 months of definition and dev, how many people were working on your team in which roles in order to accomplish this?
We used one media/content creator and one software developer. The work was carried out over the course of 9 months, but neither the content creation nor the software development role was full-time.
How were the analyses done? Observation? Performance against criteria on task?
The analyses were done based on (i) time taken for the trainee to perform certain tasks (ii) measurements of how accurately the trainee performed certain tasks and (iii) estimates of where the trainee’s attention and focus was at specific points in the procedure.
How many users has Appuccino had? What has been the impact? In how many months after development did you achieve that impact?
We’ve just completed an initial evaluation/pilot involving 15-20 trainees within a coffee shop managed by a partner training organization. At some point in February 2019 we intend to expand this to all barista trainees within that training organization.
How long did it take to create AR content for training?
The 3D models for the AR overlays were completed in approximately 1 week. The media is actually quite simple. Strong, simple content proved to be the best way to present the required information to special needs trainees who are affected by autism, Downs Syndrome or general learning difficulties. However we needed to spend much more time – about 4 months – on the instructional design and the directions, animations and transitions within the AR scenarios.
How did you already go about gathering data on ROI prior to the development (and also launch) of the app?
The data was gathered from a pilot study at one location. This was after the app was developed but before it was publicly released.
Very exciting project! Curious why you selected that small, unique demographic to start? Unique requirements? Expect to scale up to other populations?
We picked barista training as it’s an area where there is expected to be a lack of qualified employees in the future. In the UK, where I’m from, it is estimated that there is a shortfall of 40000 trained baristas. We selected special-needs barista trainees because there is a high degree of underemployment in that group and if we can create a good intuitive solution for special-needs trainees then it should scale up well to other groups and other types of catering or hospitality training applications.
Are these analytics published somewhere?
Not yet – the small pilot study that we’ve carried out gave us some indications on the training improvements and return on investment from AppUccino but we’re starting a larger evaluation in February 2019, when more trainees start to use the app at our partner training organization. We plan to publish the detailed outputs from this larger study.
Presenter 2 : Jeff Meador, Portico
Presentation Theme
Jeff presented Portico’s Diversity and Inclusion VR Seminar, which provides learners the opportunity to practice and master their skills with inclusive language. This program was designed to complement traditional classroom learning by preparing learners with practical applications of the skills discussed during classroom sessions. With Portico’s software, learners speak directly to AI-powered digital avatars and engage in simulated conversations that put their skills to test. Learners also get real-time feedback on their performance through a virtual trainer, who uses AI to guide users towards better language choices.
Webinar Questions for Jeff Meador
How did you factor in orientation to the VR experience to overcome novelty or unfamiliarity with the technology to get straight to the experience?
We didn’t want to rely on the learner having any previous experience with VR. We give users time to look around and explore prior to beginning the scenario. This gives them a chance to understand the parameters and boundaries of the exercise. We also engage in some light chit-chat (“how’s the weather” sort of stuff) to let the learner know that they are expected to talk and that the AI is listening and will respond.
Do you need a quiet room?
A quiet room certainly helps. Current speech-to-text technology is very strong, but having a quiet place for the training is very helpful. It also helps the learner concentrate on the task at hand.
Is Portico looking at including body language (proximity to client gestures, etc.)?
Yes; we’ve done some preliminary work with this and have set some behaviors that change based on the learner’s position and/ or focal point during the training.
How is the real time interaction happening? is it a sort of multiplayer training and a mentor is interacting with you? or the AI is giving you feedback?
Everything within our scenarios is driven by AI, including the feedback. Our goal is to have these role playing experiences run without the need for actors or facilitators.
Presenter 3 : Marlo Brooke, AVATAR Partners
Marlo presented a Mixed Reality Training System for aircraft and aircraft subsystem maintenance. Primarily an Augmented Reality / Mixed Reality solution, this high-fidelity, industrial-grade maintenance training system scales from a tabletop model design, to a full-sized aircraft, allowing the maintainer to train on the job, hands free, at the point of need, without any jitter or drift. The system utilizes object tracking (as opposed to stickers or markers) to ensure highly accurate lock on target, even as the user moves around the aircraft. It also integrates to any backend system to both retrieve and record/send information about the maintenance task. It supports collaborative tasks and remote assistance.
Webinar Questions for Marlo Brooke
How did you come up with your pricing for your offerings?
Pricing is based on the specific requirement of the customer. AVATAR Partners can develop the application in its entirety, and/or the customer can purchase SimplifyXR to build/extend all or some of the functionality. Furthermore, because we use open source products such as Unity, the customer does not need to purchase SimplifyXR to extend any AVATAR Partners built XR solutions.
Is SimplifyXR proprietary to you or is it 3rd party?
SimplifyXR is is developed by AVATAR Partners using open source development tools, and can integrate to any backend system through APIs. It is licensed and sold through AVATAR Partners and its certified resellers. SimplifyXR is not required to extend any AVATAR Partner XR applications.
Could you please clarify how 97% reduction was calculated? Does it mean that instead of 100 hours of training without glasses only 3 hours with glasses are required?
You are correct. The training task was calculated based on the current training method for that particular wire failure, which involved an experienced maintainer gathering information from 3 different source materials (ETM, Wire Illuminator, and 2D schematics) to trace and identify the wire failure. The current training method is documented as taking 4 hours on average. The AR training experience is 7 minutes in duration, which is a 97% reduction in time to train.
Marlo, you said you have offices around the country. I didn’t hear if Portico has the same, but do either of your companies look to establish satellite offices in more remote locations? If so, what type of environments do you look for? Developed workforce, local colleges, etc?
AVATAR Partners has regional offices in Huntington Beach CA, Arlington (Crystal City) VA, Virginia Beach VA, and Patuxent River MD, with personnel working in 8 different states within the USA. AVATAR has teaming partners with offices in Atlanta GA, Orlando FL, San DIego CA, Dayton OH, and Fort Worth, TX. We are expanding based on a variety of factors including the locations of teaming partners, certified resellers, and universities both within the USA and Allied Nations.
Marlo, how did you measure those analytics?
The performance metrics were derived from test results by AVATAR Partners based on the comparison of the current methodology versus the use of XR. These tests included accuracy, time to train, and time to complete the task. Cost savings were derived based on an assumed reduction of errors based on increased accuracy, and will vary depending on the application and cost of parts. We also discussed other studies such as the Boeing/Iowa State Study on wing assembly (Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2014).
I’m not familiar with SimplifyXR — can you describe what that is and whether that is proprietary to AVATAR?
SimplifyXR is a software product developed by AVATAR Partners that allows non-programmers to develop their own Augmented Reality, Mixed Reality, and Virtual Reality experiences. It also speeds development time by tenfold by using a user-friendly interface. The product also evaluates the user’s desired XR application and notifies the user whether or not the application is well suited for XR, or if a different training media (e.g. video training) would be more effective. SimplifyXR is owned and licensed by AVATAR Partners, and is developed using open-source tools and industry standards, and also integrates to any backend system using APIs. The product is available directly through AVATAR Partners or certified resellers.
Was there a problem using AR on an object as large as an aircraft?
We overcame several challenges to make the solution commercially usable. Using the basic development tools and markers didn’t work – it caused significant drift, jitter and an inability to accurately track the object as the maintainer walked around the aircraft. AVATAR Partners was able to overcome all of these issues by developing algorithms to lock on target without any drift or jitter to the user. Our solution being hardware agnostic, we found the Microsoft Hololens and android tablet supported higher fidelity. Also, we used Mantis Vision 3D scanners to scan the aircraft, gaining an accurate 3D model so that we could use object tracking (instead of markers). This along with our programming algorithms provides the maintainer with absolute precision as they walk around the aircraft, regardless of the unit’s size (from 1:6 the model size, to full aircraft).
Presenter 4 : Carlos Ochoa, ONE digital consulting
Presentation Theme
Carlos presented XR platform for immersive training, operation and maintenance of industry plant services, eolic farms, which includes advanced reality technologies in 3d interactive environments (extended reality, 360º video and digital data integration), methodologies in a modular architecture (on/off line). Innvestion deploy an intelligent XR ecosystem with the specialized know-how to turn unstructured data into value-adding information, for specific subject when providing services such as preventive maintenance, security management or operational management. Simulation-based training, prefaced with traditional training methods on process fundamentals, is the most efficient and fastest way to train new employees and prepare them for their daily work providing huge added value and benefits.
ONE digital consulting has been working for more than 20 years in the eLearning and Training arena, and with our selected partners network in Europe, Saudi, India, Africa and Latam.
Webinar Questions for Carlos Ochoa
What hardware configurations are everyone on the panel currently using for their XR applications?
For a full INNVESTION experience, the user only needs a mobile device and VR glasses that fit the device. The training centre will need an internet connection to download different modules.
INNVESTION can be used in a single “low cost” VR environment based on Samsung Gear Mobile VR, through the most advanced VR Oculus Rift, HTC Vive configuration. The key issue of INNVESTION is data integration in one device. From reality to virtual reality or vice versa.
INNVESTION can be used in a multi window spheric environment, for multiuser training environments.
What are KPI’s?
The acronym KPI sounds for Key Performance Indicator. This is a general business term referring to any metric used in understanding how an organization is doing. While the term can be tossed around the business world ostentatiously or irrelevantly, the fundamental rationale derives from the simple concept that if you don’t know how you did in the past, you don’t know if you have improved in the future. Because every business has its own economic niche, customer acquisition models, geographic localities, production methods, etc., each business must define for themselves what makes them successful. Therefore, the “key” in KPI, should be a metric key to your enterprise. These indicators are as varied as your imagination, but a number of familiar indicators include monthly recurring revenue (MRR), cost to acquire a customer (CAC) and the term used in this webinar’s title, Return on Investment (ROI).
When you are looking to improve your ROI in your training strategy, you need to understand clearly the training savings metrics, saved commute hours, lifelong expense of your LMS solution, overall productivity and revenue increase, other factors specific to your business and training model.
In conclusion, we can see selected companies transforming their business opportunities for the XR technology and adopt it regardless of cost and ‘clunkiness.’ Today we see leader enterprises utilizing XR for training, maintenance, engineering, and marketing purposes. These companies eventually pull everyone else in their direction as the industry caters to their operational needs and chases enterprise benefits.
Carlos J. Ochoa Fernandez 