Daytime Project

Augmented Reality Maintenance Guide

Complex machines deployed in the field require maintenance at some point in time, whether it is a planned service check-up or an emergency repair because the machine stopped working. While service engineers dispatched to perform the maintenance are trained to service the machine, they might now always be fully aware of all the different versions of machines deployed, and how they exactly differ. Even experienced engineers might need a manual to perform service. Furthermore, some more complex machines might require a type of maintenance that a specific engineer is not an expert at. For example, an electrical engineer dispatched to service an electrical cooling system might be required to perform work on the pipes and valves of the cooling system as well.

The machine can often not be used before maintenance has been performed. This down-time can be expensive for a user. Therefore, the maintenance process should be as efficient and fast as possible and mistakes leading to longer down-times should be avoided. To make maintenance time less dependent on the skill level of the dispatched engineer, PS-Tech developed an Augmented Reality (AR) app that guides engineers through the steps of performing specific maintenance tasks on specific machines.

To keep the cost of deployment of the AR app low, the app runs on a service engineer’s phone. Upon starting the app, the engineer is presented with the camera view from the phone and the outline of a specific part of the machine to be serviced. This specific, easily recognizable part is used as an initialization point. When the phone camera is pointed at that part, the AR app recognizes the machine and registers the position of the machine in the real world (Figure 1, top left). The app will indicate the estimated position of the machine by putting a virtual green boundary around it (Figure 1, right). This boundary can be used as a reference at any moment to check that the app still estimates the machine position correctly.


Figure 1 (top left). AR app detects part of the machine. (right) After machine detection, the AR app highlights the machine by adding a virtual green boundary. (bottom left) Green spheres highlight the location of the components of interest

The engineer now gets presented with a step-by-step maintenance guide that can be navigated by pressing the screen repeatedly. With each press on the screen, the next step of the guide will be explained in text and the relevant part of the machine will be highlighted in AR. When the camera is not pointed at the part in question, an arrow will appear on screen pointing in the direction the user should move their phone to get it into view. This reduces the time the engineer needs to locate specific parts of the machine. Once the part is in view, it will be highlighted by a green sphere and its name will be shown next to it (Figure 1, bottom left). The engineer can freely move their phone to get a better view of the part while keeping it highlighted.

The engineer can now follow the steps from the guide one by one and perform the maintenance. When status information is provided by the machine, it is possible to show more information on the status of specific parts or display a historic sensor log. This will be shown as an information bubble hovering next to that part.

By making use of the functionalities present on the engineer’s smartphone, it is also possible to call in assistance from an expert engineer off-site. By using a generic communication app like MS Teams or Zoom, an engineer can communicate with an off-site expert while sharing their screen and showing what they are seeing in AR (Figure 2). This simplifies calling in remote assistance and facilitates fast problem-solving.


Figure 2. An external expert can be called in using a communication app, seeing the AR view from the engineer and offering assistance.


Contact Information: