Mixed Reality (MR) technology entered the consumer market in 2016 with the commercial availability of head-mounted MR devices. Since then, it has attracted increasing interest in the architecture and construction industry due to its ability of overlaying digital building models in real construction sites. This unique ability allows    building    owners,   architects,   engineers,   and contractors to analyze proposed Building Information Modeling (BIM) models right in front of their eyes through on-site architectural visualization, construction planning, and as-built verification, which brings unparalleled value in decision-making over existing technologies such as computer screens, mobile devices, or even Virtual Reality (VR) headsets. During the early industry adoption  of  MR technology, a handful of design firms and contractors performed pilot tests with limited application uses, and only sparse preliminary reviews and demonstrations have been found. There were, however, no comprehensive reviews in existing literature reporting the use of MR technology in architecture and construction practice, and this gap in the body of knowledge has yet to be filled.

This paper presents the results of evaluating the use of current MR technology in architectural design and construction layout through a comparative review of commercially available MR applications. The study employed two HoloLens devices that were attachedto a hard hat to meet construction safety regulations. Following a literature review and Internet search, nine MR applications developed for architectural and construction purposes were identified and installed on the two HoloLensfor field testing. The capabilities of MR technology within these applications were evaluated and compared under different on-site conditions for architectural design and construction layout. The evaluated MR capabilities includedmodel formats and file access methods, user interface, different view modes, various BIM tools, collaboration features, etc. In addition, the limitations of MR technology and the head-mounted MR device for on-site uses were discussed. This paper provides timely and empirical information to help practitioners and researchers better understand MR technology for the architecture and construction industry.

BACKGROUND

MR is an emerging technology that is capable of blending physical objects with digital content in an interactive and real-time fashion (Almagor, 2016). A head-mounted MR device can present virtual 3D objects on see-through displays, allowing observations of the virtual world co-existing in the real world right in front of the user’s eyes, which makes it substantially different than the VR technology. An MR device constantly tracks the surroundings with multiple built-in sensors to determine the user’s current position and provide real-time interaction between virtual objects and the physical environment. In addition, an MR device receives input commands primarily through hand gestures and voice recognition, as opposed to the controller inputs required by VR headsets. The first commercially available MR devices include Microsoft HoloLens and Magic Leap One. Both are self-contained computer headsets that communicate with other devices wirelessly. The HoloLens runs on a Windows 10 platform while Magic Leap One operates on a Linux-based system.

Due to its ability of overlaying BIM models in real construction sites, the architecture and construction industry soon started to explore the uses of MR technology   since    the   first   MR   devices   were made commercially   available.    The    ability    of     visualizing architectural designs and construction layouts directly on-site has the potential to significantly improve the efficiency in decision-making compared with using existing technologies such as computer screens, mobile devices, or VR headsets. To bring an MR device to construction jobsite environments as a headset, safety regulations have to be complied with. Several solutions have been developed to replace the safety glasses with the HoloLens and attach it to a construction hard hat for on-site uses, as demonstrated in Figure 1. The Trimble hard hat solution for HoloLens allows the MR device to be used as certified personal protective equipment (PPE) for eye, safety, and electrical protection in construction activities (MCAA, 2018; Trimble, 2018). Due to adverse construction environments, the EBMR helmet also integrates a surveying GPS in the HoloLens hard hat solution to increase the accuracy of position tracking on-site (Xsens, 2018).

Early research efforts have also started to focus on the implementation of MR technology in architectural and industrial design as well as construction. Alsafouri and Ayer (2017) first designed a methodology for generating marker-based MR environments for various mobile computing devices, such as smart phones and tablets, to enable design and constructability review using existing BIM contents. Chalhoub and Ayer (2017) then investigated the perception of field workers in using a BIM model with HoloLens as the construction documents to assemble electrical conduits compared with traditional paper-based communication in 2D drawings. The study found that although the BIM model viewed in MR was easier to use than paper plans, it should only supplement but not replace 2D drawings. Moreover, the reliability of BIM models observed in MR was also a concern. Chalhoub and Ayer (2018) further examined the impact of MR technology on construction performance in terms of productivity and quality based on the same electrical conduit assembly and discovered that while most practitioners still preferred to use paper plans for design communication, using BIM models in MR significantly increased productivity rate, lowered the number of installation errors, and reduced the time needed to understand the design during the electrical conduit assembly process. Most recently, Chalhoub and Ayer (2019) evaluated the effects of using MR technology on the accuracy and performance of a specific construction task, electrical point layout, and concluded that the HoloLens was able to display digital contents with a locational accuracy of five centimeters from intended design both horizontally and vertically, and the accuracy decreased at a further viewing distance following a parabolic curve.

At   the  same  time, the use of MR technology has also been investigated in other areas of design and construction, such as tunneling, underground utilities, site surveying, human behaviors, and maintainability. Zhou et al. (2017) investigated the use of MR technology in inspecting segment displacement during tunneling construction by overlaying a baseline model observed in MR onto the real segment displacement for quality control purposes. The study compared the prototype MR system with a traditional method and concluded that MR technology allowed the inspections and analyses to be conducted on-site, in real-time, and at a very low cost. Fenais et al. (2018) analyzed the challenges of using MR technology in underground utility construction by synthesizing over 500 publications and summarized them as (1) data collection issues, (2) modeling alignment barriers, 3) hardware limitations, and 4) data storing and managing challenges. Chalhoub et al. (2018) presented a workflow that enabled full-size BIM models to be displayed in MR at their correct positions by leveraging existing surveying points found on-site. The study utilized markers located at existing surveying points to align BIM models to their correct on-site positions and validated the workflow for quality assurance and quality control processes. Alsafouri and Ayer (2019) investigated the impact of using different MR devices on various human behaviors while viewing the same MR contents. The study included different sizes of smartphones, tablets, and smart glasses and suggested that while there were no differences for alternative design formation, navigation of design, and problem-solving between different MR devices, users did not perform explanative, decision-making, and discussions with team members when they were allowed to select a preferred MR device. Khalek et al. (2019) examined the capabilities of MR technology in enabling design professionals to identify maintainability concerns in the design stage. The study compared student perceptions and  behaviours  of  using  both  MR technology and the traditional computer BIM models, and the indicated that while the traditional computer BIM models supported better identification of potentially problematic areas, MR technology allowed users to more consistently determine why an area was problematic.

To evaluate MR technology for architectural design and construction layout, this study employed two HoloLensand applied the Trimble hard hat solution to meet construction safety regulations, as shown in Figure 2. After being attached to a hard hat, the HoloLens itself does not make any contact with the user’s head or face, and the user wears the HoloLens hard hat the same way as wearing a regular hard hat. In addition, the hard hat solution distributes the weight of HoloLensover the user’s head more evenly with the hard hat suspension and liner, making it more comfortable to wear for longer durations.

To evaluate MR technology for architectural and construction uses at different project stages, various BIM models and site conditions were used in this study. Figure 3demonstrates an architectural design of a glass panel wall observed in MR in an interior renovation project as well as a construction layout of interior partition walls observed in MR in a new construction project. Due to the information needed for decision-making at different project stages, the number of visible objects in the BIM models was adjusted significantly to ensure a clear and unobstructed view at different site conditions, as compared in Figure 3.

Following a literature review and Internet search, nine commercially available MR applications developed for architectural and construction purposes were identified and installed on the two HoloLens for field testing under different on-site conditions to evaluate MR technology. The developer and application namesof each identified MR applications are listed in Table 1. The nine MR applications were grouped into three categories based on their primary purpose, namely basic 3D viewing, BIM use, and presentation purpose. 3D Viewer Beta is a free application provided by Microsoft to simply view any 3D models in MR. SketchUp Viewer and Trimble Connect are both developed by Trimble specifically for BIM uses. Similar MR applications also include BIM Holoview, MR Builder   Lite  by Holo Group, HoloLive 3D by VisualLive, and Fuzor AR by Kalloc. While Vyzn by Zengalt and Prism by Object Theory are developed for the specific purpose of MR presentation, they can still be applied for architecture and constructionuses (Table 1).

Although all nine MR applications were installed on the two HoloLens, only seven were tested in this study due to not being able to acquire a valid license for HoloLive 3D and Fuzor AR. As a result, the evaluation of HoloLive 3D and Fuzor AR was made solely based on online documentation and video demonstrations. In addition, while the same BIM modelswereattempted to be used to evaluate all MR applications, MR Builder Lite was not able to open any user uploaded models, and thus it was evaluated alternatively with built-in sample models. Vyzn and Prism, as MR presentation applications, also provided sample models for presentation demonstrations.

It was found that MR technology offers a variety of capabilities   among   the   evaluated   applications.  While some are features that offer basic information and navigation, several are unique but essential to provide an effective and successful on-site experience for architectural and construction uses. Therefore, the results of evaluating MR technology through the identified applications were grouped into two following categories.

Common and basic capabilities

Model format and access

MR technology supports the most widely used 3D formats in the architecture and construction industry. As listed in Table 2, five MR applications usefbx files as their native file format, which can be exported into from either a BIM program, including Autodesk Revit, Trimble SketchUp, and Autodesk Navisworks, or  a  3D  modeling program, such as Autodesk 3ds Max and Maya. SketchUp Viewer and Trimble Connect use SketchUp models as the native file format, andTrimble Connect additionally allows Revit model to be uploaded through an add-in. Fuzor AR only supports native files created by the Fuzor computer program, which is also able to import models from various BIM and 3D modeling programs. Prism, as an exception, is a demonstration MR application developed from 3D models directly instead of being able to load models.

The MR technology accesses    3D models from the HoloLens through various approaches. While 3D Viewer, SketchUp Viewer, MR Builder, and Fuzor AR can directly open 3D models stored locally in the MR device, Trimble Connect, BIM Holoview, HoloLive 3D, and Vyzn need an application login before accessing 3D models stored in the user’s account, which requires a wireless Internet connection. Additionally, 3D Viewer and MR Builder can open models stored in OneDrive while Fuzor AR can access models stored in Dropbox. When transferring BIM models from a computer to the MR applications in the MR device, in addition to application accounts and online storage services, SketchUp Viewer, HoloLive 3D, and Fuzor AR can push the models from the computer program directly to their MR application, which is quite convenient to immediately view the models in MR. Furthermore, SketchUp Viewer can access Trimble’s 3D Warehouse and download any 3D model shared by the SketchUp community. Prism, a gain  as  an exception, stores and transfers models within the MR application itself.

Interactive user interface

The MR technology provides various interactive features in the user interface to provide a better experience for a different environment, namely the ability to reset menu, scale menu, and respond to voice commands, as listed in Table 3. When using an MR device in a large and complex space, it is very common to move away from where the application menu was originally located and lose track of it. The reset menu feature, provided by MR Builder and Fuzor AR as a voice command, immediately brings the menu back right in front of the user’s sight. SketchUp Viewer allows the user to freely scale the menu size based on viewing distance when the menu is pinned in a stationary position. In addition, most MR applications support voice commands for partial or full functions, although the user needs to remember the exact voice command for each function.

Tabletop view mode

The MR technology offers various navigation tools to inspect the BIM models when they are observed at a tabletop or miniature scale, including moving, rotating, and scaling, as listed in Table 4. The most typical movement control of a BIM model in MR is to pick it up then drop at a new   position,   or   to  drag  it.  There  is  also  an alternative movement control by clicking on a present directional increment, which is however less intuitive andconvenient than the dragging method since it is difficult to reach the exact location where the model needs to be placed. Model rotation uses similar control methods as movement while dragging or directional increments change the rotation angle. HoloLive 3D, however, does not allow the model to be moved or rotated, while Prism does not allow rotation. Dragging can also be applied for scale control, but a more accurate approach is to apply either preset or custom scale. Dragging allows real-time observation of model size change but without accurate control, while a preset or custom scale allows the model to be viewed at a defined scale to its true size, such as 1:5 or 1:20. SketchUp Viewer and Trimble Connect support all three scaling methods while Vyzn and Prism do not support any model scaling. Lastly, a few MR applications provide a model reset tool that allows the model to be quickly moved in front of the user at the default scale for a new placement.

Basic BIM tools

MR technology provides a variety of utility tools that are specifically designed for architectural and construction uses. These utility tools are commonly found in BIM programs for computer and VR devices, including layer control, measurement and unit, as well as annotation, as detailed in Table 5. Typical layer control by SketchUp Viewer and Fuzor AR allows each layer or category of building components to be turned on or off individually if they are defined in the BIM model, which helps isolate key components and hide unneeded objects when viewing the model in MR. Alternatively, BIM Holoview divides a model into an exterior layer and an interior layer and controls their visibility as well as transparency and color separately. MR Builder additionally highlights selected layers with glowing effects to enhance visualization.

Measurement and unit tools allow the user to take distance measurements of BIM models with the MR device. Depending on the objects to be measured from, distance measurements can be taken either within the digital model, within the physical space, or between the digital model and the physical space. Trimble Connect is the only MR application that offers all three measurement types. While not all applications are able to measure distances, most displays either a metric or imperial unit for movement or scaling purposes, and some also support unit conversion between the two. Model annotation tools in MR include text or sketch markup, snapshot of the current view, as well as property information of the selected objects. These annotation tools assist the inspection of the model in MR by identifying the  problematic   areas   and   locating   them    during revisits.

Unique and essential capabilities

Immersive view mode

The MR technology enables an unobstructed immersive view mode that allows observation of BIM models at their full-size scale in a construction jobsite, which is unarguably the most essential toolset for architectural and construction uses. Although 3D Viewer is the only MR application not providing the immersive view as a native view mode, it can still be achieved by scaling a model to a 1:1 scale. At such scale, the user sees the BIM model at its true size and can physically walk around to inspect the model in MR when it is placed on-site or in a space large enough to accommodate the full-size model. Alternatively, as summarized in Table 6, the user can choose to virtually walk or teleport within the model when it is not aligned on-site, a navigation method used by all VR devices. In such cases, the user stays physically stationary while the selected destination point in the model moves right in front of the user, consequently moving the entire model to a new position. A  third  navigation method in the immersive view is to virtually elevate the user directly to different levels byeither raising or lowering the model. In addition, SketchUp Viewer and Trimble Connect also allow user height adjustment to accommodate various audiences for a better immersive view.

One essential capability to use MR technology on-site for architectural design and construction layout is the full-scale adjustment. Under the immersive view mode at 1:1 scale, the BIM models are expected to match their true size on-site, but discrepancies exist at a small percentage between 1-2% so that the model looks slightly smaller when overlaid directly on its physical counterpart, and the discrepancy varies between each MR device. This variance in size could be negligible for small 3D objects, but for BIM models it can easily exceed tolerances. Full-scale adjustment allows the user to manually compensate for the discrepancy when overlaying and comparing a BIM model on-site to its physical counterpart. SketchUp Viewer and Trimble Connect are the only two MR applications supporting full-scale adjustment.

Another capability of MR technology that can significantly improve the efficiency of its on-site use for architectural design and construction layout  is  automatic  model  alignment.  Under the immersive view mode at 1:1scale, the first thing to do is to ensure the alignment of the BIM model to the physical site before any review or inspection can happen. Manual alignment takes a considerable amount of time and effort, and often results in inaccurate alignment due to the observation-based approach and leads to great frustration due to the difficulty in dragging a full-size model. Automatic model alignment allows a full-size model to be immediately placed at its correct position on-site after registering the required reference targets.

Five MR applications offer automatic model alignment through different methods, including two-point registration, one-point registration, and QR code scan. Each application applies the two-point registration approach differently, as detailed in Table 7. Trimble Connect asks the user to identify two walls and the floor in the model and their counterparts in the physical site. BIM Holoview requires the user to insert a provided model component with two predefined points in the BIM model, and then register these two points in the physical space. MR Builder allows the user to choose any two points in the model and register their counterparts in the physical space, as demonstrated in Figure 4. HoloLive 3D requires the user to register one point on the floor and then identify the  second  pointing  the  north  direction. Fuzor AR uses a similar approach except that the second point on the floor could be pointing any of the four directions. Fuzor AR also uses a one-point registration method where only one point needs to be registered on a wall assuming all walls are plumb. The QR code alignment method requires a QR code label model component to be inserted in the BIM model first, and then the same QR code label needs to be printed and affixed to the same location in the physical space, either a wall or floor. After scanning the affixed QR code label with the MR device, the model can be immediately aligned. Lastly, a few MR applications allow pivot points to be selected for fine adjustment of movement and rotation if needed.

Multiuser collaboration

The MR technology provides the opportunity for multiple MR device users to inspect the same BIM model together through a collaborative session, which is an essential part of decision-making for a project team between the building owners, architects, engineers, and contractors. Abasic multiuser collaboration session starts with a session host whose MR device will upload the BIM  model  to  the  sharing  service.  When  other participants join the session, the model will be downloaded from the sharing service to their MR devices for viewing. In such case, all session participants can inspect the same model together, but they do not know where each other is located relative to the model position. In order to share this information, a co-located collaboration session is required where all participants will specify a common reference point in the same physical space to create a shared coordinate system. Once the shared coordinate system is established across all MR devices in the session, each participant will be represented as an avatar to indicate their position relative to the model, and their focus point will be presented with a laser beam. Figure 5demonstrates the comparison between a basic collaboration session in Trimble Connect and a co-located collaboration session in SketchUp Viewer. Both snapshots were captured by a third MR device in the same session.Table 8 summarizes the multiuser collaboration capabilities of MR technology.

BIM enhancement tools

In addition to the basic BIM tools, MR technology provides other tools to enhance design visualization  and  model   interaction  to   facilitate decision-making in architectural design and construction layout. These BIM enhancement tools can either animate constructionsequence or schedule, switch between different preset scenarios, or change the design by adding or relocating model components, as shown in Table 9. 3D Viewer and Vyzn support object animation such as flames, waves, or wind effects, while SketchUp Viewer allows automatic rotation of the model. SketchUp Viewer and Fuzor AR can also present construction schedules when different views are capturedin a logical sequence. As mainly presentation applications, Vyzn and Prism allow the model to contain different scenario views, such as day views vs. night views, architectural viewsvs structural views, or even between different models. In addition, MR Builder and Vyzn allow real-time design changes in MR by adding or relocating predefined building components to explore alternative designs and layouts.    Overall,     these     BIM    enhancement    tools complement the basic tools such as measurement and annotation to better utilize the potential of MR technology in architectural design and construction layout.

Through the field testing of the various MR applications in this study, several limitations of MR technology and the MR device in on-site uses have been identified, including the model drifting issue, automatic alignment issue, and co-located collaboration issue. These issues fall within the essential capabilities of MR technology and are expected to improve and hopefully be eliminated over the development of MR technology and devices.

The model drifting issue can be easily observed in the immersive view mode when the BIM model has been aligned   with   its   physical   counterpart.    Under    such conditions, when the user wearing the MR device moves away from the original position where the model has been aligned from, the model also drifts towards the same direction at a small rate instead of holding stationary. The drifting distance is proportional to the distance that the user has traveled, which consequently causes model misalignment. To compensate for this misalignment during the user’s movement, manual realignment is needed depending on the distance traveled.

The automatic alignment issue exists due to the fact that during either the two-point registration or one-point registration approach, the locations of the registration points in the BIM model must exactly match their physical counterparts. These physical locations are selected manually with hand gestures by observing a focus point through the MR device. With the device being mounted on the user’s head, this focus point can hardly stay still, always resulting in an inaccurate selection of the registration point location. Further fine adjustments through pivot points are therefore needed for a better model alignment.

The co-located collaboration issue happens more often under complicated on-site conditions where many building components are exposed and building materials are stored disorderly. Under such conditions, it is difficult for the MR devices to map the same physical space and recognize a common reference point on-site in order to establish the required shared coordinate system. As a result, an expected co-located collaboration session will often present the wrong locations for the participants’avatars.

To better overcome these issues that can happen on-site and hinder the successful use of MR   technology   for   architectural   design    and construction layout, one alleviation is to allow enough time for the MR devices to fully scan and map the entire physical space before using any BIM models on-site. These current issues will hopefully be eliminated with the further development of MR technology, such as continuous model alignment and automatic recognition of multiple QR codes.

This study was sponsored by the Thomas Glavinich ELECTRI International 2018 Early Career Award. The author would like to thank ELECTRI International and Thompson Electric Company for their support.

The author has not declared any conflict of interests.