Evaluating mixed reality technology for architectural design and construction layout

Mixed reality (MR) technology has attracted increasing interest in the architecture and construction industry since the commercial availability of head-mounted MR devices in 2016. With its ability of overlaying digital building models in real construction sites, building owners, architects, engineers, and contractors have quickly recognized the value of MR technology in on-site architectural visualization, construction planning, and as-built verification. Although a few industry practitioners have piloted some initial tests, no thorough reviews on such uses have been reported in the existing literature. To bridge this gap in the body of knowledge, this study aims to evaluate the use of MR technology in architectural design and construction layout through a comparative review of commercially available MR applications. The study employed two HoloLens devices that were attached to a hard hat to meet construction safety regulations and identified nine MR applications developed for architectural and construction purposes. The capabilities of MR technology within these applications were evaluated and compared under different on-site conditions for architectural design and construction layout. It was found that among the various MR capabilities, while some are common features that offer basic information and navigation, several are unique but essential to provide an effective and successful onsite experience for architectural and construction uses. In addition, the limitations of MR technology and the head-mounted MR device for on-site uses were discussed. This paper provides timely and empirical information to help practitioners and researchers better understand MR technology for the architecture and construction industry.


INTRODUCTION
Mixed Reality (MR) technology entered the consumer market in 2016 with the commercial availability of headmounted MR devices. Since then, it has attracted increasing interest in the architecture and construction industry due to its ability of overlaying digital building models in real construction sites. This unique ability allows building owners, architects, engineers, and contractors to analyze proposed Building Information Modeling (BIM) models right in front of their eyes through on-site architectural visualization, construction planning, and as-built verification, which brings unparalleled value in decision-making over existing technologies such as computer screens, mobile devices, or even Virtual Reality (VR) headsets. During the early industry adoption of MR E-mail: yilei.huang@sdstate.edu Tel: +1 (605) 688-4160.
Author(s) agree that this article remain permanently open access under the terms of the Creative Commons Attribution License 4.0 International License technology, a handful of design firms and contractors performed pilot tests with limited application uses, and only sparse preliminary reviews and demonstrations have been found. There were, however, no comprehensive reviews in existing literature reporting the use of MR technology in architecture and construction practice, and this gap in the body of knowledge has yet to be filled. This paper presents the results of evaluating the use of current MR technology in architectural design and construction layout through a comparative review of commercially available MR applications. The study employed two HoloLens devices that were attached to a hard hat to meet construction safety regulations. Following a literature review and Internet search, nine MR applications developed for architectural and construction purposes were identified and installed on the two HoloLensfor field testing. The capabilities of MR technology within these applications were evaluated and compared under different on-site conditions for architectural design and construction layout. The evaluated MR capabilities included model formats and file access methods, user interface, different view modes, various BIM tools, collaboration features, etc. In addition, the limitations of MR technology and the head-mounted MR device for on-site uses were discussed. This paper provides timely and empirical information to help practitioners and researchers better understand MR technology for the architecture and construction industry.

BACKGROUND
MR is an emerging technology that is capable of blending physical objects with digital content in an interactive and real-time fashion (Almagor, 2016). A head-mounted MR device can present virtual 3D objects on see-through displays, allowing observations of the virtual world coexisting in the real world right in front of the user's eyes, which makes it substantially different than the VR technology. An MR device constantly tracks the surroundings with multiple built-in sensors to determine the user's current position and provide real-time interaction between virtual objects and the physical environment. In addition, an MR device receives input commands primarily through hand gestures and voice recognition, as opposed to the controller inputs required by VR headsets. The first commercially available MR devices include Microsoft HoloLens and Magic Leap One. Both are self-contained computer headsets that communicate with other devices wirelessly. The HoloLens runs on a Windows 10 platform while Magic Leap One operates on a Linux-based system.
Due to its ability of overlaying BIM models in real construction sites, the architecture and construction industry soon started to explore the uses of MR technology since the first MR devices were made commercially available. The ability of visualizing architectural designs and construction layouts directly on-site has the potential to significantly improve the efficiency in decision-making compared with using existing technologies such as computer screens, mobile devices, or VR headsets. To bring an MR device to construction jobsite environments as a headset, safety regulations have to be complied with. Several solutions have been developed to replace the safety glasses with the HoloLens and attach it to a construction hard hat for on-site uses, as demonstrated in Figure 1. The Trimble hard hat solution for HoloLens allows the MR device to be used as certified personal protective equipment (PPE) for eye, safety, and electrical protection in construction activities (MCAA, 2018;Trimble, 2018). Due to adverse construction environments, the EBMR helmet also integrates a surveying GPS in the HoloLens hard hat solution to increase the accuracy of position tracking onsite (Xsens, 2018).

LITERATURE REVIEW
Early research efforts have also started to focus on the implementation of MR technology in architectural and industrial design as well as construction. Alsafouri and Ayer (2017) first designed a methodology for generating marker-based MR environments for various mobile computing devices, such as smart phones and tablets, to enable design and constructability review using existing BIM contents. Chalhoub and Ayer (2017) then investigated the perception of field workers in using a BIM model with HoloLens as the construction documents to assemble electrical conduits compared with traditional paper-based communication in 2D drawings. The study found that although the BIM model viewed in MR was easier to use than paper plans, it should only supplement but not replace 2D drawings. Moreover, the reliability of BIM models observed in MR was also a concern. Chalhoub and Ayer (2018) further examined the impact of MR technology on construction performance in terms of productivity and quality based on the same electrical conduit assembly and discovered that while most practitioners still preferred to use paper plans for design communication, using BIM models in MR significantly increased productivity rate, lowered the number of installation errors, and reduced the time needed to understand the design during the electrical conduit assembly process. Most recently, Chalhoub and Ayer (2019) evaluated the effects of using MR technology on the accuracy and performance of a specific construction task, electrical point layout, and concluded that the HoloLens was able to display digital contents with a locational accuracy of five centimeters from intended design both horizontally and vertically, and the accuracy decreased at a further viewing distance following a parabolic curve.
At the same time, the use of MR technology has also been investigated in other areas of design and construction, such as tunneling, underground utilities, site surveying, human behaviors, and maintainability. Zhou et al. (2017) investigated the use of MR technology in inspecting segment displacement during tunneling construction by overlaying a baseline model observed in MR onto the real segment displacement for quality control purposes. The study compared the prototype MR system with a traditional method and concluded that MR technology allowed the inspections and analyses to be conducted on-site, in real-time, and at a very low cost. Fenais et al. (2018) analyzed the challenges of using MR technology in underground utility construction by synthesizing over 500 publications and summarized them as (1) data collection issues, (2) modeling alignment barriers, 3) hardware limitations, and 4) data storing and managing challenges.  presented a workflow that enabled full-size BIM models to be displayed in MR at their correct positions by leveraging existing surveying points found on-site. The study utilized markers located at existing surveying points to align BIM models to their correct on-site positions and validated the workflow for quality assurance and quality control processes. Alsafouri and Ayer (2019) investigated the impact of using different MR devices on various human behaviors while viewing the same MR contents. The study included different sizes of smartphones, tablets, and smart glasses and suggested that while there were no differences for alternative design formation, navigation of design, and problem-solving between different MR devices, users did not perform explanative, decisionmaking, and discussions with team members when they were allowed to select a preferred MR device. Khalek et al. (2019) examined the capabilities of MR technology in enabling design professionals to identify maintainability concerns in the design stage. The study compared student perceptions and behaviours of using both MR technology and the traditional computer BIM models, and the indicated that while the traditional computer BIM models supported better identification of potentially problematic areas, MR technology allowed users to more consistently determine why an area was problematic.

METHODOLOGY
To evaluate MR technology for architectural design and construction layout, this study employed two HoloLensand applied the Trimble hard hat solution to meet construction safety regulations, as shown in Figure 2. After being attached to a hard hat, the HoloLens itself does not make any contact with the user's head or face, and the user wears the HoloLens hard hat the same way as wearing a regular hard hat. In addition, the hard hat solution distributes the weight of HoloLensover the user's head more evenly with the hard hat suspension and liner, making it more comfortable to wear for longer durations.
To evaluate MR technology for architectural and construction uses at different project stages, various BIM models and site conditions were used in this study. Figure 3 demonstrates an architectural design of a glass panel wall observed in MR in an interior renovation project as well as a construction layout of interior partition walls observed in MR in a new construction project. Due to the information needed for decision-making at different project stages, the number of visible objects in the BIM models was adjusted significantly to ensure a clear and unobstructed view at different site conditions, as compared in Figure 3.
Following a literature review and Internet search, nine commercially available MR applications developed for architectural and construction purposes were identified and installed on the two HoloLens for field testing under different on-site conditions to evaluate MR technology. The developer and application names of each identified MR applications are listed in Table 1. The nine MR applications were grouped into three categories based on their primary purpose, namely basic 3D viewing, BIM use, and presentation purpose. 3D Viewer Beta is a free application provided by Microsoft to simply view any 3D models in MR. SketchUp Viewer and Trimble Connect are both developed by Trimble specifically for BIM uses. Similar MR applications also include BIM Holoview, MR Builder Lite by Holo Group, HoloLive 3D by VisualLive, and Fuzor  AR by Kalloc. While Vyzn by Zengalt and Prism by Object Theory are developed for the specific purpose of MR presentation, they can still be applied for architecture and construction uses (Table 1).
Although all nine MR applications were installed on the two HoloLens, only seven were tested in this study due to not being able to acquire a valid license for HoloLive 3D and Fuzor AR. As a result, the evaluation of HoloLive 3D and Fuzor AR was made solely based on online documentation and video demonstrations. In addition, while the same BIM models were attempted to be used to evaluate all MR applications, MR Builder Lite was not able to open any user uploaded models, and thus it was evaluated alternatively with built-in sample models. Vyzn and Prism, as MR presentation applications, also provided sample models for presentation demonstrations.

RESULTS
It was found that MR technology offers a variety of capabilities among the evaluated applications. While some are features that offer basic information and navigation, several are unique but essential to provide an effective and successful on-site experience for architectural and construction uses. Therefore, the results of evaluating MR technology through the identified applications were grouped into two following categories.

Model format and access
MR technology supports the most widely used 3D formats in the architecture and construction industry. As listed in Table 2, five MR applications use fbx files as their native file format, which can be exported into from either a BIM program, including Autodesk Revit, Trimble SketchUp, and Autodesk Navisworks, or a 3D modeling

Interactive user interface
The MR technology provides various interactive features in the user interface to provide a better experience for a different environment, namely the ability to reset menu, scale menu, and respond to voice commands, as listed in Table 3. When using an MR device in a large and complex space, it is very common to move away from where the application menu was originally located and lose track of it. The reset menu feature, provided by MR Builder and Fuzor AR as a voice command, immediately brings the menu back right in front of the user's sight. SketchUp Viewer allows the user to freely scale the menu size based on viewing distance when the menu is pinned in a stationary position. In addition, most MR applications support voice commands for partial or full functions, although the user needs to remember the exact voice command for each function.

Tabletop view mode
The MR technology offers various navigation tools to inspect the BIM models when they are observed at a tabletop or miniature scale, including moving, rotating, and scaling, as listed in Table 4. The most typical movement control of a BIM model in MR is to pick it up then drop at a new position, or to drag it. There is also an alternative movement control by clicking on a present directional increment, which is however less intuitive andconvenient than the dragging method since it is difficult to reach the exact location where the model needs to be placed. Model rotation uses similar control methods as movement while dragging or directional increments change the rotation angle. HoloLive 3D, however, does not allow the model to be moved or rotated, while Prism does not allow rotation. Dragging can also be applied for scale control, but a more accurate approach is to apply either preset or custom scale. Dragging allows real-time observation of model size change but without accurate control, while a preset or custom scale allows the model to be viewed at a defined scale to its true size, such as 1:5 or 1:20. SketchUp Viewer and Trimble Connect support all three scaling methods while Vyzn and Prism do not support any model scaling. Lastly, a few MR applications provide a model reset tool that allows the model to be quickly moved in front of the user at the default scale for a new placement.

Basic BIM tools
MR technology provides a variety of utility tools that are specifically designed for architectural and construction uses. These utility tools are commonly found in BIM programs for computer and VR devices, including layer control, measurement and unit, as well as annotation, as detailed in Table 5. Typical layer control by SketchUp Viewer and Fuzor AR allows each layer or category of building components to be turned on or off individually if they are defined in the BIM model, which helps isolate key components and hide unneeded objects when viewing the model in MR. Alternatively, BIM Holoview divides a model into an exterior layer and an interior layer and controls their visibility as well as transparency and color separately. MR Builder additionally highlights selected layers with glowing effects to enhance visualization. Measurement and unit tools allow the user to take distance measurements of BIM models with the MR device. Depending on the objects to be measured from, distance measurements can be taken either within the digital model, within the physical space, or between the digital model and the physical space. Trimble Connect is the only MR application that offers all three measurement types. While not all applications are able to measure distances, most displays either a metric or imperial unit for movement or scaling purposes, and some also support unit conversion between the two. Model annotation tools in MR include text or sketch markup, snapshot of the current view, as well as property information of the selected objects. These annotation tools assist the inspection of the model in MR by identifying the problematic areas and locating them during revisits.

Immersive view mode
The MR technology enables an unobstructed immersive view mode that allows observation of BIM models at their full-size scale in a construction jobsite, which is unarguably the most essential toolset for architectural and construction uses. Although 3D Viewer is the only MR application not providing the immersive view as a native view mode, it can still be achieved by scaling a model to a 1:1 scale. At such scale, the user sees the BIM model at its true size and can physically walk around to inspect the model in MR when it is placed on-site or in a space large enough to accommodate the full-size model. Alternatively, as summarized in Table 6, the user can choose to virtually walk or teleport within the model when it is not aligned on-site, a navigation method used by all VR devices. In such cases, the user stays physically stationary while the selected destination point in the model moves right in front of the user, consequently moving the entire model to a new position. A third navigation method in the immersive view is to virtually elevate the user directly to different levels byeither raising or lowering the model. In addition, SketchUp Viewer and Trimble Connect also allow user height adjustment to accommodate various audiences for a better immersive view.
One essential capability to use MR technology on-site for architectural design and construction layout is the full-scale adjustment. Under the immersive view mode at 1:1 scale, the BIM models are expected to match their true size onsite, but discrepancies exist at a small percentage between 1-2% so that the model looks slightly smaller when overlaid directly on its physical counterpart, and the discrepancy varies between each MR device. This variance in size could be negligible for small 3D objects, but for BIM models it can easily exceed tolerances. Full-scale adjustment allows the user to manually compensate for the discrepancy when overlaying and comparing a BIM model on-site to its physical counterpart. SketchUp Viewer and Trimble Connect are the only two MR applications supporting full-scale adjustment.
Another capability of MR technology that can significantly improve the efficiency of its on-site use for architectural design and construction layout is automatic model alignment. Under the immersive view mode at 1:1scale, the first thing to do is to ensure the alignment of the BIM model to the physical site before any review or inspection can happen. Manual alignment takes a considerable amount of time and effort, and often results in inaccurate alignment due to the observation-based approach and leads to great frustration due to the difficulty in dragging a fullsize model. Automatic model alignment allows a full-size model to be immediately placed at its correct position on-site after registering the required reference targets. Five MR applications offer automatic model alignment through different methods, including two-point registration, one-point registration, and QR code scan. Each application applies the twopoint registration approach differently, as detailed in Table 7. Trimble Connect asks the user to identify two walls and the floor in the model and their counterparts in the physical site. BIM Holoview requires the user to insert a provided model component with two predefined points in the BIM model, and then register these two points in the physical space. MR Builder allows the user to choose any two points in the model and register their counterparts in the physical space, as demonstrated in Figure 4. HoloLive 3D requires the user to register one point on the floor and then identify the second pointing the north direction.
Fuzor AR uses a similar approach except that the second point on the floor could be pointing any of the four directions. Fuzor AR also uses a onepoint registration method where only one point needs to be registered on a wall assuming all walls are plumb. The QR code alignment method requires a QR code label model component to be inserted in the BIM model first, and then the same QR code label needs to be printed and affixed to the same location in the physical space, either a wall or floor. After scanning the affixed QR code label with the MR device, the model can be immediately aligned. Lastly, a few MR applications allow pivot points to be selected for fine adjustment of movement and rotation if needed.

Multiuser collaboration
The MR technology provides the opportunity for multiple MR device users to inspect the same BIM model together through a collaborative session, which is an essential part of decision-making for a project team between the building owners, architects, engineers, and contractors. A basic multiuser collaboration session starts with a session host whose MR device will upload the BIM model to the sharing service. When other participants join the session, the model will be downloaded from the sharing service to their MR devices for viewing. In such case, all session participants can inspect the same model together, but they do not know where each other is located relative to the model position. In order to share this information, a co-located collaboration session is required where all participants will specify a common reference point in the same physical space to create a shared coordinate system. Once the shared coordinate system is established across all MR devices in the session, each participant will be represented as an avatar to indicate their position relative to the model, and their focus point will be presented with a laser beam. Figure 5 demonstrates the comparison between a basic collaboration session in Trimble Connect and a co-located collaboration session in SketchUp Viewer. Both snapshots were captured by a third MR device in the same session. Table 8 summarizes the multiuser collaboration capabilities of MR technology.

BIM enhancement tools
In addition to the basic BIM tools, MR technology provides other tools to enhance design visualization and model interaction to facilitate  decision-making in architectural design and construction layout. These BIM enhancement tools can either animate construction sequence or schedule, switch between different preset scenarios, or change the design by adding or relocating model components, as shown in Table 9. 3D Viewer and Vyzn support object animation such as flames, waves, or wind effects, while SketchUp Viewer allows automatic rotation of the model. SketchUp Viewer and Fuzor AR can also present construction schedules when different views are captured in a logical sequence. As mainly presentation applications, Vyzn and Prism allow the model to contain different scenario views, such as day views vs. night views, architectural view svs structural views, or even between different models. In addition, MR Builder and Vyzn allow real-time design changes in MR by adding or relocating predefined building components to explore alternative designs and layouts. Overall, these BIM enhancement tools complement the basic tools such as measurement and annotation to better utilize the potential of MR technology in architectural design and construction layout.

DISCUSSION
Through the field testing of the various MR applications in this study, several limitations of MR technology and the MR device in on-site uses have been identified, including the model drifting issue, automatic alignment issue, and co-located collaboration issue. These issues fall within the essential capabilities of MR technology and are expected to improve and hopefully be eliminated over the development of MR technology and devices.
The model drifting issue can be easily observed in the immersive view mode when the BIM model has been aligned with its physical counterpart. Under such  conditions, when the user wearing the MR device moves away from the original position where the model has been aligned from, the model also drifts towards the same direction at a small rate instead of holding stationary. The drifting distance is proportional to the distance that the user has traveled, which consequently causes model misalignment.
To compensate for this misalignment during the user's movement, manual realignment is needed depending on the distance traveled.
The automatic alignment issue exists due to the fact that during either the two-point registration or one-point registration approach, the locations of the registration points in the BIM model must exactly match their physical counterparts. These physical locations are selected manually with hand gestures by observing a focus point through the MR device. With the device being mounted on the user's head, this focus point can hardly stay still, always resulting in an inaccurate selection of the registration point location. Further fine adjustments through pivot points are therefore needed for a better model alignment.
The co-located collaboration issue happens more often under complicated on-site conditions where many building components are exposed and building materials are stored disorderly. Under such conditions, it is difficult for the MR devices to map the same physical space and recognize a common reference point on-site in order to establish the required shared coordinate system. As a result, an expected co-located collaboration session will often present the wrong locations for the participants' avatars.
To better overcome these issues that can happen on-site and hinder the successful use of MR technology for architectural design and construction layout, one alleviation is to allow enough time for the MR devices to fully scan and map the entire physical space before using any BIM models on-site. These current issues will hopefully be eliminated with the further development of MR technology, such as continuous model alignment and automatic recognition of multiple QR codes.

Conclusions
While MR technology has attracted increasing interest in the architecture and construction industry, no comprehensive reviews in existing literature have been found to report the use of MR technology in architecture and construction practice. To bridge this gap in the body of knowledge, this paper presents the results of evaluating the use of current MR technology in architectural design and construction layout through a comparative review of commercially available MR applications. The study employed two HoloLens devices that were attached to a hard hat to meet construction safety regulations. Following a literature review and Internet search, nine MR applications developed for architectural and construction purposes were identified and installed on the two HoloLens for field testing. The capabilities of MR technology within these applications were evaluated and compared under different on-site conditions for architectural design and construction layout. The evaluated MR capabilities included model formats and file access methods, user interface, different view modes, various BIM tools, collaboration features, etc. In addition, the limitations of MR technology and the head-mounted MR device for on-site uses were discussed. This paper provides timely and empirical information to help practitioners and researchers better understand MR technology for the architecture and construction industry.
This study was sponsored by the Thomas Glavinich ELECTRI International 2018 Early Career Award. The author would like to thank ELECTRI International and Thompson Electric Company for their support.