A mobile augmented reality-based assembly guidance application for LEGO

Usually LEGO structures are created by following an instruction manual in paper or digital form that depicts the LEGO model at different stages of assembly. Some instructions can be very complex and difficult to represent using plain illustrations. Consequently, this makes it difficult to follow and assemble a LEGO structure. Augmented reality (AR) is a technology that adds perceptual information into the physical world and it has recently been gaining popularity. This study explores the use of AR for assembly through ARGO, a mobile application for Android smartphones that serves as an assembly guide for LEGO structures. It can enhance instructions and provide more details that assist users during the construction process. The system can recognize the current step of the LEGO assembly using data captured via Microsoft Kinect while providing visual feedback through the mobile application. To identify the current step, the system obtains the color and depth data from the Kinect camera and converts the data into a point cloud. The point cloud undergoes segmentation to isolate the physical LEGO model and compares it to the point clouds of a preloaded model to check if the brick is placed correctly. The system was evaluated using 3 models and can recognize when to advance to the next step during assembly and estimate the orientation of the step differing an average angle of 48.27 degrees. For each step, it takes an average of a few seconds to decide on the recognition, leading to a short delay for the feedback to appear on the phone.