A machine learning approach to improve Kinect-based motion capture in manual material handling

In response to rising labor costs, workforce shortages, and the prevalence of musculoskeletal disorders in warehouse environments, companies are looking for cost-effective motion capture (MoCap) solutions to assess and mitigate posture-related risks. Although professional MoCap systems offer high accuracy, they are often expensive, making their purchase particularly difficult for small and medium-sized companies. This study investigates the use of a markerless MoCap system based on Azure Kinect enhanced with machine learning (ML) to avoid occlusion-related tracking errors, improve tracking accuracy and reliability, and achieve cost benefits. By applying design science research methodology, a conceptual three-layer architectural model is developed comprising (1) sensor setup and data collection, (2) model development and testing, and (3) model evaluation for decision-making. First, an experimental approach is used to quantify the proportion of artefacts. Then, a convolutional neural network is trained using benchmark data from an inertial measurement unit-based system (Captiv) to improve the tracking accuracy of the Kinect-based system. Our evaluation shows that the ML-corrected Kinect data significantly reduces deviations in key joint angles (e.g., back, elbow, and shoulder) compared to the original Kinect data, as measured by multiple performance metrics. Furthermore, the Ovako working posture assessment system is used to test the usefulness of the ML model in assessing postural risk. ML-corrected Kinect data is found to align more closely with the Captiv data while showing a higher sensitivity to hazardous postures.