The integration of cutting-edge technologies, such as wearables, in complex systems is crucial for enhancing collaboration between humans and machines in the era of Industry 5.0. However, this increased interaction also introduces new challenges and risks, including the potential for human errors. A thorough analysis of the literature reveals an absence of studies that have quantified these risks, underscoring the utmost importance of this research. To address the above gap, the present study introduces the STPA-PSO methodology, which aims to quantify the risks associated with the use of smart glasses in complex systems, with a specific focus on human error risks. The proposed methodology leverages the Systems-Theoretic Process Analysis (STPA) approach to proactively identify hazards, while harnessing the power of the Particle Swarm Optimization (PSO) algorithm to accurately calculate and optimize risks, including those related to human errors. To validate the effectiveness of the methodology, a case study involving the assembly of a refrigerator was conducted, encompassing various critical aspects, such as the Industrial, Financial, and Occupational Health and Safety (OHS) aspects. The results provide evidence of the efficacy of the STPA-PSO approach in assessing, quantifying, and managing risks during the design stage. By proposing a robust and comprehensive risk quantification framework, this study makes a significant contribution to the advancement of system safety analysis in complex environments, providing invaluable insights for the seamless integration of wearables and ensuring safer interactions between humans and machines.