Optical spectroscopy and imaging techniques play important roles in many fields such as disease diagnosis, biological study, information technology, optical science, and materials science. Over the past decade, machine learning (ML) has proved promising in decoding complex data, enabling rapid and accurate analysis of optical spectra and images. This review aims to shed light on various ML algorithms for optical data analysis with a focus on their applications in a wide range of fields. The goal of this work is to sketch the validity of ML-based optical data decoding. The review concludes with an outlook on unaddressed problems and opportunities in this emerging subject that interfaces optics, data science and ML.