The outbreak of Coronavirus Disease 2019 (COVID-19) has pervasive implications for the well-being of people, especially for the social withdrawn individuals. The present study examined changes of well-being among people in distinct subgroups of social withdrawal - shyness, unsociability, and social avoidance -in different phases of the COVID-19 pandemic using six-wave longitudinal data in China ( N = 222; 54.50% female). Results showed that, in general, well-being sharply decreased from the initial phase to the peak phase of the pandemic, but steadily recovered after the peak phase. People in different withdrawal groups displayed different levels and trajectories of well-being during a period of six months. The current study has implications for developing targeted interventions for vulnerable people in public health crisis.