Sexual and gender minority (SGM) adults experience poor health outcomes, in part due to frequent avoidance of necessary health care. Little is known, however, about factors contributing to patterns of health care utilization in this population. Using national data from the All of Us Research Program, this study evaluated the prevalence of care avoidance due to patient-clinician identity discordance (PCID) and its association with health care discrimination among SGM adults. Sexual minority (20.0% vs 9.4%; adjusted rate ratio [aRR] = 1.58; 95% CI, 1.49-1.67, P <0.001) and gender minority adults (34.4% vs 10.3%; aRR = 2.00; 95% CI, 1.79-2.21, P <0.001) were significantly more likely than their non-SGM counterparts to report care avoidance due to PCID. Exposure to health care discrimination was also more prevalent in this population and was dose-dependently associated with significantly higher rates of PCID-based care avoidance. Study findings highlight the importance of diversifying the health care workforce, expanding SGM-related clinical training, and preventing health care discrimination against SGM patients.