Adenoid cystic carcinoma (ACC) is an extremely rare type of breast cancer. The role of adjuvant treatments for ACC remains controversial. Patients with a histology-confirmed diagnosis of ACC of the breast were identified based on the SEER (Surveillance, Epidemiology and End Results) database. Propensity score matching (PSM) was performed to balance the baseline characteristics. The Kaplan-Meier method and Cox regression models were performed to determine the impact of the adjuvant chemotherapy (CT) and radiotherapy (RT) associated with breast cancer-specific survival (BCSS) and overall survival (OS). A total of 1036 patients with ACC of the breast were included. After a median follow-up of 11.3 years, the 10-year OS and BCSS rate was 76.2% and 92.6%, respectively. After PSM, adjuvant CT converted into neither OS (Log-rank p = 1.000) nor BCSS (Log-rank p = 0.900) benefits in patients with ACC of the breast. Adjuvant RT also did not improve OS (Log-rank p = 0.060) and BCSS (Log-rank p = 0.400). According to the univariate stratified analysis, adjuvant RT was favorable for OS in patients who underwent breast-conserving surgery (HR 0.66, 95% CI 0.45, 0.99, p = 0.042). From the multivariate analysis, histology grade and nodal status were independent prognostic factors for both OS and BCSS. In conclusion, ACC of the breast presented a favorable prognosis. Adjuvant treatment, especially adjuvant CT, might not be essential for patients with this disease.