Residential area characteristics and discrimination have been associated with psychological distress. Differences in these relationships across racial groups are not well understood. We examined the relative role of perceived discrimination, neighborhood problems and neighborhood cohesion/trust in explaining differences in psychological distress (indicated by anxiety and depressive symptoms) between 224 African American and 225 White smokers (income ≤ 400% federal poverty level) in a smoking cessation intervention study. Surveys were linked to US census-tract data. We conducted random intercept Poisson multi-level regression models and examined interactions between race and neighborhood experiences. African Americans had greater risk of anxiety and depressive symptoms and greater individual and neighborhood disadvantage than Whites. Controlling for objective neighborhood characteristics, when perceived discrimination and perceived neighborhood characteristics were added to the regression models the association between anxiety symptoms and race were no longer statistically significant; the association between depressive symptoms and race decreased but remained statistically significant. Lower neighborhood social cohesion/trust and greater neighborhood problems increased depressive symptoms for African Americans, but not for Whites. Perceived discrimination and neighborhood social cohesion/trust outweighed the importance of race in explaining anxiety symptoms. These findings underscore the need for multi-level interventions addressing social and environmental contexts.