Although harmful substance use is common and represented by shared symptom features and high genetic correlations, the underlying genetic relationships between substance use traits have not been fully explored. We have investigated the genetic architecture of substance use traits through exploratory and confirmatory factor analyses using genomic structural equation modeling (Genomic SEM), and explored genetic correlations between different aspects of substance use and mental health-related traits. Genomic SEM was used to identify latent factors representing the relationships between 14 substance use traits (alcohol, nicotine, cannabis and opioid use), and to confirm or modify existing latent factors for 38 mental health-related traits. A bi-factor model best explained the genetic overlap between substance use traits, including a general substance use factor and two sub-factors representing genetic liability specific to alcohol use or smoking. The SNP-based heritability of these factors ranged from 2 to 7 % and each factor had 10 or more independent significant SNPs identified. Bivariate correlations revealed patterns of genetic overlap with other mental health-related factors unique to each substance use factor. Variations in the genetic overlap between psychiatric traits and different aspects of substance use can be used to further investigate the pleiotropy present between these traits, and explore commonalities in etiology.