Motivated by the need for understanding neurological disorders, large-scale imaging genetic studies are being increasingly conducted. A salient objective in such studies is to identify important neuroimaging biomarkers such as the brain functional connectivity, as well as genetic biomarkers, which are predictive of disorders. However, typical approaches for estimating the group level brain functional connectivity do not account for potential variation, resulting from demographic and genetic factors, while usual methods for discovering genetic biomarkers do not factor in the influence of the brain network on the imaging phenotype. We propose a novel semiparametric Bayesian conditional graphical model for joint variable selection and graph estimation, which simultaneously estimates the brain network after accounting for heterogeneity, and infers significant genetic biomarkers. The proposed approach specifies priors on the regression coefficients, which clusters brain regions having similar activation patterns depending on covariates, leading to dimension reduction. A novel graphical prior is proposed, which encourages modularity in brain organization by specifying denser and sparse connections within and across clusters, respectively. The posterior computation proceeds via a Markov chain Monte Carlo. We apply the approach to data obtained from the Alzheimer's disease neuroimaging initiative and demonstrate numerical advantages via simulation studies.