Six poly(phenylene-alt-dithienobenzothiadiazole)-based polymers have been synthesized for application in polymer-fullerene solar cells. Hydrogen, fluorine, or nitrile substitution on benzo-thiadiazole and alkoxy or ester substitution on the phenylene moiety are investigated to reduce the energy loss per converted photon. Power conversion efficiencies (PCEs) up to 6.6% have been obtained. The best performance is found for the polymer-fullerene combination with distinct phase separation and crystalline domains. This improves the maximum external quantum efficiency for charge formation and collection to 66%. The resulting higher photocurrent compensates for the relatively large energy loss per photon (Eloss = 0.97 eV) in achieving a high PCE. By contrast, the poly-mer that provides a reduced energy loss (Eloss = 0.49 eV) gives a lower photocurrent and a reduced PCE of 1.8% because the external quantum efficiency of 17% is limited by a suboptimal morphology and a reduced driving force for charge transfer.