Computer vision and classification methods have become increasingly wide-spread in recent years due to ever-increasing access to computation power. Advances in semiconductor devices are the basis for this growth, but few publications have probed the benefits of data-driven methods for improving a critical component of semiconductor manufacturing, the detection and inspection of defects for such devices. As defects become smaller, intensity threshold-based approaches eventually fail to adequately discern differences between faulty and non-faulty structures. To overcome these challenges we present machine learning methods including convolutional neural networks (CNN) applied to image-based defect detection. These images are formed from the simulated scattering of realistic geometries with and without key defects while also taking into account line edge roughness (LER). LER is a known and challenging problem in fabrication as it yields additional scattering that further complicates defect inspection. Simulating images of an intentional defect array, a CNN approach is applied to extend detectability and enhance classification to these defects, even those that are more than 20 times smaller than the inspection wavelength.