The current study investigated the phonetic adjustment mechanisms that underlie perceptual adaptation in first and second language (Dutch-English) listeners by exposing them to a novel English accent containing controlled deviations from the standard accent (e.g. /i/-to-/ɪ/ yielding /krɪm/ instead of /krim/ for 'cream'). These deviations involved contrasts that either were contrastive or were not contrastive in Dutch. Following accent exposure with disambiguating feedback, listeners completed lexical decision and word identification tasks. Both native and second language listeners demonstrated adaptation, evidenced by higher lexical endorsement rates and word identification accuracy than untrained control listeners for items containing trained accent patterns. However, for L2 listeners, adaptation was modulated by the phonemic contrast, that is, whether or not it was contrastive in the listeners' native language. Specifically, the training-induced criterion loosening for the L2 listeners was limited to contrasts that exist in both their L1, Dutch, and L2, English. For contrasts that are either absent or neutralized in Dutch, the L2 listeners demonstrated relatively loose pre-training criteria compared to L1 listeners. The results indicate that accent exposure induces both a general increase in tolerance for atypical speech input as well as targeted adjustments to specific categories for both L1 and L2 listeners.