With the development of infrared polarization sensors, image enhancement algorithms have been developed. Although using polarization information quickly distinguishes man-made objects from natural backgrounds, cumulus clouds would become detection noise because of their similar characteristics to targets in the sky scene. In this paper, we propose an image enhancement algorithm based on polarization characteristics and the atmospheric transmission model. The algorithm utilizes the principle of polarization imaging and atmospheric transmission theory to enhance the target in the image while suppressing the interference of clutter. We compare with other algorithms through the data we collected. The experimental results show that our algorithm significantly improves the target brightness and reduces clutter at the same time with real-time performance.