Immune checkpoint inhibitors (ICIs) are used to relieve and refuel anti-tumor immunity by blocking the interaction, transcription, and translation of co-inhibitory immune checkpoints or degrading co-inhibitory immune checkpoints. Thousands of small molecule drugs or biological materials, especially antibody-based ICIs, are actively being studied and antibodies are currently widely used. Limitations, such as anti-tumor efficacy, poor membrane permeability, and unneglected tolerance issues of antibody-based ICIs, remain evident but are thought to be overcome by small molecule drugs. Recent structural studies have broadened the scope of candidate immune checkpoint molecules, as well as innovative chemical inhibitors. By way of comparison, small molecule drug-based ICIs represent superior oral bioavailability and favorable pharmacokinetic features. Several ongoing clinical trials are exploring the synergetic effect of ICIs and other therapeutic strategies based on multiple ICI functions, including immune regulation, anti-angiogenesis, and cell cycle regulation. In this review we summarized the current progression of small molecule ICIs and the mechanism underlying immune checkpoint proteins, which will lay the foundation for further exploration.