Biomarkers for immune checkpoint inhibitors (ICIs) are limited in gastrointestinal cancer. Peripheral blood lymphocyte subset and associated dynamic changes were retrospectively analyzed in patients with gastrointestinal cancer treated with ICIs. Cox regression and Kaplan-Meier analyses were conducted for survival. A total of 80 patients were enrolled. Baseline CD4+/CD8+ T cells were lower in patients who experienced tumor progression by 6 months than in patients who did not (1.160 ± 0.652 vs 1.705 ± 0.924, respectively; p = 0.003). In multivariate analyses, decline in CD4+ T cells after the first dose of ICIs (CD4-C1-decline) was an independent prognostic factor for overall survival (hazard ratio: 13.00; 95% CI: 2.24-75.54; p = 0.004). Furthermore, CD4-C1-decline was a preferable indicator for progression in patients with deficient mismatch repair/microsatellite instability-high (p = 0.027). Early change in CD4+ T cell counts in peripheral blood may act as a prognostic biomarker for gastrointestinal cancer patients treated with ICIs.