Patterns of sea surface temperature (SST) anomalies of the Indian Ocean Dipole (IOD) exhibit strong diversity, ranging from being dominated by the western tropical Indian Ocean (WTIO) to the eastern tropical Indian Ocean (ETIO). Whether and how the different types of IOD variability patterns affect the variability of Antarctic sea ice is not known, nor is how the impact may change in a warming climate. Here, we find that the leading mode of austral spring Antarctic sea ice variability is dominated by WTIO SST variability rather than ETIO SST or El Niño-Southern Oscillation. WTIO warm SST anomalies excite a poleward-propagating Rossby wave, inducing a tri-polar anomaly pattern characterized by a decrease in sea ice near the Amundsen Sea but an increase in regions on both sides. Such impact has been weakening in the two decades post-2000, accompanied by weakened WTIO SST variability. Under greenhouse warming, climate models project a decrease in WTIO SST variability, suggesting that the reduced impact on Antarctic sea ice from the IOD will likely to continue, facilitating a fast decline of Antarctic sea ice.