"This court has now determined that Mr Zuma's release on medical parole was unlawful," the supreme court of appeal said, sending the former president back to jail.
Zuma, 80, had been handed a 15-month term in June 2021 for contempt of court, a move that triggered deadly unrest.
But he served only two months before being given parole for medical reasons that remain unclear.
Parole was granted by the head of South Africa's prison service, despite an opinion by the service's medical committee that Zuma did not meet the required conditions.
The order by the supreme court of appeal is unlikely to be the final word in the long-running saga, as Zuma still has recourse to the Constitutional Court, the highest judicial instance in the land.
Zuma, a prominent figure in the fight against apartheid, became president in 2009 and was forced to step down in disgrace by the ANC in 2018 following mounting corruption allegations.
But he remains a revered figure among grassroots members of the ruling African National Congress (ANC), who perceive him as a defender of the poor and oppressed.
When his jail term was announced, violent protests broke out that spiraled into looting, leaving 350 people dead.
Zuma was jailed for refusing to testify to a high-level inquiry into massive state corruption that unfolded under his presidency.
He also faces separate corruption charges over an arms deal dating back more than two decades.