It was not immediately clear whether Zuma would return to jail, with prison authorities saying they were studying the judgement and would seek legal advice.
Zuma was sentenced in June 2021 after refusing to testify before a panel probing financial sleaze and cronyism under his presidency — but was freed on medical parole just two months into his term.
He started serving the sentence early in July.
His jailing sparked riots that descended into looting and left more than 350 dead in the worst violence to hit the country since the advent of democracy in South Africa.
The following month, he was admitted to hospital for an undisclosed condition before being granted medical parole.
In November last year an appeals court found the release was illegally granted and ordered the now 81-year-old back to the Estcourt Correctional Center in the eastern KwaZulu-Natal province.
South Africa's prison service, which had granted Zuma's conditional release, appealed the decision, but the bid was dismissed by the Constitutional Court.
It ruled on Thursday that it had "no reasonable prospects of success" and ordered the prison services and Zuma, an intervening applicant, to pay legal costs.
The prison services declared in October last year that Zuma had officially ended his sentence and had been released from their system.
But in November the appeals court described the pronouncement as "premature."
On Thursday, the prison services said they were "studying the Constitutional Court Judgment for the review application in relation to the medical parole placement for the former President."
They added in a statement that they were "seeking legal advice and will comment further in due course."
Zuma's spokesman Mzwanele Manyi referred requests for comment to prison authorities.
South Africa's leading opposition party, the Democratic Alliance described the latest ruling as a "victory for the rule of law."
"Zuma must now go back to jail!" the party wrote on Twitter.