A total of 4,270 inmates, mostly men, were let out, according to Zimbabwe's correctional service, which described the reprieve as a "noble gesture" by the president.
The amnesty was granted to various categories of prisoners including those who have served at least three-quarters of their sentence, or one tenth if over the age of 60.
Violent criminals as well as those serving time for robbery, treason and public order and security offences were excluded.
Those released will be able to vote in presidential and legislative elections that are to be held in August, although no date has been announced yet.
"We would like to appeal to... society at large to embrace and accept the inmates who have been released," Zimbabwe Prisons and Correctional Service (ZPCS) said in a statement.
"Those aggrieved are encouraged to forgive them."
The move brings down overcrowding in the country's more than 50 detention centers, which have capacity for about 17,000 people but held more than 22,000 before the amnesty.
Yet, ZPCS spokeswoman Meya Khanyezi told AFP "it was not about decongestion."
President Emmerson Mnangagwa is struggling to ease entrenched poverty, end chronic power cuts and rein in inflation.
He granted a similar amnesty at the height of the Covid-19 pandemic in 2020 in a bid to curtail the spread of the virus in detention facilities.