"The peace negotiation that will be held with (the Oromo Liberation Army) will start the day after tomorrow (April 25) in Tanzania," Abiy said Sunday.
"The Ethiopian government and people will need this negotiation very much.
"I call upon everybody to play (their) part."
The OLA has been fighting the Ethiopian federal government since it split in 2018 with the historic Oromo Liberation Front (OLF) when it renounced armed struggle.
Abiy spoke at a gathering of parties to the Tigray peace process which saw a deal on November 2 end a two-year conflict between the federal government and rebelling regional authorities.
Neither Abiy nor the OLA gave details on the format of the new talks, who would mediate or where they would be held.
The OLA confirmed in a statement "that the Ethiopian regime has accepted our terms for peace negotiations, which include the involvement of an independent third-party mediator and a commitment to maintain transparency throughout the process.
"This is a crucial and positive step towards establishing a lasting peace in the region," it said.
"Since the beginning of this conflict, the OLA has consistently called for peaceful dialogue as the only viable solution... It is heartening to see the regime finally come to the same realization."
The split of the Oromia rebel groups gave rise to a string of armed groups claiming to be one, but with loose ties.
The OLA's strength, estimated at a few thousand men in 2018, has increased significantly in recent years, though observers believe it is insufficiently organized or well-armed to pose a real threat to the federal government.
Oromia has suffered ethnic massacres in recent years carried out by unknown groups.
The OLA has been repeatedly accused by Abiy's government of being responsible for these massacres, which it denies.
The government is accused of waging an indiscriminate crackdown that has fueled Oromo resentment against the central government.
Addis Opens Oromo Rebel Talks
ADDIS ABABA - Ethiopian Prime Minister Abiy Ahmed said his government had opened dialogue with a rebel group active in the Oromia region, the country's largest and most populous region, surrounding the capital Addis Ababa.