"The trial chamber finds Mr. Kabuga is no longer capable of meaningful participation in his trial," a decision published on the Hague court's website said.
The decision came after doctors found Kabuga suffered from dementia.
It is rare for suspects before international courts to be declared mentally unfit to stand trial, though many defendants try.
Instead of halting the trial, the judges said they would set up an "alternative finding procedure that resembles a trial as closely as possible, but without the possibility of a conviction."
It was not immediately clear what form such proceedings will take or what will happen to Kabuga who is in the court's detention center in The Hague.
The former coffee and tea tycoon has denied the charges of genocide and crimes against humanity. Prosecutors say Kabuga promoted hate speech through his broadcaster, Radio Television Libre des Milles Collines (RTLM), and armed ethnic Hutu militias.
Kabuga has been on trial at The Hague branch of the United Nations mechanism that took over operations of the International Criminal Tribunal for Rwanda since September last year. His lawyers have tried since his arrest to get the case dropped over health concerns due to his advanced age.
Kabuga is in his late 80s, though his precise date of birth is disputed. He was arrested in France in 2020 after more than 20 years on the run.
The former businessman and radio station owner was one of the last suspects sought by the tribunal prosecuting crimes committed in the 1994 genocide, when ruling Hutu majority extremists killed more than 800,000 minority Tutsis and Hutu moderates in 100 days.