Wearing a blue jacket, black trousers and spectacles, and flanked by armed officers with helmets and bullet proof vests, Kayishema identified himself before the packed courtroom.
The 62-year-old appeared calm and composed as a state prosecutor read out the accusations against him.
They include "genocide" and conspiracy to commit genocide "relating to the slaughter of more than 2,000 people in Rwanda in 1994," prosecutor Nathan Adriaanse told the court.
At the end of a short hearing, magistrate Ronel Oliver remanded him into custody, with a next appearance scheduled for June 2.
Kayishema and others allegedly murdered more than 2,000 men, women and children who had taken refuge in a Catholic church in Nyange in Kivumu district.
"Kayishema directly participated in the planning and execution of this massacre, including by procuring and distributing petrol to burn down the church with the refugees inside," the International Residual Mechanism for Criminal Tribunals [MICT] said on Thursday, announcing his arrest.
"When this failed, Kayishema and others used a bulldozer to collapse the church, burying and killing the refugees inside."
"Kayishema and others then supervised the transfer of corpses from the church grounds into mass graves."
The former Rwandan police inspector was arrested on Wednesday at a grape farm in Paarl, 60 kilometers north of Cape Town, according to U.N. investigators.
South African police said the arrest was made in response to an Interpol red notice, and the suspect had been living under the false name of Donatien Nibashumba.
The International Residual Mechanism for Criminal Tribunals (MICT) - the successor to the U.N. court that prosecuted scores of major suspects - describes him as "one of the world's most wanted genocide fugitives."
He faces charges of genocide, complicity in and conspiracy to commit genocide, and crimes against humanity.
Around 800,000 Rwandans, most of them ethnic Tutsis, were slaughtered over 100 days at the hands of Hutu extremists.
The hunt for Kayishema spanned countries across Africa, targeting a man who used many aliases and false documents and relied on a "network of trusted supporters," the MICT said.
These included family members, members of the former Rwandan armed forces, the Democratic Forces for the Liberation of Rwanda (FDLR) rebel group "and those aligned with the genocidal Hutu Power ideology."
The United States had offered up to $5 million reward for information leading to Kayishema's arrest, transfer or conviction, but there have been no details as to whether this played a role in the capture.
The MICT in 2015 took over the work of the International Criminal Tribunal for Rwanda (ICTR), set up by U.N. after the genocide.
Rwanda started trying genocide suspects in 1996, and on a single day in April 1998 had 22 of them executed by firing squad.
It abolished the death penalty in 2007, lifting the main obstacle for the ICTR to extradite genocide suspects to Rwanda for trial.
Between 2005 and 2012, more than 12,000 community-based courts put nearly two million people on trial and convicted 65 percent, sending most to prison.
Other convictions have been handed down in the former colonial power Belgium as well as in France, Sweden, Finland, Norway, Germany, the Netherlands, the United States and Canada.