Twenty-two people were rescued from the site and sent to hospitals, the municipality of George, a coastal city east of Cape Town, said in a statement.
Western Cape premier Alan Winde said two among the retrieved had died, while 53 remain trapped under the rubble.
"Seventy-five members of the construction crew have been confirmed on-site at the time of the incident," he said.
CCTV footage obtained by Reuters showed a cloud of dust as the building crumbled on Monday afternoon.
"I saw one guy was working and then 'boom,' and I saw the whole building collapse. ... I'm also traumatized. It is very sad," Theresa Jeyi, a local councilor, told reporters at the scene.
The provincial government said police and rescue teams with sniffer dogs were on-site.
Rescue workers on-site were trying to recall where they heard calls from people, George Mayor Leon van Wyk told SABC, while rescue equipment was on its way.
"This is going to be a really lengthy exercise and probably going to go right through the night," he said.