Attackers from the al Shabab group charged into the block that houses the office of Mogadishu's mayor around noon and got caught in a firefight with security forces, the ministry and witnesses said.
Security killed six of the militants and cleared the area by about 6 p.m., the ministry said on its Facebook page.
"We were in the office and we were deafened by a blast. We ran out. Gunfire followed," Farah Abdullahi, who works in the mayor's office, told Reuters.
Abdikadir Abdirahman, director of Aamin Ambulance Services, gave a higher estimate of the death count, saying it had transported the bodies of eight civilian victims.
At least another 16 people were injured in the attack, he added.
The mayor's office is in the local government headquarters building in a well-guarded area of Mogadishu.
Roads in the area have concrete barriers and multiple roadblocks. The building is about 1.5 km away from Villa Somalia, the president's office.
Al Shabab said in a statement its suicide bombers struck, "then foot fighters entered the building after killing the building guards."
In a sign the government was expanding its offensive against the group into the country's south, state TV for Jubbaland, one of the country's semi-autonomous states, reported on Sunday that regional and Somalia federal forces had launched attacks on al Shabab and taken control of the town of Janay Abdale from the militants.
Al Shabab has stepped up attacks in a show of resilience since President Hassan Sheikh Mohamud's government launched an offensive against the al Qaeda-linked group in August.
The movement, which has been fighting the government since 2006 and seeking to impose its own interpretation of Islamic law, frequently carries out bombings and gun attacks across the country.