The attacker, a National Guard member, arrived on a quad bike and was wearing body armor, said Rene Trabelsi, Tunisia's former tourism minister who organized the pilgrimage, adding that the two cousins had tried to hide behind a bus outside the synagogue.
The shooter killed a colleague at a naval installation, then travelled to the synagogue where an annual Jewish pilgrimage was taking place and opened fire on police and visitors before he was shot dead.
Two Jewish cousins, one French and the other Israeli-Tunisian, were killed, along with one police officer who died at the scene and another in hospital on Wednesday.
Four more police were injured, one critically, hospital sources said, along with four other visitors.
Worshippers attending the pilgrimage described a scene of panic after gunshots rang out, as people tried to hide in different rooms of the synagogue.
"People were happy and dancing until we heard a lot of gunfire. Everyone ran away... some hid in my office and others in the other rooms. There was lots of fear," said Peres Trabelsi, head of Djerba's Jewish community.
Authorities have not yet outlined a motive or said if anybody else was involved in the attack, which is the deadliest in years and took place on an island that is a major destination for Tunisia's vital tourism industry.
The pilgrimage to Africa's oldest synagogue regularly draws hundreds of Jews from Europe and Israel to Djerba, located just off the coast about 500 km from the capital Tunis.
The pilgrimage has had tight security since al Qaeda militants attacked the synagogue in 2002 with a truck bomb, killing 21 Western tourists. Mainly Muslim Tunisia is home to one of North Africa's largest Jewish communities with about 1,800 members.
Any impact on Tunisia's tourism sector, a major source of foreign currency, will be closely watched in a year when the government is seeking financial help to avert a crisis in public finances.
Tourism Minister Moez Belhassine was visiting the site and Djerba hotels to reassure people on security.
Israel's Foreign Ministry described the attack as a "lethal shooting incident," with a spokesperson saying it was still under investigation.
The United States and France said Tunisian security forces had responded quickly.
Your browser doesn’t support HTML5
Police Deploy Near Tunisia Synagogue After Deadly Attack