The Gambia, a state along the Gambia River that is almost entirely surrounded by Senegal, is being especially badly hit, NGO Amnesty International said.
Large industrial trawlers, many of them foreign owned, are scooping up small fish such as sardinella and bonga to be turned into meal for fish farming.
Amnesty's report estimated that illegal and excessive fishing off The Gambia, as well as Mauritania, Senegal, Guinea-Bissau, Guinea and Sierra Leone, amounted to a total loss of $2.3 billion per year.
The report was based on an investigation that included research in the capital Banjul and testimonies from fishermen, vendors and restaurant owners in the coastal region of Sanyang.
"Local communities are being deprived of their right to a decent standard of living as well as their right to health and food," said Samira Daoud, Amnesty's regional director West and Central Africa.
"The Gambian authorities must take urgent action to both better protect the environment and the fundamental rights of these communities."