Le Graet's resignation comes 13 days after the publication of a damning report into management practices at the Paris-based FFF which had been commissioned by the sports ministry.
"Considering his conduct towards women, his public comments and the governance failings of the FFF, Mr. Le Graet no longer has the necessary legitimacy to run and represent French football," the report said.
Le Graet had already agreed in January to step down pending the outcome of the audit, which concluded that he should not return to the role because his "behavioral excesses are incompatible with the carrying out of his functions".
It was revealed last month that Le Graet, whose mandate was due to run until 2024, was being investigated for sexual and psychological harassment following allegations made by a female football agent, Sonia Souid.
He was already under pressure after making dismissive remarks in a radio interview in early January about France legend Zinedine Zidane's potential interest in coaching the national team.
That was after long-serving coach Didier Deschamps had his contract extended until 2026.
"I wouldn't even have taken his call," Le Graet told RMC radio when asked about Zidane.
Despite his resignation, Le Graet will continue to work for FIFA's office in Paris.
Football's world governing body did not confirm that 81-year-old Le Graet, who has faced allegations of sexual and psychological harassment, would continue to occupy a role which is already among his duties.
But several members of the FFF's executive committee said after the meeting on Tuesday when Le Graet announced he was standing down from the federation after 11 years in charge that he would maintain his links with FIFA in the French capital.
Le Graet announced his departure at an executive committee meeting of the federation on Tuesday, with members of the committee telling AFP that Philippe Diallo would remain in the role of interim president.
Diallo, a federation vice-president who had initially moved up to fill in for Le Graet when he first stepped aside in January, is expected to remain in charge at least until a permanent successor is voted for in June.
"The FFF recognizes the remarkable sporting and economic performance of Noel Le Graet," the federation said in a statement, before hitting out at the authors of the audit.
"The report does not mention any systematic failing," it added. "The FFF nevertheless notes that the report is based less on objective facts than on comments that have sometimes led to exaggerated bad-mouthing of the body."
- Pressure on women's coach Diacre -
Le Graet, a former socialist mayor of the small Brittany town of Guingamp, oversaw the rise of the local football team that became a top-flight force during his time as club president.
He then became president of the FFF in 2011, at a time when the French game was reeling in the wake of the national team's disastrous showing at the 2010 World Cup, when the players went on strike.
Le Graet oversaw the appointment of Deschamps as coach in 2012, and France reached the Euro 2016 final as hosts before their back-to-back World Cup final appearances.
He also oversaw France's successful hosting of the 2019 Women's World Cup.
"He is a great leader. This federation is one of the best-run federations," insisted the veteran Lyon president Jean-Michel Aulas, an influential member of the FFF executive committee, on Tuesday.
However, Le Graet's departure also comes amid a deepening crisis within the France women's team.
Several star players, led by captain Wendie Renard, last week announced that they would no longer play for France under the current set-up, with just five months to go until the Women's World Cup in Australia and New Zealand.
That has placed the future of coach Corinne Diacre, who had the backing of Le Graet, in serious doubt.
Speaking on Tuesday, the federation's interim chief Diallo said a decision on Diacre's future would be taken "very soon."