Despite Nigeria having the better of the chances, the European Champions — reduced to 10 players after 87 minutes — snuck home in front of 50,000 spectators after the match had finished 0-0 at the end of extra time.
England held on after playmaker Lauren James received a red card for a needless stamp and will know they have to improve if they want to add the World Cup to their European crown.
They will face Colombia or Jamaica on Saturday for a place in the semifinals.
England got off to a terrible start in the penalty shootout when Georgia Stanway fired wide, but Desire Oparanozie couldn't take advantage and missed with an almost identical effort.
Beth England made no mistake, but Nigeria's Michelle Alozie shot way over top to give England an advantage they never surrendered, sealing the win when Chloe Kelly calmly converted.
"The first thing is we really stuck together as a team," said England coach Sarina Wiegman. "It was a very, very hard game, then we had a red card, but we made it through extra-time and then won on penalties. How we did that, and how the team just kept going, I'm so proud of them."
The Lionesses, who last year won the European Championship at a packed Wembley Stadium in London, were expected to win comfortably against the world's 40th ranked team.
The Nigerians, however, have shown in this World Cup that they have the game to challenge anybody.
They stunned hosts Australia in the group stage, taking advantage of their speed in the transition to score a 3-2 win.
And they employed the same tactics against England, whose back three were regularly exposed by the Nigerian attack.
Nigeria 'unlucky'
"They're very organized, very transitional, very physical and that's exactly what we saw," said Wiegman of Nigeria. I know you "But we made it through and I’m really happy."
Nigeria coach Randy Waldrum was proud of his side.
"To come in here and play the way they did tonight, I though we were every bit as good (as England)," he said. "We had every opportunity to get the result, and unfortunately we didn't get it done. Obviously, we didn't execute on our penalties, and when it gets to that point, it's anyone's game."
"We had the best chances, we hit the crossbar twice in regulation play. We were a bit unlucky not to get something out of it. But credit to England, they are a great side."
England started the match strongly and rattled Nigeria in the opening skirmishes as the Lionesses found plenty of space out wide.
They also controlled possession, completing 173 passes in the first half to Nigeria's 83.
But it was the Nigerians who looked the most dangerous, and they gave the European champions a real fright on a number of occasions.
England's forward #09 Rachel Daly (L) and Nigeria's midfielder #18 Halimatu Ayinde compete for the ball during the Australia and New Zealand 2023 Women's World Cup round of 16 football match between England and Nigeria at Brisbane Stadium in Brisbane on August 7, 2023.
England-born Ashleigh Plumptre had two golden chances to open the scoring, the first coming in the 16th minute when a strike from outside the box crashed into the crossbar.
A minute later she forced a fine diving save from Mary Earps.
The Nigerians had eight shots at goal in the first half, three more than England, whose best chance fell to Alessia Russo in the 23rd minute.
The second half followed the same pattern, but England had a golden chance to seal the match with 15 minutes left only for Chiamaka Nnadozie to pull off a fine save from Rachel Daly's powerful header.
In extra-time, Nigeria again looked most likely, Michele Alozie and Oshoala both going close, but neither team could break the deadlock.