Two goals in as many first-half minutes had given Newport a 2-0 lead at half-time, but they were stunned by a second-half fightback from Jim Bentley's side.
Kevin Ellison followed up a superb 25-yard free-kick in the 58th minute by pulling the hosts level with a header five minutes later.
Paul Mullin then headed home unmarked six minutes from time to seal a remarkable victory.
Wycombe Wanderers - tipped by many to struggle this season - picked up their second-successive win by beating Carlisle United 3-1 at Adams Park.
Aaron Pierre headed the home side in front, but Brad Potts provided an almost-instant reply in the form of an equalising volley.
Paul Hayes put Wycombe back in front from the penalty spot after Sean O'Hanlon was penalised for handball, and Peter Murphy converted the third in the 80th minute to make sure of the three points.
Carlisle's day went from bad to worse when Alex Marrow was dismissed for a second bookable offence.
The newly-promoted clubs - Luton Town and Cambridge United - were consigned to League Two defeat for the first time since their return to the division.
AFC Wimbledon saw off Luton courtesy of a Matt Tubbs strike just before the break, while a Chris Dunn own goal cost Cambridge in a 2-1 defeat to 10-man Portsmouth.
There are now five teams tied on six points at the summit, with Southend United, Burton Albion and Cheltenham Town joining Morecambe and Wycombe also winning their first two matches.
Southend picked up a 2-0 home victory over Stevenage, Burton were 2-1 winners over Dagenham and Redbridge, while Cheltenham beat Accrington Stanley by the same scoreline.
Hartlepool United and Oxford United are among six sides awaiting their first points.
Hartlepool suffered a 2-0 defeat at the hands of Bury, while Oxford were on the receiving end of a 2-1 loss against Mansfield Town.
Plymouth Argyle cruised to a 3-0 victory in the south-west derby against Exeter City, while Shrewsbury Town netted two late goals in a 2-1 win over Tranmere Rovers.
In the day's other fixture, York City and Northampton Town played out a 1-1 draw.