Having dominated his French opponent in the opening set of the ATP 1000 clay-court encounter, fourth seed Federer - making his first appearance since wife Mirka gave birth to a second set of twins earlier this month - struggled to build on his early lead.
A single break of serve was enough for Chardy to level matters with victory in the second set, before the third went down to a tie-breaker in the blustery Roman conditions.
And, after squandering a match point at 6-5, the Swiss lost his two subsequent serves to leave Chardy serving for one of the most memorable wins of his career.
Top seed Rafael Nadal edged past Gilles Simon 7-6 (7-1) 6-7 (4-7) 6-2 in Wednesday's final second-round encounter.
Nadal, a seven-time champion in Rome, traded two breaks with the Frenchman in set one before ruthlessly winning the match's first tie-break.
The world number one looked on course for a straight-sets victory when he broke early in the second and moved 3-1 up, but Simon battled back valiantly to force another tie-break, which he duly got the better of.
Nadal, fresh from his triumph in Madrid last week, set his stall out from the off in the decider by breaking in the very first game, but then quickly dropped his own serve to leave the score at 2-2.
A marathon game on the Simon serve ensued, with Nadal eventually taking it on his third break point to move 3-2 to the good.
The Spaniard then held serve twice, sandwiching a further break, to move into round three, where he will meet Mikhail Youzhny.
Britain's Andy Murray safely negotiated his way into the third round with a 6-2 7-5 win over Marcel Granollers.
After wasting two early break points in the opening set, Murray - forced to retire hurt at this stage against the same opponent last year - eventually made his mark by winning four games on the bounce to put himself in front.
Both players failed to hold their opening service games of the second set, but the Scot edged a closely-fought final game to avoid a tie-breaker.
Number 11 seed Jo-Wilfried Tsonga took both sets down to the wire as he triumphed 7-6 (16-14) 7-6 (7-5) over South Africa's Kevin Anderson, while Andrey Golubev retirement saw Russia's Youzhny through.
Elsewhere, Jurgen Melzer beat Marin Cilic to set up a third-round meeting with Murray, Tommy Haas will face Stanislas Wawrinka after overcoming Igor Sijsling, and Ivan Dodig meets Chardy thanks to a 6-1 6-2 demolition of Lukas Rosol.
Ernests Gulbis, Grigor Dimitrov and Tomas Berdych are also in the next round after wins over Stephane Robert, Ivo Karlovic and Dmitry Tursonov respectively.