The former Milan striker has made an impressive start to his coaching career but his side had to withstand plenty of pressure in a match that saw both teams finish with 10 men.
Giacomo Bonaventura opened the scoring on his Milan debut before Antonio Cassano levelled with a goal against his old club.
Keisuke Honda and a Jeremy Menez penalty put Milan back in the driving seat before the break but they were pegged back early in the second half as Felipe fired home from close range.
Daniele Bonera's sending off just before the hour gave Parma a way back into the match but it was Milan who struck next through Nigel de Jong.
Remarkably, there was further drama to come as Alessandro Lucarelli netted for the home side, Felipe was given his marching orders and Menez scored again to make it 5-3.
And to cap it all off, Mattia De Sciglio scored a farcical own goal from around 30 yards out as Diego Lopez failed to control the defender's backpass.
All of which leaves Roberto Donadoni's men still looking for their first point.
The visitors, who were without new signing Fernando Torres due to a sprained ankle, made a positive start to the game with Ignazio Abate and Honda causing problems down the right.
And they got their rewards when Bonaventura converted from Honda's pass in the 25th minute, allowing the ball to go through his own legs before firing back across Antonio Mirante.
Milan's lead lasted just two minutes, though, as Cassano came back to haunt his former club.
The 32-year-old striker ghosted into the area to meet a left-wing cross from Cristobal Jorquera and head beyond Lopez from six yards.
Inzaghi's men regained the lead in the 37th minute as Abate once again made a nuisance of himself, this time delivering perfectly for Honda to nod in his second goal of the campaign.
And in first-half stoppage time the visitors netted a third goal after being awarded a penalty for a foul by Lucarelli on Menez.
Having dusted himself down the France international duly converted the spot-kick to make Inzaghi's half-time team talk much easier.
Parma came out with renewed vigour after the break and were rewarded on 51 minutes when Felipe met a skewed Jorquera shot and volleyed the ball home at the back post.
Seven minutes later Milan's task became tougher with the dismissal of Bonera having picked up a second yellow card for blocking a Jorquera shot with his hand.
Rather than buckle under the pressure Milan came roaring back through De Jong, the Netherlands international racing clear to calmly slot past Mirante.
Jorquera picked up his third assist of the day in the 73rd minute as the hosts refused to lie down, this time Lucarelli rising unmarked to power a header beyond Lopez from a right-wing corner.
However, Parma's momentum disappeared when Felipe received his second yellow after catching Menez late with 77 minutes gone.
The Frenchman made him pay two minutes later as his cheeky back heel added to the glut of goals.
With full-time approaching the game had its ninth and final goal, De Sciglio's pass somehow eluding his goalkeeper, who appeared to pull a muscle as he let the ball skid past him and into the net.