The head coach recorded his second competitive win since replacing Clarence Seedorf thanks to Jeremy Menez's brace and goals from Keisuke Honda, Nigel de Jong and debutant Giacomo Bonaventura.
However, it was tough for Milan as they continually allowed Parma back into the match, including a red card to Daniele Bonera and Mattia De Sciglio's comical own goal.
The latter saw the full-back send a 30-yard back pass towards Diego Lopez, but the goalkeeper misjudged the flight and allowed it to roll into his net to give Parma a late boost.
Milan held on and Inzaghi was delighted with the battling qualities they showed to stay top of the table.
"We must be very, very happy, as this is an extraordinary group," he told Sky Sport Italia. "Let's not forget where we are coming from.
"I saw the moves that we tried out in training used to score great goals, but I also saw that we conceded on every chance that we allowed. It was a very open game, but I think there can be few doubts that Milan deserved to win.
"At the final whistle I was thinking it would've been a shame to not win a game we were always in control of.
"This is the spirit of Milan. I don't know how far we can go, but I am very proud to be the coach of this team."
The victory ensures Milan are one of just three teams to have a 100 per cent record, along with Roma and Juventus, through two league matches.