The Proteas chased down 230 with 11 balls and six wickets to spare as Duminy and captain AB de Villiers combined for a match-winning partnership of 139.
Luke Ronchi's stubborn 99 helped the hosts recover from 68-5 to post a competitive total in Mount Maunganui.
However, with victory in the three-match series in sight at the same venue on Friday, Duminy believes the tourists can improve further on a strong win.
"It was a really good bowling performance," Duminy said of the first ODI.
"If we are really hard on ourselves in any way, we could be better towards the backend, but that is a work in progress.
"We haven't played together in five weeks so I guess it's about finding out the areas where we can improve and that is probably the one area.
"The batting went according to plan but in an ideal world you probably want to finish the match with only two or three wickets down.
I'm obviously quite happy with the partnership that AB and I put together. If we had lost another wicket the game would have been in the balance."
Though confident of seeing an improvement, Duminy is wary that a backs-to-the-wall scenario could play into the hands of New Zealand captain Brendon McCullum.
He explained: "It's a tick in their box, the way they go about their business in terms of their aggressive nature in the field.
"As a batter you always have that intimidated feeling if fielders are in your face and I think that is what he [McCullum] is trying to employ there."
McCullum will once again be without key batsman Kane Williamson, who has now been ruled out fo the remainder of the series due to a wrist injury.
Despite New Zealand looking rusty in their first 50-over contest since January, captain McCullum was pleased with his side's "fighting qualities" in defeat.
"There were some fighting qualities there but unfortunately the class of AB and JP towards the end meant we couldn't win," he said.
McCullum also indicated that left-armer Mitchell McClenaghan could be drafted into the hosts' line-up, while James Neesham is expected to retain his spot at the top of the order.
"I thought he looked pretty good today considering a lot of other people struggled to time the ball," McCullum said.
"He showed some good intent, some skills that he's got against quick bowling and that he is not out of his depth in that position."