The Croatian came through 6-7 (1-7) 7-6 (11-9) 7-6 (7-2) in a gruelling encounter that lasted two hours and 25 minutes and saw him rack up 29 aces.
Karlovic's reward for his victory is a second-round clash with Jurgen Melzer, who made a winning return to his home-town tournament by beating lucky loser Norbert Gombos 3-6 6-4 6-3.
Melzer, winner of the ATP 250 hard-court event in 2009 and 2010, missed last year's tournament due to a shoulder injury, but delighted the home fans on his comeback by displaying great resiliency to fight back against his Slovakian opponent.
There was disappointment for seeds Guillermo Garcia-Lopez and Dominic Thiem, however, as they fell at the first hurdle.
Seventh seed Garcia-Lopez's tournament lasted just an hour and nine minutes, the Spaniard losing 6-3 6-4 to Jan-Lennard Struff of Germany, while home hope and eighth seed Thiem was beaten 6-3 3-6 6-3 by 2013 finalist Robin Haase of Netherlands.
Viktor Troicki produced a hugely impressive showing to overcome Dominican Victor Estrella 6-0 6-3, the Serbian losing just five points on serve and knocking down 14 aces in a dominant performance.
Elsewhere, Benjamin Becker, Thomaz Bellucci and Vasek Pospisil emerged victorious.