The Olympic gold medallist had been scheduled to face Gael Monfils in the first round of the $1.28 million hardcourt event but the Frenchman pulled out with a knee injury.
Japan's Kei Nishikori survived a fright against countryman Go Soeda, the eighth seed squeaking through 4-6 6-2 6-3 to the relief of Tokyo organisers in two hours, 29 minutes.
Murray got a workout from 2.08 metres Karlovic and was forced into a tiebreak in the first set which the Scot took 9-7 with a superb running forehand pass down the line.
A quick change of wardrobe and Murray, now in all black, waited for his moment in the second as Karlovic slammed down 18 aces, converting his third match point with a vicious backhand to the Croatian's body after one hour, 38 minutes.
Nishikori, who reached his first semi-final of the year in Kuala Lumpur last week, next plays Spaniard Tommy Robredo, a 6-2 6-4 winner over Finland's Jarkko Nieminen.
Big serving Milos Raonic hit 10 aces in a 6-4 6-4 victory over Czech Radek Stepanek, sending the Canadian through to face Serbia's Viktor Troicki.
