The Croatian goalscorer, 28, arrived at the Vicente Calderon in place of Costa, who departed for Chelsea.
Diego Simeone's men enjoyed stunning success in 2013-14, breaking Real madrid and Barcelona's duopoly in La Liga by claiming the title before falling agonisingly short of winning the UEFA Champions League.
Costa was a huge part of Atletico's meteoric rise, scoring 36 goals in all competitions last term - including 27 in the league.
However, Mandzukic was quick to stress he was not a Costa clone, after being unveiled in the Spanish capital on Thursday.
"Costa is Costa, and I'm Mario Mandzukic," the former Bayern Munich man said.
"Everyone has their own style, and I'm going to follow the instructions of my new boss, El Cholo Simeone."
Mandzukic added he was thrilled to be linking up with Simeone, claiming the pair share a likeness.
"I'm really pleased that El Cholo will be my new coach," Mandzukic said.
"I know his footballing past, he was a great player. He shows the same as a coach, he lives football 100 per cent. We are quite similar."