Serge Gnabry and Niclas Fullkrug struck either side of a Kai Havertz brace for the European side, while Yeltsin Tejeda's goal and German goalkeeper Manuel Neuer's own goal had once put Costa Rica 2-1 ahead.
The result means Germany finished third in Group E, two points behind leaders Japan and trailing second-placed Spain on goal difference.
RELATED:
Japan Beat Spain To Win Group E on Dramatic Night in World Cup
It marks the second consecutive time that Germany has failed to reach the World Cup knockout stage following their dismal showing in Russia four years ago when they finished last in their group as defending champions.
Germany showed their attacking intent in the first minute at Al Bayt Stadium when Jamal Musiala's long-range effort forced a diving save from Costa Rica goalkeeper Keylor Navas.
Hansi Flick's men went ahead shortly after when Gnabry headed home after David Raum's cross from the left flank.
Costa Rica equalized just before the hour after Kendall Waston's header was directed straight at Neuer, who failed to keep the ball in his grasp. The ball fell kindly for Tejeda, who fired the rebound into the bottom corner.
The Central American side took the lead in the 70th minute through an own goal from Neuer.
Both teams had a slew of scoring chances as they surged forward in numbers in a bid to stay alive in the tournament.
Havertz put his team on level terms in the 73rd minute after running onto Fullkrug's pass and coolly chipping a shot over Navas.
Germany went agonizingly close to scoring again when Navas tipped Fullkrug's powerful effort over the bar.
The intensity was unrelenting and Germany restored their lead five minutes from time through Havertz, who bundled home from close range after Gnabry's cross.
Fullkrug then made amends for his earlier miss by latching onto Leroy Sane's pass and sending a shot beyond a flat-footed Navas.