Merkel, 64, had previously indicated that she planned to seek another two-year term as leader of her Christian Democratic Union at a December party congress, but appeared to be moving quickly to acknowledge pressure for renewal as her fourth-term government struggles to gain traction.
She said Monday that it's "time to start a new chapter". The CDU is due to elect a new party chair at the conference in December.
Merkel's preferred successor as leader of her conservatives, CDU Secretary General Annegret Kramp-Karrenbauer, has also warned the SPD that pulling out of the ruling coalition after the Hesse vote would trigger a federal election.
Hesse's conservative governor, Volker Bouffier, complains that "the election campaign has been completely overshadowed by Berlin".
This weekend's election was a second blow to Merkel's fragile "grand coalition" government.
During that momentous upheaval, Merkel, who holds a doctorate in quantum chemistry and was working in a lab, joined a pro-democracy group that would merge with the Christian Democrats of chancellor Helmut Kohl.
Annegret Kramp-Karrenbauer, secretary general of the CDU, is seen October 21 in Berlin during the election campaign.
She has also ruled out running for a fifth term as chancellor in 2021.
But by giving up the CDU leadership she has effectively put an end date on her time in power, as the party will expect to fight the next election under its new leader.
During a press conference Monday in Berlin, Merkel announced she will not be seeking reelection as leader of Germany's Christian Democratic Party.
Either way, Merkel stepping down will mark a major transition for a continent she has shaped for the past 13 years with her handling of multiple debt crises, her decisions on nuclear energy and, most of all, her fateful choice to allow more than one million asylum-seekers to enter Germany.
Merkel, her party and its allies in a governing coalition have been under attack by far-right politicians, who have been able to translate tensions over Germany receiving refugees from Syria, Iraq and other countries into election wins and a growing base of support.
Merkel's first step towards the exit door is likely to send ripples across the European Union, where she has served a beacon of stability as bloc grapples with multiple global crises, Brexit and an unpredictable ally in the White House.
Railing against the newcomers, the far-right anti-immigrant AfD is now the biggest opposition party in the Bundestag, and has seats in all of Germany's state parliaments.
The parties also had a poor result in Bavarian elections earlier this month.
When Kohl became embroiled in a campaign finance scandal in 1999, Merkel openly urged her party to drop the self-declared "old warhorse".
The Frankfurter Allgemeine Zeitung said on Sunday it would be "a mistake" for Merkel to cling to power.