Russia, has resigned.
Mr Rosenstein, who had a fraught working relationship with Mr Trump, will step down on 11 May in a departure that had been expected for months.
In his letter, he paid tribute to Mr Trump, even praising the president's "courtesy and humour".
Mr Trump once tweeted an image showing Mr Rosenstein jailed for treason.
Mr Rosenstein - who was originally appointed by Republican President George W Bush - had been expected to resign in March following the appointment of William Barr as attorney general.
But the justice department second-in-command stayed in the job a while longer to help Mr Barr manage a redacted public release of special counsel Robert Mueller's investigation.
The report did not determine that the Trump campaign had conspired with Russia to sway the 2016 election, and did not find sufficient evidence for an obstruction-of-justice case against the president.
What does Rosenstein's resignation letter say?
In his letter, Mr Rosenstein praises some of what he calls the Department of Justice's achievements and its employees' "devotion to duty".
"I am grateful to you for the opportunity to serve; for the courtesy and humour you often display in our personal conversations; and for the goals you set in your inaugural address: patriotism, unity, safety, education, and prosperity," he wrote, addressing the president directly.
"The Department of Justice pursues those goals while operating in accordance with the rule of law. The rule of law is the foundation of America. It secures our freedom, allows our citizens to flourish, and enables our nation to serve as a model of liberty and justice for all."
He goes on to say that "truth is not determined by opinion polls".
"We ignore fleeting distractions and focus our attention on the things that matter, because a republic that endures is not governed by the news cycle."
In his conclusion, he echoes one of Mr Trump's campaign slogans: "We keep the faith, we follow the rules, and we always put America first."
The White House said Mr Trump had already nominated Deputy Transportation Secretary Jeffrey Rosen to replace Mr Rosenstein.
A not-so-subtle rebuke
Rod Rosenstein is leaving the justice department, and he's doing it on his own terms.
Not too long ago, such a dignified exit would have come as a real surprise to most observers.
Donald Trump frequently seemed to view Mr Rosenstein as an internal threat. It got so bad that Mr Rosenstein reportedly expressed fear that he would be unceremoniously sacked via presidential tweet.
Rosenstein managed to navigate the hazardous terrain of the Trump administration, however, seeing the special counsel investigation that he initiated to its conclusion. It is that Russia inquiry, with its no-Trump-conspiracy conclusion and open question on presidential obstruction, that will be Mr Rosenstein's legacy.
Mr Rosenstein announced his departure with a six-paragraph mini-lecture on the importance of a justice department free of political influence and respect for the rule of law.
That could be viewed as a not-so-subtle rebuke of the president's near-constant badgering of justice department officials, although it will almost certainly fall on deaf ears.
Perhaps, however, it will allow the long-time government lawyer a measure of satisfaction, believing he had the last word.
In this administration, that is a not unremarkable achievement.
Why were Rosenstein and Trump at odds?
In May 2017, Mr Rosenstein was left in charge of appointing someone to oversee the Russia investigation after Mr Trump fired FBI Director James Comey.
Jeff Sessions, who was attorney general at the time, had already recused himself, meaning that his deputy had to take on the responsibility.
Mr Rosenstein surprised the White House by appointing Mr Mueller, a former FBI director.
The deputy attorney general spent the past two years overseeing Mr Mueller's work and defending the inquiry from attacks by Mr Trump, who continually condemned the probe as a "witch hunt".
Mr Rosenstein's departure seemed imminent after the New York Times last September reported that he had discussed ousting Mr Trump.
Citing anonymous sources, the newspaper said Mr Rosenstein had suggested secretly recording the president in order to prove he was dysfunctional.
He had supposedly cited the 25th amendment of the constitution, which allows for the removal of a president if deemed unfit for office.
Mr Rosenstein dismissed the claims as "inaccurate and factually incorrect", and a justice department source told the BBC at the time that the comment had been sarcastic.
But former acting FBI chief Andrew McCabe said in a TV interview that Mr Rosenstein was serious when he suggested wearing a wire inside the White House to covertly tape the president.
Mr Trump accused Mr Rosenstein on Twitter of "illegal and treasonous" activity.
While the deputy attorney general managed to weather that controversy, Mr Trump in November retweeted an image of the deputy attorney general behind prison bars.