(The Center Square) – President Donald Trump denied reports Wednesday that he plans to fire Federal Reserve Chairman Jerome Powell, but said he wouldn’t rule it out completely.

“We’re not planning on doing anything,” Trump told reporters at the White House.

Later, the president added, “I don’t rule out anything, but I think it’s highly unlikely, unless he has to leave for fraud.”

Trump said Tuesday that overruns on a Federal Reserve office renovation project could lead to Powell’s ouster. On Wednesday, Trump dismissed media reports that he was gearing up to eliminate Powell.

Trump has wanted to fire the central banker for years, but hasn’t taken action.

On Tuesday, a top Wall Street banker warned Trump not to interfere with the central bank’s independence.

“Playing around with the Fed could have adverse consequences, the absolute opposite of what you might be hoping for,” JPMorgan Chase CEO Jamie Dimon said during an earnings call. “It is important that they be independent.”

While that warning came from Wall Street, the Supreme Court suggested in a May ruling that it wouldn’t let Trump boot Powell. The nation’s highest court said the president and the Federal Reserve’s relationship differed from other independent agencies.

“The Federal Reserve is a uniquely structured, quasi-private entity that follows in the distinct historical tradition of the First and Second Banks of the United States,” the high court wrote in an emergency ruling.

Trump has repeatedly pushed Powell to lower interest rates, but the Fed has kept rates flat.

Trump has nicknamed the Fed Chairman “Too Late Powell” and said he’s not smart. The president has even suggested appointing himself chairman of the Federal Reserve.

Trump has called for aggressive cuts. Powell and the Federal Open Market Committee, which sets rates, have taken a wait-and-see approach on interest rates, partly due to concerns about how Trump’s tariffs will affect the U.S. economy and the Fed’s mission for stable prices and maximum employment.