The International Atomic Energy Agency says Director General Yukiya Amano died last Thursday, aged 72.
The global nuclear watchdog announced Amano's death on Monday. Sources say he was in poor health since last year and had returned to Japan frequently.
He was reportedly preparing to notify the IAEA Board of Governors that he was stepping down.
Amano became the first Japanese to lead the IAEA in December 2009 by emphasizing he was from the only country in the world to have suffered atomic bombings.
He also stressed that Japan had promoted peaceful use of nuclear power.
He oversaw measures agreed in the 2015 nuclear deal with Iran to limit the country's nuclear development, including inspection of its nuclear facilities.
US National Security Advisor John Bolton paid tribute to Amano, saying his commitment to nuclear nonproliferation and championing of peaceful nuclear energy were unparalleled in the decade he led the IAEA.
Iran's deputy foreign minister Abbas Araqchi also expressed his condolences, and commended his skillful and professional performance as the director general.
UN Secretary-General Antonio Guterres tweeted that Amano's remarkable contributions to the peaceful use of nuclear technology and his commitment to multilateralism would be remembered by all of those who had the privilege of knowing him.
Amano's third term as IAEA chief was due to run until 2021.
The agency plans to decide in coming days how to elect his successor.