Honesty hour. Prince Harry said being in England was “toxic” for him, wife Meghan Markle and their son, Archie.

“It was a really difficult environment as I think a lot of people saw,” the Duke of Sussex, 36, revealed during an appearance on The Late Late Show With James Corden on Thursday, February 25. “We all know what the British press can be like. It was destroying my mental health.” He continued, “This is toxic, so I did what any husband and what any father would do which is, ‘I need to get my family out of here,’ but we never walked away.”

Airing just days after it was announced the couple officially would not return as working royals, the prince made it clear that he will always be there for his family.

“It was never walking away, it was stepping back rather than stepping down,” he told James as the pair did a tour around Los Angeles in an open-air bus. “I’ll always be contributing. My life is public service, so wherever I am in the world it’s going to be the same thing.”

Prince Harry Meghan Markle Archie Photos
HENK KRUGER/POOL/EPA-EFE/Shutterstock

On Friday, February 19, Buckingham Palace confirmed that Harry and Meghan, who is pregnant with their second child, won’t be returning as working royals. The announcement came just over a year after the Sussexes first shared their plans to step down, which came with a one-year review period. As part of stepping down, they will no longer keep their patronages, their royal involvement with numerous U.K. charities.

“Following conversations with The Duke, The Queen has written confirming that in stepping away from the work of The Royal Family it is not possible to continue with the responsibilities and duties that come with a life of public service,” the palace statement said.

In response, Harry and Meghan pushed back with a statement that specifically referenced the Queen’s. “We can all live a life of service. Service is universal.”

Amid their rift, a source told Us Weekly that Queen Elizabeth has seemingly been putting on a positive front despite drama.

“It’s been a rough few weeks for the queen, that’s for sure,” the insider explained following husband Prince Philip‘s hospitalization. “The situation with Harry and Meghan has been challenging, and on top of that, she’s distraught over her beloved husband’s health setback.”

For access to all our exclusive celebrity videos and interviews – Subscribe on YouTube!