The royals denied there was any “bullying” behind the scenes, but In Touch has learned Prince Harry and Meghan Markle‘s decision to take a step back from their duties as part of the ruling family was directly related to the former Duke of Sussex’s rocky relationship with Prince William.

“He’s over living in William’s shadow,” an inside source exclusively revealed to In Touch. “The Queen stressed the importance of her family putting on a united front, and she made this very clear in her statement, but his relationship with William is still strained. The way things are going, it’s unlikely they’ll ever be close again. Like they used to be.”

In addition to being sick of all the “rules and regulations,” the redheaded royal wanted some distance from his brother — and his brother’s reputation. Instead, Harry, 35, and Meghan, 38, are looking forward to getting a “fresh start” as they settle into their new lives in Canada. Though a source previously told In Touch that William, 37, is “not interfering and is letting his brother get on with it,” that seems to come not from a place of benevolence but indifference. “William and Harry still don’t get on,” they said. “There’s 100 percent sibling rivalry between them.”

In December, an insider even told In Touch the older brother was “secretly relieved” the Sussex couple had left the European continent for Christmas. “William’s grown tired of Harry feeling sorry for himself,” they revealed. With duo out of the country, the father of three could focus less on keeping the peace and more on spending time with his own family.

But there may still be hope for the brothers’ relationship. Conflict resolution expert Damali Peterman gave In Touch some insight into how the duo could mend their rift. “There’s always something in the family,” Peterman said. “There’s always some sibling rivalry. There are typically some situations when you bring someone new to your family — that changes the dynamics. … Communication is key, because communication is at the heart of conflict in any family. If someone doesn’t like what someone said, what someone did, … the way they communicate that is impacted by it, whether they stop talking to each other or the conversation is strained.”

For access to all our exclusive celebrity videos and interviews – Subscribe on YouTube!