The truth is coming back to haunt Matt Lauer. For years, rumors that he was cheating on his wife, Annette, seemed to permeate the halls of 30 Rock, with whispers that he was secretly courting women behind his wife’s back. And now, in the wake of news that he has been terminated from NBC following allegations of sexual assault, there’s one secret that he fears will destroy him: Some NBC insiders say that he secretly fathered two children with two former co-workers.

“All of Matt’s secrets are coming back to haunt him, and the most damamging is one that he’s terrified of facing: that he has children out there with other women,” an NBC insider tells In Touch exclusively. “The thought of having to do DNA tests to prove paternity scares him to death. He stands to lose everything.”

Matt — who has three kids, Jack, 16, Romy, 14, and Thijs, 11, with his wife of nearly 20 years — is already doing damage control. A rep for the disgraced host tells In Touch, “This is completely and provably false.”

And saying he has a lot to lose is an understatement; it has already been reported that he won’t get the balance of his $25 million yearly salary following his dismissal, and it seems unlikely his career will ever recover from this hit. “He’s lost his cachet as a journalist, and that pains him greatly,” the insider says.

It would appear his marriage is taking a hit too. A source tells Life & Style that Annette is telling her friends that divorce is inevitable at this point. “She’s furious with Matt for leaving this legacy for their kids,” the source explains. “Matt let her down, yes, but their children [having to deal with it] — that is what makes her especially angry.”