If I had to guess based on what I've read and seen of their relationship so far, I'm guessing theirs was a political marriage, not a love match.
If they were bethrothed early in life, they likely would have spent a lot of time together (hence the emperor started calling her Noona) and grown to care for each other, but as close friends and not as lovers. My take is that the empress isn't mad because in her mind it's not "On, the man I love romantically betrayed me" but rather "My friend had a kid and now I'm going to support him in raising her"