He didn't ignore nor neglect her. He did that to the original character, and it's an insult to the original character who suffered that neglect for FL or us to think of it as something she endured. From the first moment FL showed up in the world, he gave her everything she wanted (except the divorce) and supported her. The rest of your words is just as absurd, you're calling FL "inappropriate" to diminish why he likes her, call "overpower" the rumors he changed about her in order for her to keep the empress position by his side, etc. Literally how I called it from my first comment: you make up stuff about him just to make the whole pointless drama more logic.
Lastly I know that's yur opinion and everyone's opinions should be respected as they are, but it's one thing when someone forms an opinion based on the actual story, and another thing when they change the da** story by making up stuff. That can hardly be called an opinion. It's a fantasy or fanfiction at this point. Peace out.