Imagine being so selfish, you're oblivious to the fact you are the cause of people's suffering, lowkey stalking/chasing and abducting them from their peaceful lives, risking their lives in a capital where nobles are all out for blood, with everyone imposing themselves on them like this no-longer-brother went to see a kid without notice... Now wanting to restrain them there ? You're mad.
Mind you, you're also the dude who made the girl throw her youth away to get educated as the empress-to-be, only to throw her away (after having your fun, of course), causing her to be expelled from her family.
Ignorance really is bliss. Unless you're really a total piece of sh**, how would you cope with that ? On top of the rest, since based on that it's safe to assume he doesn't manage the empire so well either. (well, between the rebellions and epidemics we've seen, it's not really an assumption, is it)
This story would be really interesting if it wasn't shoujo manga, for which anything and everything is only a pretense to draw characters blushing with a flowery background, and nothing more than that.