@Abiss to be fair the guy was basically forced to be with her since from the start the guy was brought into a different world with no way back home because of the saintess and because of that he got paired with someone he didn't love at the time, due to the fact that he also knew the problems if he didn't keep her, he had to keep her close to him at all times, so him acting like that in the past makes perfect sense in my opinion.
I'm serious Rocks Talk