My guess is that the girls fall in love with the "hero" and were willing to do anything to stay with him, even if it meant destroying their own home and the mc in the other timeline found out, but at that point it was too late for anyone to believe him.