I'm not sure either, maybe it's because the director is making decisions to "repay" him without considering his feelings, because seojin doesn't remember the past, and doesnt want to re-live it either, for him the boss is his family, he loves the director and wants to start a new page with him as seojin not lee junyoung.