Comment by Sera on Happy Together - Chapter 17

Why does it feel like he means late from past like,i am sure their past is connected and late from there