Spoiler!
He's very innocent and that could come off as stupid but he didn't want to have s*x with the neighbor. Taeju called him loose so he got insecure and asked the neighbor to check if he's really loose. He thought that was the reason why Taeju stopped visiting as often as he usually does.