The father couldn't have his son in that type of situation again that's why he moved him with his mother permentaly. The boy didn't understand that and when he finally came to visit, seeing a boy in a sense replace him was like a slap in the face. But the boy didn't know that Clay wasn't a replacement son.