It was the grandfather who did it not the father. All the children have different fathers but the same mother. The mother belongs to the Garth family that's why all the children have red eyes. Only Shuden have both blonde hair and red eyes. That's why he was taken and the other was considered trash.
Even after the youngest had died the grandfather fed Leo a type of poison that causes his eyes to turn color because he was disgusting that his family blood runs through his veins.