This is probably a play on the Dames de voyage and femmes en caoutchouc. Realistic dolls have been around for a very long time. My guess is the wives left him because of the dolls, not because of his skin colour. After all, you could absolutely put up with having a blue husband. But a husband who travels around with all his s*x dolls would be incredibly awkward.