@DK00 this is exactly it. The goddess never gave him a curse, she just refused to give someone that she thought was ugly any power.
The entire thing about him being considered ugly is just based on the fact that the world he is currently in has extra high beauty standards.