In world, they are a frontier problem. You have to make city people who probably never will see a goblin in their lives to care....
Most times we see goblins in a major city, they are being led by another bigger treat.
Yeah goblins are nasty little $%#%$ but you have dark elfs, undead, crazy wizards towers, ogres, beholders and probably a lot more treats tan can puverise an entire city or a kingdom alone to worry about.
In the scope of things they are a little problem. That's why they get mostly ignored and are deal as rockies quest.
I mean, mostly, to a proper village, they are a nuisance and will only take some crops and smal animals.
To a frontier village trying to establish itself, they became a bigger treat.
and again, when they have the numbers to be a treat is rarely lead by goblins, mostly there is a bigger treat to lead them. because of their nature of betrayal and fight for power.