Memory lost or even hurting noble, they asked him to fight while they held a real sword against his wooden sword. They started in private and public but get angry? He did not damage the nobility as one stated, that would be stupid, by witness account he just reacted. Meaning it is him vs their arrogance and pride nothing more.