Okay hear me out! I read the novel and I think the manhwa did him dirty here. In the novel he asked him politely to step out and he seemed more mad at the others for not noticing how shy shushu is. I recommend reading the novel because it's way more detailed!