That's how I interpreted the oaths as well. But it's a big leap from oath of silence to her controlling all those soldiers and the grey hair girl with it. They said in an earlier chapter she used the oath and took control of a bunch of troops and then handed that control over to someone to be less of a burden.
Read Cursed Princess Club, it's finished and fantastic.