both the current emperor and the crown prince (aka future emperor) each have their own harem. The 4 girls that fell for MC are all from the crown prince's harem. Each concubine comes from a specific palace, and are usually the daughters of the clans in change of said palaces. The emperor is technically free to select any woman he wants to be his concubine and similarly he is technically free to relieve them of their duties as concubines and set them free. However just because the emperor technically can do that doesnt mean he would even if he doesnt have any interest in the harem. Why? Obviously, politics. For example, if an emperor were to show favor to a servant girl of no renown and make her his concubine, that can displease the nobility, and while they cant go against the emperor's orders directly, they may be provoked into secretly acting against the emperor and even try to overthrow him. Same goes for relieving a concubine of her duties on a whim rather than as a punishment or something like that. The nobles could consider this as the emperor disrespecting the noble clans. It doesn't matter if the man want's to give his concubine as a reward to a renowned general of whatever, that is still disrespectful towards the noble clans because the concubine is considered to be a part of the royal family while she's still within the harem. To relieve her of duty is cast her down from royalty back to nobility. Technically it's even worse because a noble clan wouldn't be interested in accepting a failed concubine back into the clan or the palace. It also severs the link between the clan and the royal family through which the clan could've influenced the emperor's choices, which means the clan isn't just disrespected, but also loses some of their political power as well.
Anyway, point is, concubines are concubines for life, or death (if they fu** up badly enough).