Be clear about what you are taking on. Self-hosting a voice stack means owning turn-taking bugs at 2am, GPU capacity planning, and the model bill. That is a team, not a licence, and it is worth saying out loud before a compliance requirement makes the decision for you.
The hardest part is not the models. It is interruption handling, which the hosted platforms have spent years tuning and which you will not match in a sprint. An agent that talks over a caller reads as broken in a way a slightly slow one does not.
Also worth checking: several vendors elsewhere on this site will say they can discuss deployment options and mean a dedicated tenant in their own cloud. That is a different thing from on-premise, and if your requirement is real the distinction is the whole point.