The vast majority of people using "end to end encrypted" messaging systems fail to verify the identity of their contacts. So those running the servers can fairly trivially MITM the messages. So in practice it does matter who controls the servers.
The good thing is that verifying the other contact is invisible to the server in Signal. This means that it's stochastically sufficient that a few people do check their contacts in order to see whether there is any widespread MITMing going on.
