I don't know, from experience Huf Haus is (while being very expensive) offering a level of quality (both in the construction and in the details/personalization) that I haven't seen in other EU firms, it is very possible that there are US firms at the same level, though from the little I have seen they are on par on the "other" EU ones (possibly good, but not exceptionally good).
If I recall correctly, Huf Haus is around since forever, before WW1, and they continuously evolved, their solutions are more than well tested, as "perfect" as they can be.
The great advantage of the "fully integrated pre-fab" (such as Huf House and similar) is that IF/WHEN the project is done, the result will be exactly what has been designed, the great disadvantage is that once the project has been approved, NO changes are possible (and again from experience it is a rare case that the customer/owner is capable of fully understand the project in all its details).
Back to the OP question the issue (generally speaking and in my experience) is about coordination of the various figures involved, typically here (EU, Italy) you will have several people:
1) land surveyor
2) architect (here the architect is more about design than building)
3) engineer (structural)
4) engineer (hydraulics)
5) engineer (electric)
6) general contractor or builder
7) various sub-contractors (electrician, plumber, etc.)
The common issue is that each of them (usually) have a somewhat "narrow" view, limited to their particular field, and often (please read as "always") they don't communicate properly among them and with the customer/owner.
The exception (as well usually) is, or should be, the general contractor, but an experienced enough one is unlikely to be available for a single house building or will dedicate to it too little time.
Mind you each of the figures I mentioned are in most cases very good at what they do, but, in the absence of someone who coordinates the project and execution, it is common that the result is not as good as it could be.
There are (were) people (shameless plug I am or was one of them) that while not specialists in anything have the experience and capability to coordinate these people and - also important - "translate" and "interpret" the desires of the customer/owner, the "project manager" and/or "site manager" that could be either paid by the general contractor or by the customer/owner.
Sometimes (not often) the architect (or the builder) has these capabilities, or is part of a largish firm that can provide such a resource for the project.
Since this has a cost, and building is generally a cost competitive field, this added cost is one of those that in recent years has been (generally speaking) either cut or reduced to the minimum (like giving - say - five or six building sites to the same person, which won't have the needed time to follow in detail each of them).
So, in a nutshell (and as always) it all depends on the people, you might be able to find a builder (usually small firms, almost familiar ones) that can do this or you might be able to find an architect studio (as well a smallish, but not too small one) that can provide you with this service, or maybe you can find and hire yourself such a manager/coordinator.
I agree with everything you said, this is why I think pre-fab is the way to go. It's like ordering a car, high industrial quality and few/no surprises. Other readers should keep in mind the pre-fabs we are talking have nothing to do with park homes. It's more like a Lego that is assembled onsite.
Regarding Huf Haus alternatives, there are a few German brands that are quite similar. Baufritz for instance is also old, and manufactures outstanding houses: https://www.baufritz.com/uk. The problem is that, unlike Huf Haus, they are not willing to do cheap (<€800k) projects. Huf can do smaller bungalows, which will be expensive per sqm but still less than a half out what Baufritz would be willing to consider.
For smaller stuff, in the UK there is https://wudl.co.uk, which builds pretty nice things. I also love https://www.echoliving.co.uk, although this is just a high-end cabin builder. Very different.
I'm not familiar with the American market, but I'm willing to bet there are some similar options.