I think regulations can work when they enforce very well-supported, long-established best practices, such that if you don't do them it amounts to negligence.

They might also work better if they say “you can't do it that way, which is known to be unsafe,” as opposed to “you must do it this way, which is the only safe thing.”

Note also that regulatory standards are not the only mechanism in the law to create safety. Liability law can be very effective at creating safety, by giving the right incentives to the right parties, but liability law doesn't tell anyone what to do—only what will happen to you if you cause harm.

Companies are very good at making slight changes to bad ideas and calling them "new ideas". The plastics industry runs on this.

Yeah, this "we can't build anymore" shit is _often_ just people complaining about not being able to externalize costs that they don't want to pay for.

There must be ways to write regulations in meaningful ways. Like a train bridge must meet specification a, b and c and pass tests d, e and f, instead of saying it must be made of wrought iron.

Agreed, it's a fine anecdote but easy to draw the wrong conclusion of "rules hamper progress and counterproductive". I think the main challenge is to attract the right talent to the rule-making body.

This is the exact direction I think we should reshape political systems. Perhaps even make empowering experts the main point. Too much is legislated and enforced by people who are overly self-interested or out of touch. It's a huge missed opportunity that experts are forced to watch representatives fumble around or make wrong decisions.

A long those lines, governance power should not require being a celebrity either.

Applying this post to the topic of AI alignment, I'd like a democratic option for large entitlements of tax funds to be applied as Yudkowsky sees fit, if enough people vote to appoint him as AI Minister, and vote to give him governance teeth against big tech and the thousands of startups driving Moloch via the standard economic paradigm.

You have that option. It’s called have Yudkowsky run for office.

Most public offices he could run for would put him in a generalist position, with various public facing and non-technical aspects that he is not suited to or qualified to handle. I want him to be able to completely and wholly run the show for just his one narrow specialty.

This isn't an option at all because our institutions don't adapt to the territory we often find ourselves in. We have a political alignment problem that exacerbates the technical alignment problems.

I would like to have something like the concept of senators, but not as location-based representatives, who are strictly assigned nothing more than to lead senate committees related to issues that we separately apportion funds and power to via direct democracy.