[Crossposted on Windows On Theory]
Throughout history, technological and scientific advances have had both good and ill effects, but their overall impact has been overwhelmingly positive. Thanks to scientific progress, most people on earth live longer, healthier, and better than they did centuries or even decades ago.
I believe that AI (including AGI and ASI) can do the same and be a positive force for humanity. I also believe that it is possible to solve the “technical alignment” problem and build AIs that follow the words and intent of our instructions and report faithfully on their actions and observations.
I will not defend these two claims here. However, even granting these optimistic premises, AI’s positive impact is not guaranteed. In this essay, I will:
1. Define what I mean by “solving” the technical alignment problem via training AIs to be faithful and obedient.
2. Discuss some of the potential risks that still remain even if we do solve this problem.
3. Talk about some scenarios that could make these risks more likely, and how do we avoid them.
In the next decade, AI progress will be extremely rapid, and such periods of sharp transition can be risky. What we — in industry, academia, and government— do in the coming years will matter a lot to ensure that AI’s benefits far outweigh its costs.
Faithful, obedient, superintelligent.
Currently, it is very challenging to train AIs to achieve tasks that are too hard for humans to supervise. I believe ...
Make this part of your paper trail
Save this paper to a shelf, write a review, and keep your own notes.