🍈 Zettelkasten

AGI Risk Case

1 min read

An argument of Existential Catastrophe arising from all intances of AGI

Argument

  1. If superhuman AI is built, any given system is likely to be goal oriented
    1. Goal directed behavior is likely to be valuable (economically, or otherwise)
    2. Goal directed entities arise from machine learning processes
  2. if goal directed superhuman AI is built, their output will be as bad as an empty universe by human right
    1. Hard to come up with goals that will be good for the AI, and ensure it doesn’t come up with its own goals later
  3. If most goal directed ASI will hav bad goals, the future will be very bad

Counter Arguments

  • AGI Is Not Goal Directed
  • Goal directed systems may not always be bad
    • Small differences in utility functions may not be catastrophic
    • Human and AI values may be similar
    • Maybe value isnt fragile
  • Human success isnt from individual intelligence
  • AI agents may not be radically superior to combinations of humans and non-agentic machines
  • Intelligence may not be an overwhelming advantage
  • Unclear if many goals incentive taking over the universe
  • Speed of intelligence growth is ambiguous
  • Argument proves too much about corporations

Graph View

Backlinks