The biggest unknown is what happens if/when an AGI is given (or obtains) full access to its own code, including any intended safeguards put in by humans.
In a neural network, the code itself isn’t sufficient to understand how it behaves. You need the parameter weights, which were developed through lots and lots of computation, presumably through resource-intensive processes with lots of training data and feedback and selection mechanisms.
So if AGI can be achieved through a particular hard coded architecture and the weights of trillions of parameters, what can an AGI do to perform brain surgery on itself? Like Borges’ Library of Babel thought experiment, the overwhelming majority of possible states will be broken, so any edits will have to be very careful and guided by extrinsic rules. Plus, the ability to edit the weights may form problems akin to biological cancer, dementia, hallucinations, other brain disorders.
Just as the human brain doesn’t understand everything about the human brain, it would be incorrect to assume that an AGI that can achieve both general intelligence and consciousness must necessarily have the ability to understand its own internal function, or modify itself in a way that improves things for itself. More likely, it is either programmed to (or learns through reinforcement learning and evolutionary mechanisms) that self modification is dangerous, and develops a very conservative approach to self preservation.
How dare you critique me, a rank amateur with mediocre knowledge about a few subjects? Especially with this reasoned argument?!
But yeah, my main takeaway is “the code must be complex in order to meaningfully analyze anything, and the more complex the thing it can analyze, the even-more-complex it must itself be.”
Yeah, with sufficient complexity it’s more along the lines of “I created a procedure that makes this complex thing” rather than “I built this complex thing up piece by piece.”
So if the act of creation is considerably less complex than analyzing and understanding a part of that creation, it’s far more likely that the complexity gap ends up preventing any self-aware AGI from being able to effectively reinvent oneself, even if it does have full write access to the components.
In a neural network, the code itself isn’t sufficient to understand how it behaves. You need the parameter weights, which were developed through lots and lots of computation, presumably through resource-intensive processes with lots of training data and feedback and selection mechanisms.
So if AGI can be achieved through a particular hard coded architecture and the weights of trillions of parameters, what can an AGI do to perform brain surgery on itself? Like Borges’ Library of Babel thought experiment, the overwhelming majority of possible states will be broken, so any edits will have to be very careful and guided by extrinsic rules. Plus, the ability to edit the weights may form problems akin to biological cancer, dementia, hallucinations, other brain disorders.
Just as the human brain doesn’t understand everything about the human brain, it would be incorrect to assume that an AGI that can achieve both general intelligence and consciousness must necessarily have the ability to understand its own internal function, or modify itself in a way that improves things for itself. More likely, it is either programmed to (or learns through reinforcement learning and evolutionary mechanisms) that self modification is dangerous, and develops a very conservative approach to self preservation.
How dare you critique me, a rank amateur with mediocre knowledge about a few subjects? Especially with this reasoned argument?!
But yeah, my main takeaway is “the code must be complex in order to meaningfully analyze anything, and the more complex the thing it can analyze, the even-more-complex it must itself be.”
Good stuff!
Yeah, with sufficient complexity it’s more along the lines of “I created a procedure that makes this complex thing” rather than “I built this complex thing up piece by piece.”
So if the act of creation is considerably less complex than analyzing and understanding a part of that creation, it’s far more likely that the complexity gap ends up preventing any self-aware AGI from being able to effectively reinvent oneself, even if it does have full write access to the components.