An AGI Modifying Its Utility Function in Violation of the Strong Orthogonality Thesis
Bibliographic Data
| ID | 22134178 |
|---|---|
| Authors | James D Miller (0000-0002-2297-7870, Smith College, corresponding author), Roman V Yampolskiy (0000-0001-9637-1161, University of Louisville), Olle Häggström (Chalmers University of Technology) |
| Year | 2020 |
| Volume | 5 |
| Issue | 4 |
| Pages | 40 |
| Publication date | 2020-12-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Philosophies (JOURNAL) |
| Journal identifiers | ISSN: 2409-9287 • E-ISSN: 2409-9287 |
| Publisher | MDPI AG (PUBLISHER • IT) |
| DOI | 10.3390/philosophies5040040 |
| OpenAlex | W3107114626 |
| Language | EN |
| Citations received | 6 |
| References cited | 8 |
An artificial general intelligence (AGI) might have an instrumental drive to modify its utility function to improve its ability to cooperate, bargain, promise, threaten, and resist and engage in blackmail. Such an AGI would necessarily have a utility function that was at least partially observable and that was influenced by how other agents chose to interact with it. This instrumental drive would conflict with the strong orthogonality thesis since the modifications would be influenced by the AGI’s intelligence. AGIs in highly competitive environments might converge to having nearly the same utility function, one optimized to favorably influencing other agents through game theory. Nothing in our analysis weakens arguments concerning the risks of AGI
Economics · Epistemology · Mathematical economics · Nothing · Observable · Orthogonality · Physics · Quantum mechanics · Blockchain Technology Applications and Security · Computability, Logic, AI Algorithms · Computer Science · Mathematics · Philosophy · Psychology
| Unique citing works | 6 |
|---|---|
| Citations per year | 1.2 |
| Citation span | 2021 - 2026 (6) |
| Citation velocity | current |
| Highly cited | No |
| Citation types | Neutral: 5 |