“An Intelligence Explosion”

      A few months ago, six highly respected individuals in the tech world, among them Scott Alexander, published a detailed prediction on the future of artificial intelligence. It is entitled, AI 2027.

      Among the most dramatic of their prognostications is the arrival in 2027 of what they refer to as “an intelligence explosion.”

      That is much sooner than expected—cheery news. The scary news in the AI 2027 report, of course, is that all that intelligence will be gained by computers, and we might not be intelligent enough to make sure it helps humans.

The authors of that report end up presenting two different scenarios for the near future: one ends with a kind of utopia, the other with mass death.

      The term “intelligence explosion” was coined in 1965 by I. J. Good, a mathematician born into a Polish Jewish family in England, who had participated in Alan Turing’s successful effort to crack the German code during World War II. Good was the computer expert Stanley Kubrick relied upon while making 2001: A Space Odyssey.

        Here is what I.J. Good wrote in 1965:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion,” and the intelligence of man would be left far behind... Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.

       Sixty years after I.J. Good wrote that, the AI 2027 effort was organized to predict when that “information explosion” might actually happen and how “docile” it might be. The organizer of AI 2027, Daniel Kokotajlo, had made a similar prediction about AI developments in 2021: “What 2026 Looks Like.” It has proven pretty damn accurate. After he released his previous attempt to peek into the future, Kokotajlo was hired by OpenAI.

      Another of the participants in this new effort at prediction was the renowned blogger on matters tech and intellectual, Scott Alexander.

      And they conclude that I.J. Good’s “intelligence explosion”—essentially super-smart computers writing the code to design even super-smarter computers—is nigh: “In 2027, coding agents will finally be good enough to substantially boost AI R&D itself, causing an intelligence explosion that plows through the human level sometime in mid-2027 and reaches superintelligence by early 2028,” Alexander writes, in a summary of their report on his “Astral Codex Ten” blog.

      Computers will not only be able to beat us in chess and write novels, but will become smarter then us about everything, then smarter still, then even smarter. That’s superintelligence.

      This, of course, raises I.J. Good’s good question of whether such ever-more intelligent computers will remain “docile enough” to play nicely with us humans—or whether, our computer overlords will have no use for us silly humans and, consequently, as Eliezer Yudkowsky, a thoroughly alarmed computer wiz, puts it: “We’re all gonna die.”

      This is where I, as a media historian whose digital understandings do not extend beyond Microsoft Word and Adobe Premiere Pro, might actually have something to add.

      First, I will note that the Faustian fear that great gifts exact terrible costs is an old fear. The technologies that inspired it go back at least to agriculture. That’s why various holy books have their protagonists leaving cultivated areas and returning to the “wilderness” in search of a spiritual boost.

      The notion that new inventions can, Frankenstein-like, escape human control, also seems elemental. Maybe it, too, is a nightmare from which we ought to wake up.

      The authors of the AI 2027 report do briefly allude to one earlier period of rapid technological innovation: the Industrial Revolution. But they don’t mention the two human inventions that came closest to setting off, in their times, “intelligence explosions”: writing and the printing press.

      But, wow, did they explode and elevate human understandings:

      Writing brought calculations of eclipses. It brought Sophocles. It brought Plato and Aristotle. It introduced to the world much of what we would call today “intelligence.”

     And writing also brought—in the role of Yudkowsky—Socrates, who certainly was literate but did not write anything, and whom Plato, who wrote quite a bit, quotes as insisting that writing provides only a “semblance” of “wisdom,” not “truth,” because the written word comes without “a teacher’s instruction.”

      A piece of writing, in other words, has gained independence from “a teacher,” from even the person who wrote it, from a human. Does that not seem, in its way, a Yudkowsky-like fear?

      The printing press brought newspapers and novels and accurate maps and Newton and the Encyclopédie and an explosive growth in the amount, availability and level of human knowledge.    

      The printing press also brought Tolstoy, who might have made as good use of the printing press as anyone else ever, but who also insisted, near the end of War and Peace, that printing is “ignorance’s weapon.”

      And again, we have the Yudkowsky-like nightmare of a force that is up to no good—ignorance, for Tolstoy—taking control of a powerful and relatively new invention.

      It should have been pretty clear that the AI 2027 report was going to be sensitive to such fears about AI. In particular that report would be expected to air the fear that AI—or the much more powerful AGI, artificial general intelligence, that will follow—will not be properly “aligned” with humanity’s needs and goals. For Daniel Kokotajlo, who headed the AI 2027 team, had left OpenAI and accused the company of “recklessly racing” to build artificial general intelligence.

     The AI 2027 report ends up finessing the question of whether the intelligence explosion they see rapidly approaching will be a force for much good or for much ill: They simply give their prediction two different endings.       In scenario #1, entitled “Slowdown,” the company leading in developing AGI in the United States is ordered by the government to take a step back and, despite competition from China, shut down its most advanced model and suspend progress until some safety measures can be devised and installed and AGI can become better aligned with human needs and values. Those safety measures include insisting that the hordes of iterations of its AGI models communicate with each other in English, not some unintelligible computer code, so humans can monitor what they are up to.

      The changes work in this scenario. China eventually comes to an agreement with the US on AGI. And AGI basically takes over—and frees humans from—all the work that needs to be done on the planet. Yet everyone gets a good-sized guaranteed income and, with the help of AGI advisers, figures out what they want to do with their lives.

      In scenario #2, entitled “Race,” AGI development is allowed to proceed without a pause for better “alignment.” Continued competition with China encourages haste. And, the upshot is, that in mid-2030, the AGI—”well beyond human control—releases a dozen quiet-spreading biological weapons in major cities, lets them silently infect almost everyone, then triggers them with a chemical spray. Most are dead within hours . . .”

      So it’s pretty clear that the authors of AI 2027 have some Socrates-like, Tolstoy-like fears about AGI.

      Myself, I think we ought to keep in mind that in their pessimism about the new technologies of their time Socrates and Tolstoy proved wrong. Information explosions have, so far, been good things. I would expect the same from AI, even AGI.