Like AI, We Get Stuck on Old Models

      This is an excerpt from Ian Leslie’s Substack, where he lists—based on the ideas of Andrej Karpathy, a founder of OpenAI—four other “failure modes” shared by AI and humans.

A machine learning model gets trained on human data—photographs; Reddit forums; Amazon reviews; books and music and so on. We’re already at the stage where the big models have hoovered so much of this data into their maws that finding fresh sources to train on is hard.

      AI models have no problem producing data, however, so the obvious answer is to train on data from other models. Indeed, this is inevitable, as the internet becomes swamped by LLM-generated text and images.
      But there’s a problem with this. Model-generated data tends to be more predictable and less diverse than human data, as it imposes statistical patterns on the infinite variety of human outputs. Over time, this creates a feedback loop. Each new iteration of the model inherits the biases and errors of the previous one, but with less variety and signal.

      Eventually, the model converges towards—or collapses into—the generic and repetitive. It loses accuracy, nuance, creativity until only a thin and colorless monoculture remains. This is “model collapse.”
      Damn those machines, erasing the incorrigible plurality of human minds! …

      And yet I think Karpathy is right when he points out this is also something we do to ourselves: “Humans collapse during their course of their lives.”

      We, too, learn patterns in the data of experience on which we then become over-dependent, favoring the model in our heads over the flux of reality. As we get older, we become rigid in our thinking, set in our ways. We talk to the same friends and use the same information sources. We return to the same old thoughts and say the same old stuff.
      We are also inviting model collapse at the cultural level, as art forms turn in on themselves and stop looking outwards for inspiration. Much new pop music sounds like a karaoke version of what came before it. Songwriters write for the streaming algorithms. Hollywood movies have become increasingly predictable and formulaic. Visual artists offer thin imitations of twentieth century innovators. This has been going for a long time, pre-dating the internet; post-modernism is pretty much the cultural theory of model collapse.
      How do you slow or prevent this kind of decay? You raise your quality control standards. The AI companies are filtering out AI-generated content and privileging human data. OpenAI is hiring experts in various domains to create content exclusively for their models. AI companies are also making efforts to make sure that rare or anomalous data—texts which don’t fit mainstream opinion or don’t follow standard formats and styles—doesn’t get excluded from the training. The aim is to filter out nonsense, like QAnon-style conspiracy theories, without eliminating everything except for the safe, conventional and generic.

      The equivalent for the rest of us is to devote more time and effort to the curation of our information diet, rather than passively hoovering up whatever we’re served. As the rest of the world is increasingly training itself on social media slop, there are increasing returns to reading great books. We should be seeking out sources of knowledge and insight which most people aren’t looking at.

      We should expose ourselves to novelty, privileging new art (or at least, art that is new to us), and people from outside our usual circles. We should look for contrarian perspectives from people who really know what they’re talking about.