Large Language Models
Over the last few months, the internet has practically gone mad over neural networks. Artists have buried their profession several times because of Midjourney, programmers have begun eyeing Copilot nervously, and journalists are once again telling us that artificial intelligence is just around the corner.
In my view, the excitement over neural networks overstates some capabilities and misses others. Language models are a form of artificial intelligence, but I would not equate them with a general human-like mind. Their basic training objective is to predict the next token from context, using patterns learned from large amounts of text.
A token may be a word, part of a word, or punctuation. This training objective alone does not settle what counts as understanding, thought or intention, and it does not demonstrate self-awareness. Models can solve some physics, mathematics and programming problems, but their explanations can still be inconsistent or wrong. I would assess them by the tasks they can reliably perform and by the errors they make.
It is simply incredibly good at guessing which piece of text should follow the previous one. And this is where things become interesting. Despite the simplicity of the basic principle, some very strange properties start emerging once a certain scale is reached.
The ability to generalise appears. Elements of logical reasoning appear. The ability to write code, explain concepts, and hold a conversation appears. In effect, quantity starts turning into quality.
And that is extremely intriguing. Over the past twenty years, I have seen an enormous number of technological hype cycles. Expert systems. The semantic web. Blockchain. Big Data. IoT.
Most promises eventually run into reality. But the situation with neural networks is somewhat different. For the first time in a very long while, we are seeing a technology that is actually beginning to automate intellectual work.
Still within very narrow limits. Still very clumsily. But it has begun. I would even say that programmers do not yet fully grasp the scale of what is happening. Most of our work is not about creating new algorithms.
We do not invent a new method for solving NP-complete problems every day. Most of the time, we:
- search for information;
- read documentation;
- write routine code;
- transform data;
- generate boilerplate;
- write SQL queries;
- refactor old code;
- move data between systems.
And this is exactly where large language models can be extremely useful. Not as a replacement for a programmer. But as an amplifier. An exoskeleton for the intellect. A compiler for human thoughts.
The problem is that most people are currently discussing entirely the wrong questions. Everyone is arguing about whether neural networks will replace artists or programmers. I think the question should be framed differently.
What happens when one engineer can do the work of a small team? What happens when the barrier to creating complex systems drops by an order of magnitude? What happens when generating code becomes cheaper than reading it?
The amount of software being created is already beginning to grow faster than people's ability to understand it. In a few years, the situation could become very interesting indeed.
We could see an avalanche of growing system complexity. The amount of code will grow faster than the number of engineers capable of maintaining it. Legacy code will become even more monstrous.
The number of vulnerabilities will increase. The amount of information garbage will increase. The amount of mediocre code will increase. Because producing code will become much easier.
I certainly do not rule out the possibility that, before long, a programmer's main skill will be understanding vast automatically generated systems rather than writing code. There is, however, an even more interesting aspect.
Large language models are effectively a new interface for interacting with information. For the last fifty years, people have had to learn the interfaces of machines. Command lines.
Programming languages. SQL. REST APIs. Query languages. Now, for the first time, the machine is beginning to learn human language. And this could turn out to be much more significant than it currently seems.
Perhaps in ten years, most interaction with computers will take place through dialogue rather than a GUI. Not because it is fashionable. But because natural language is a more efficient interface for an enormous range of tasks.
Of course, the models still make a huge number of mistakes. They hallucinate. They confidently invent facts. They confuse cause and effect. Sometimes they write outright nonsense. But we need to understand one simple thing.
We are now seeing roughly the same stage of development that search engines went through in the early 2000s. People laughed at them then, too. Then it turned out that they had changed the world.
I think most people today seriously underestimate the long-term consequences in particular. There is no guarantee that we will have true AI in five years. Most likely, we will not.
But intellectual work may well look completely different in ten or twenty years. And if that really happens, the question will no longer be whether a neural network will replace a programmer.
The question will be what remains an exclusively human activity at all.
