You're drifting and you don't know it

The more I dig into AI, the more I find so many wonderful metaphors for thinking about human thinking. One of these is something called context drift.

For those who aren't familiar, each LLM has something called a context window, which is a certain number of tokens (a token is a certain way that the data is split up and rendered as it's passed into the LLM). In any given session, the AI can only handle a certain amount of tokens, i.e. a certain amount of context for that conversation. I guess you might think of this as akin to working memory or short-term memory in a human being. What can we keep in mind and keep relevant to us whilst we are performing a task?

So obviously this presents a problem for longer sessions with an LLM. It used to be that you would run to the end of this context window and then you would have to start a new conversation. This is probably how many people have experienced talking to ChatGPT or Claude over very long conversations. At some point the model will tell you that the conversation's become too long and you need to start anew. With the advent of the more premium models like Sonnet and Opus, and with the advent of coding agents that can run autonomously for hours, the question has arisen—how do you carry context over from one token window to the next, such that the model is able to continuously execute on a task over multiple iterations of a context window.

There are various solutions to these problems and they're not really relevant to go into here, but suffice it to say that with the advent of these solutions has arisen a new problem and that is something called “context drift”. This is where by the third or fourth context window that you're on with this agent, you'll find that it starts to drift—again, depending on your compaction strategy and the way that you preserve the memories through the different context windows, it'll either drift a little bit or a lot, but drift it will—from the original intent of the conversation or task that you started with.

This is because the way that you preserve memory from one context window to the other one is summarisation. And the choice is always what to summarize and what to leave out, what details are important and what details aren't important, what can survive a synthesis and what cannot. And ultimately, over these two, three, four context window summarisations and compactions, you end up getting something akin to Chinese whispers where the summary ends up becoming a summary of a summary of a summary of a summary and the original context gets lost.

Fascinatingly, human memory works a lot like this too.

We obviously don't encode our experiences and our knowledge with 100% fidelity to the source. To begin with, the knowledge is filtered through our pre-existing personality, knowledge, and ability to understand. You might think of this as akin to the quality of the model that you're using—some people use a cheap or free model and they get output that's commensurate with that level. Just so, some people's minds and brains are not trained well in the arts of thought or learning or knowledge growth. To begin with, therefore, the context that they're fed is not fully understood, is not fully rendered, is not fully translated into a high fidelity model in their minds.

And even when you're dealing with a very high-powered model—and this is true with AI, and it's true with human beings—you have certain preconceived patterns of thought or knowledge—schemas we might call them—that whatever it is that our learning gets passed through, which also acts to warp and distort the original knowledge. The knowledge might not fit 100% to those schemas, but because that's the way that the mind works, it tries to relate to new things by old things that it understands. The input to begin with might not be a 100% accurate representation of the source material.

And then of course you get to the idea that memory, whilst theoretically infinite in storage, is very lossy in retrieval. And the difficulty is—again, both with the models and with the humans—that that lossiness is not fully recognized or understood, and many times is even completely unconsciously ignored.

So, with humans, just like with the AI, you get context drift. Either you'll misunderstand the thing that you were trying to do, or you'll lose sight of the original goal, or you'll get distracted by a shiny object, or you'll develop an unconscious affinity for part of the knowledge or the project that you're working on and leave other parts of the project out. And unfortunately in school, we are not taught patterns of thought. We are not taught systems of accurate and rigorous thinking over time, of summarisation, of memory, of retrieval.

And so you end up getting in AI what we call “hallucinations” where the model, and human beings do this too, will confidently state something as fact or as true and it will be completely false. It won't even have any bearing on reality, but because they've lost the thread and don't know that they've lost the thread, it's said with a degree of confidence that most people just take as a given.

And then the question becomes, and this is literally the million or even billion dollar question, how to structure memory in a way that prevents context drift? How to encode information in a way that's as high fidelity as possible, how to keep the models and ourselves on track towards the original intended goal?

And there's one thing to me that comes out of this whole thing which is, funnily enough, strikingly human. These agents do better with shorter sessions, with fewer compactions and fewer summarisations. There's less context-drift when there's not as much distance between the execution and the original intent. Ephemeral containers and conditions may be a feature, not a bug. Anthropic's own guidance (the company behind Claude.ai) suggests fresh context and filesystem discovery outperforms session resumption.

And so with humans. We're not built to work uninterrupted on one task for many hours. We need breaks, we need refreshment, we need reminding, we need to come back to our north star, to remember again and call back into memory what it was that we were trying to achieve in the first place.

How ironic that the age of AI may help us to become more human.

(If this one hit home and you want to talk about where context drift might be showing up in your own work, reply to this email.)