Context Windows And Conversation History

We've previously covered how a Large Language Model (LLM) can predict the next word in a string of text and how you can control that prediction using a system prompt. Now, let's talk about a fundamental concept that underpins how we interact with them: context windows and history.

One really important thing to remember about large language models is that they are stateless. To explain what I mean by stateless, I'll use a C# analogy that should be familiar to many developers.

The Stateless Nature of LLMs

As you probably know, there are a couple of different patterns you can use when writing C# code. You can use object-oriented programming, or you can use a more functional approach. This distinction is the perfect way to understand how LLMs operate.

A C# Analogy: Object-Oriented vs. Functional Programming

Let's look at two different ways to write code that accomplishes the same task.

First, consider the object-oriented programming (OOP) approach. In OOP, you create instances of a class that capture some kind of internal state. You can then modify that state with public methods and properties.

Here, the MyObject class holds an internal state, a private field named _items. The GetLongest() method operates on this internal state without needing it to be passed in.

class MyObject
{
    string[] _items = new string[0];

    string GetLongest()
    {
        return _items.OrderByDescending(s => s.Length).FirstOrDefault();
    }
}

Now, let's look at another pattern that is becoming increasingly popular in modern C#: functional programming. In functional programming, you focus on using pure functions. These are functions that take an input and produce an output without any side effects. They don't reference external things or rely on stored memory like private member variables.

In this functional example, GetLongest is a pure function. Because it doesn't maintain any internal state, we must pass the array of strings directly into it every time we call it.

static class MyFunctions
{
    static string GetLongest(string[] items)
    {
        return items.OrderByDescending(s => s.Length).FirstOrDefault();
    }
}

So, what does this have to do with LLMs? Well, LLMs work like the functional example.

How LLMs Resemble Functional Programming

Everything you need the model to know about needs to get passed into it in every single call. They're stateless. In fact, it can be useful to think of a call to an LLM just like calling a pure function. Imagine a function called CompleteTextUsingLLM.

The input to this function would be an array representing all the messages that have come before. You have to send all of these messages every time you call the LLM, in every single request.

This means everything the LLM needs to know must be included:

• Your system prompts.
• Your tool definitions (which we'll cover later).
• Your entire conversation history.

All of this information needs to be passed into the LLM with every single request. You can't have a state that's managed inside the LLM itself. It's essentially a pure function that takes in a bunch of text (which could be JSON, tool responses, or anything else) and produces a new piece of text.

Whenever you're working with LLMs, think of them in this functional programming model. Think of them as stateless and atomic operations that happen on a long piece of text to generate a new piece of text off the end of it. We're not doing object-oriented programming with LLMs; we're doing functional programming.

The Role of an AI Agent in Managing State

When we come to build an agent later on, a lot of the work our agent does is actually about managing, summarizing, and keeping control of this conversation history. It also manages the tools and all the other variables and things we are putting inside that big piece of text.

A lot of the work an agent does is to manage that context. This ensures that when it's making LLM calls, it's sending everything off in the most efficient way that will give it the highest chance of success when it gets a response back.

Conclusion

The key takeaway is that LLMs are stateless. Just like a pure function in programming, they don't remember past interactions. Every call to an LLM is an independent event, and you must provide the entire context, including the full conversation history, with each request. Understanding this stateless nature is critical for building sophisticated and effective AI agents.

What's Next?

Now that we understand the stateless nature of LLMs and the importance of context, we can begin to explore how to build agents that effectively manage this context to create seamless and intelligent conversational experiences. Thanks for reading