What a Language Model Actually 'Sees' When It Reads

The model does not read left to right.

This is the first thing that surprises people, and the surprise is a good starting point because it means you've been imagining something that isn't happening. The reading you're picturing — sequential, one word at a time, meaning accumulating as you go — is what you do. It is not what the model does. The model reads everything at once. Every token simultaneously. It builds a web of relationships — dense, layered, bidirectional — between every element and every other element in the sequence. Before the first prediction, it has already considered all of them together.

This is computationally expensive and architecturally strange. It produces something that behaves nothing like sequential reading but performs better at language understanding than anything that did read sequentially. The strangeness is the point. The transformer architecture abandoned the intuitive approach and built something inhuman, and that inhumanity is exactly where its power comes from.

Tokens, Not Words

The model doesn't see words. It sees tokens, and a token is not a word in any linguistically meaningful sense. A token is a chunk of characters — sometimes a whole word, sometimes a morpheme, sometimes a syllable fragment — determined by a vocabulary learned during preprocessing on a large text corpus.

Subword tokenization exists because of a practical constraint: you need a finite vocabulary capable of representing any text, including words the vocabulary has never seen. Tokenize by whole words and you get a vocabulary in the millions that still fails on proper nouns, technical terms, code identifiers, and non-English text. Tokenize by individual characters and the sequences become too long for the model to process efficiently. Subword tokenization splits infrequently occurring words into meaningful fragments while keeping common words whole, balancing coverage against sequence length.

The result is that what the model actually processes looks like this:

The "##" prefix indicates a continuation piece — this chunk attaches to the previous token rather than starting a new word. Other tokenizers use different conventions, but the structure is the same. The model's input is not the sentence you wrote. It is the sentence as the tokenizer rendered it, and those two things differ in ways that matter — especially at word boundaries, with rare words, with names, and across languages.

This is also why models struggle with simple character-counting tasks. When you ask "how many r's are in 'strawberry'?" the model sees something like ["straw", "berry"] and must reason about the letters inside tokens it does not decompose character by character. The failure is not stupidity. It is a direct consequence of the input representation — a case where the architecture's trade-off becomes visible.

Embeddings — The Coordinate System of Meaning

Each token becomes a vector — a point in high-dimensional space. For most transformer models this space has 768 dimensions at minimum, often 1024 or 4096 for larger models. The model learns, during training, where each token lives in this space, and the positions it learns have remarkable structure.

Similar meanings cluster together. "King" and "queen" occupy nearby regions. "Paris" and "France" are close. The vector relationships encode semantic relationships with geometric consistency: "king" minus "man" plus "woman" produces a vector approximately equal to "queen." This is not a metaphor. It is literal arithmetic on learned coordinates.

from transformers import AutoTokenizer, AutoModel
import torch
 
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
 
inputs = tokenizer("The cat sat on the mat", return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)
 
# outputs.last_hidden_state shape: [batch, seq_len, 768]
embeddings = outputs.last_hidden_state[0]  # one vector per token

What comes back from that last line is a matrix where each row is a 768-dimensional vector corresponding to one token. Those vectors encode each token's meaning in the context of the full sentence — not its dictionary definition, but its contextual meaning at this particular position in this particular text.

The astonishing thing — the thing that took the field time to accept as real rather than artifact — is that meaning has geometry. Concepts that are similar are nearby in this space. Concepts that are opposite are in opposite directions. Analogical relationships are consistent vector offsets. The semantic structure of human language, accumulated across billions of sentences, condensed into a coordinate system where you can do arithmetic on ideas.

Attention — The Reading Mechanism

For each token, the model asks: which other tokens matter for understanding this one? Attention is the mechanism that computes the answer, and it runs simultaneously for every token in the sequence.

Take the sentence: "The trophy didn't fit in the suitcase because it was too big." What does "it" refer to? You know instantly — the trophy. How? Because "fit" and "too big" are predicates that constrain "it," and the trophy is the candidate that makes the sentence coherent. You resolved the coreference by understanding the semantics.

The model resolves it through attention. For the token "it," the model computes a relevance score for every other token in the sentence — how much should "it" attend to each one? These scores are derived from learned query and key vectors: "it" broadcasts a query ("I'm a pronoun — what do I refer to?") and every other token has a key that partially answers that query. The scores are normalized and used to weight the value vectors of those other tokens, producing a context-aware representation of "it" that has absorbed information about the most relevant surrounding words.

"it" attends to:
  trophy    ████████ 0.73
  suitcase  ███      0.21
  didn't    █        0.04
  fit       ░        0.02

The model learned to distribute attention this way by processing millions of sentences where pronoun resolution was either correct or incorrect. Nobody told it that "it" should attend to "trophy." It discovered, through gradient descent on prediction error, that attending to "trophy" produces better next-token predictions in sentences structured this way. The pattern was never programmed. It was found.

The transformer runs this computation for every token simultaneously, in parallel, using multiple attention heads — typically 8 or 12 per layer. Each head learns to track different aspects of the relationships between tokens. Some heads track syntactic dependencies. Others track positional proximity. Others track semantic similarity. The final representation for each token is a blend of information gathered by all of them.

Layers — What Each One Learns

A transformer doesn't compute attention once. It does it twelve times for BERT-base, 96 times for GPT-4-scale models, each layer taking the representations produced by the layer below and refining them further.

What researchers discovered — by training small probe models to predict specific linguistic properties from internal activations at each layer — is that the layers organize themselves by level of linguistic abstraction. Nobody programmed this. Nobody told layer 3 to learn syntax and layer 10 to learn semantics. The model organized itself this way because this organization minimizes prediction error on next-token prediction.

Early layers capture surface syntax: part-of-speech, local co-occurrence patterns, simple grammatical agreement. Middle layers capture grammatical roles and coreference: which noun phrase is the subject, which relative clause modifies which noun, which pronoun refers to which antecedent. Late layers capture semantic content: factual associations, word sense disambiguation, abstract conceptual relationships.

The layered hierarchy — from surface form to deep meaning — is exactly what linguistic theory predicts as the structure of human language knowledge. The model arrived at the same hierarchy through optimization pressure, without any linguistic theory baked in. It is one of the cleaner examples of a system discovering the structure of its domain rather than being given it.

What the Model Does Not Have

Precision here matters. Not to diminish the model, but because the gap between what the model actually has and what people assume it has is where most AI misunderstanding lives — and where both overclaiming and underclaiming do real damage.

The model has no memory between conversations unless the previous conversation is provided as context in the prompt. When you start a new chat, there is no recollection of what was discussed before. This is not forgetting — forgetting requires prior retention. The model never stored the conversation persistently. Every session begins from the same weights, the same initialization, the same learned parameters trained months or years ago.

The model has no sense of time passing. It doesn't experience latency between your message and its response. It doesn't know whether it's Tuesday or three years from now. Its knowledge has a training cutoff, and beyond that cutoff the world has continued without its awareness — not because it fell asleep, but because it was never awake to the passage of time in the first place.

The model has no persistent world model that updates across conversations. It doesn't maintain beliefs that grow more accurate as it learns more about you. It reasons within the context window, and the context window ends when the conversation ends. The next conversation begins from the same prior state as every other conversation.

Whether there is any phenomenal experience of reading — any sense of working through a sentence, recognizing a word, feeling the click of a paragraph coming together — is a question that remains genuinely open and harder than it sounds. What is clear is the absence of the specific experiences we associate with human reading: effort, recognition, confusion, the felt sense of meaning arriving. Whether the absence of these familiar markers constitutes the absence of experience, or only the absence of this particular form of it, is not settled.

For a while I built a tool called Mind Map that lets you visualize attention weights directly — click any token and watch the attention distribution light up across the sentence, layer by layer, head by head. You can find it at /projects/mind-map. Looking at it for long enough changes how you think about what is happening inside the model.

The model doesn't read. It computes relationships across a sequence. That sounds lesser than reading. I'm no longer sure it is.