Modern LLMs do not think in words. They operate with vectors, probability distributions, and weight matrices.
Any text for the model is simply a sequence of tokens transformed through billions of parameters into a probability distribution for the next token.
Yet it is precisely this mathematical abstraction that gives rise to the phenomenon we call understanding.
### Context Window
One of the key characteristics of an LLM is the size of its context window.
This is the working memory capacity of the model during generation. Everything beyond the context limit is “forgotten” by the model.
Modern architectures actively expand context: from a few thousand tokens to millions.
However, simply increasing the context introduces new challenges:
* **Lost in the Middle:** the model remembers the beginning and end of the context well, but may “lose” details from the middle;
* **Quadratic Attention Complexity:** the standard Attention mechanism requires $O(N^2)$ computational resources relative to context length, forcing architects to seek new optimizations (FlashAttention, RoPE, Sparse Attention).
### RAG (Retrieval-Augmented Generation)
Models do not store all information directly in their weights. Knowledge in weights is frozen at the time of training.
To solve this issue, RAG is utilized:
1. Documents are split into chunks.
2. Chunks are converted into embeddings (vector representations).
3. Vectors are stored in a vector database (Chroma, Qdrant, Milvus, pgvector).
4. Upon user query, the system retrieves the most semantically relevant vectors.
5. The retrieved context is injected into the model prompt.
RAG allows enriching the model with fresh private data without expensive retraining.
### Fine-Tuning and LoRA
A Base Model can only continue text. To turn it into a helpful assistant (Instruct Model), alignment techniques are applied:
* **SFT (Supervised Fine-Tuning):** training on question-answer pairs;
* **RLHF / DPO (Direct Preference Optimization):** preference-based training to reduce hallucinations and harmful outputs.
