What is an LLM? | Large Language Models explained

A large language model is an AI system trained on massive amounts of text data to understand, process, and generate texts. Well-known examples include:

• ChatGPT (OpenAI)
• Gemini (Google DeepMind)
• Claude (Anthropic)

These models identify patterns, grammatical structures, and semantic meaning in language. They “learn” by analyzing billions of text passages, extracting underlying rules and relationships. The goal is to generate responses that are statistically likely, relevant, and coherent for any given text input.

Andrej Karpathy describes an LLM as a kind of “zip file of the internet”. In other words, an LLM holds a compressed, statistical snapshot of the world’s textual knowledge – up to the point it was trained. However, since this knowledge isn’t always up to date, modern language models are increasingly combined with web search capabilities to access real-time information.

LLMs are incredibly powerful – but far from perfect. Karpathy uses the metaphor of Swiss cheese: the model performs impressively in many areas, but still has “holes” –unexpected weaknesses.

• Versatility in language and style
• High-level natural language processing capabilities
• Broad general knowledge (as of the training cutoff)
• Applicable in a wide range of use cases: content creation, customer support, data analysis

• No guaranteed factual accuracy (“hallucinations”)
• Outdated knowledge without internet access
• Failure on surprisingly simple tasks
• Not truly “intelligent”—just statistical pattern prediction

Thanks to ongoing reinforcement learning, better data sources, and human feedback loops, LLMs are improving at a rapid pace.

In this initial phase, the model is trained on massive amounts of text data from the internet. These texts are broken down into tokens—the smallest meaningful units of language. The goal: to train the model to predict which token (word or phrase) is statistically most likely to come next.

The result of this tokenization and pretraining phase is a model that can generate coherent text based on the patterns it has learned from data.

Example: Given the prompt “The weather today is very ___”, the model might predict completions like nice, warm, or rainy, based on context.

Reasoning abilities—developed during pretraining and refined through various training methods—allow an LLM to think logically, draw conclusions, and solve problems. These capabilities also enable the model to analyze its own output and identify areas for improvement.

Pro tip: If you want to dive deeper into how the neural network is trained, check out the video „Deep Dive into LLMs like ChatGPT“ starting at minute 15.

After pretraining, the model enters a fine-tuning stage where it learns from examples rated and curated by humans. This step teaches the model how to respond appropriately in real-world conversations. The aim here is to improve response quality—making outputs more relevant, safe, and accurate. Human annotators play a critical role by providing direct feedback, which becomes an essential part of the learning process.

In the final phase, the model is further refined using reinforcement learning based on human feedback. It learns to favor helpful or high-quality responses while avoiding poor ones. 

RLHF is one of the key reasons behind the impressive quality of today’s LLM.

Inference refers to the moment when a fully trained model responds to a new input:

1. Input: You type a question or prompt.
2. Tokenization: The text is broken down into individual tokens.
3. Prediction: The model calculates the most likely next token.
4. Output: The response is generated token by token—each word depends on the preceding context.

The model doesn’t “think” like a human. Instead, it follows probability patterns it learned during training.

Large language models (LLM) like ChatGPT or Gemini mark the beginning of a new era in human–machine interaction. Their multimodal capabilities—combining text, images, audio, and video—and the constant stream of user feedback are making them more powerful by the day.

The range of applications is nearly limitless: from personalized assistants to automated content generation to data-driven decision-making in business.

Understanding how an LLM works today puts you ahead of the curve—whether you're a content creator, a developer, or a decision-maker navigating digital transformation. I hope this guide helped you gain a clearer understanding of how LLM like ChatGPT actually work! :-)