It Depends on the Context 🤔

If you’ve ever asked a tricky question—at work, in life, or to an AI—you’ve probably heard the phrase: "It depends on the context."

This phrase is more relevant than ever in the age of large language models (LLMs). As we saw in the previous post, Truth is Cold, LLMs can be precise and factual when set to a low temperature. But even the coldest truth is only as good as the information the model has access to. And that’s where context comes in.

Why Context Matters for LLMs

LLMs are trained on vast swathes of the internet, but their knowledge is frozen at a certain point in time. They can hallucinate, misrepresent facts, or even make things up—especially when asked about recent events or niche topics. The solution? Provide the right context.

When you give an LLM the right background, documents, or data, you unlock its full reasoning power. The model can combine its learned knowledge with your provided context to generate more accurate, relevant, and trustworthy answers.

How to Provide Context to LLMs

Here are proven techniques to help LLMs deliver the truth:

1. Prompt Engineering

Craft your prompt to include all relevant details, background, and constraints. The more specific you are, the better the answer.

2. Retrieval Augmented Generation (RAG)

Use tools or pipelines that fetch up-to-date or domain-specific documents and inject them into the prompt. This is essential for question answering, chatbots, and knowledge-grounded tasks.

3. Reranking and Reordering

When you retrieve multiple documents, use reranking methods to order them by relevance before passing them to the LLM. This helps the model focus on the most important information first.

4. Query Rewriting

If a user’s question is ambiguous, rewrite it to be more specific. This ensures the retrieval system and the LLM get the context they need.

5. Multi-hop Reasoning

For complex questions, chain together multiple pieces of context, allowing the LLM to reason across several documents or facts.

6. Feedback and Logging

Log user interactions and collect feedback to iteratively improve your context selection and prompt design.

7. Context Window Management and Chunking

For long documents or large datasets, split content into manageable chunks that fit within the LLM’s context window. Use smart chunking strategies (e.g., by semantic boundaries or sections) and assemble relevant chunks dynamically at query time.

8. Embedding-Based Semantic Search

Leverage vector databases and embedding models to perform semantic search over your knowledge base. Retrieve the most relevant passages or documents based on similarity to the user’s query, then inject these into the LLM prompt.

9. Tool Use and Function Calling

Integrate the LLM with external tools, APIs, or plugins. Enable the model to call functions or fetch real-time data, calculations, or perform actions, then use the results as context for further reasoning.

10. Contextual Memory and Session Management

Implement memory modules or session state to persist relevant information across multi-turn conversations. This allows the LLM to maintain context and continuity, improving the quality of responses in chat or agent scenarios.

11. Automated Context Selection Pipelines

Develop pipelines that automatically select, filter, and rank context from multiple sources (databases, APIs, files) using rules, ML models, or hybrid approaches. This engineering ensures only the most relevant and up-to-date information is provided to the LLM.

12. Custom Middleware and Orchestration

Build middleware layers that orchestrate context retrieval, chunking, reranking, and prompt assembly. This enables scalable, modular, and maintainable LLM applications that can adapt to changing requirements and data sources.

Truth, Temperature, and Context

In Truth is Cold, we learned that a low temperature setting makes LLMs more factual and deterministic. But even with the perfect temperature, the model can only work with the information it has. Context is the bridge between cold truth and useful answers.

"Well, it depends on the context." 😊

By providing the right context, you help the LLM use all its learned data and reasoning ability to deliver answers that are not just plausible, but actually true.

Practical Example: Context in Action

Suppose you ask an LLM, "What’s the best way to evaluate AI agents?" Without context, it might give a generic answer. But if you provide a snippet or link to Evaluating AI Agents with RAGAS, the model can reference the latest frameworks and best practices.

Conclusion

The next time you interact with an LLM, remember: truth isn’t just cold—it’s contextual.

Be specific.
Provide relevant documents or data.
Use retrieval and reranking techniques.
Iterate with feedback.

That’s how you turn "It depends on the context" from a shrug into a superpower for AI.

Want to go deeper? Explore more on context, evaluation, and metric-driven development in these posts:

Have thoughts or questions? Connect with me—let’s talk about building more reliable, context-aware AI!