If you’re running LLM-powered features in production, you’ve probably noticed something: a huge chunk of your API calls are repeated or near-identical. A customer support bot answering “how do I reset my password” for the hundredth time. A coding assistant explaining the same Python concept. A summarizer processing the same document twice.

Every one of those duplicate calls costs you money and adds latency. The fix is caching — but LLM caching isn’t as simple as slapping a CDN in front of your API. You need a layered approach. This guide covers three caching strategies that, combined, can cut your LLM API bill by 80% or more.

Layer 1: Exact-Match Caching

The simplest and most reliable layer. Hash the prompt and model parameters, check if you’ve seen this exact request before, and return the cached response.

How It Works

You create a deterministic key from the full request — model name, prompt text, temperature, max tokens, and any other parameters. Store the response in Redis or another fast key-value store. On the next identical request, skip the API call entirely.

Implementation

import hashlib
import json
import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)
CACHE_TTL = 86400  # 24 hours

def cache_key(model: str, messages: list, **params) -> str:
    payload = json.dumps({"model": model, "messages": messages, **params}, sort_keys=True)
    return f"llm:{hashlib.sha256(payload.encode()).hexdigest()}"

def cached_completion(model: str, messages: list, **params):
    key = cache_key(model, messages, **params)
    if cached := r.get(key):
        return json.loads(cached)

    response = call_llm_api(model, messages, **params)
    r.setex(key, CACHE_TTL, json.dumps(response))
    return response

Hit Rates and Tradeoffs

Exact-match caching works best with temperature=0 and deterministic prompts. Expect hit rates of 30–60% for customer-facing apps with common queries. The tradeoff is obvious: “How do I reset my password?” and “how do i reset my password” are different hashes. You get zero fuzzy matching. That’s where layer 2 comes in.

Layer 2: Semantic Caching

Semantic caching solves the near-duplicate problem. Instead of hashing the exact text, you embed the query into a vector and search for similar cached queries within a similarity threshold.

How It Works

When a request comes in, generate an embedding of the user’s query. Search your vector store for cached entries above a similarity threshold (typically 0.95+). If a match is found, return the cached response. If not, call the LLM, cache both the embedding and the response.

Implementation with GPTCache

from gptcache import cache
from gptcache.adapter import openai
from gptcache.embedding import OpenAI as OpenAIEmbedding
from gptcache.manager import CacheBase, VectorBase, get_data_manager
from gptcache.similarity_evaluation import SearchDistanceEvaluation

embed = OpenAIEmbedding()
cache_base = CacheBase("sqlite")
vector_base = VectorBase("faiss", dimension=embed.dimension)

cache.init(
    embedding_func=embed.to_embeddings,
    data_manager=get_data_manager(cache_base, vector_base),
    similarity_evaluation=SearchDistanceEvaluation(),
)

# Now use the adapted client — cache hits are automatic
response = openai.ChatCompletion.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "How do I reset my password?"}],
)

GPTCache handles the embedding, similarity search, and cache management. You can also build this yourself with any vector database and embedding model.

Hit Rates and Tradeoffs

Semantic caching catches the queries that exact-match misses. Combined with layer 1, hit rates can reach 60–80% for apps with repetitive query patterns. The tradeoffs:

• Similarity threshold tuning — too low and you return wrong answers; too high and you miss valid cache hits. Start at 0.95 and adjust based on your domain.
• Embedding cost — every query still requires an embedding call, though embedding models are ~100x cheaper than completion models.
• Latency — the vector search adds 5–20ms, which is still far less than a full LLM call (500ms–3s).

Layer 3: Provider-Level Prompt Caching

This layer operates at the API provider level. You don’t cache full responses — instead, the provider caches the processing of your prompt prefix, so repeated calls with the same system prompt or context pay reduced input token costs. For a deeper dive, see how prompt caching works.

Anthropic’s cache_control

Anthropic lets you explicitly mark which parts of your prompt to cache. Cached input tokens cost 90% less on subsequent requests.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "You are a support agent for Acme Corp. Here is the full product manual: [... 5000 tokens of context ...]",
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": "How do I reset my password?"}],
)
# First call: full price. Subsequent calls with same system prompt: 90% off input tokens.

OpenAI Prefix Caching

OpenAI automatically caches prompt prefixes of 1024+ tokens. There’s nothing to configure — if your requests share a long common prefix (system prompt, few-shot examples, document context), cached tokens are billed at 50% off.

This works best when you structure prompts so the static content comes first and the variable user query comes last.

Hit Rates and Tradeoffs

Provider-level caching is nearly free to implement and has no accuracy risk — you get the exact same response you would without caching. The savings depend on your prompt structure:

• High savings: Long system prompts, RAG with shared document context, few-shot examples
• Low savings: Short prompts, highly variable context per request

The main limitation is that this only reduces input token costs, not output tokens. And cache lifetime is short — typically 5–10 minutes of inactivity before eviction.

Combining All Three Layers

The real power comes from stacking these layers in your AI application architecture. Route every request through them in order:

1. Exact-match — cheapest check, highest confidence. If the hash matches, return immediately.
2. Semantic cache — catches near-duplicates that exact-match misses. Slightly more expensive (embedding call + vector search).
3. Provider-level caching — for cache misses that actually hit the LLM, at least the prompt prefix is cached.

An AI gateway is the natural place to implement layers 1 and 2, sitting between your application and the LLM provider.

Cache Invalidation

LLM caches need invalidation strategies just like any other cache:

• TTL-based — simplest approach. Set a 24-hour TTL for most use cases. Shorter for time-sensitive content.
• Version-based — include your prompt template version in the cache key. When you update your system prompt, old cache entries are automatically bypassed.
• Manual purge — when your knowledge base or product data changes, invalidate related cache entries. Semantic caching makes this harder since you can’t just delete a single key.

A practical pattern: use short TTLs (1–6 hours) for semantic cache entries and longer TTLs (24–72 hours) for exact-match entries.

Monitoring Cache Performance

You can’t optimize what you don’t measure. Track these metrics:

• Hit rate per layer — exact-match hits, semantic hits, and total misses. If your semantic hit rate is low, your threshold may be too high.
• Cost savings — compare actual API spend against estimated spend without caching.
• Latency distribution — cache hits should be 10–100x faster than misses. If they’re not, your cache infrastructure needs tuning.
• False hit rate — sample semantic cache hits and verify the returned response actually answers the new query. This is the metric most teams forget to track.

# Simple cache metrics tracking
import time

class CacheMetrics:
    def __init__(self):
        self.exact_hits = self.semantic_hits = self.misses = 0

    def record(self, layer: str):
        if layer == "exact": self.exact_hits += 1
        elif layer == "semantic": self.semantic_hits += 1
        else: self.misses += 1

    @property
    def hit_rate(self) -> float:
        total = self.exact_hits + self.semantic_hits + self.misses
        return (self.exact_hits + self.semantic_hits) / total if total else 0

What to Do Next

Start with exact-match caching — it takes an afternoon to implement and immediately cuts costs for your most common queries. Add semantic caching once you’ve measured your exact-match hit rate and want to capture more. Enable provider-level prompt caching by restructuring your prompts (static content first, variable content last).

The combination of all three layers is how production teams are hitting that 80% cost reduction number. Your specific savings depend on your query distribution, but even a 40% reduction pays for the caching infrastructure many times over.