Most AI applications go to production with zero visibility into what’s actually happening. A user complains about slow responses and you’re left guessing — was it the LLM call? The retrieval step? The embedding generation? OpenTelemetry solves this by giving you distributed tracing across your entire AI pipeline, without locking you into a single vendor.

This tutorial walks you through instrumenting a Python AI application with OpenTelemetry, from installing the SDK to viewing traces in Jaeger.

Why OpenTelemetry for AI Monitoring

The observability landscape for AI apps is fragmented. Dedicated platforms like Langfuse are excellent, but they’re another vendor dependency. OpenTelemetry (OTel) takes a different approach:

• Vendor-neutral: Export traces to Jaeger, Grafana Tempo, Datadog, Honeycomb, or any OTLP-compatible backend. Switch providers without changing instrumentation code.
• Standard semantic conventions: The OTel community has published GenAI semantic conventions that standardize how LLM calls are described — model name, token counts, finish reason, and more.
• Full pipeline visibility: Trace a request from your API endpoint through retrieval, embedding, LLM generation, and back. One trace, every hop.
• You already use it: If your backend services are instrumented with OTel, adding AI-specific spans slots right into your existing traces.

This fits naturally into a broader AI app architecture where the LLM is one component among many — and each component needs monitoring.

What to Instrument

A typical RAG pipeline has several stages worth tracing:

Stage	Key Metrics
Total pipeline	End-to-end latency, success/failure
Embedding call	Model, input token count, latency
Retrieval	Number of results, relevance scores, latency
LLM call	Model, prompt/completion tokens, latency, cost, finish reason
Post-processing	Parsing time, validation results

Knowing what to log in AI systems is half the battle. The other half is structuring that data so you can actually query it.

Python Setup

Install the OpenTelemetry SDK and the OTLP exporter:

pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc openai

Configure the tracer provider to export spans over gRPC to a local collector (or directly to Jaeger):

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

resource = Resource.create({"service.name": "ai-pipeline"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True))
)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("ai-pipeline")

That’s the foundation. Every span you create from here will be exported to whatever backend is listening on port 4317.

Custom Spans for LLM Calls

Here’s where it gets useful. Wrap your LLM calls in spans that capture the metrics that matter:

import time
import openai

client = openai.OpenAI()

def call_llm(messages: list[dict], model: str = "gpt-4o") -> str:
    with tracer.start_as_current_span("llm.chat_completion") as span:
        span.set_attribute("gen_ai.system", "openai")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.request.temperature", 0.7)

        start = time.perf_counter()
        response = client.chat.completions.create(model=model, messages=messages)
        latency = time.perf_counter() - start

        usage = response.usage
        span.set_attribute("gen_ai.response.model", response.model)
        span.set_attribute("gen_ai.usage.prompt_tokens", usage.prompt_tokens)
        span.set_attribute("gen_ai.usage.completion_tokens", usage.completion_tokens)
        span.set_attribute("gen_ai.response.finish_reason", response.choices[0].finish_reason)
        span.set_attribute("llm.latency_seconds", round(latency, 3))

        # Estimate cost (example rates for gpt-4o)
        cost = (usage.prompt_tokens * 2.50 + usage.completion_tokens * 10.0) / 1_000_000
        span.set_attribute("llm.estimated_cost_usd", round(cost, 6))

        return response.choices[0].message.content

The attribute names follow OTel’s GenAI semantic conventions (gen_ai.*). This means dashboards and queries work consistently regardless of which LLM provider you use.

Apply the same pattern to embedding calls:

def get_embeddings(texts: list[str], model: str = "text-embedding-3-small") -> list:
    with tracer.start_as_current_span("llm.embeddings") as span:
        span.set_attribute("gen_ai.system", "openai")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.usage.input_count", len(texts))

        response = client.embeddings.create(model=model, input=texts)

        span.set_attribute("gen_ai.usage.prompt_tokens", response.usage.prompt_tokens)
        return [item.embedding for item in response.data]

Tracing the Full Pipeline

Now tie it together. A parent span wraps the entire request, and each step becomes a child span:

def rag_pipeline(query: str) -> str:
    with tracer.start_as_current_span("rag.pipeline") as span:
        span.set_attribute("rag.query", query)

        # Step 1: Embed the query
        query_embedding = get_embeddings([query])[0]

        # Step 2: Retrieve context (simplified)
        with tracer.start_as_current_span("rag.retrieval") as retrieval_span:
            docs = retrieve_similar_docs(query_embedding, top_k=5)
            retrieval_span.set_attribute("rag.results_count", len(docs))

        # Step 3: Generate response
        context = "\n".join(docs)
        messages = [
            {"role": "system", "content": f"Answer using this context:\n{context}"},
            {"role": "user", "content": query},
        ]
        answer = call_llm(messages)

        span.set_attribute("rag.response_length", len(answer))
        return answer

In your trace viewer, you’ll see a waterfall: rag.pipeline → llm.embeddings → rag.retrieval → llm.chat_completion, each with its own timing and attributes. This is the kind of visibility that makes debugging production issues possible instead of painful.

Exporting to Jaeger

Jaeger accepts OTLP natively since v1.35. Start it with Docker in a single command:

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  jaegertracing/all-in-one:latest

Port 16686 is the Jaeger UI. Port 4317 is the OTLP gRPC endpoint your Python code is already pointing at. Open http://localhost:16686, select the ai-pipeline service, and you’ll see your traces immediately.

Click into any trace and you get the waterfall view: the parent rag.pipeline span at the top, with llm.embeddings, rag.retrieval, and llm.chat_completion nested below it. Each span shows its duration, and clicking on one reveals all the attributes you set — token counts, model name, estimated cost. This is where you start spotting patterns: maybe your embedding calls are consistently 200ms but your LLM calls swing between 800ms and 4 seconds.

No code changes are needed to switch backends. To export to Grafana Tempo instead, just change the OTLP endpoint URL. To send to Datadog, use their OTLP intake URL. To fan out to multiple backends simultaneously, put an OpenTelemetry Collector in front and configure multiple exporters. This is the core value of OpenTelemetry — the instrumentation stays the same regardless of where the data goes.

Dashboard Examples

Once traces are flowing, build dashboards around the metrics that matter:

• P50/P95/P99 LLM latency — Filter spans where name = llm.chat_completion, chart llm.latency_seconds as a histogram. Spot regressions immediately.
• Token usage over time — Sum gen_ai.usage.prompt_tokens and gen_ai.usage.completion_tokens per hour. Correlate with traffic spikes.
• Cost tracking — Aggregate llm.estimated_cost_usd by model. Know exactly which model is burning your budget.
• Error rate by stage — Count spans with status = ERROR grouped by span name. Find out if failures cluster in retrieval vs. generation.
• Retrieval quality — Track rag.results_count alongside user feedback signals. Low retrieval counts often correlate with poor answers.

In Grafana, you can build these using TraceQL queries against Tempo. In Jaeger, use the search and compare features to diff slow traces against fast ones.

Connecting the Dots

If you’re running an AI gateway in front of your LLM providers, instrument that too — it becomes another span in the trace, showing you routing decisions, retries, and fallback behavior.

For a deeper look at what metrics and logs to capture beyond traces, see the guide on LLM observability for developers. Traces show you where time is spent; logs and metrics show you why.

What’s Next

Start with the basics: instrument your LLM calls, export to Jaeger, and look at the traces. You’ll immediately see things you didn’t know about your application — cold starts, unexpectedly large prompts, retry storms from flaky provider APIs.

From there, add span attributes for your domain-specific context: user tier, feature flag state, prompt template version, A/B test variant. The more context you attach to spans, the faster you can diagnose issues when they inevitably show up at 2 AM. Consider adding span events for intermediate steps too — span.add_event("cache_hit", {"cache.key": cache_key}) gives you even more granularity without creating separate spans.

If you’re processing high volumes, configure sampling in your TracerProvider to keep costs manageable. A TraceIdRatioBased sampler at 10% still gives you statistical significance while cutting storage by 90%.

OpenTelemetry won’t tell you if your AI is giving good answers — that’s a separate evaluation problem. But it will tell you exactly how your system behaves under load, where the bottlenecks are, and how much each request costs. That’s the foundation everything else is built on.