Your agent answers wrong and you have no idea which step broke. Was it the prompt, a tool called with bad arguments, or the model picking the wrong function? With a single LLM call, debugging means inspecting one request. With an agent, the loop runs several times, calls tools in between, and the failure often only shows up at the end. Print statements get you part of the way, then you spend an hour stitching timestamps together by hand.
This tutorial fixes that. You will self-host Langfuse, an open source LLM observability platform, with Docker Compose, then instrument a minimal tool-calling agent with the Python SDK so every model call and tool execution lands in one trace tree with latency, token usage, and cost per step.
Prerequisites
- Docker and Docker Compose (Docker Desktop works on macOS and Windows)
- Python 3.10 or newer
- An OpenAI API key, or any OpenAI-compatible endpoint (Ollama works too)
- About 15 minutes
Step 1: Run Langfuse with Docker Compose
Langfuse is a full platform, and for local development the repo ships a docker-compose.yml that starts everything: the web app, a background worker, Postgres, ClickHouse, Redis, and MinIO for blob storage.
git clone https://github.com/langfuse/langfuse.git
cd langfuse
docker compose up
Wait until the langfuse-web-1 container logs "Ready". On a first start that takes 2-3 minutes. Then open http://localhost:3000.
Create an account, create a project, and open the project settings to generate API keys. You get two: a public key (pk-lf-...) and a secret key (sk-lf-...). The secret key is a password. Keep it out of git.
One thing before you expose this beyond your laptop: the compose file marks every secret with # CHANGEME. Rotate them if the instance will be reachable from a network.
Step 2: Build a Minimal Tool-Calling Agent
The agent is a loop. Call the model with the conversation and the tool list, check whether the response contains tool calls, execute them, append the results, and repeat until the model answers without calling a tool.
import json
from openai import OpenAI
client = OpenAI()
TOOLS = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city",
"parameters": {
"type": "object",
"properties": {
"city": {"type": "string"}
},
"required": ["city"],
},
},
},
{
"type": "function",
"function": {
"name": "get_exchange_rate",
"description": "Get the USD exchange rate for a currency",
"parameters": {
"type": "object",
"properties": {
"currency": {"type": "string"}
},
"required": ["currency"],
},
},
},
]
def get_weather(city: str) -> str:
# Replace with a real weather API call in your app
return json.dumps({"city": city, "temperature_c": 31, "condition": "partly cloudy"})
def get_exchange_rate(currency: str) -> str:
# Replace with a real rates API call in your app
return json.dumps({"currency": currency.upper(), "usd": 16250})
TOOL_IMPL = {
"get_weather": get_weather,
"get_exchange_rate": get_exchange_rate,
}
def run_agent(user_input: str) -> str:
messages = [
{"role": "system", "content": "You answer questions using the provided tools."},
{"role": "user", "content": user_input},
]
for _ in range(5): # safety cap against infinite loops
response = client.chat.completions.create(
model="gpt-4o-mini", # any current chat model works
messages=messages,
tools=TOOLS,
)
message = response.choices[0].message
messages.append(message)
if not message.tool_calls:
return message.content or ""
for tool_call in message.tool_calls:
result = TOOL_IMPL[tool_call.function.name](
**json.loads(tool_call.function.arguments)
)
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": result,
})
return "Reached max steps without a final answer"
if __name__ == "__main__":
print(run_agent("How warm is Jakarta today, and what is one dollar in rupiah?"))
Run it once to confirm it works. Then break a tool on purpose (raise an exception inside get_weather) and ask the agent something that needs it. The error you get is useless: you see the traceback, but not what the model was trying to do or what arguments it sent. That is the problem observability solves.
Step 3: Instrument with the Python SDK
Install the SDK and point it at your local instance:
pip install langfuse
export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...
export LANGFUSE_BASE_URL=http://localhost:3000
The Python SDK v4 is built on OpenTelemetry. It gives you three ways to create observations: the @observe() decorator for whole functions, start_as_current_observation() as a context manager for blocks of code, and manual start_observation(). They nest, so you can mix them freely.
Here is the instrumented agent. The decorator wraps the whole run, each LLM call becomes a generation observation, and each tool execution becomes a span:
from langfuse import get_client, observe, propagate_attributes
langfuse = get_client()
@observe(name="agent-run")
def run_agent(user_input: str) -> str:
propagate_attributes(user_id="demo-user", session_id="demo-session")
messages = [
{"role": "system", "content": "You answer questions using the provided tools."},
{"role": "user", "content": user_input},
]
for _ in range(5):
with langfuse.start_as_current_observation(
as_type="generation",
name="llm-call",
model="gpt-4o-mini",
input=messages,
) as generation:
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=TOOLS,
)
generation.update(output=response.choices[0].message.content)
message = response.choices[0].message
messages.append(message)
if not message.tool_calls:
return message.content or ""
for tool_call in message.tool_calls:
with langfuse.start_as_current_observation(
as_type="span",
name=f"tool-{tool_call.function.name}",
) as tool_span:
result = TOOL_IMPL[tool_call.function.name](
**json.loads(tool_call.function.arguments)
)
tool_span.update(input=tool_call.function.arguments, output=result)
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": result,
})
return "Reached max steps without a final answer"
if __name__ == "__main__":
print(run_agent("How warm is Jakarta today, and what is one dollar in rupiah?"))
langfuse.flush()
The changes are small. The decorator captures the function's input, output, timing, and exceptions automatically. Each generation records the exact messages sent and the model's reply. Each tool span records the parsed arguments and the result.
langfuse.flush() matters in this script. The SDK batches events and sends them in the background, so a script that exits immediately can drop the last traces. Long-running services do not need it.
Run the script, then open the Traces page. You should see one trace per run with a tree like this:
agent-run
├── llm-call (generation)
├── tool-get_weather (span)
├── llm-call (generation)
├── tool-get_exchange_rate (span)
└── llm-call (generation)
Click any node to see the prompt, the output, token usage, and cost. Langfuse ships price tables for common models (OpenAI, Anthropic, Google), so cost appears without extra configuration. The user id and session id you set with propagate_attributes become filters. When you have thousands of traces, that is how you find the ones that matter.
Step 4: The OpenTelemetry Path for Non-Python Stacks
The Python SDK is convenient, but the protocol underneath is OpenTelemetry, and Langfuse accepts OTLP directly at /api/public/otel. Any exporter in any language can send traces:
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:3000/api/public/otel"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n 'pk-lf-...:sk-lf-...' | base64 -w 0),x-langfuse-ingestion-version=4"
Two details worth knowing. First, auth is HTTP Basic with the public key as username and the secret key as password. Second, the x-langfuse-ingestion-version: 4 header makes directly ingested spans appear in real time; without it they can be delayed up to 10 minutes.
If your spans follow the OpenTelemetry GenAI semantic conventions (gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, and the rest), Langfuse renders them as LLM generations with token and cost tracking, same as the SDK path. This is the route for vendor-neutral telemetry you could also send to Jaeger or Tempo, or for agents written in Go, Rust, or TypeScript where you do not want a Python SDK in the critical path.
Langfuse vs Alternatives
| Tool | Strengths | Watch out for |
|---|---|---|
| Langfuse | Open source (MIT core), self-hostable, OTel-native, prompt management and evals included | The Compose setup is single-node; scaling means Kubernetes |
| LangSmith | Deepest LangChain and LangGraph integration, strong dataset and eval tooling | SaaS-first; self-hosting is enterprise-only |
| Arize Phoenix | Local-first, excellent for notebook debugging, free | Not a full platform: no prompt management, thinner production features |
| Helicone | Proxy-based, zero code changes for OpenAI SDK users | Observability is limited by what the proxy sees; custom spans are clunkier |
For a solo project or a team that wants data on its own infrastructure, Langfuse is the most complete open option. If you live inside LangChain and want the tightest integration, LangSmith is the comfortable choice. If your need is debugging traces in a notebook this afternoon, Phoenix gets you there fastest.
Common Pitfalls
Missing traces from short scripts. Forgetting flush() drops the tail of the batch. Call it before exit.
Capturing huge payloads. Traces with megabyte-sized inputs are slow to render and eat storage. Disable IO capture where it does not help: @observe(capture_input=False) or the LANGFUSE_OBSERVE_DECORATOR_IO_CAPTURE_ENABLED env var.
Exposing default secrets. The # CHANGEME values in the compose file are public knowledge. Rotate them before the instance is reachable from anything other than localhost.
No sampling at volume. At high traffic, storing 100% of traces gets expensive. Langfuse supports sampling, so keep a configurable percentage and drop the rest.
Summary and Next Steps
You now have a local observability stack that shows every LLM call and tool execution in a trace tree, with tokens and cost per step. "The agent is broken somewhere" turns into "the second llm-call sent the wrong arguments to get_weather". From here:
- Group multi-turn conversations with
session_idviapropagate_attributes - Add user feedback buttons and send the ratings to Langfuse
- Build LLM-as-a-judge evals on top of logged traces, using datasets from real runs
- Route the same OTLP pipeline through a collector so logs, metrics, and traces share one path