Tracely is an open-source (MIT) observability and evaluation platform for LLM and AI agent applications, and a self-hostable alternative to Langfuse, LangSmith and Braintrust.
Every eval tool asks you to hand-author a dataset: invent questions, write ideal answers, keep them current as the product changes. That dataset is a guess about what might break. Production already handed you the real thing — a trace of the exact run that failed, with the exact input, tool calls and model responses.
Traces arrive over OpenTelemetry (OTLP) from the Python SDK or any OTel-compatible instrumentation. Evaluators — deterministic structural checks or LLM-as-a-judge rubrics — grade each trace as it lands. Failures are automatically clustered into issues so you see the pattern rather than a wall of individual bad runs. Any cluster can then be frozen into a hermetic, replayable regression case that runs in CI and blocks the pull request that would reintroduce it.
Key features:
- Full trace and span tracing for agents, tools and LLM calls, with conversation replay
- Automatic evaluation on ingest: structural evaluators and LLM-as-a-judge
- Failure clustering and trend analysis across releases
- Hermetic regression cases replayed in CI, with a pass/fail commit status and PR comment
- Judge-vs-human calibration to measure how much you can trust your evaluators
- Scenario simulation, including adversarial runs
- MCP server, so coding agents can query your traces directly
- Bring your own model key; no vendor lock-in
Self-host the entire stack — API, worker, UI, Postgres, ClickHouse, Redis and object storage — with Docker Compose or a one-click Railway template. A hosted free tier is available at 20,000 traces per month.
Jwuthri 
