LLM observability and evaluation tools, with reproducible numbers
64 tools for LLM tracing, evaluation, guardrails, prompt management and gateways. Pricing, licence, self-hosting and OpenTelemetry support are checked on each vendor's own pages. Tools with a mapped SDK also show weekly PyPI and npm package downloads and GitHub stars, each linked to the public query that reproduces it.
Package downloads are not people or customers: CI runs and packages installed as dependencies count too. Python, npm and stars are separate columns and are never added together. How we measure · Monthly movers 2026-09
Week 2026-W40 (2026-09-28 to 2026-10-04, UTC), measured 2026-10-07 20:16 UTC.
Rankings, week 2026-W40
PyPI weekly downloads
| # | Tool | Package | PyPI downloads (week) | 12 weeks | vs prior week | Licence | Self-host | OTel | Entry price |
|---|---|---|---|---|---|---|---|---|---|
| 1 | LiteLLMGateways with observability | litellmbundled | 22,710,323 | -0.1% | MIT (open core) | Yes | Exporter / bridge | Contact sales | |
| 2 | LangSmithTracing & observability | langsmithbundled | 18,866,051 | +8.1% | Proprietary | Enterprise only | Native (OTLP) | $39/seat/mo | |
| 3 | LangfuseTracing & observability | langfuse | 6,306,471 | +12.7% | MIT (open core) | Yes | Native (OTLP) | $29/mo | |
| 4 | MLflowTracing & observability | mlflow-tracingbundled | 3,616,315 | +3.7% | Apache-2.0 | Yes | Native (OTLP) | Free | |
| 5 | Pydantic LogfireTracing & observability | logfirebundled | 3,488,038 | +14.3% | Proprietary | Enterprise only | Native (OTLP) | $49/mo | |
| 6 | Pydantic EvalsEvaluation | pydantic-evalsbundled | 2,121,173 | +10.5% | MIT | Yes | Native (OTLP) | Free | |
| 7 | BraintrustTracing & observability | braintrust | 1,650,096 | 0.0% | Proprietary | Enterprise only | Native (OTLP) | $249/mo | |
| 8 | Inspect AIEvaluation | inspect-ai | 907,448 | +33.1% | MIT | Yes | Unknown | Free | |
| 9 | PortkeyGateways with observability | portkey-ai | 765,740 | -26.5% | MIT (open core) | Yes | Native (OTLP) | $49/mo | |
| 10 | DeepEval / Confident AIEvaluation | deepeval | 686,214 | +9.7% | Apache-2.0 (open core) | Enterprise only | Native (OTLP) | $200/mo | |
| 11 | Traceloop / OpenLLMetryTracing & observability | traceloop-sdk | 455,110 | +7.6% | Apache-2.0 (open core) | Enterprise only | Native (OTLP) | Contact sales | |
| 12 | OpikTracing & observability | opik | 437,574 | +2.0% | Apache-2.0 | Yes | Native (OTLP) | $19/mo | |
| 13 | RagasEvaluation | ragas | 279,757 | +9.5% | Apache-2.0 | Yes | Unknown | Free | |
| 14 | lm-evaluation-harnessEvaluation | lm-eval | 270,051 | +10.3% | MIT | Yes | Unknown | Free | |
| 15 | W&B WeaveTracing & observability | weave | 239,276 | +28.3% | Apache-2.0 (open core) | Yes | Native (OTLP) | $60/mo | |
| 16 | Evidently AIEvaluation | evidently | 198,068 | +7.3% | Apache-2.0 (open core) | Yes | Native (OTLP) | Free | |
| 17 | Arize PhoenixTracing & observability | arize-phoenix | 161,687 | +13.6% | Elastic-2.0 (source-available) | Yes | Native (OTLP) | Free | |
| 18 | NeMo GuardrailsGuardrails & safety | nemoguardrails | 148,111 | +64.0% | Apache-2.0 | Yes | Exporter / bridge | Free | |
| 19 | Arize AXTracing & observability | arize-otel | 84,266 | -5.1% | Proprietary | Enterprise only | Native (OTLP) | $50/mo | |
| 20 | OpenLITTracing & observability | openlit | 72,503 | +14.5% | Apache-2.0 | Yes | Native (OTLP) | Free | |
| 21 | LunaryTracing & observability | lunary | 49,317 | +22.9% | Unknown | Enterprise only | Native (OTLP) | $20/seat/mo | |
| 22 | OpenlayerEvaluation | openlayer | 37,719 | -16.7% | Proprietary | Enterprise only | Native (OTLP) | Contact sales | |
| 23 | Azure AI Evaluation SDK (Microsoft Foundry)previewEvaluation | azure-ai-evaluation | 33,482 | -4.5% | MIT | Yes | Unknown | not published in USD | |
| 24 | Orq.aiGateways with observability | orq-ai-sdk | 30,379 | +14.1% | Proprietary | Enterprise only | Native (OTLP) | not published in USD | |
| 25 | AgentOpsTracing & observability | agentops | 28,508 | +84.6% | MIT | Yes | Native (OTLP) | $40/mo | |
| 26 | PromptLayerPrompt management | promptlayer | 23,137 | +10.5% | Proprietary | Enterprise only | Native (OTLP) | $49/mo | |
| 27 | Judgment Labs (Judgeval)Tracing & observability | judgeval | 19,443 | +39.3% | Apache-2.0 (open core) | No | Native (OTLP) | Contact sales | |
| 28 | garakEvaluation | garak | 11,417 | +25.8% | Apache-2.0 | Yes | Unknown | Free | |
| 29 | AgentaPrompt management | agenta | 10,423 | +38.6% | MIT (open core) | Yes | Native (OTLP) | $29/mo | |
| 30 | GalileoEvaluation | galileo | 9,821 | +46.8% | Proprietary | Enterprise only | Native (OTLP) | Contact sales | |
| 31 | TruLensEvaluation | trulens | 6,198 | +28.0% | MIT | Yes | Native (OTLP) | Free | |
| 32 | GiskardEvaluation | giskard | 5,667 | +93.3% | Apache-2.0 (open core) | Yes | Unknown | Contact sales | |
| 33 | HeliconeTracing & observability | helicone-helpers | 4,840 | +20.7% | Apache-2.0 | Yes | Unknown | $79/mo | |
| 34 | PromptfooEvaluation | promptfoo | 3,451 | +13.7% | MIT (open core) | Yes | Native (OTLP) | Contact sales | |
| 35 | HoneyHiveTracing & observability | honeyhive | 3,245 | +38.9% | Proprietary | Enterprise only | Native (OTLP) | Contact sales | |
| 36 | LatitudeTracing & observability | latitude-telemetry | 2,379 | -18.5% | MIT | Yes | Native (OTLP) | $99/mo | |
| 37 | GalteaEvaluation | galtea | 2,373 | +23.0% | Proprietary | Enterprise only | Native (OTLP) | Contact sales | |
| 38 | OkareoEvaluation | okareo | 2,051 | +0.3% | Proprietary | Enterprise only | Native (OTLP) | $199/mo | |
| 39 | LlamaFirewallGuardrails & safety | llamafirewall | 1,471 | +53.9% | MIT | Yes | Unknown | Free | |
| 40 | Last9 (AI / LLM observability)Tracing & observability | last9-genai | 918 | -16.0% | Proprietary | Unknown | Native (OTLP) | not published in USD | |
| 41 | Fiddler AITracing & observability | fiddler-otel | 791 | -21.0% | Proprietary | Enterprise only | Native (OTLP) | not published in USD | |
| 42 | ScorecardEvaluation | scorecard-ai | 621 | +3.8% | Proprietary | Unknown | Native (OTLP) | $299/mo | |
| 43 | UpTrainEvaluation | uptrain | 423 | +46.4% | Apache-2.0 | Yes | Unknown | Contact sales | |
| 44 | AI Verify MoonshotbetaEvaluation | aiverify-moonshot | 370 | +65.9% | Apache-2.0 | Yes | Unknown | Free | |
| 45 | Scorable (formerly Root Signals)Evaluation | scorable | 118 | +57.3% | Proprietary | Enterprise only | Native (OTLP) | $19/seat/mo | |
| – | AgentEvalsEvaluation | – | SDK not mapped | MIT | Yes | Unknown | Free | ||
| – | Arthur EngineEvaluation | – | SDK not mapped | MIT (open core) | Yes | Native (OTLP) | $60/mo | ||
| – | Cloudflare AI GatewayGateways with observability | – | SDK-less | Proprietary | Unknown | Exporter / bridge | Free | ||
| – | Coralogix AI Observability (AI Center)Tracing & observability | – | SDK-less | Proprietary | Unknown | Native (OTLP) | not published in USD | ||
| – | Datadog LLM ObservabilityTracing & observability | – | proxy-only | Proprietary | Unknown | Native (OTLP) | $160/mo | ||
| – | Deepchecks LLM EvaluationEvaluation | – | no PyPI SDK | Proprietary | Enterprise only | Unknown | Contact sales | ||
| – | Future AGIEvaluation | – | SDK not mapped | Apache-2.0 (open core) | Yes | Native (OTLP) | $250/mo | ||
| – | Guardrails AIGuardrails & safety | – | SDK not mapped | Apache-2.0 | Yes | Exporter / bridge | Free | ||
| – | Invariant GuardrailsGuardrails & safety | – | SDK not mapped | Apache-2.0 | Yes | Unknown | Free | ||
| – | Lakera GuardGuardrails & safety | – | no PyPI SDK | Proprietary | Enterprise only | Unknown | Contact sales | ||
| – | LaminarTracing & observability | – | SDK not mapped | Apache-2.0 (open core) | Yes | Native (OTLP) | $30/mo | ||
| – | LangWatchTracing & observability | – | SDK not mapped | Apache-2.0 (open core) | Yes | Native (OTLP) | not published in USD | ||
| – | Maxim AIEvaluation | – | SDK not mapped | Proprietary | Enterprise only | Native (OTLP) | Contact sales | ||
| – | OpenEvalsEvaluation | – | SDK not mapped | MIT | Yes | Unknown | Free | ||
| – | OpenObserve (LLM observability)Tracing & observability | – | SDK-less | AGPL-3.0 (open core) | Yes | Native (OTLP) | not published in USD | ||
| – | Patronus AIEvaluation | – | no PyPI SDK | Proprietary | Enterprise only | Unknown | not published in USD | ||
| – | RaindropTracing & observability | – | SDK not mapped | Proprietary | Unknown | Native (OTLP) | $299/mo | ||
| – | RespanTracing & observability | – | SDK not mapped | Proprietary | Enterprise only | Native (OTLP) | $199/mo | ||
| – | SigNoz (LLM observability)Tracing & observability | – | SDK-less | MIT (open core) | Yes | Native (OTLP) | $49/mo |
npm weekly downloads
| # | Tool | Package | npm downloads (week) | 12 weeks | vs prior week | Licence | Self-host | OTel | Entry price |
|---|---|---|---|---|---|---|---|---|---|
| 1 | LangSmithTracing & observability | langsmith | 8,568,634 | +17.4% | Proprietary | Enterprise only | Native (OTLP) | $39/seat/mo | |
| 2 | LangfuseTracing & observability | @langfuse/tracing | 3,268,181 | +12.2% | MIT (open core) | Yes | Native (OTLP) | $29/mo | |
| 3 | BraintrustTracing & observability | braintrust | 2,046,537 | +2.7% | Proprietary | Enterprise only | Native (OTLP) | $249/mo | |
| 4 | PromptfooEvaluation | promptfoo | 1,020,946 | +26.9% | MIT (open core) | Yes | Native (OTLP) | Contact sales | |
| 5 | W&B WeaveTracing & observability | weave | 623,550 | +6.9% | Apache-2.0 (open core) | Yes | Native (OTLP) | $60/mo | |
| 6 | PortkeyGateways with observability | portkey-ai | 280,520 | +29.2% | MIT (open core) | Yes | Native (OTLP) | $49/mo | |
| 7 | OpenEvalsEvaluation | openevals | 161,288 | +13.3% | MIT | Yes | Unknown | Free | |
| 8 | Traceloop / OpenLLMetryTracing & observability | @traceloop/node-server-sdk | 147,877 | -0.2% | Apache-2.0 (open core) | Enterprise only | Native (OTLP) | Contact sales | |
| 9 | RaindropTracing & observability | raindrop-ai | 99,912 | -13.8% | Proprietary | Unknown | Native (OTLP) | $299/mo | |
| 10 | LaminarTracing & observability | @lmnr-ai/lmnr | 77,663 | +4.2% | Apache-2.0 (open core) | Yes | Native (OTLP) | $30/mo | |
| 11 | Arize PhoenixTracing & observability | @arizeai/phoenix-otel | 73,285 | -37.5% | Elastic-2.0 (source-available) | Yes | Native (OTLP) | Free | |
| 12 | LangWatchTracing & observability | langwatch | 50,105 | +4.9% | Apache-2.0 (open core) | Yes | Native (OTLP) | not published in USD | |
| 13 | Orq.aiGateways with observability | @orq-ai/node | 43,521 | -41.8% | Proprietary | Enterprise only | Native (OTLP) | not published in USD | |
| 14 | OpikTracing & observability | opik | 42,992 | -8.3% | Apache-2.0 | Yes | Native (OTLP) | $19/mo | |
| 15 | Judgment Labs (Judgeval)Tracing & observability | judgeval | 34,312 | -5.8% | Apache-2.0 (open core) | No | Native (OTLP) | Contact sales | |
| 16 | Pydantic LogfireTracing & observability | @pydantic/logfire-node | 31,486 | -59.7% | Proprietary | Enterprise only | Native (OTLP) | $49/mo | |
| 17 | LatitudeTracing & observability | @latitude-data/telemetry | 7,507 | +43.1% | MIT | Yes | Native (OTLP) | $99/mo | |
| 18 | PromptLayerPrompt management | promptlayer | 6,509 | +31.4% | Proprietary | Enterprise only | Native (OTLP) | $49/mo | |
| 19 | HeliconeTracing & observability | @helicone/helpers | 4,098 | -2.5% | Apache-2.0 | Yes | Unknown | $79/mo | |
| 20 | ScorecardEvaluation | scorecard-ai | 2,117 | +2.7% | Proprietary | Unknown | Native (OTLP) | $299/mo | |
| 21 | GalileoEvaluation | galileo | 1,810 | +32.3% | Proprietary | Enterprise only | Native (OTLP) | Contact sales | |
| 22 | OpenLITTracing & observability | openlit | 1,231 | -43.1% | Apache-2.0 | Yes | Native (OTLP) | Free | |
| 23 | OpenlayerEvaluation | openlayer | 308 | -50.0% | Proprietary | Enterprise only | Native (OTLP) | Contact sales | |
| 24 | HoneyHiveTracing & observability | @honeyhive/api-client | 71 | -65.0% | Proprietary | Enterprise only | Native (OTLP) | Contact sales | |
| 25 | RespanTracing & observability | @respan/respan | 69 | -12.7% | Proprietary | Enterprise only | Native (OTLP) | $199/mo | |
| 26 | AgentOpsTracing & observability | agentops | 25 | +38.9% | MIT | Yes | Native (OTLP) | $40/mo | |
| 27 | Future AGIEvaluation | @future-agi/ai-evaluation | 3 | -66.7% | Apache-2.0 (open core) | Yes | Native (OTLP) | $250/mo | |
| 28 | OkareoEvaluation | okareo-ts-sdk | 3 | -81.2% | Proprietary | Enterprise only | Native (OTLP) | $199/mo | |
| – | AgentaPrompt management | – | no npm SDK | MIT (open core) | Yes | Native (OTLP) | $29/mo | ||
| – | AgentEvalsEvaluation | – | SDK not mapped | MIT | Yes | Unknown | Free | ||
| – | AI Verify MoonshotbetaEvaluation | – | no npm SDK | Apache-2.0 | Yes | Unknown | Free | ||
| – | Arize AXTracing & observability | – | no npm SDK | Proprietary | Enterprise only | Native (OTLP) | $50/mo | ||
| – | Arthur EngineEvaluation | – | no npm SDK | MIT (open core) | Yes | Native (OTLP) | $60/mo | ||
| – | Azure AI Evaluation SDK (Microsoft Foundry)previewEvaluation | – | no npm SDK | MIT | Yes | Unknown | not published in USD | ||
| – | Cloudflare AI GatewayGateways with observability | – | SDK-less | Proprietary | Unknown | Exporter / bridge | Free | ||
| – | Coralogix AI Observability (AI Center)Tracing & observability | – | SDK-less | Proprietary | Unknown | Native (OTLP) | not published in USD | ||
| – | Datadog LLM ObservabilityTracing & observability | – | proxy-only | Proprietary | Unknown | Native (OTLP) | $160/mo | ||
| – | Deepchecks LLM EvaluationEvaluation | – | no npm SDK | Proprietary | Enterprise only | Unknown | Contact sales | ||
| – | DeepEval / Confident AIEvaluation | – | SDK not mapped | Apache-2.0 (open core) | Enterprise only | Native (OTLP) | $200/mo | ||
| – | Evidently AIEvaluation | – | no npm SDK | Apache-2.0 (open core) | Yes | Native (OTLP) | Free | ||
| – | Fiddler AITracing & observability | – | no npm SDK | Proprietary | Enterprise only | Native (OTLP) | not published in USD | ||
| – | GalteaEvaluation | – | no npm SDK | Proprietary | Enterprise only | Native (OTLP) | Contact sales | ||
| – | garakEvaluation | – | no npm SDK | Apache-2.0 | Yes | Unknown | Free | ||
| – | GiskardEvaluation | – | no npm SDK | Apache-2.0 (open core) | Yes | Unknown | Contact sales | ||
| – | Guardrails AIGuardrails & safety | – | no npm SDK | Apache-2.0 | Yes | Exporter / bridge | Free | ||
| – | Inspect AIEvaluation | – | no npm SDK | MIT | Yes | Unknown | Free | ||
| – | Invariant GuardrailsGuardrails & safety | – | no npm SDK | Apache-2.0 | Yes | Unknown | Free | ||
| – | Lakera GuardGuardrails & safety | – | no npm SDK | Proprietary | Enterprise only | Unknown | Contact sales | ||
| – | Last9 (AI / LLM observability)Tracing & observability | – | no npm SDK | Proprietary | Unknown | Native (OTLP) | not published in USD | ||
| – | LiteLLMGateways with observability | – | no npm SDK | MIT (open core) | Yes | Exporter / bridge | Contact sales | ||
| – | LlamaFirewallGuardrails & safety | – | no npm SDK | MIT | Yes | Unknown | Free | ||
| – | lm-evaluation-harnessEvaluation | – | no npm SDK | MIT | Yes | Unknown | Free | ||
| – | LunaryTracing & observability | – | SDK not mapped | Unknown | Enterprise only | Native (OTLP) | $20/seat/mo | ||
| – | Maxim AIEvaluation | – | SDK not mapped | Proprietary | Enterprise only | Native (OTLP) | Contact sales | ||
| – | MLflowTracing & observability | – | no npm SDK | Apache-2.0 | Yes | Native (OTLP) | Free | ||
| – | NeMo GuardrailsGuardrails & safety | – | no npm SDK | Apache-2.0 | Yes | Exporter / bridge | Free | ||
| – | OpenObserve (LLM observability)Tracing & observability | – | SDK-less | AGPL-3.0 (open core) | Yes | Native (OTLP) | not published in USD | ||
| – | Patronus AIEvaluation | – | no npm SDK | Proprietary | Enterprise only | Unknown | not published in USD | ||
| – | Pydantic EvalsEvaluation | – | no npm SDK | MIT | Yes | Native (OTLP) | Free | ||
| – | RagasEvaluation | – | no npm SDK | Apache-2.0 | Yes | Unknown | Free | ||
| – | Scorable (formerly Root Signals)Evaluation | – | SDK not mapped | Proprietary | Enterprise only | Native (OTLP) | $19/seat/mo | ||
| – | SigNoz (LLM observability)Tracing & observability | – | SDK-less | MIT (open core) | Yes | Native (OTLP) | $49/mo | ||
| – | TruLensEvaluation | – | no npm SDK | MIT | Yes | Native (OTLP) | Free | ||
| – | UpTrainEvaluation | – | no npm SDK | Apache-2.0 | Yes | Unknown | Contact sales |
GitHub stars
| # | Tool | Repository | GitHub stars | 12 weeks | Licence | Self-host | OTel | Entry price |
|---|---|---|---|---|---|---|---|---|
| 1 | LiteLLMGateways with observability | BerriAI/litellm | 60,298 | series starts 2026-W40 | MIT (open core) | Yes | Exporter / bridge | Contact sales |
| 2 | LangfuseTracing & observability | langfuse/langfuse | 35,491 | series starts 2026-W40 | MIT (open core) | Yes | Native (OTLP) | $29/mo |
| 3 | SigNoz (LLM observability)Tracing & observability | SigNoz/signoz | 32,308 | series starts 2026-W40 | MIT (open core) | Yes | Native (OTLP) | $49/mo |
| 4 | PromptfooEvaluation | promptfoo/promptfoo | 25,794 | series starts 2026-W40 | MIT (open core) | Yes | Native (OTLP) | Contact sales |
| 5 | OpikTracing & observability | comet-ml/opik | 22,436 | series starts 2026-W40 | Apache-2.0 | Yes | Native (OTLP) | $19/mo |
| 6 | OpenObserve (LLM observability)Tracing & observability | openobserve/openobserve | 22,287 | series starts 2026-W40 | AGPL-3.0 (open core) | Yes | Native (OTLP) | not published in USD |
| 7 | DeepEval / Confident AIEvaluation | confident-ai/deepeval | 18,682 | series starts 2026-W40 | Apache-2.0 (open core) | Enterprise only | Native (OTLP) | $200/mo |
| 8 | RagasEvaluation | vibrantlabsai/ragas | 15,952 | series starts 2026-W40 | Apache-2.0 | Yes | Unknown | Free |
| 9 | lm-evaluation-harnessEvaluation | EleutherAI/lm-evaluation-harness | 14,147 | series starts 2026-W40 | MIT | Yes | Unknown | Free |
| 10 | PortkeyGateways with observability | Portkey-AI/gateway | 13,144 | series starts 2026-W40 | MIT (open core) | Yes | Native (OTLP) | $49/mo |
| 11 | Arize PhoenixTracing & observability | Arize-ai/phoenix | 11,744 | series starts 2026-W40 | Elastic-2.0 (source-available) | Yes | Native (OTLP) | Free |
| 12 | garakEvaluation | NVIDIA/garak | 9,490 | series starts 2026-W40 | Apache-2.0 | Yes | Unknown | Free |
| 13 | Evidently AIEvaluation | evidentlyai/evidently | 7,975 | series starts 2026-W40 | Apache-2.0 (open core) | Yes | Native (OTLP) | Free |
| 14 | Guardrails AIGuardrails & safety | guardrails-ai/guardrails | 7,495 | series starts 2026-W40 | Apache-2.0 | Yes | Exporter / bridge | Free |
| 15 | Traceloop / OpenLLMetryTracing & observability | traceloop/openllmetry | 7,474 | series starts 2026-W40 | Apache-2.0 (open core) | Enterprise only | Native (OTLP) | Contact sales |
| 16 | NeMo GuardrailsGuardrails & safety | NVIDIA-NeMo/Guardrails | 7,260 | series starts 2026-W40 | Apache-2.0 | Yes | Exporter / bridge | Free |
| 17 | HeliconeTracing & observability | Helicone/helicone | 6,206 | series starts 2026-W40 | Apache-2.0 | Yes | Unknown | $79/mo |
| 18 | AgentOpsTracing & observability | AgentOps-AI/agentops | 5,887 | series starts 2026-W40 | MIT | Yes | Native (OTLP) | $40/mo |
| 19 | GiskardEvaluation | Giskard-AI/giskard-oss | 5,869 | series starts 2026-W40 | Apache-2.0 (open core) | Yes | Unknown | Contact sales |
| 20 | LangWatchTracing & observability | langwatch/langwatch | 4,923 | series starts 2026-W40 | Apache-2.0 (open core) | Yes | Native (OTLP) | not published in USD |
| 21 | AgentaPrompt management | Agenta-AI/agenta | 4,814 | series starts 2026-W40 | MIT (open core) | Yes | Native (OTLP) | $29/mo |
| 22 | LatitudeTracing & observability | latitude-dev/latitude-llm | 4,714 | series starts 2026-W40 | MIT | Yes | Native (OTLP) | $99/mo |
| 23 | Pydantic LogfireTracing & observability | pydantic/logfire | 4,509 | series starts 2026-W40 | Proprietary | Enterprise only | Native (OTLP) | $49/mo |
| 24 | TruLensEvaluation | truera/trulens | 3,592 | series starts 2026-W40 | MIT | Yes | Native (OTLP) | Free |
| 25 | LaminarTracing & observability | lmnr-ai/lmnr | 3,360 | series starts 2026-W40 | Apache-2.0 (open core) | Yes | Native (OTLP) | $30/mo |
| 26 | Inspect AIEvaluation | UKGovernmentBEIS/inspect_ai | 2,953 | series starts 2026-W40 | MIT | Yes | Unknown | Free |
| 27 | OpenLITTracing & observability | openlit/openlit | 2,829 | series starts 2026-W40 | Apache-2.0 | Yes | Native (OTLP) | Free |
| 28 | UpTrainEvaluation | uptrain-ai/uptrain | 2,366 | series starts 2026-W40 | Apache-2.0 | Yes | Unknown | Contact sales |
| 29 | Future AGIEvaluation | future-agi/future-agi | 2,116 | series starts 2026-W40 | Apache-2.0 (open core) | Yes | Native (OTLP) | $250/mo |
| 30 | OpenEvalsEvaluation | langchain-ai/openevals | 1,216 | series starts 2026-W40 | MIT | Yes | Unknown | Free |
| 31 | W&B WeaveTracing & observability | wandb/weave | 1,137 | series starts 2026-W40 | Apache-2.0 (open core) | Yes | Native (OTLP) | $60/mo |
| 32 | LangSmithTracing & observability | langchain-ai/langsmith-sdk | 1,069 | series starts 2026-W40 | Proprietary | Enterprise only | Native (OTLP) | $39/seat/mo |
| 33 | Judgment Labs (Judgeval)Tracing & observability | JudgmentLabs/judgeval | 1,063 | series starts 2026-W40 | Apache-2.0 (open core) | No | Native (OTLP) | Contact sales |
| 34 | PromptLayerPrompt management | MagnivOrg/prompt-layer-library | 785 | series starts 2026-W40 | Proprietary | Enterprise only | Native (OTLP) | $49/mo |
| 35 | AgentEvalsEvaluation | langchain-ai/agentevals | 740 | series starts 2026-W40 | MIT | Yes | Unknown | Free |
| 36 | Invariant GuardrailsGuardrails & safety | invariantlabs-ai/invariant | 467 | series starts 2026-W40 | Apache-2.0 | Yes | Unknown | Free |
| 37 | AI Verify MoonshotbetaEvaluation | aiverify-foundation/moonshot | 355 | series starts 2026-W40 | Apache-2.0 | Yes | Unknown | Free |
| 38 | Arthur EngineEvaluation | arthur-ai/arthur-engine | 91 | series starts 2026-W40 | MIT (open core) | Yes | Native (OTLP) | $60/mo |
| 39 | RespanTracing & observability | respanai/respan | 48 | series starts 2026-W40 | Proprietary | Enterprise only | Native (OTLP) | $199/mo |
| 40 | BraintrustTracing & observability | braintrustdata/braintrust-sdk-javascript | 28 | series starts 2026-W40 | Proprietary | Enterprise only | Native (OTLP) | $249/mo |
| 41 | OpenlayerEvaluation | openlayer-ai/openlayer-python | 18 | series starts 2026-W40 | Proprietary | Enterprise only | Native (OTLP) | Contact sales |
| 42 | OkareoEvaluation | okareo-ai/okareo-python-sdk | 8 | series starts 2026-W40 | Proprietary | Enterprise only | Native (OTLP) | $199/mo |
| – | Arize AXTracing & observability | – | no public repo | Proprietary | Enterprise only | Native (OTLP) | $50/mo | |
| – | Azure AI Evaluation SDK (Microsoft Foundry)previewEvaluation | – | no public repo | MIT | Yes | Unknown | not published in USD | |
| – | Cloudflare AI GatewayGateways with observability | – | SDK-less | Proprietary | Unknown | Exporter / bridge | Free | |
| – | Coralogix AI Observability (AI Center)Tracing & observability | – | SDK-less | Proprietary | Unknown | Native (OTLP) | not published in USD | |
| – | Datadog LLM ObservabilityTracing & observability | – | proxy-only | Proprietary | Unknown | Native (OTLP) | $160/mo | |
| – | Deepchecks LLM EvaluationEvaluation | – | closed-source | Proprietary | Enterprise only | Unknown | Contact sales | |
| – | Fiddler AITracing & observability | – | no public repo | Proprietary | Enterprise only | Native (OTLP) | not published in USD | |
| – | GalileoEvaluation | – | no public repo | Proprietary | Enterprise only | Native (OTLP) | Contact sales | |
| – | GalteaEvaluation | – | no public repo | Proprietary | Enterprise only | Native (OTLP) | Contact sales | |
| – | HoneyHiveTracing & observability | – | no public repo | Proprietary | Enterprise only | Native (OTLP) | Contact sales | |
| – | Lakera GuardGuardrails & safety | – | closed-source | Proprietary | Enterprise only | Unknown | Contact sales | |
| – | Last9 (AI / LLM observability)Tracing & observability | – | no public repo | Proprietary | Unknown | Native (OTLP) | not published in USD | |
| – | LlamaFirewallGuardrails & safety | – | no public repo | MIT | Yes | Unknown | Free | |
| – | LunaryTracing & observability | – | no public repo | Unknown | Enterprise only | Native (OTLP) | $20/seat/mo | |
| – | Maxim AIEvaluation | – | no public repo | Proprietary | Enterprise only | Native (OTLP) | Contact sales | |
| – | MLflowTracing & observability | – | no public repo | Apache-2.0 | Yes | Native (OTLP) | Free | |
| – | Orq.aiGateways with observability | – | no public repo | Proprietary | Enterprise only | Native (OTLP) | not published in USD | |
| – | Patronus AIEvaluation | – | closed-source | Proprietary | Enterprise only | Unknown | not published in USD | |
| – | Pydantic EvalsEvaluation | – | no public repo | MIT | Yes | Native (OTLP) | Free | |
| – | RaindropTracing & observability | – | no public repo | Proprietary | Unknown | Native (OTLP) | $299/mo | |
| – | Scorable (formerly Root Signals)Evaluation | – | no public repo | Proprietary | Enterprise only | Native (OTLP) | $19/seat/mo | |
| – | ScorecardEvaluation | – | no public repo | Proprietary | Unknown | Native (OTLP) | $299/mo |
Only each tool's headline package is ranked; its other packages are on its own page. Tools without a number are listed by name: no PyPI SDK / no npm SDK means the vendor publishes no package in that registry; SDK not mapped means its package's registry metadata does not link the vendor, so it is not counted; SDK-less means the product ingests OpenTelemetry or proxies requests without its own SDK; proxy-only means no package measures the product alone; closed-source means there is no public repository. None of these is a zero.
Browse by category
- Tracing & observability (45) Record traces, spans, costs and latency of LLM calls and agent runs.
- Evaluation (53) Score LLM and agent outputs with datasets, metrics, LLM judges or human review.
- Guardrails & safety (19) Check prompts and responses at runtime for injection, leakage, toxicity or policy violations.
- Prompt management (21) Version, test and deploy prompts outside application code.
- Gateways with observability (10) Proxy LLM API calls and log every request, cost and latency on the way.
Vendor and want a richer profile? See sponsorship. It never changes ranks, numbers or facts. Data refreshed weekly.