Evaluation
Score LLM and agent outputs with datasets, metrics, LLM judges or human review. 53 listed tools are in this category (a tool can be in several). Tables are ordered by package downloads or stars, never by payment.
Week 2026-W40 (2026-09-28 to 2026-10-04, UTC), measured 2026-10-07 20:16 UTC.
PyPI weekly downloads
| # | Tool | Package | PyPI downloads (week) | 12 weeks | vs prior week | Entry price |
|---|---|---|---|---|---|---|
| 2 | LangSmithTracing & observability | langsmithbundled | 18,866,051 | +8.1% | $39/seat/mo | |
| 3 | LangfuseTracing & observability | langfuse | 6,306,471 | +12.7% | $29/mo | |
| 4 | MLflowTracing & observability | mlflow-tracingbundled | 3,616,315 | +3.7% | Free | |
| 5 | Pydantic LogfireTracing & observability | logfirebundled | 3,488,038 | +14.3% | $49/mo | |
| 6 | Pydantic EvalsEvaluation | pydantic-evalsbundled | 2,121,173 | +10.5% | Free | |
| 7 | BraintrustTracing & observability | braintrust | 1,650,096 | 0.0% | $249/mo | |
| 8 | Inspect AIEvaluation | inspect-ai | 907,448 | +33.1% | Free | |
| 10 | DeepEval / Confident AIEvaluation | deepeval | 686,214 | +9.7% | $200/mo | |
| 11 | Traceloop / OpenLLMetryTracing & observability | traceloop-sdk | 455,110 | +7.6% | Contact sales | |
| 12 | OpikTracing & observability | opik | 437,574 | +2.0% | $19/mo | |
| 13 | RagasEvaluation | ragas | 279,757 | +9.5% | Free | |
| 14 | lm-evaluation-harnessEvaluation | lm-eval | 270,051 | +10.3% | Free | |
| 15 | W&B WeaveTracing & observability | weave | 239,276 | +28.3% | $60/mo | |
| 16 | Evidently AIEvaluation | evidently | 198,068 | +7.3% | Free | |
| 17 | Arize PhoenixTracing & observability | arize-phoenix | 161,687 | +13.6% | Free | |
| 19 | Arize AXTracing & observability | arize-otel | 84,266 | -5.1% | $50/mo | |
| 20 | OpenLITTracing & observability | openlit | 72,503 | +14.5% | Free | |
| 21 | LunaryTracing & observability | lunary | 49,317 | +22.9% | $20/seat/mo | |
| 22 | OpenlayerEvaluation | openlayer | 37,719 | -16.7% | Contact sales | |
| 23 | Azure AI Evaluation SDK (Microsoft Foundry)previewEvaluation | azure-ai-evaluation | 33,482 | -4.5% | not published in USD | |
| 24 | Orq.aiGateways with observability | orq-ai-sdk | 30,379 | +14.1% | not published in USD | |
| 26 | PromptLayerPrompt management | promptlayer | 23,137 | +10.5% | $49/mo | |
| 27 | Judgment Labs (Judgeval)Tracing & observability | judgeval | 19,443 | +39.3% | Contact sales | |
| 28 | garakEvaluation | garak | 11,417 | +25.8% | Free | |
| 29 | AgentaPrompt management | agenta | 10,423 | +38.6% | $29/mo | |
| 30 | GalileoEvaluation | galileo | 9,821 | +46.8% | Contact sales | |
| 31 | TruLensEvaluation | trulens | 6,198 | +28.0% | Free | |
| 32 | GiskardEvaluation | giskard | 5,667 | +93.3% | Contact sales | |
| 33 | HeliconeTracing & observability | helicone-helpers | 4,840 | +20.7% | $79/mo | |
| 34 | PromptfooEvaluation | promptfoo | 3,451 | +13.7% | Contact sales | |
| 35 | HoneyHiveTracing & observability | honeyhive | 3,245 | +38.9% | Contact sales | |
| 36 | LatitudeTracing & observability | latitude-telemetry | 2,379 | -18.5% | $99/mo | |
| 37 | GalteaEvaluation | galtea | 2,373 | +23.0% | Contact sales | |
| 38 | OkareoEvaluation | okareo | 2,051 | +0.3% | $199/mo | |
| 41 | Fiddler AITracing & observability | fiddler-otel | 791 | -21.0% | not published in USD | |
| 42 | ScorecardEvaluation | scorecard-ai | 621 | +3.8% | $299/mo | |
| 43 | UpTrainEvaluation | uptrain | 423 | +46.4% | Contact sales | |
| 44 | AI Verify MoonshotbetaEvaluation | aiverify-moonshot | 370 | +65.9% | Free | |
| 45 | Scorable (formerly Root Signals)Evaluation | scorable | 118 | +57.3% | $19/seat/mo | |
| – | AgentEvalsEvaluation | – | SDK not mapped | Free | ||
| – | Arthur EngineEvaluation | – | SDK not mapped | $60/mo | ||
| – | Coralogix AI Observability (AI Center)Tracing & observability | – | SDK-less | not published in USD | ||
| – | Datadog LLM ObservabilityTracing & observability | – | proxy-only | $160/mo | ||
| – | Deepchecks LLM EvaluationEvaluation | – | no PyPI SDK | Contact sales | ||
| – | Future AGIEvaluation | – | SDK not mapped | $250/mo | ||
| – | LaminarTracing & observability | – | SDK not mapped | $30/mo | ||
| – | LangWatchTracing & observability | – | SDK not mapped | not published in USD | ||
| – | Maxim AIEvaluation | – | SDK not mapped | Contact sales | ||
| – | OpenEvalsEvaluation | – | SDK not mapped | Free | ||
| – | OpenObserve (LLM observability)Tracing & observability | – | SDK-less | not published in USD | ||
| – | Patronus AIEvaluation | – | no PyPI SDK | not published in USD | ||
| – | RaindropTracing & observability | – | SDK not mapped | $299/mo | ||
| – | RespanTracing & observability | – | SDK not mapped | $199/mo |
npm weekly downloads
| # | Tool | Package | npm downloads (week) | 12 weeks | vs prior week | Entry price |
|---|---|---|---|---|---|---|
| 1 | LangSmithTracing & observability | langsmith | 8,568,634 | +17.4% | $39/seat/mo | |
| 2 | LangfuseTracing & observability | @langfuse/tracing | 3,268,181 | +12.2% | $29/mo | |
| 3 | BraintrustTracing & observability | braintrust | 2,046,537 | +2.7% | $249/mo | |
| 4 | PromptfooEvaluation | promptfoo | 1,020,946 | +26.9% | Contact sales | |
| 5 | W&B WeaveTracing & observability | weave | 623,550 | +6.9% | $60/mo | |
| 7 | OpenEvalsEvaluation | openevals | 161,288 | +13.3% | Free | |
| 8 | Traceloop / OpenLLMetryTracing & observability | @traceloop/node-server-sdk | 147,877 | -0.2% | Contact sales | |
| 9 | RaindropTracing & observability | raindrop-ai | 99,912 | -13.8% | $299/mo | |
| 10 | LaminarTracing & observability | @lmnr-ai/lmnr | 77,663 | +4.2% | $30/mo | |
| 11 | Arize PhoenixTracing & observability | @arizeai/phoenix-otel | 73,285 | -37.5% | Free | |
| 12 | LangWatchTracing & observability | langwatch | 50,105 | +4.9% | not published in USD | |
| 13 | Orq.aiGateways with observability | @orq-ai/node | 43,521 | -41.8% | not published in USD | |
| 14 | OpikTracing & observability | opik | 42,992 | -8.3% | $19/mo | |
| 15 | Judgment Labs (Judgeval)Tracing & observability | judgeval | 34,312 | -5.8% | Contact sales | |
| 16 | Pydantic LogfireTracing & observability | @pydantic/logfire-node | 31,486 | -59.7% | $49/mo | |
| 17 | LatitudeTracing & observability | @latitude-data/telemetry | 7,507 | +43.1% | $99/mo | |
| 18 | PromptLayerPrompt management | promptlayer | 6,509 | +31.4% | $49/mo | |
| 19 | HeliconeTracing & observability | @helicone/helpers | 4,098 | -2.5% | $79/mo | |
| 20 | ScorecardEvaluation | scorecard-ai | 2,117 | +2.7% | $299/mo | |
| 21 | GalileoEvaluation | galileo | 1,810 | +32.3% | Contact sales | |
| 22 | OpenLITTracing & observability | openlit | 1,231 | -43.1% | Free | |
| 23 | OpenlayerEvaluation | openlayer | 308 | -50.0% | Contact sales | |
| 24 | HoneyHiveTracing & observability | @honeyhive/api-client | 71 | -65.0% | Contact sales | |
| 25 | RespanTracing & observability | @respan/respan | 69 | -12.7% | $199/mo | |
| 27 | Future AGIEvaluation | @future-agi/ai-evaluation | 3 | -66.7% | $250/mo | |
| 28 | OkareoEvaluation | okareo-ts-sdk | 3 | -81.2% | $199/mo | |
| – | AgentaPrompt management | – | no npm SDK | $29/mo | ||
| – | AgentEvalsEvaluation | – | SDK not mapped | Free | ||
| – | AI Verify MoonshotbetaEvaluation | – | no npm SDK | Free | ||
| – | Arize AXTracing & observability | – | no npm SDK | $50/mo | ||
| – | Arthur EngineEvaluation | – | no npm SDK | $60/mo | ||
| – | Azure AI Evaluation SDK (Microsoft Foundry)previewEvaluation | – | no npm SDK | not published in USD | ||
| – | Coralogix AI Observability (AI Center)Tracing & observability | – | SDK-less | not published in USD | ||
| – | Datadog LLM ObservabilityTracing & observability | – | proxy-only | $160/mo | ||
| – | Deepchecks LLM EvaluationEvaluation | – | no npm SDK | Contact sales | ||
| – | DeepEval / Confident AIEvaluation | – | SDK not mapped | $200/mo | ||
| – | Evidently AIEvaluation | – | no npm SDK | Free | ||
| – | Fiddler AITracing & observability | – | no npm SDK | not published in USD | ||
| – | GalteaEvaluation | – | no npm SDK | Contact sales | ||
| – | garakEvaluation | – | no npm SDK | Free | ||
| – | GiskardEvaluation | – | no npm SDK | Contact sales | ||
| – | Inspect AIEvaluation | – | no npm SDK | Free | ||
| – | lm-evaluation-harnessEvaluation | – | no npm SDK | Free | ||
| – | LunaryTracing & observability | – | SDK not mapped | $20/seat/mo | ||
| – | Maxim AIEvaluation | – | SDK not mapped | Contact sales | ||
| – | MLflowTracing & observability | – | no npm SDK | Free | ||
| – | OpenObserve (LLM observability)Tracing & observability | – | SDK-less | not published in USD | ||
| – | Patronus AIEvaluation | – | no npm SDK | not published in USD | ||
| – | Pydantic EvalsEvaluation | – | no npm SDK | Free | ||
| – | RagasEvaluation | – | no npm SDK | Free | ||
| – | Scorable (formerly Root Signals)Evaluation | – | SDK not mapped | $19/seat/mo | ||
| – | TruLensEvaluation | – | no npm SDK | Free | ||
| – | UpTrainEvaluation | – | no npm SDK | Contact sales |
GitHub stars
| # | Tool | Repository | GitHub stars | 12 weeks | Entry price |
|---|---|---|---|---|---|
| 2 | LangfuseTracing & observability | langfuse/langfuse | 35,491 | series starts 2026-W40 | $29/mo |
| 4 | PromptfooEvaluation | promptfoo/promptfoo | 25,794 | series starts 2026-W40 | Contact sales |
| 5 | OpikTracing & observability | comet-ml/opik | 22,436 | series starts 2026-W40 | $19/mo |
| 6 | OpenObserve (LLM observability)Tracing & observability | openobserve/openobserve | 22,287 | series starts 2026-W40 | not published in USD |
| 7 | DeepEval / Confident AIEvaluation | confident-ai/deepeval | 18,682 | series starts 2026-W40 | $200/mo |
| 8 | RagasEvaluation | vibrantlabsai/ragas | 15,952 | series starts 2026-W40 | Free |
| 9 | lm-evaluation-harnessEvaluation | EleutherAI/lm-evaluation-harness | 14,147 | series starts 2026-W40 | Free |
| 11 | Arize PhoenixTracing & observability | Arize-ai/phoenix | 11,744 | series starts 2026-W40 | Free |
| 12 | garakEvaluation | NVIDIA/garak | 9,490 | series starts 2026-W40 | Free |
| 13 | Evidently AIEvaluation | evidentlyai/evidently | 7,975 | series starts 2026-W40 | Free |
| 15 | Traceloop / OpenLLMetryTracing & observability | traceloop/openllmetry | 7,474 | series starts 2026-W40 | Contact sales |
| 17 | HeliconeTracing & observability | Helicone/helicone | 6,206 | series starts 2026-W40 | $79/mo |
| 19 | GiskardEvaluation | Giskard-AI/giskard-oss | 5,869 | series starts 2026-W40 | Contact sales |
| 20 | LangWatchTracing & observability | langwatch/langwatch | 4,923 | series starts 2026-W40 | not published in USD |
| 21 | AgentaPrompt management | Agenta-AI/agenta | 4,814 | series starts 2026-W40 | $29/mo |
| 22 | LatitudeTracing & observability | latitude-dev/latitude-llm | 4,714 | series starts 2026-W40 | $99/mo |
| 23 | Pydantic LogfireTracing & observability | pydantic/logfire | 4,509 | series starts 2026-W40 | $49/mo |
| 24 | TruLensEvaluation | truera/trulens | 3,592 | series starts 2026-W40 | Free |
| 25 | LaminarTracing & observability | lmnr-ai/lmnr | 3,360 | series starts 2026-W40 | $30/mo |
| 26 | Inspect AIEvaluation | UKGovernmentBEIS/inspect_ai | 2,953 | series starts 2026-W40 | Free |
| 27 | OpenLITTracing & observability | openlit/openlit | 2,829 | series starts 2026-W40 | Free |
| 28 | UpTrainEvaluation | uptrain-ai/uptrain | 2,366 | series starts 2026-W40 | Contact sales |
| 29 | Future AGIEvaluation | future-agi/future-agi | 2,116 | series starts 2026-W40 | $250/mo |
| 30 | OpenEvalsEvaluation | langchain-ai/openevals | 1,216 | series starts 2026-W40 | Free |
| 31 | W&B WeaveTracing & observability | wandb/weave | 1,137 | series starts 2026-W40 | $60/mo |
| 32 | LangSmithTracing & observability | langchain-ai/langsmith-sdk | 1,069 | series starts 2026-W40 | $39/seat/mo |
| 33 | Judgment Labs (Judgeval)Tracing & observability | JudgmentLabs/judgeval | 1,063 | series starts 2026-W40 | Contact sales |
| 34 | PromptLayerPrompt management | MagnivOrg/prompt-layer-library | 785 | series starts 2026-W40 | $49/mo |
| 35 | AgentEvalsEvaluation | langchain-ai/agentevals | 740 | series starts 2026-W40 | Free |
| 37 | AI Verify MoonshotbetaEvaluation | aiverify-foundation/moonshot | 355 | series starts 2026-W40 | Free |
| 38 | Arthur EngineEvaluation | arthur-ai/arthur-engine | 91 | series starts 2026-W40 | $60/mo |
| 39 | RespanTracing & observability | respanai/respan | 48 | series starts 2026-W40 | $199/mo |
| 40 | BraintrustTracing & observability | braintrustdata/braintrust-sdk-javascript | 28 | series starts 2026-W40 | $249/mo |
| 41 | OpenlayerEvaluation | openlayer-ai/openlayer-python | 18 | series starts 2026-W40 | Contact sales |
| 42 | OkareoEvaluation | okareo-ai/okareo-python-sdk | 8 | series starts 2026-W40 | $199/mo |
| – | Arize AXTracing & observability | – | no public repo | $50/mo | |
| – | Azure AI Evaluation SDK (Microsoft Foundry)previewEvaluation | – | no public repo | not published in USD | |
| – | Coralogix AI Observability (AI Center)Tracing & observability | – | SDK-less | not published in USD | |
| – | Datadog LLM ObservabilityTracing & observability | – | proxy-only | $160/mo | |
| – | Deepchecks LLM EvaluationEvaluation | – | closed-source | Contact sales | |
| – | Fiddler AITracing & observability | – | no public repo | not published in USD | |
| – | GalileoEvaluation | – | no public repo | Contact sales | |
| – | GalteaEvaluation | – | no public repo | Contact sales | |
| – | HoneyHiveTracing & observability | – | no public repo | Contact sales | |
| – | LunaryTracing & observability | – | no public repo | $20/seat/mo | |
| – | Maxim AIEvaluation | – | no public repo | Contact sales | |
| – | MLflowTracing & observability | – | no public repo | Free | |
| – | Orq.aiGateways with observability | – | no public repo | not published in USD | |
| – | Patronus AIEvaluation | – | closed-source | not published in USD | |
| – | Pydantic EvalsEvaluation | – | no public repo | Free | |
| – | RaindropTracing & observability | – | no public repo | $299/mo | |
| – | Scorable (formerly Root Signals)Evaluation | – | no public repo | $19/seat/mo | |
| – | ScorecardEvaluation | – | no public repo | $299/mo |