Skip to content
LLM Observability Index

Methodology

What is measured

  • PyPI weekly downloads per package, from the public ClickPy dataset (PyPI download data via ClickPy, run by ClickHouse), read with one SQL query per week against ClickHouse's documented public read-only endpoint:
    SELECT project, toMonday(date) AS week, sum(count) FROM pypi.pypi_downloads_per_day
WHERE date BETWEEN <Monday> AND <Sunday> AND project IN (<mapped packages>) GROUP BY project, week
    We never crawl ClickPy's web pages.
  • npm weekly downloads per package, from the npm registry's documented downloads API (/downloads/range/): one bulk call for unscoped packages and one call per scoped package, at least 2 seconds apart.
  • GitHub stars of each tool's main public repository, from GET /repos/{owner}/{repo} (GitHub REST API, unauthenticated, one request at a time, within the published rate-limit headers).

A week is Monday to Sunday (UTC). The weekly refresh (Tuesday 07:30 UTC) measures the last complete week whose Sunday is at least two days old, because ClickPy lags about a day. Every number on the site links to a public query that reproduces it.

Latest complete week: 2026-W40 (2026-09-28 to 2026-10-04), measured 2026-10-07.

History

The first build pulled 12 weeks of PyPI and npm history from the same daily data; those weeks are marked backfilled in the data files. GitHub has no star history in this API, so the stars series starts on the first measured week and nothing earlier is made up.

Data note (2026-08-24): Downloads of many packages fell sharply from the week of 2026-08-24 in both ClickPy and PyPI Stats (litellm, mirrors excluded: 177,335,553 in the week of 2026-08-17, 37,111,274 in the week of 2026-08-24, about 19-23 million a week since). The same step appears in lmnr, inspect-ai, arize-phoenix and others, so August-to-September changes largely reflect a change in automated download traffic, not in how much the products are used. source (checked 2026-10-07)

What the numbers are not

  • Not people, customers or deduplicated installs. One team can download a package thousands of times.
  • Inflated by CI and dependencies. Continuous-integration runs and packages installed automatically by other packages count every time.
  • Not quality. A package being installed says nothing about whether the product is good.
  • Not market share. PyPI, npm and stars are separate columns and are never summed or blended into a score, and one tool's packages are never added together because they can be installed together.
  • Stars are bookmarks, not usage.

Packages flagged bundled are required dependencies of other popular packages (from their PyPI metadata): arize-phoenix-otel, langsmith, litellm, mlflow-tracing, pydantic-evals, logfire.

Package mapping

A package counts for a tool only when its registry metadata (PyPI project_urls/home_page, npm repository/homepage) points to the vendor's own domain or GitHub organisation. Generic names without such a link are not mapped. Each tool has at most one headline package per registry, and only headline packages are ranked; its current, legacy and integration packages appear on its page with their own series. Mappings are re-checked every 90 days. No other evidence counts: an author e-mail address, npm's bugs field, PyPI's download_url, or a vendor's docs naming the package are not enough on their own. 23 packages named in vendor docs fail this rule today; each is listed on its tool page as not measured, with its registry link.

Of 64 listed tools, 51 have mapped packages; 5 publish an SDK whose registry metadata does not link the vendor, so they are listed by name without a number; 4 are SDK-less (they ingest OpenTelemetry or proxy requests), 1 are proxy-only and 3 are closed-source. Tools without a number are never shown as 0.

Cross-check

Every week, 5 headline PyPI packages (rotating through all of them) are compared with PyPI Stats' daily downloads without known mirrors for the same 7 days. A difference within 5% is shown as within 5%; anything larger is shown as mismatch, never hidden. Requests are at least 15 seconds apart, cached, and stop at the first rate-limit answer (the rest are marked unavailable). Both services process the same PyPI download logs with different filters, so this checks processing, not independent measurement.

So far: 5 checks completed, 2 mismatches (40%), 0 unavailable.

Failures

A failed request is stored as empty, never as 0. If ClickHouse, npm or GitHub refuses a request (rate limit or quota), that source stops for the run, the week is saved as partial, and the site keeps the last complete week with a stale badge. Complete weekly snapshots are immutable.

Facts: pricing, licence, self-hosting, OpenTelemetry

Every fact comes from the vendor's own website, docs or repository, with the source link and the date it was checked. We do not copy prices or features from review sites. Facts older than 30 days are re-checked by hand. When a fact is not published, it is shown as unknown with the page we checked. Non-USD prices are shown as published, without conversion.

Licence: the SPDX ID of the product's own LICENSE file, or proprietary. The licence filter uses four classes: open source (OSI) only for OSI-approved licences (MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, MPL-2.0 and similar); source-available for published code under non-OSI terms such as Elastic-2.0, BUSL-1.1, SSPL-1.0; proprietary; and unknown when the vendor does not publish a licence. Open core means some parts sit under a separate commercial licence. A free price does not imply an open-source licence.

OpenTelemetry: native means the product ingests OpenTelemetry (OTLP) traces or its SDK is built on OpenTelemetry; exporter / bridge means it emits OpenTelemetry data or accepts it only through a documented bridge.

A tool is listed when its main function includes LLM or agent tracing, evaluation, guardrails, prompt management or an LLM gateway with request logging, and it is available in 2026. Broad ML or APM platforms are listed only through their LLM module. Generic LLM libraries are not listed. Listing is free and is not for sale.

Integrity rules for sponsorship

  • Payment never changes ranks, numbers, facts or which tools are listed.
  • Sponsored cards carry a “Sponsored” label and sit outside the ranked tables.
  • No paid tier is called “verified”.

Disclosure: PyPI download data comes from ClickPy, which ClickHouse runs. ClickHouse also owns Langfuse, which is listed here. ClickHouse has no say over this site, and Langfuse is ranked by the same rules as every other tool.

Details: advertise.

Data sources and privacy

We store only aggregate counts and, for repositories, the name, star count, licence ID, archived flag and last push time. No user lists, contributors, owners or e-mail addresses are stored; an automated test fails the build if a data file contains such fields. This site does not resell API access or data.