Dataset

A datasheet for release 2606

This page describes what release 2606 contains, how it was produced, how to load it, and where it should not be trusted. Its structure follows the datasheet model for datasets. The authoritative copy of each figure is the dataset card on Hugging Face, which is rebuilt from the published files.

Composition

The release has eleven configurations: ten metadata tables with 20,637,822 rows in total, and one tier of article text. Each has one grain, one schema, and a checksum for every file.

Table 2. Configurations on Hugging Face. Row counts are exact.
Configuration Rows One row is
articles 5,135,607 One article: its citation counts and parser-quality signals.
articles-text-p0 908,270 One article's retained Markdown text, categories, and provenance hashes.
categories 529,402 One category: article and citation totals across its members.
clusters 1,511,504 One deduplicated citation record and the exact-match cluster it belongs to.
publishers 20,000 One normalized publisher string among the most cited.
templates 37 One citation-template family and parser outcome.
refs 256,564 One dump block: raw counts of <ref> tags and citation templates.
streams 256,564 One dump block: coverage and custody signals.
redirects 11,894,129 One redirect edge, with flags for a missing target or a disambiguation target.
disambig-nodes 310,529 One disambiguation page and its base title.
disambig-links 723,486 One candidate link from a disambiguation page to an article.

The text of all 7,073,577 retained articles is in a public bucket, in 1,003 shards. Articles are assigned to one of four delivery priorities (Table 3). The dataset repository carries the first priority inline as articles-text-p0. Disambiguation pages are excluded from the article text and kept as relation tables instead.

Table 3. Article text by delivery priority. Priorities are exclusive and advisory.
Priority Articles Share (%)
p0 908,270 12.8
p1 4,665,480 66.0
p2 1,208,553 17.1
p3 291,274 4.1

Collection and processing

The source is the English Wikipedia XML dump of 1 June 2026 (enwiki-20260601): 8,531,070 pages in 256,564 compressed blocks. The Wikiparse package parsed each block once into a read-only store, which is the source of record for every published table. No MediaWiki software or API was used. Each text row carries the hash of its article and of its source archive, so any row can be traced back to the dump.

Citations

Each {{cite …}} template and <ref> tag becomes one record that belongs to its article. Repeated citations within an article collapse into one record. The parser labels every record with an outcome (Table 4).

Table 4. Parser outcomes for 45,003,332 citation records.
Outcome Records Share (%) Meaning
Clean 22,571,325 50.2 Every field the parser expected was present and normalized.
Flagged 22,359,745 49.7 Readable, but a field was missing or unusual, most often the author.
Failed 72,262 0.2 Citations on pages whose markup the parser could not read.

An outcome is the parser's own operational label, not a judgment of the cited source. A flag lets you exclude uncertain rows before an analysis instead of discovering them afterward. Failures are recorded per page: one unreadable page removes every citation on it. The citation records cover 254,939 of the 256,564 blocks (99.37%). The streams table lists the 1,625 blocks without coverage.

The records are published as per-article counts in articles, as template-family totals in templates, and as 1,511,504 exact-match clusters in clusters. A cluster groups citations whose normalized fields match exactly. It does not certify that they cite the same work.

Categories, redirects, and disambiguation

The categories table gives article and citation totals for 529,402 categories, which makes it a map of where citation data is dense or sparse. Categories overlap: an article can belong to dozens, so category totals must not be summed into a population count.

Redirects (11,894,129 edges) and disambiguation pages (310,529 pages and 723,486 candidate links) are relation tables, not article text. The links come from fixed matching rules without human review. Use them as an index, not as a finished taxonomy.

Subject bundles

The bundles/ folder holds 44 Markdown collections: 38 single fields and 6 broader divisions, about 4.8 GB in total. They cover 415,766 articles, 99.47% of a 418,016-title selection, together with each article's citations. Every article states whether it is a biography.

Membership comes from category-graph closure plus lead-sentence and infobox rules, with no model calls. Each bundle is scored against a fixed set of sentinel articles, but no domain expert has reviewed any field. Read a bundle's own README before relying on its membership.

Access

Every configuration loads with the Hugging Face datasets library, and the Parquet files read directly in pandas, PyArrow, or DuckDB:

from datasets import load_dataset

refs = load_dataset("robbd/wikiparse-2606", "refs", split="train")
text = load_dataset("robbd/wikiparse-2606", "articles-text-p0",
                    split="train", streaming=True)
import pandas as pd

refs = pd.read_parquet(
    "hf://datasets/robbd/wikiparse-2606/data/refs/0000.parquet")
SELECT count(*)
FROM 'hf://datasets/robbd/wikiparse-2606/data/articles-text-p0/…/p0.parquet';

The … in the DuckDB recipe stands for one shard range (for example shards/000000-000256); point DuckDB at one shard or a narrow range. A glob across all 1,003 shards goes through the Hub's tree API and hits its rate limit. For a full scan, use the streaming datasets recipe. Three text rows exceed the Hub viewer's 2 MB row limit. They load normally in every library.

Limitations

License and provenance

The data is derived from English Wikipedia and distributed under CC BY-SA 4.0. Credit Wikipedia's contributors and share derived data under the same license. The Wikimedia Foundation does not endorse this release.

checksums.sha256 covers every file in the dataset repository, and provenance.json binds each public file to its row count and Arrow schema. The parser and the release tooling are in the source repository.