Dataset
A datasheet for release 2606
This page describes what release 2606 contains, how it was produced, how to load it, and where it should not be trusted. Its structure follows the datasheet model for datasets. The authoritative copy of each figure is the dataset card on Hugging Face, which is rebuilt from the published files.
Composition
The release has eleven configurations: ten metadata tables with 20,637,822 rows in total, and one tier of article text. Each has one grain, one schema, and a checksum for every file.
| Configuration | Rows | One row is |
|---|---|---|
articles |
5,135,607 | One article: its citation counts and parser-quality signals. |
articles-text-p0 |
908,270 | One article's retained Markdown text, categories, and provenance hashes. |
categories |
529,402 | One category: article and citation totals across its members. |
clusters |
1,511,504 | One deduplicated citation record and the exact-match cluster it belongs to. |
publishers |
20,000 | One normalized publisher string among the most cited. |
templates |
37 | One citation-template family and parser outcome. |
refs |
256,564 | One dump block: raw counts of <ref> tags and citation templates. |
streams |
256,564 | One dump block: coverage and custody signals. |
redirects |
11,894,129 | One redirect edge, with flags for a missing target or a disambiguation target. |
disambig-nodes |
310,529 | One disambiguation page and its base title. |
disambig-links |
723,486 | One candidate link from a disambiguation page to an article. |
The text of all 7,073,577 retained articles is in a
public bucket, in 1,003 shards. Articles are
assigned to one of four delivery priorities (Table 3). The dataset
repository carries the first priority inline as
articles-text-p0. Disambiguation pages are excluded from
the article text and kept as relation tables instead.
| Priority | Articles | Share (%) |
|---|---|---|
p0 |
908,270 | 12.8 |
p1 |
4,665,480 | 66.0 |
p2 |
1,208,553 | 17.1 |
p3 |
291,274 | 4.1 |
Collection and processing
The source is the English Wikipedia XML dump of 1 June 2026
(enwiki-20260601): 8,531,070 pages in 256,564 compressed
blocks. The Wikiparse package parsed each block once into a read-only
store, which is the source of record for every published table. No
MediaWiki software or API was used. Each text row carries the hash of
its article and of its source archive, so any row can be traced back to
the dump.
Citations
Each {{cite …}} template and <ref>
tag becomes one record that belongs to its article. Repeated citations
within an article collapse into one record. The parser labels every
record with an outcome (Table 4).
| Outcome | Records | Share (%) | Meaning |
|---|---|---|---|
| Clean | 22,571,325 | 50.2 | Every field the parser expected was present and normalized. |
| Flagged | 22,359,745 | 49.7 | Readable, but a field was missing or unusual, most often the author. |
| Failed | 72,262 | 0.2 | Citations on pages whose markup the parser could not read. |
An outcome is the parser's own operational label, not a judgment of
the cited source. A flag lets you exclude uncertain rows before an
analysis instead of discovering them afterward. Failures are recorded
per page: one unreadable page removes every citation on it. The
citation records cover 254,939 of the 256,564 blocks (99.37%). The
streams table lists the 1,625 blocks without coverage.
The records are published as per-article counts in
articles, as template-family totals in
templates, and as 1,511,504 exact-match clusters in
clusters. A cluster groups citations whose normalized
fields match exactly. It does not certify that they cite the same work.
Categories, redirects, and disambiguation
The categories table gives article and citation totals for
529,402 categories, which makes it a map of where citation data is
dense or sparse. Categories overlap: an article can belong to dozens,
so category totals must not be summed into a population count.
Redirects (11,894,129 edges) and disambiguation pages (310,529 pages and 723,486 candidate links) are relation tables, not article text. The links come from fixed matching rules without human review. Use them as an index, not as a finished taxonomy.
Subject bundles
The bundles/ folder holds 44 Markdown collections: 38
single fields and 6 broader divisions, about 4.8 GB in total. They
cover 415,766 articles, 99.47% of a 418,016-title selection, together
with each article's citations. Every article states whether it is a
biography.
Membership comes from category-graph closure plus lead-sentence and infobox rules, with no model calls. Each bundle is scored against a fixed set of sentinel articles, but no domain expert has reviewed any field. Read a bundle's own README before relying on its membership.
Access
Every configuration loads with the Hugging Face datasets
library, and the Parquet files read directly in pandas, PyArrow, or
DuckDB:
from datasets import load_dataset
refs = load_dataset("robbd/wikiparse-2606", "refs", split="train")
text = load_dataset("robbd/wikiparse-2606", "articles-text-p0",
split="train", streaming=True)
import pandas as pd
refs = pd.read_parquet(
"hf://datasets/robbd/wikiparse-2606/data/refs/0000.parquet")
SELECT count(*)
FROM 'hf://datasets/robbd/wikiparse-2606/data/articles-text-p0/…/p0.parquet';
The … in the DuckDB recipe stands for one shard range (for example
shards/000000-000256); point DuckDB at one shard or a narrow range. A
glob across all 1,003 shards goes through the Hub's tree API and hits
its rate limit. For a full scan, use the streaming
datasets recipe. Three text rows exceed the Hub viewer's
2 MB row limit. They load normally in every library.
Limitations
- Citation counts measure how often Wikipedia cites a source, not the source's importance or reliability.
- Parser outcomes and detector flags are operational signals, not adjudicated labels.
- Publisher and author names are normalized strings. They are not matched to real organizations or people.
- Citations are article-level records. The release does not record which sentence a citation supports.
-
The
sciencelabel on article text is a routing hint from an advisory index. Its precision has not been measured. - 1,625 of the 256,564 blocks have no citation records.
License and provenance
The data is derived from English Wikipedia and distributed under CC BY-SA 4.0. Credit Wikipedia's contributors and share derived data under the same license. The Wikimedia Foundation does not endorse this release.
checksums.sha256 covers every file in the dataset
repository, and provenance.json binds each public file to
its row count and Arrow schema. The parser and the release tooling are
in the source repository.