Wikiparse

A parser and open dataset for English Wikipedia

Abstract

Wikiparse is a Python package that converts MediaWiki wikitext into Markdown with YAML metadata, and reads each citation into a structured, deduplicated record. It is an independent implementation and does not call the MediaWiki API. Applied to the complete English Wikipedia dump of 1 June 2026, which holds 8,531,070 pages, it produced release 2606: 45,003,332 citation records, category and redirect relations, and the text of 7,073,577 articles. The release is distributed as Parquet tables under CC BY-SA 4.0. Each file has a row count, a schema, and a SHA-256 checksum.

Dataset
robbd/wikiparse-2606 on Hugging Face: eleven Parquet configurations. The text of every retained article is in a separate public bucket.
Package
wikiparse on PyPI, for Python 3.13 or newer: pip install wikiparse.
Documentation
wikiparse.docs.doering.ai: package guide and API reference.
Source
gitlab.com/doering-ai/apps/wiki-parse, under GPL-2.0.

The release

The dump was parsed once into a read-only store, which remains the source of record for every published table. Table 1 lists the main quantities. The dataset page describes each table, its grain, and its known limits.

Table 1. Release 2606 in counts. Every count is exact, not a sample or an estimate.
QuantityCount
Pages in the dump 8,531,070
Compressed blocks (multistreams) in the dump 256,564
Citation records extracted by the parser 45,003,332
Articles with retained text 7,073,577
Redirect edges 11,894,129
Disambiguation pages 310,529
Categories with citation aggregates 529,402
Rows in the ten published metadata tables 20,637,822

The parser's 45,003,332 citation records are published as per-article counts and quality signals and as 1,511,504 citation clusters. They are not published as one row per citation.

An example

The parser separates an article into metadata and body text. Links become counts, and each citation becomes a record with a canonical identifier built from its author, year, and title, or from its publisher or site when it has no author. Repeated citations within an article collapse into one record, and identical citations in different articles receive the same identifier.

Input wikitext

'''Coffee''' is a [[drink]] prepared
from roasted [[coffee bean]]s.<ref>{{cite book
 |last=Pendergrast |first=Mark
 |title=Uncommon Grounds
 |publisher=Basic Books |year=2010}}</ref>

Output of wikiparse.parse_article

---
internal_links:
    coffee-bean: 1
    drink: 1
citations:
    book:
        - pendergrast-mark__2010__uncommon-grounds
---
**Coffee** is a drink prepared
from roasted coffee beans.
Example. A short article before and after parsing. The output is abridged: empty metadata fields are omitted. The citation's full record (title, publisher, and names) is stored separately under its identifier.

Contents of this site

Citing

If the dataset is useful in your work, please cite it and attribute Wikipedia's contributors:

@misc{wikiparse2606,
  title  = {wikiparse 2606: citation, category, and article-text tables from the
            English Wikipedia 2026-06-01 dump},
  author = {Doering, Robb},
  year   = {2026},
  url    = {https://huggingface.co/datasets/robbd/wikiparse-2606},
  note   = {Derived from English Wikipedia; CC BY-SA 4.0.}
}

Robb Doering (2026). wikiparse 2606: citation, category, and article-text tables from the English Wikipedia 2026-06-01 dump. Hugging Face dataset. The data is derived from English Wikipedia under CC BY-SA 4.0. Wikimedia does not endorse it.