Wikiparse
A parser and open dataset for English Wikipedia
Abstract
Wikiparse is a Python package that converts MediaWiki wikitext into Markdown with YAML metadata, and reads each citation into a structured, deduplicated record. It is an independent implementation and does not call the MediaWiki API. Applied to the complete English Wikipedia dump of 1 June 2026, which holds 8,531,070 pages, it produced release 2606: 45,003,332 citation records, category and redirect relations, and the text of 7,073,577 articles. The release is distributed as Parquet tables under CC BY-SA 4.0. Each file has a row count, a schema, and a SHA-256 checksum.
- Dataset
- robbd/wikiparse-2606 on Hugging Face: eleven Parquet configurations. The text of every retained article is in a separate public bucket.
- Package
-
wikiparse on PyPI, for Python 3.13 or newer:
pip install wikiparse. - Documentation
- wikiparse.docs.doering.ai: package guide and API reference.
- Source
- gitlab.com/doering-ai/apps/wiki-parse, under GPL-2.0.
The release
The dump was parsed once into a read-only store, which remains the source of record for every published table. Table 1 lists the main quantities. The dataset page describes each table, its grain, and its known limits.
| Quantity | Count |
|---|---|
| Pages in the dump | 8,531,070 |
| Compressed blocks (multistreams) in the dump | 256,564 |
| Citation records extracted by the parser | 45,003,332 |
| Articles with retained text | 7,073,577 |
| Redirect edges | 11,894,129 |
| Disambiguation pages | 310,529 |
| Categories with citation aggregates | 529,402 |
| Rows in the ten published metadata tables | 20,637,822 |
The parser's 45,003,332 citation records are published as per-article counts and quality signals and as 1,511,504 citation clusters. They are not published as one row per citation.
An example
The parser separates an article into metadata and body text. Links become counts, and each citation becomes a record with a canonical identifier built from its author, year, and title, or from its publisher or site when it has no author. Repeated citations within an article collapse into one record, and identical citations in different articles receive the same identifier.
Input wikitext
'''Coffee''' is a [[drink]] prepared
from roasted [[coffee bean]]s.<ref>{{cite book
|last=Pendergrast |first=Mark
|title=Uncommon Grounds
|publisher=Basic Books |year=2010}}</ref>
Output of wikiparse.parse_article
---
internal_links:
coffee-bean: 1
drink: 1
citations:
book:
- pendergrast-mark__2010__uncommon-grounds
---
**Coffee** is a drink prepared
from roasted coffee beans.
Contents of this site
- Dataset: a datasheet for release 2606, covering its tables, citation quality, categories, redirects, access recipes, and limitations.
- Parser: installation, parsing one article, exporting a bibliography, and processing a full dump.
- People: people in English Wikipedia by birth year, from Wikipedia's own birth-year categories.
- Figures: interactive drawings from the release — citations, categories, people, and text.
- Sitemap: every page and external resource.
Citing
If the dataset is useful in your work, please cite it and attribute Wikipedia's contributors:
@misc{wikiparse2606,
title = {wikiparse 2606: citation, category, and article-text tables from the
English Wikipedia 2026-06-01 dump},
author = {Doering, Robb},
year = {2026},
url = {https://huggingface.co/datasets/robbd/wikiparse-2606},
note = {Derived from English Wikipedia; CC BY-SA 4.0.}
}
Robb Doering (2026). wikiparse 2606: citation, category, and article-text tables from the English Wikipedia 2026-06-01 dump. Hugging Face dataset. The data is derived from English Wikipedia under CC BY-SA 4.0. Wikimedia does not endorse it.