Parser

The wikiparse Python package

Wikiparse converts MediaWiki wikitext into Markdown with YAML metadata. It identifies templates and references with composed regular expressions, in Python, without the MediaWiki API. The package is on PyPI and its source is on GitLab under GPL-2.0. The full guide and API reference are in the documentation.

Installation

Wikiparse requires Python 3.13 or newer.

python -m venv .venv
. .venv/bin/activate
python -m pip install wikiparse

Converting HTML tables and lists also requires the Pandoc binary on PATH.

Parsing one article

parse_article returns an Article. Its serialized form is YAML metadata followed by a Markdown body. The metadata holds aliases, link counts, categories, citation identifiers, and images.

import wikiparse

article = wikiparse.parse_article(wikitext, title="Coffee")
print(article.serialize())  # YAML metadata, then the Markdown body

The default corpus profile produces text for extraction and training: links become counts, and sections such as “See also” are dropped. The document profile keeps ordinary links, those sections, and the supported emphasis, lists, tables, code, and math.

from wikiparse import ArticleProfile, parse_article

article = parse_article(
    wikitext, title="Coffee", profile=ArticleProfile.DOCUMENT)

Citations and bibliographies

Each citation becomes a record whose identifier is built from its author, year, and title, or from its publisher or site when it has no author. Repeated citations collapse into one record. The records export as CSL-JSON, with the same identifiers, so Pandoc can render a bibliography for the article. Save article.serialize() as article.md first:

from pathlib import Path
from wikiparse import csl

Path("refs.json").write_text(csl.dumps(article.citation_cache.values()))
printf 'nocite: |\n  @*\n' > nocite.yaml
pandoc --citeproc --bibliography=refs.json --metadata-file=nocite.yaml \
       -f markdown -t docx article.md -o article.docx

Citations are article-level records. The parser does not reconstruct which sentence each reference supported. Templates it does not recognize are not expanded. They remain visible in failure records and in the unhandled-template census.

Processing a dump

The console script reads a raw Wikipedia XML dump:

wikiparse --path ~/data/wikipedia/enwiki.xml

Output goes to $MY_DATA/wikipedia/: one Markdown file per article in articles/, with categories/ and citations/ for the category graph and the canonical references. A resumed run skips processed chunks. The README describes resumable runs across several machines and on Modal, which is how release 2606 was produced.