Parser
The wikiparse Python package
Wikiparse converts MediaWiki wikitext into Markdown with YAML metadata. It identifies templates and references with composed regular expressions, in Python, without the MediaWiki API. The package is on PyPI and its source is on GitLab under GPL-2.0. The full guide and API reference are in the documentation.
Installation
Wikiparse requires Python 3.13 or newer.
python -m venv .venv
. .venv/bin/activate
python -m pip install wikiparse
Converting HTML tables and lists also requires the
Pandoc binary on
PATH.
Parsing one article
parse_article returns an Article. Its
serialized form is YAML metadata followed by a Markdown body. The
metadata holds aliases, link counts, categories, citation identifiers,
and images.
import wikiparse
article = wikiparse.parse_article(wikitext, title="Coffee")
print(article.serialize()) # YAML metadata, then the Markdown body
The default corpus profile produces text for extraction and training: links become counts, and sections such as “See also” are dropped. The document profile keeps ordinary links, those sections, and the supported emphasis, lists, tables, code, and math.
from wikiparse import ArticleProfile, parse_article
article = parse_article(
wikitext, title="Coffee", profile=ArticleProfile.DOCUMENT)
Citations and bibliographies
Each citation becomes a record whose identifier is built from its
author, year, and title, or from its publisher or site when it has no
author. Repeated citations collapse into one record. The records export
as CSL-JSON, with the same identifiers, so Pandoc can render a
bibliography for the article. Save article.serialize() as
article.md first:
from pathlib import Path
from wikiparse import csl
Path("refs.json").write_text(csl.dumps(article.citation_cache.values()))
printf 'nocite: |\n @*\n' > nocite.yaml
pandoc --citeproc --bibliography=refs.json --metadata-file=nocite.yaml \
-f markdown -t docx article.md -o article.docx
Citations are article-level records. The parser does not reconstruct which sentence each reference supported. Templates it does not recognize are not expanded. They remain visible in failure records and in the unhandled-template census.
Processing a dump
The console script reads a raw Wikipedia XML dump:
wikiparse --path ~/data/wikipedia/enwiki.xml
Output goes to $MY_DATA/wikipedia/: one Markdown file per
article in articles/, with categories/ and
citations/ for the category graph and the canonical
references. A resumed run skips processed chunks. The
README
describes resumable runs across several machines and on Modal, which
is how release 2606 was produced.