Wikimedia Dumps Schemas and Formats: XML, SQL, JSON

You want to analyze Wikipedia's entire history. Maybe you're building a search engine, training an AI model on human knowledge, or just curious about how edit wars unfold over decades. But when you hit the Wikimedia Dumps, you're not downloading a single file. You're facing a maze of formats: XML, SQL, and JSON. Each one solves different problems, breaks in different ways, and requires different tools to parse.

Most tutorials gloss over this. They say "download the dump" and leave you stranded with gigabytes of data that don't fit into your RAM. This guide cuts through the noise. We'll look at what each format actually contains, why MediaWiki uses them, and which one you should pick for your specific job. No fluff, just the technical reality of handling petabytes of open knowledge.

Key Takeaways

• SQL dumps are best for relational analysis and direct database imports but require significant setup and disk space.
• XML dumps (specifically current and revision histories) offer structured metadata and are ideal for text mining and NLP tasks.
• JSON dumps are newer, easier to parse programmatically, and increasingly preferred for modern web applications and API integrations.
• The choice depends entirely on your infrastructure: do you have a MySQL server ready? Or are you processing streams in Python?
• Always check the timestamp. Wikimedia dumps are generated periodically, so your data is always slightly stale compared to live Wikipedia.

The MediaWiki Schema: What Are You Actually Downloading?

Before touching a file, you need to understand the underlying structure. MediaWiki is the software running Wikipedia. It stores content in a relational database with specific tables like page, revision, and text.

The page table holds metadata: title, namespace, and whether it's a redirect. The revision table links pages to specific versions of content. The text table stores the actual wikitext blob. When you download a dump, you're exporting snapshots of these tables. Understanding this EAV-like structure (Entity-Attribute-Value across tables) helps you decode why some formats feel fragmented.

For example, if you want to track who edited a page last, you join page to revision using page_id, then grab the rev_user. If you ignore the schema, you'll end up writing complex parsing logic that could have been a simple SQL query.

SQL Dumps: The Heavyweight Champion

If you need to run complex queries-like "find all articles edited by bots in the last year"-SQL Dumps are your friend. These are standard MySQL-compatible .sql.gz files containing CREATE TABLE statements and INSERT INTO commands.

Why use them? Because they preserve referential integrity. You can import them directly into MariaDB or MySQL and start querying immediately. The downside? They are massive. A full English Wikipedia SQL dump can exceed 100GB uncompressed. Importing it takes hours, even on decent hardware.

There are two main types:

• Current Pages: Only the latest version of every article. Smaller, faster to process.
• Revision History: Every edit ever made. This is huge. For English Wikipedia, this exceeds 2TB compressed.

Pro tip: Don't try to load the full revision history unless you have a dedicated server. Start with the "current pages" dump. It gives you the state of knowledge right now, which is often enough for static analysis.

XML Dumps: Structured Data for Text Mining

When researchers talk about "Wikipedia datasets," they usually mean XML Dumps. Specifically, the pages-meta-current.xml and pages-articles.xml files. These aren't just raw text; they include rich metadata wrapped in tags.

Each <page> element contains the title, namespace, and ID. Inside, <revision> elements hold the timestamp, contributor info, and comment. The <text> tag wraps the wikitext. This structure makes it easy to extract citations, categories, or infoboxes without parsing HTML.

However, XML is verbose. Parsing it requires streaming libraries like lxml in Python or SAX parsers because loading the whole DOM into memory will crash your machine. Use xml.etree.ElementTree.iterparse to handle large files efficiently.

One common pitfall: Wikitext isn't HTML. It has templates, magic words, and internal links ([[Link]]). If you feed raw wikitext into a standard NLP pipeline, your results will be noisy. You need a parser like wikitextparser or mwparsing to clean it first.

JSON Dumps: The Modern Alternative

Enter JSON Dumps. Generated via the CirrusSearch backend, these files represent pages as JavaScript Object Notation structures. Why switch from XML? Simplicity. Most programming languages have native JSON support. You don't need specialized parsers.

JSON dumps flatten the hierarchy. Instead of nested XML tags, you get key-value pairs. For example, categories become an array of strings. External links become a list of objects. This makes it incredibly fast to filter data. Want all pages with "Physics" in their categories? One line of code.

Note that JSON dumps often lack the full revision history. They focus on the current state optimized for search indexing. If you need every edit comment, stick to XML or SQL.

Choosing the Right Format for Your Project

Let's make this practical. Here is a decision tree based on common developer needs.

• Building a Search Engine? Use JSON. It mirrors how Elasticsearch indexes documents. You can ingest it directly into OpenSearch or Elastic.
• Analyzing Edit Patterns? Use SQL. You need timestamps and user IDs linked together. SQL handles this naturally.
• Training a Language Model? Use XML (pages-articles). It provides clean wikitext separated by page boundaries. You can strip markup later.
• Quick Prototype? Use JSON. Load a small sample into Python, inspect the keys, and build your MVP fast.

Remember, you can convert between formats. There are open-source tools like mediawiki-dump-parser that read XML and output JSON. But conversion takes time. Pick the source format that matches your final destination to save CPU cycles.

Handling Large Files: Tools and Tactics

You cannot open a 50GB XML file in Notepad++. You need command-line tools. gzip is your first friend. All dumps come compressed. Decompress only what you need.

For SQL, use mysqlimport or pipe directly into the database client: zcat enwiki-latest-pages-current.sql.gz | mysql -u root -p wikidb. This avoids creating intermediate uncompressed files.

For XML, avoid DOM parsers. Use iterparse in Python. It yields elements one by one, keeping memory usage constant regardless of file size. Here’s a rough pattern:

for event, elem in iterparse(file):
    if elem.tag == 'page':
        process_page(elem)
        elem.clear() # Free memory

For JSON, consider ijson or streaming parsers if the file exceeds your RAM. Otherwise, standard json.load works fine for smaller language editions.

Next Steps and Troubleshooting

Start small. Download the "Simple English" Wikipedia dump. It's tiny, fast to process, and lets you test your pipeline without waiting hours for downloads. Once your script works there, scale up to the main English edition.

If you encounter encoding errors, ensure your scripts explicitly handle UTF-8. Wikipedia supports almost every character set. Default ASCII assumptions will break your parser.

Finally, respect the bandwidth. Wikimedia offers these dumps for free, but they serve millions of requests. Use the rsync protocol or download during off-peak hours if possible. Happy hacking.