Data

Identifiers — the public contract

From docs/id-scheme.md, rendered as committed.

This document describes the one thing in OpenTome that can never change.

Consumers (Mangarr, Kavita, Komga, Suwayomi) store these identifiers in their own databases. The moment they do, the scheme stops being ours to revise: churn breaks every consumer simultaneously and silently. IMDb and TheTVDB are defensible because of their identifiers, not their bytes — the IDs are the moat.

Treat everything below as frozen.

Form

w_a3f21c9d4b07     work         — the canonical creative work, medium- and market-neutral
rl_88e0d2114f6a    release line — one medium, one market, one publisher
v_5c7be0913a2d     volume       — one physical/digital unit within a release line
ch_1f04a7cd88b2    chapter      — one canonical serialized unit

Prefix names the entity type; the remainder is 12 hex characters, opaque.

Typed prefixes, not branded ones — this follows IMDb (tt/nm/co) and MusicBrainz (bare UUIDs). The brand belongs in the field name a consumer stores, not inside the identifier.

Canonical consumer field name

tome_id

Use tome_id alongside the fields integrators already carry (tvdb_id, tmdb_id, anilist_id). Where a consumer must disambiguate entity types, tome_work_id, tome_volume_id etc. are the sanctioned forms.

Guarantees

1. Never reused. A retired identifier is never issued to a different entity, ever.
2. Never re-keyed. An entity's id does not change because a fact about it was corrected. Ids are opaque precisely so that no correction can force a re-key.
3. Always resolvable. A merged or superseded id resolves through id_redirect forever. It must never 404 — consumers holding it get the successor, not an error.
4. Idempotent generation. Ids derive from a stable natural key by hash, so re-ingesting the same source produces the same ids. Verified: reloading a populated database leaves the volume count unchanged. Since 2026-09-27 an id is no longer derivable from the natural key alone (docs/krcn-design.md §8–9). A work or line that shipped keeps its published id even when a later build derives a different natural key for it: a library line whose lowest member LCCN / ark changes, or a library-born work that a Wikipedia work later covers. The build looks the entity up in the carry (the previous artifact) first and mints from the natural key only when nothing carried matches. Re-running the same build on the same carry still gives the same ids.

id_redirect is a pipeline table (schema/schema.sql), and since 2026-09-24 the published manga-metadata.sqlite carries it too: id_redirect(old_tome_id, new_tome_id, entity, reason, old_series_id, new_series_id), chains collapsed to an id present in that file. Each build re-reads the previous artifact's rows, so a redirect is never lost. A retired volume with no successor volume resolves to its line (reason retired).

Since 2026-09-25 the pipeline writes those rows for every market and every entity, work included (tier0/carried_ids.py, docs/carried-ids.md): a carried id the build lost goes to the work / line / volume now holding its evidence (ISBNs, else dated volumes, volumes by number), and the contract test fails on any carried id, in any market, left without a present target.

Integer ids (series.gcd_series_id) follow the same rule: a successor line with no integer of its own takes the retired line's; a successor that already has one keeps it, the retired integer resolves through id_redirect.old_series_id -> new_series_id, and it stays reserved in id_map (kind retired) so it is never issued again. A German line the DNB linker stops linking stays published under its published work (role kept) rather than disappearing.

What is not guaranteed

• Ordering. Ids are opaque. Do not sort by them, parse them, or infer recency.
• Meaning. The hex carries no information. Do not derive anything from it.
• Stability of the natural key. If a work's natural key genuinely changes (a merge, a split), a new id is issued and the old one is written to id_redirect (unless the carry keeps the published id, Guarantee 4). The old id keeps resolving; it simply points somewhere new.

Merges and splits

Deletion is not a supported operation. Nothing is ever removed from id_redirect -- with one exception (2026-09-25, docs/carried-ids.md): a row whose OLD id is present again (a re-key reverted, "s X" -> "Les X" -> "s X") is dropped, because the id now resolves by being present, and keeping the row would send a live id elsewhere and close a cycle.

Why this is written down

The project's own history motivates it. The predecessor's blueprint records the lesson in its own words — do not big-bang re-key existing series — after an ID migration nearly orphaned a live library. That was one person's instance. A published scheme has the same failure mode multiplied by every consumer, and no way to walk it back.

Schema design notes

From docs/schema-v1.md, rendered as committed. These are the pipeline's design notes; the tables of the published file are listed above. The "commercially clean" figure in the last table was later retracted in docs/legal-position.md — the licence section above states the verified position.

Every table traces to an empirical finding in step0-findings.md or tier1-crossverify.md (or an early private smoke test). Nothing here is speculative.

The four decisions that matter

1. IDs are a public contract

Opaque, prefixed (w_, rl_, v_, ch_), never reused, never re-keyed. Merges write id_redirect; retired ids resolve forever and never 404. This is the commercial moat — IMDb and TheTVDB are defensible because of their identifiers, not their bytes. Once Mangarr or Kavita stores one, churn breaks every consumer simultaneously.

Ids are derived from a stable natural key by hash so loads are idempotent, but they are stored opaquely and never parsed. Verified: re-running the loader against a populated DB produced 116 volumes before and after.

2. release_line is the unit of volume numbering

Independence is forced by the data, not chosen: Re:Zero Chapter 3 vol 1 and Chapter 4 vol 1 are both volume 1. Merging lines corrupts numbering — the only large errors in cross-verification were flattened franchises (SAO "v6" was actually Progressive 3 (light novel)).

medium is open-ended TEXT, not an enum. Solo Leveling in the reference library is already manhwa, so a fixed manga/light-novel pair is under-general on day one. Consumer preference ("I only want manga") shapes queries, never storage.

3. Dates carry three qualifiers

release_date + _precision + _type. Finding: 49 of 50 cross-source disagreements were exact multiples of seven days, 45 in the same direction — two sources answering different questions (publication vs retail on-sale), not one being wrong. A bare date silently mixes them.

4. claim / resolution split — provenance per field

claim stores every source's assertion; resolution records which won, with a confidence and a basis. Disagreement is preserved rather than discarded, because the disagreement is the confidence signal.

Commercially this is the load-bearing table. licence per claim drives the clean_claim view — the subset filtered to unencumbered provenance. Google Books and MangaDex forbid database-building outright, so anything sourced from them can never enter a paid product. Per-field provenance is cheap now and unrecoverable later.

One primitive, three problems

composition expresses all of these with one relation:

• an omnibus contains volumes [1,2,3]
• a French volume contains Japanese volumes [1,2]
• any volume contains chapters [1..7]

Verified end-to-end

Loaded Chainsaw Man (en) and Attack on Titan (fr) — 4 release lines, 116 volumes, 228 claims:

Retracted: the Commercially clean claims row in this table was later found to be circular (the licence labels it counted were assigned without reading the sources' terms) — see docs/legal-position.md; the verified position is in LICENSE-DATA.md.

(Written before the retraction above — kept as the record of the smoke test's reasoning.) That last row was a direct consequence of dropping Google Books: with Wikipedia used as a citation index rather than a data source — every date attributed to the publisher URL the article itself cites — 40% of claims resolve to publisher provenance and nothing is restricted.

Language support is data, not code

DIALECTS in tier0/wikipedia_volumes.py maps per-wiki template and field names:

German is not a dialect — de.wikipedia uses wikitables (9 tables, 138 rows, {{DatumZelle}}), so it needs a separate table extractor.

Correction to an earlier assumption

The plan predicted French would diverge structurally from Japanese volume splits. For Attack on Titan it does not — French tracks Japanese 1:1. The divergent market for this title is German, whose Carlsen omnibus editions run 450–472pp against ~190pp elsewhere. Cross-market divergence is real, but it is per-title and per-market, not a property of French.

2026-09-21 — volume titles, per-line status, orig_series_id

volumes.title is the Wikipedia row title (number-only titles, e.g. "Volume 3", are dropped rather than exported as if they said something).

series.status is now per LINE, not per work, for a licensed market: completed | ongoing | stalled | NULL. The rule lives in export/line_status.py — a line is stalled when it is at least two volumes behind its origin-market counterpart and nothing has shipped in 24 months while the origin kept going. A line that is NOT the work's main line (an arc, a side story) and has reached its origin's top volume, with neither market shipping for 24 months, is completed even while the work itself is ongoing — a finished arc of a running series. A main line in the same position keeps the work's status (a hiatus is still ongoing). Origin-market and omnibus lines take the work's own status as before; release_line.status in the pipeline DB is not consulted for a licensed line with a resolved origin (it is the column a future corrections-only cancelled would use).

series.orig_series_id is the origin counterpart line's id for a licensed line — the same-work, same-medium line the status rule and a cross-market join compare against.

A consumer that maps status to its own enum must treat stalled as neither completed nor ongoing. Mangarr's MapGcdStatus does not recognize it yet and falls back to AniList, which is safe but loses the signal.

2026-09-24 — display_anilist_id / display_anilist_via (display only)

Two additive series columns (Mangarr selects named columns, so an older Mangarr never sees them):

• display_anilist_id INTEGER — an AniList id to take a cover / synopsis from, for an English line the resolver left with anilist_id NULL. It is never a binding: set only where anilist_id IS NULL, never copied into anilist_id, never a source of aliases. A consumer must not present it as the line's AniList id (the site links only anilist_id).
• display_anilist_via TEXT — how it was found: - parent: the line's name cut at its first (, : , - or / key-equals the name of a bound English line of the same work (tome_work_id, any medium) — that line's anilist_id (Re:Zero (Truth of Zero) → the bound Re:Zero). Never across works, never when the same-work bound lines of that name carry different ids. - medium: a novel / light_novel line whose own (format: NOVEL) AniList page has no title-equal candidate at all, but whose manga-family page for the same name gives the resolver's pick() a candidate — AniList lists the adaptation, not the novel (Otherside Picnic, Bungo Stray Dogs).

Both columns are NULL or both set. export/test_artifact.py asserts: display ids only on English lines with a NULL anilist_id; via in {parent, medium}; medium only on novel mediums; every parent id is the unambiguous anilist_id of its same-work parent line. Written by export/resolve_anilist.py --display in stage 8a, after corrections/anilist.json's pins and before --covers-only, which fills build/anilist-covers.json for display ids too.

2026-09-24 — the German market from DNB: projected dates, volumes.release_date_type

volume.release_date_type gains projected: a planned publication month (MARC 263, YYYYMM) from a DNB announcement record -- a book announced but not yet deposited. It is always month precision, it is only used for a volume that has no published / on-sale date from any source, and an announcement whose year is after the current year produces no volume at all (2027-2030 placeholders are often cancellations). DNB's deposited records give published at year precision (the 008 year). See docs/dnb-design.md.

The artifact's volumes table gains release_date_type TEXT (additive; Mangarr selects named columns): the milestone release_date_raw states -- published, on_sale, projected, unknown -- and NULL exactly when the volume is undated. Without it a projected month would ship indistinguishable from a published one.

Claims: a DNB published year is field='release_date'; a projected month is its own field, projected_date (the claim key is (entity, entity_id, field, source), and one DNB volume can carry both kinds of record). In tier2/resolve.py a bare-year claim never beats a finer claim it disagrees with, whatever the source precedence says -- a late-December release is catalogued by DNB under the next year.

Pipeline staging tables, written by tier0/build_dnb.py and never exported: dnb_line (every DNB line, exported or not: its key, the rl_ id it is or would be, the linker's tier and work, its role -- merged / sibling / linked / review / unlinked / absorbed -- and, for lines that share ISBNs with a German Wikipedia line, that line's work as ground truth) and dnb_member (every DNB volume record kept: its line, number, ISBN, the volume it reached and its fate). The contract test and the measure gate read them.