The Living Library

Part one What it is

The collection is not the Library's

Worth establishing before anything else, because the framework gets the attention and the collection is the part that took twenty years. Roosevelt's record is held by dozens of separate institutions — the Library of Congress, Harvard, national parks, state historical societies, private collectors. Very little of it belongs to any one place.

Gathering it, cataloguing it item by item, and putting it online is the work of the Theodore Roosevelt Center at Dickinson State University, going on since 2007, much of it done by archivists, student interns, and volunteers working one record at a time. A retrieval index is only as good as the collection beneath it, and this one was built by people, by hand, first. The breakdown of who actually holds what is further down.

What it looks like in use

Before the architecture, the thing itself. Here the historian Doris Kearns Goodwin puts questions to the Talk to TR avatar in the Library's exhibit space, alongside Microsoft vice chair and president Brad Smith.

The problem it starts from

Roosevelt's record does not live in one building, and digitization alone does not fix that. Item-level cataloguing is manual and inconsistent across eras of practice, so backlogs grow alongside acquisition. And a scanned page behind a search box is still not an accessible one: a visitor has to already know what to search for, in a vocabulary the archive happens to share, before the archive will answer.

So the paper sets itself a narrow, testable question: how might institutions make vast, fragmented, and partially catalogued collections universally accessible, searchable, and interpretable — without sacrificing historical integrity?

Four layers, three tools, one corpus

The answer is a stack. Each layer is useful on its own and depends only on the one beneath it. Read from the bottom up.

1. 1

   Digitization and corpus creation

   Material moves into institution-controlled preservation storage with source identifiers and rights status preserved, so later stages never depend on a fragile upstream path.

2. 2

   AI-powered processing

   OCR and metadata extraction. Original metadata stays immutable and separate from anything a model generated. This layer produces the Archivist App, where curators correct what the model got wrong.

3. 3

   Retrieval and reasoning

   A hybrid index over the governed corpus. Questions are interpreted, real material is retrieved, and a model composes an answer attributable to actual documents. This layer alone is Campfire.

4. 4

   Embodied conversational interface (optional)

   Voice, avatar, and physical presence over the very same corpus. This layer is Talk to TR — and the paper is deliberate about calling it optional.

That last word is the most useful thing on this page for anyone weighing cost. Layers one through three already turn a fragmented collection into a unified, governed, searchable resource. An institution can stop there, take the entire research benefit, and decide about a conversational layer later — or never.

Review that does not become a bottleneck

This is the governance decision most worth stealing. Most review workflows are admission gates: nothing reaches the index until a human signs off, which with a 300,000-record backlog means the collection stays dark for years.

The Living Library inverts it. Processed records publish to the index continuously, carrying their review status and OCR confidence. What the Archivist App adds is curatorial control over that index rather than a precondition for entering it — a curator can push a corrected record in or withdraw a problematic one at any time. Edits are non-destructive and versioned, the original model output is preserved, and hard metadata from the source system stays immutable, so human correction never overwrites the institutional record. The collection becomes searchable immediately; review raises its quality over time instead of blocking it.

Part two How it works

The engineering half. If you came for the decision rather than the machinery, skip to part three — nothing below changes the answer, it only explains what the answer costs.

Answering questions the archive never anticipated

In plain terms: when a visitor asks about something that did not exist in your subject's lifetime, the system finds the closest thing that did, and answers through that instead of guessing.

A century-old archive cannot answer a question about social media. Unconstrained generation would answer, but invites anachronism and fabrication. Refusing is accurate and deadening. The paper's central technique, Cross-Era Analogical Grounding, takes a third path: it reframes the contemporary question as a retrieval for a historically attested analog.

A mid-tier model picks an era-appropriate theme, selects a story from a resident catalog of 108 curated narratives, emits a retrieval query, and attaches a one-line rationale — a curator hint — explaining why that story fits. Real archive evidence comes back, and the speaking model answers the modern question through the analog, in period and in voice. Stories rotate, so no visitor hears the same one twice.

Visitor asks

“What do you think about social media?”

Theme chosen

Reaching the people directly, over the gatekeepers of the day

Story selected

Words Sharper Than Swords — how Roosevelt moved the public with his voice and pen

Retrieval query

Roosevelt / the press / the ‘bully pulpit’ / appealing directly to the people

Evidence returned

Passages on Roosevelt's use of the presidency as a “bully pulpit” to reach citizens over the party bosses

What comes back is a generated response, not a historical quotation. The system composes a new sentence in Roosevelt's register, grounded in real retrieved passages, containing no reference postdating 1919. It is never presented as something Roosevelt said, and visitors are told the responses are AI-generated. The paper treats that gap — between a source-grounded analog and an utterance the man never spoke — as a real and unresolved concern.

Staying fast enough to feel like conversation

In plain terms: a visitor will forgive a slightly worse answer, but not a long silence.

The measured figure is the delay from a visitor releasing the push-to-talk button to the first synthesized word, across one exhibition period:

End-to-end first-token latency, 457 completed answers

Mean	2.80 s
Median	2.55 s
Maximum	7.14 s
Under 5 seconds	97%
Under 3 seconds	69%

Of 653 total push-to-talk releases, 457 ran to a completed answer; the rest were interruptions, repeat requests, or held-button timeouts.

Three choices buy that. Speech recognition runs locally and incrementally while the visitor is still talking. Retrieval takes two paths at once — the speaking model decides for itself whether the current turn needs the knowledge base, while a second path prefetches evidence for the next turn. And the whole chain streams: recognition into model into speech synthesis into avatar frames, each stage starting before the previous finishes, so total latency approaches the slowest single stage rather than the sum of all of them.

The finding underneath is worth noting for anyone sizing an index: the dominant cost is whether a turn retrieves synchronously at all, not how large the index is.

Safety that never stalls the exhibit

In plain terms: children will try to make the exhibit say something awful, and an avatar that freezes mid-sentence in front of a crowd is its own kind of failure.

The kiosk is public and includes children, so it has to resist prompt injection and steer away from improper content — under one strict rule that inverts the usual design: a safety check must never make the avatar stall or fall silent.

So the stack is three layers, each heavier and later than the last, none blocking the turn in flight. A fast pattern screen runs on visitor input almost instantly; on a hit it does not end the session but injects an in-character deflection into the next turn. A small-model classifier runs asynchronously and fails open. A mid-tier model reviews the avatar's own outgoing line in parallel with the stream, defaulting to observe rather than block.

That design came from a real failure: an early substring filter matched “kill” inside the benign phrase “killer view” and terminated the session irrecoverably. The current version deflects in character and matches on word boundaries.

Running all day without a babysitter

In plain terms: it has to open at nine and still work at five without staff restarting it, and no visitor should see the previous visitor's conversation.

The exhibit holds one long-lived session open all day rather than rebuilding per visitor, so nobody pays a cold-start cost. A moving watermark means each visitor sees only history from after they arrived; a short bounded window of recent ambient content bleeds across so someone arriving mid-story can refer to it; and a watchdog periodically discards raw transcript and transient state while preserving a running summary.

Watchdogs cover every part likely to fail — avatar generation, dialogue, transport, rendering — and escalate from the least disruptive recovery to the most: rebuild the avatar session, restart the agent, and only then restart the whole show. The exhibit engaged close to 5,000 visitors in its first two weeks of public operation in July 2026.

Part three Doing it yourself

The six-step process

Steps one to four are the load-bearing, broadly transferable core. They need governance and engineering effort proportional to collection size, but no persona design and no real-time systems expertise.

1. Assess collection readiness. Inventory assets across your own fragmented repositories and identify digitization gaps. Policy and inventory work, not engineering — and its outcome scopes everything after it.
2. Build the corpus. Digitize and aggregate into institution-controlled storage. Establish governance — source identifiers, rights status, and a schema separating immutable metadata from later enrichment — before any AI processing begins.
3. Apply AI processing. Run OCR and metadata enrichment as a model-pluggable stage. Validate the model choice against a source-grounded evaluation on a stratified sample rather than by inspection, and plan for expert review.
4. Implement retrieval. Index for hybrid search and expose it through a governed contract. This step alone gives you a researcher-facing tool, whether or not a conversational layer ever follows.
5. Add a conversational layer. Integrate a model against that retrieval contract and define the interaction patterns: persona grounding, analogical reframing, and a non-blocking safety stack.
6. Add an avatar — optional. Only where an institution wants embodied, real-time presence, under the same latency and autonomous-operation discipline.

Five principles the authors think transfer

What the paper does not claim

The authors are unusually direct about this, and it is worth repeating rather than glossing. This is a deployment experience report, not a controlled evaluation, and the evidence is largely observational over two weeks of public operation.

• Three qualities central to this class of system remain open: whether grounding keeps synthesis faithful, how often the figure lapses into anachronism, and how strong the visitor's sense of presence actually is.
• OCR errors are reduced by expert review but not eliminated, and residual errors can propagate into retrieval.
• A corpus centred on one figure's correspondence over-represents that figure's perspective. Curatorial review can mitigate that bias but not remove it.
• The boundary between inference and fabrication is inherently imperfect. Even a source-grounded analog remains a novel utterance the historical figure never spoke.
• The study covers a single figure and a single archive.

For a presidential library, that candour is the point. A framework that overclaimed would be the wrong thing to hand another institution.

Privacy, as deployed

Talk to TR is built for anonymous, walk-up use. The vision subsystem estimates presence but performs no identity or facial recognition — camera frames, images, face embeddings and biometric templates are not retained, and the only identifier it emits is transient and non-biometric with roughly a 30-second lifetime. Microphone audio is processed as a transient stream, never retained as a recording. Spoken participation is opt-in through the push-to-talk control, on-site notice tells visitors they are interacting with an AI-generated persona, and operational logs are kept for up to 30 days. No visitor-linked memory persists across visits.

The collection, in detail

Back to where this started. Here is the Theodore Roosevelt Center's digital library broken out by the institution that actually holds the material — the clearest available picture of how distributed a presidential record really is.

• Library of Congress Manuscript Division119,80485.7%
• Library of Congress Prints and Photographs5,3393.8%
• Harvard College Library3,3682.4%
• Theodore Roosevelt National Park2,9322.1%
• Theodore Roosevelt Association Journal1,7581.3%
• Sagamore Hill National Historic Site1,1000.8%
• Charles C. Myers Collection9530.7%
• Theodore Roosevelt Inaugural National Historic Site5880.4%
• Arizona Historical Society5180.4%
• Dickinson State University3760.3%
• Theodore Roosevelt Birthplace National Historic Site3630.3%
• Fritz R. Gordner Collection3180.2%

…and 40 more collections, 2,297 records between them — state historical societies, national parks and monuments, university libraries, and private collections given or loaned for digitization.

Read this chart for one thing only: who holds what. It is a snapshot of the Center's collection facet, harvested 19 July 2026 by the Library's own TRC Search Widget, and counts move as cataloguing continues. It is not a measure of any institution's holdings — each figure counts only what that institution has catalogued into this particular index. The Library's own Roosevelt material is substantially larger than its line here suggests and is mostly catalogued elsewhere, and the roughly 300,000-record corpus the paper describes is broader still than this digital library.

The people doing the work

• Michael Patrick Cullinane, PhDCo-Director; Lowman Walton Chair of Theodore Roosevelt Studies
• Erik Johnson, MA, MLISCo-Director
• William J. Hansard, PhDPublic Historian
• Alexandra Hecht, MA, MLISDigital Collections Cataloger and Archivist
• Gemma Koontz, MSDigital Collections Cataloger
• Rachel Lane, MAResearcher
• Valerie Naylor, MSNational Parks Researcher

Staff as listed by the Center. Roles change — the current roster is on their staff page.

Where this sits in the Lab's work

The paper is a Microsoft AI for Good Lab publication — ten of its eleven authors are Microsoft, among them the Lab's senior director, Laura Hoffman, and its director and co-founder, Juan Lavista Ferres. When the Library opened in July 2026, Microsoft said it planned to publish a paper documenting how the technology works and to open source the software. This paper is the first half of that; the catalog on this site is the second.

It is also not the Lab's only run at this problem. Its closest sibling is Ukuvula, built with the Nelson Mandela Foundation — a pipeline making large oral-history archives searchable, generating transcripts, named entities, and summaries from liberation-era recordings. Different medium, different continent, same shape of problem: a collection that exists but cannot be asked a question.

The short of it

An archive that cannot be questioned is, for most people, an archive that does not exist. The Living Library is one worked answer to that — not the only one, and not a finished one. Its most transferable parts are the least glamorous: publish records continuously and let curators correct them in place, keep every claim traceable to a real document, and decide honestly whether you need the avatar at all.

If your institution is somewhere in this, the Library would like to hear about it. hello@trlibrary.com reaches a person.

Everything above is drawn from the paper. The full text adds the architecture diagrams, the OCR model evaluation, and the complete references — read it on arXiv.