The idea for this experiment did not begin with a PDF. It began with a few conversations at CyberCon26.
I spoke with people who had asked LLMs questions about IAEA safety and nuclear security guidance and had come away with an understanding that the publications did not really support. The answers were not necessarily obvious hallucinations. Some drew on genuine passages but lost the distinctions that tell an experienced reader what those passages mean and how much weight to give them.
That made me curious. Could I improve the outcome not simply by writing a cleverer prompt, but by giving the model a better representation of the source material? I had previously worked on a deterministic parser to support the production of IEC 63096 (Annex 1 was deterministically generated from the body content), so maybe I could apply a similar concept for this.
Uploading a collection of PDFs is easy. Getting useful and defensible answers from them is not.
The difficulty appears as soon as a question depends on more than finding the right words. Is the passage a requirement, a recommendation, an example in an annex or a qualification in a footnote? Does it come from the publication being discussed, or from a contents page that repeats the same heading? Is it paragraph 4.32, a row in a table that happens to begin with 4.32, or a page number that has wandered into the extracted text?
Those distinctions are familiar to an experienced reader used to using the IAEA guidance. An LLM will not reliably recover them from a flattened stream of PDF text.
I built the IAEA Guidance Parser to address that problem. It turns local folders of IAEA publications into structured, searchable knowledge files for a LLM. The aim is not simply to make the PDFs searchable. It is to carry enough of the publication’s structure, provenance and status into the search material that a model has a reasonable chance of using a passage correctly.
There was another reason I cared about the problem. I had previously worked as the scientific secretary for publications in the Nuclear Security Series, where I saw how much work goes into choosing a term, placing a paragraph and resolving comments on draft text. A paragraph is not merely a convenient container for sentences: its location in a publication, its relationship to other publications and its status within the document all affect how it should be read. Those distinctions can seem incidental once the final PDF exists, but they are part of the substance created through and upheld by the international consensus process.
The problem was not search, It was meaning
My first approach was the obvious one: extract the text, or take the raw PDFs, and upload them into an LLM as knowledge.
That works in the same way that cutting a book into equal piles of words works. All the words are still there, but a chapter heading may end up in one pile, the paragraph it governs in another, and the footnote explaining an exception in a third. A blind search may retrieve the right pile while leaving behind the information needed to interpret it.
A PDF makes this harder because it is often closer to a set of instructions for placing text on a page than an easy lookup of paragraphs, headings and footnotes. The page can look perfectly ordered to a person while its text layer is a jumble of lines in an inconvenient reading order. Many of the semantic relationships are conveyed visually rather than encoded explicitly.
A plain extraction therefore tends to lose the signals a careful reader uses almost automatically:
- the source publication and its place in the series;
- the numbered paragraph, requirement, table or figure;
- the section and document region in which it appears;
- whether the material is integral, supporting or just navigational;
- the physical PDF page needed to verify it; and
- whether the parser was confident about the boundary it detected.
Without those signals, retrieval can be technically successful and still produce a substantively misleading or ill-informed answer. Finding a relevant sentence is not the same as knowing what weight to give it.
What the parser actually does
One point is worth making very clearly: the parser itself is not an LLM.
It is a deterministic Python program. It reads the PDF text layer, applies explicit recognition and repair rules, maintains state while moving through the publication, and emits typed records. The LLM only enters the process later, after those records have been exported and uploaded as knowledge.
That separation is deliberate. I do not want a model to look at a sentence and guess whether it sounds like a requirement or whether a block looks like an annex. Those decisions can be made more consistently from the publication’s explicit structure leading to a much better outcome from the probabilistic generation of an LLM. Ideally this would be performed by a human, or automatically output from the IAEA mastering software, but let’s not let that get in the way of a good experiment!
1. Extracting the text without pretending the PDF is made for parsing
The parser opens each PDF and extracts text page by page. For every page it keeps the physical PDF page number and, where it can identify one safely, the printed page number shown in the publication.
It then performs some conservative clean-up. It normalises whitespace, removes recognised page mastering artefacts, joins lines that appear to be artificial PDF wraps and repairs ordinary words split by end-of-line hyphenation. At the same time, it preserves dashes used in publication identifiers and annex paragraph numbers, where changing the character could change the identifier.
This stage is less like proofreading and more like sweeping confetti back into sensible colour-coded piles. The parser is not rewriting the publication. It is trying to undo damage introduced when a visually mastered page is converted back into a sequence of text lines.
The current parser reads the PDF’s existing text layer. It does not perform OCR. If an older PDF contains damaged font mappings, or a scanned page has no usable text layer, the parser cannot magically reconstruct the intended characters. It can flag suspicious output, but the source still needs review or a separate OCR process.
2. Rebuilding structure with a small state machine
After extraction, the parser walks through the cleaned lines in order. While doing that, it keeps a small working memory of where it is in the publication:
- the current region, such as front matter, body, appendix, annex or references;
- the current major heading and subheading;
- whether it is still inside the contents pages;
- the active paragraph or other prose block; and
- whether a table is currently open across one or more pages.
This is the equivalent of reading with one finger holding your place at the opening of the current chapter while you read through it page-by-page.
The order of the tests matters. Tables are considered before ordinary paragraphs because a table cell can begin with something that looks exactly like a paragraph number. Region transitions are handled before prose so that an Annex heading changes the status of everything that follows. Requirement headings are separated before a preceding paragraph can absorb them. Footnotes are extracted without automatically ending the paragraph they interrupt, because the main sentence may continue after the footnote on the same or next page.
The parser recognises the numbering patterns used across different generations of the series, including forms such as:
1.1.
101.
101.1.
A.64.
III–21.
Requirement 7:
TABLE II–3.
FIG. 4.
It also accepts several visually similar dash characters. That may sound fussy, but the PDFs routinely contain hyphens, non-breaking hyphens, figure dashes, en dashes, and em dashes (yes, even before the age of the LLM) that look almost identical on the page and behave differently in text matching.
Multi-line headings are joined before they update the section path. Contents entries are suppressed rather than mistaken for the beginning of the body. Running headers and page numbers are removed before paragraph reconstruction. Some annexes contain curriculum outlines or report templates that are visually tables but have no conventional TABLE N. caption; the parser can create a synthetic table record for those blocks so that their outline numbers do not become dozens of fake paragraphs.
As the parser reaches a new structural boundary it finalises the active element and outputs a record. This is an important difference from chunking text on an arbitrary boundary such as a set number of characters or a sentence. The record follows the series reference units: one numbered paragraph, one requirement, one table, one footnote or one heading.
3. Inferring metadata conservatively
The parser also needs to know what publication it is reading. It tries to infer the title, series number, document family, category, type, year and other identifiers from the first pages of the PDF. It avoids generic pages that list every category in a series, because those pages can otherwise be mistaken for the identity of the particular publication.
When the PDF is ambiguous, it can fall back to the filename. A YAML configuration can override either source for documents that remain difficult to identify. The resulting metadata records where each important value came from: configuration, PDF inference, filename or fallback.
Each source PDF is also hashed. This does not say that the PDF is correct, but it does let me establish exactly which file produced a particular set of records for provenance purposes.
4. Turning structural elements into self-describing records
Each record contains fields such as the document identity and type, element type and number, source region, status and the reason for that status, section path, physical and printed pages, text, parser confidence, notes and links between elements such as a footnote and its paragraph.
That machine-readable index is useful for auditing, testing and building other tools. For an LLM, the parser produces a more compact Markdown version. A record looks roughly like this:
---
doc: GSR-PART-2
record: paragraph 4.32
status: Normative
region: Body
pdf: 27
section: 4. LEADERSHIP FOR SAFETY
...paragraph text...
The labels are repeated beside every record. That repetition is intentional. Each record carries its own metadata, so if it is retrieved on its own it still says where it came from and how it was classified.
The compact Markdown does not contain every field from the record. Parser confidence, detailed status reasons and diagnostic notes remain available in the full structural output and QA reports. The LLM-facing version keeps the high-signal information most useful during retrieval and citation.
Status is data, not decoration
The most important design decision was to preserve the status of the text.
The Nuclear Security Series has its hierarchy of Fundamentals, Recommendations, Implementing Guides and Technical Guidance. Similarly, the Safety Standards Series is organised into Safety Fundamentals, Safety Requirements and Safety Guides. At different levels in the hierarchies stronger language is used. As an example, requirements use “shall” while guides use “should” for recommended measures or equivalent alternatives.
There is also an important distinction within a publication. The SPESS C explains that an appendix is integral to a publication and has the same status as the body. Annexes and footnotes provide practical examples, additional information or explanation and are not integral parts of the main text. Section 1 establishes the background, objective, scope and structure rather than carrying the primary requirements, recommendations or guidance.
The parser applies that distinction through three deliberately simple labels:
- Normative — substantive body content from Section 2 onwards, together with substantive appendix material;
- Informative — substantive annex material and footnotes; and
- Informational — front matter, Section 1 context, headings, references, glossary material, publication metadata and back matter.
This classification is structural. The parser does not ask an AI whether a sentence sounds forceful. It asks where the element sits, what type of element it is and which section or region governs it. The rule is reproducible and can be tested against SPESS C.
“Normative” here describes consensus status within the publication. It does not mean that every passage is automatically binding law in every State, or that a Safety Guide has somehow become a national regulation. Legal effect depends on the applicable national and international framework.
This distinction is particularly important for an LLM. Models are very good at combining related passages into fluent prose. Fluency can hide the fact that two equally relevant passages do not have equal status. An example in an annex may be extremely useful, but it should not quietly reappear in an answer as a requirement. A footnote may clarify the intended reading of a paragraph, but it should not become a new recommendation. On the other side, excluding all supporting material would throw away valuable explanation.
With the status label present, the model can answer more like an experienced reader: the main text establishes this expectation; an annex illustrates one possible application; a footnote adds this qualification.
What an LLM does with the records
It is tempting to imagine that, after I upload the Markdown files, the model opens them all and reads the whole collection from beginning to end for every question. That is not how a knowledge-enabled LLM normally works.
The uploaded files are processed into a retrieval index. When a user asks a question, the platform searches that index and supplies a smaller selection of relevant excerpts to the LLM as additional context. The model then produces an answer from the user question, its instructions and the retrieved material. This is often called retrieval-augmented generation, or RAG: retrieval supplies a temporary reading pack, and generation writes the answer from that pack.
The labels are not magic fields with constraints, I cannot tell an LLM to guarantee that one of my Markdown records will always remain one indivisible internal chunk (at least not without costly vector stores). To the model, they are still text, broken into tokens like the paragraph that follows. Their value comes from being short, consistent and located immediately beside the source passage. The model can use the pattern that status: Informative should be treated differently from status: Normative, particularly when the system instructions explicitly tell it to preserve that distinction.
A common implementation of semantic retrieval represents each text chunk as an embedding: a long list of numbers that captures patterns in its meaning. The user’s question is represented in the same way, and the system looks for nearby vectors.
A helpful, although simplified, way to picture this is as a map of meaning. Passages about similar concepts are placed near one another even when they do not use exactly the same words. A question about leadership responsibilities may therefore retrieve a paragraph about senior management without needing a literal phrase match. Exact terms still matter as well: document identifiers, paragraph numbers and specialised terminology provide strong anchors for keyword-style matching.
This creates a two-stage process:
- Retrieval decides what the model gets to see. A relevant record that is not retrieved cannot contribute to that answer.
- Generation decides what the model says about it. The model must compare the retrieved records, follow the instructions, preserve qualifications and present citations accurately.
The parser improves both stages, but it cannot guarantee either. Clear record text and repeated metadata improve retrieval. Explicit status and provenance improve the model’s ability to interpret the excerpts. Neither prevents a retrieval miss or an unsupported synthesis.
This is also why I keep the instructions fairly strict. They tell the model to cite the document, record, section and PDF page; distinguish requirements, recommendations, guidance, examples and background; avoid making the source sound more mandatory than it is; and say when the uploaded material does not answer the question.
A structure-aware export gives the model better evidence. Instructions tell it how to handle that evidence. The official PDF remains the source to which important conclusions must return.
So what’s the benefit then?
The real benefit appears when the publications are loaded as a series rather than opened one PDF at a time.
IAEA guidance is intentionally interdependent. A Safety Guide may explain ways to meet requirements established in several Requirements publications. An Implementing Guide may draw from the Fundamentals, Recommendations and other topic-specific Technical Guidance. Definitions and cross-cutting concepts recur at different levels of abstraction.
That creates practical questions that are awkward to answer with ordinary document search:
- What do the overarching requirements say about an issue, and how do the relevant guides elaborate on it?
- Which publications discuss the same responsibility from the perspectives of government, regulator and operator?
- Where does a recommendation originate, and which implementing or technical guidance develops it further?
- Do several publications use a term consistently?
- Which passages establish expectations, and which provide examples of implementation?
With the series available as structured knowledge, a single question can retrieve records from several publications. The model can then follow a chain from principle to requirement or recommendation to more detailed guidance, while retaining the document, paragraph, section, page and status of each step.
This is where the tool becomes more than another search box. Search finds occurrences. A structure-aware assistant can help map relationships between occurrences, provided every step remains traceable to the publications.
Tables and figures are where honesty matters most
Tables were the most persistent parsing problem.
A table is a relationship between rows, columns, headings and cells. PDF text extraction often returns only a reading order. It is like hearing somebody read every cell aloud without telling you which column they are pointing at. All the words may be present while the relationship between them has disappeared.
Continued tables can repeat headings. Cell labels can look like paragraph numbers. A paragraph immediately after a table can be swallowed into the table record. Some annexes contain course curricula or report outlines that are clearly tables to the eye but have no metadata in the text layer.
The parser handles the boundaries conservatively and can recognise several outline-style blocks as synthetic tables. It then preserves the table as raw extracted text and records the page range. It does not invent a clean grid that the PDF extraction did not reliably provide. For a simple table, the text may be enough. Where row and column relationships affect the meaning, the page image has to be checked.
Figures receive the same honest treatment. The parser captures a figure identifier, caption and page. It does not turn a diagram into authoritative prose. A caption is a road sign pointing to the figure; it is not the map itself.
Building in reasons not to trust the output blindly
For a high-consequence subject, a useful tool needs to show where it may be wrong.
Each run produces manifests, source checksums, a parser version, a run identifier and QA reports alongside the knowledge files. Automated checks look for problems such as:
- requirement markers left inside ordinary paragraphs;
- footnote bodies contaminating substantive text;
- page headers or footers that leaked into records;
- wrapped headings that were not rejoined;
- contents-page entries mistaken for body content;
- table cells misidentified as paragraphs;
- status labels inconsistent with document regions;
- suspicious encoding or damaged characters;
- duplicate record identifiers; and
- disagreement between the records and the series manifest.
The findings include a severity and a suggested point of review. They are an alarm, not a certification of correctness. No finding means that the deterministic checks did not detect one of the problems they know how to recognise. It does not mean the source has been perfectly reconstructed.
What the tool still cannot promise
The parser has limits, and using them in a Custom GPT or NotebookLM adds another set of limits on top.
The parser can miss a structural boundary. A damaged text layer can change a symbol whose exact value matters. A complex table can lose the relationship between a heading and a cell. A figure caption cannot substitute for the figure. A low-confidence footnote boundary may need manual review. Metadata inference can be wrong, which is why overrides and source tracking exist.
The retrieval system can fail to surface a relevant record. It can surface a passage without enough neighbouring context. The LLM can misunderstand a label, over-combine passages or produce a citation that does not support every part of its sentence. Structured inputs reduce these risks; they do not remove them.
For important interpretations, decisions or exact quotations, the answer must be checked against the official PDF. The parser and LLM are navigation aids, not an alternative publication channel and not an official IAEA service.
What I learned
Building this parser reinforced something I had already learned while working on Nuclear Security Series publications: document structure is part of meaning and it is not always perfect.
The challenge was never to put more words in front of an LLM. It was to carry across the distinctions experts use when they read: the hierarchy of publications, the boundary of a numbered paragraph, the difference between integral and supporting material, the section that gives a statement it’s context, and the provenance needed to verify it.
There is a useful analogy with trying to restore a library after every book has been emptied into a single pile of pages. A LLM may be remarkably good at reading the pages, but somebody still has to put the page numbers, chapter dividers, catalogue records and shelf labels back. That is what I was trying to do.
Done carefully, an LLM can make a large and interconnected body of guidance much easier to explore. It can surface related passages that would otherwise require opening several publications, following references and repeating searches. It can help a reader see how fundamentals, requirements, recommendations and detailed guidance fit together.
Its value is not that it replaces the publication process or the published text. Its value is that it makes the results of that process easier to navigate while preserving enough status, provenance and uncertainty to lead the reader back to the source.
