When people describe an LLM that "reads insurance policies," they usually mean something that can be shown a PDF and asked questions about it. That capability is real, and it is genuinely useful for certain tasks. It is not sufficient for driving automated claims decisions. The gap between a model that can answer policy questions and an extraction pipeline that can reliably produce structured coverage outputs with auditable confidence scores is the gap that Livia spent the first year of Insurteam's existence designing across. This post describes the failure modes we observed in general-purpose document parsing and the architecture we built to address them.
How general-purpose LLMs fail on insurance contracts
A general-purpose LLM given a well-structured travel policy document can answer questions like "does this policy cover flight delays?" with reasonable accuracy most of the time. The failure modes appear precisely on the cases that matter most for a claims decision pipeline: policies with non-standard document structures; endorsements that modify base coverage terms without appearing in the main coverage narrative; benefit clauses that use defined terms from a separate definitions section, where the defined term meaning is narrower than the natural language meaning; and documents where the benefit schedule is in a table that the model does not reliably connect to the coverage narrative it read earlier.
Insurance contracts are intentionally precise documents. The meaning hinges on defined terms, cross-references between sections, and exclusion clauses that explicitly carve out territory that the coverage narrative appears to include. A model generating a summary of coverage terms based on the surface narrative, without tracking these cross-references, produces extractions that are correct for the simple cases and wrong for exactly the edge cases that require careful analysis.
The practical consequence in a claims pipeline is not a visible error rate on standard claims. It is a low-visibility error rate on the more complex claims, the ones involving endorsement modifications, unusual policy structures, or exclusion clauses that require cross-referencing the definitions section to interpret correctly. Standard claims process fine; edge cases get coverage determinations that are plausible but wrong.
The three-pass extraction architecture
Our extraction pipeline runs three passes over each policy document before producing a structured coverage output for the claim type being evaluated.
The first pass is structural. The pipeline maps the document: identifies its sections and their hierarchy, locates any tables, enumerates defined terms and their locations, and identifies any endorsements and their scope. This pass does not extract coverage logic. It builds a navigable map of the document that subsequent passes can use to resolve cross-references correctly.
The second pass extracts the coverage logic for the specific claim type being processed. For a flight-delay claim, this means extracting: the triggering definition (what qualifies as a covered delay, measured from what reference time, using what data source), the benefit schedule (payment amount by delay duration tier, or a formula reference), the exclusion set (delay causes that remove coverage, such as delays within a defined minimum period or delays caused by specified carrier categories), and any policyholder conditions (notification timeframes, documentation requirements). This extraction produces structured output against a predefined schema, not a free-form narrative summary.
The third pass resolves cross-references. When the coverage clause uses a defined term, the pipeline retrieves the definition from the definitions section and substitutes the operative text. When a benefit limit references a schedule table, the pipeline locates the relevant row and returns the specific figure. When an endorsement modifies any extracted value, the pipeline applies the modification and marks the affected field as endorsement-overridden. This third pass is where the pipeline diverges most sharply from general-purpose document Q&A: it explicitly follows the reference graph of the document rather than relying on the model's attention mechanism to implicitly connect disparate sections.
Confidence scoring and routing
Every extracted value carries a confidence score. The score combines the extraction model's probability signal for the specific value with a structural coherence check: did the cross-references resolve without ambiguity? Did the benefit schedule match the expected structure for this product type? Does the extracted exclusion set contain values within the expected distribution for travel policies?
Extractions below a confidence threshold route to the adjuster queue. The claim arrives with the extraction output highlighted for verification, and the adjuster reviews the specific extracted value rather than re-reading the full policy. This is operationally important: adjuster review of a pre-extracted coverage determination takes two to three minutes; adjuster review of a raw policy document for the same information takes seven to fifteen.
We tune the confidence threshold per policy product type rather than uniformly. Policy products with consistent, well-structured documents have a stable extraction accuracy that warrants a lower threshold for routing to human review. Products with variable document structures or frequent endorsement modifications require a higher threshold to maintain the same accuracy on automated decisions. A uniform threshold applied across all product types over-routes claims from simple products and under-routes claims from complex ones.
What the pipeline does not attempt
The extraction architecture is built for the travel insurance claim types where we have sufficient training signal to calibrate it: flight delay, flight cancellation, and missed connection. Within those claim types, the architecture handles the full range of policy structures we have encountered across more than 150 distinct product variants.
The pipeline does not attempt to extract coverage logic for claim types outside this scope. It does not attempt to interpret policy language that does not match any pattern in the extraction schema. In both cases, the claim routes to adjuster review with the best available partial extraction flagged. Expanding to new claim types requires dedicated architecture work: schema design for the new claim type, extraction training on representative policy samples, and calibration against adjuster decisions on a validation set. It is not a configuration change; it is an engineering project. We say this clearly because the alternative, a system that attempts coverage determinations on claim types it was not built for, produces confident-looking outputs that are wrong in non-obvious ways.
