AI Pipeline Lineage: Govern the Record, Not the Run

An AI pipeline is the automated path a piece of data takes from where it is created to where a decision gets acted on: ingest, normalise, enrich, run a model, do something, keep a record. Six hops. Most teams can describe all six and cannot trace one record through them.

That gap is the argument of this piece. The industry governs the run — did the job succeed, how long did it take, what did it cost — and almost nobody governs the record. Ask which rows a retrieval index contains and a team can usually show you a schema. Ask which artifacts descend from one specific customer record and the answer is typically a query console and a few days of somebody's time. In February 2024 a developer opened LangChain issue #17340 on GitHub asking how to "delete particular file embeddings" from a store holding many files. It is a question with no clean answer in the framework, and the same question a data protection officer asks in a different accent. A practitioner posting to Hacker News on 29 July 2026 described the common shortcut plainly: people "store all the embeddings once and call it a day."

The short answer: An AI pipeline is only governed when you can trace one real record forward to every artifact it produced and backward from one output to every input that shaped it — and hop 3, embedding, is where that ability is usually lost for good.

Last updated: July 30, 2026.

The six hops, and the two directions you have to be able to walk them.

We have not run a controlled lineage rebuild inside a customer environment and published the numbers, so nothing below is presented as a measurement of ours. Every figure is either cited to a named source or labelled as illustrative arithmetic you can substitute your own inputs into.

What an AI pipeline actually is

An AI pipeline is a sequence of automated steps that moves data from a source system into a model and then into an action, keeping the data in a usable state at each step. IBM defines the related term as "a system for ingesting, transforming and continuously delivering data for the development, training and deployment of artificial intelligence (AI) models," and notes that modern versions "enrich data with metadata, lineage, business definitions and governance rules," according to IBM's AI data pipeline explainer. That last clause is doing more work than most readers notice. Lineage is not an add-on to the definition. It is in the definition.

The plain version: something happens in the world, a system writes it down, your pipeline picks it up, changes its shape a few times, shows some version of it to a model, and the model's answer causes a second thing to happen. The pipeline is the connective tissue between those two events, and everything interesting about governing AI lives in the tissue rather than the model.

The engineering literature splits the term further. Hopsworks' MLOps dictionary defines an AI pipeline as "a program that takes input and produces one or more ML artifacts as output" and names three canonical types: feature, training and inference pipelines. Those are useful for people building models. They are not the shape most enterprises operate in 2026, because most enterprises are not training anything — they call a vendor model and wire its answer into a business system.

So this article uses the operating definition rather than the training one. An AI pipeline in the enterprise sense is the ingest-to-action path around a model you did not train, and the thing you are accountable for is not the model's weights but the data you fed it and the actions it triggered. Most published AI pipeline architecture diagrams draw the same five or six boxes, interface, orchestration, model, data connectivity, output — and the diagram is not wrong. It is simply answering a build question when the operating question is custody.

That reframing changes what "monitoring the pipeline" has to mean. Latency, throughput and error rate describe the machine. They say nothing about which specific rows crossed which boundary. A pipeline can be green on every dashboard while quietly copying a revoked customer record into four systems you have not inventoried.

The three questions a pipeline definition has to survive

Where does the data change custody, so you know where a policy has to exist? Which hops are reversible, given that dropping a row from a landing table is easy and withdrawing a sentence from a vendor's logs is not? And what is the smallest unit you can name, because if it is "the nightly job", you cannot answer a question about one person. Most pipeline diagrams answer none of the three, since they are drawn as boxes describing stages rather than custody. The diagram above is drawn the other way round.

What an AI pipeline is not: three things that get mislabelled

Three adjacent things get called an AI pipeline and are not one. Getting the distinction right saves an argument in every architecture review, because each of the three has a different owner, a different failure mode and a different regulator.

It is not a machine learning pipeline. A machine learning pipeline is scoped to producing and updating a model: feature engineering, training, evaluation, deployment, retraining. It ends when the model artifact ships. An AI pipeline in the operating sense begins roughly where that one ends and continues into the business system. A company can have zero machine learning pipelines and a dozen AI pipelines, which is the common 2026 configuration: no in-house training, a great deal of inference. Conflating the two is why data science teams get handed governance questions they have no visibility into.

It is not a workflow builder. A workflow tool chains steps and handles retries, branching and approvals. It is the orchestration layer, and it is genuinely part of the pipeline — but the pipeline includes the data substrate underneath it, which the builder usually does not model at all. A workflow run record tells you step 4 called the model. It does not tell you which 200 chunks were retrieved to build step 4's prompt. We covered the orchestration half of this in our earlier analysis of why AI-powered workflows break at the commit step; this piece is about the substrate the workflow sits on.

It is not an agent. An agent chooses its own sequence; a pipeline has one chosen in advance. "We replaced the pipeline with an agent" usually means the sequence became dynamic, which makes lineage harder rather than unnecessary: the set of records touched now varies per run and can only be known afterwards, from the log. Agentic pipelines need more record-level tracing, not less.

A fourth thing worth naming: a data pipeline with no model in it already had lineage problems. AI did not create the discipline. AI made the consequences of skipping it irreversible, which is a different claim and the one this article defends.

The six hops a record takes through an AI pipeline

Every AI pipeline, whatever the vendor's architecture diagram calls its layers, moves a record through six hops. The hops are the same across the standard use cases, lead scoring, ticket triage, document extraction, forecasting — because the custody changes are a property of the data, not of the industry. The hop names vary; the custody changes do not. What follows is the frame this article uses throughout, and it is the frame the worked example later applies to a single support ticket.

Hop 1 — Ingest. The record leaves its system of origin and lands somewhere you control. A copy now exists in a landing zone, a bucket, a queue. This hop is fully reversible: you can delete the copy and be back where you started.

Hop 2 — Normalise. The record is cleaned, joined, deduplicated, typed. New derived rows exist. Still reversible, because the transformation is deterministic and re-runnable from the source.

Hop 3 — Enrich. The record is chunked, embedded, indexed, summarised, or turned into a feature. This is the hop where reversibility quietly stops being free. An embedding is a lossy, one-way projection of the text; the text can often be partially reconstructed from it, but the embedding cannot be cleanly "un-made" from the index in the way a row can be dropped from a table. More on this below, because it is the single most under-appreciated fact in AI pipeline design.

Hop 4 — Infer. Some representation of the record is placed in a prompt and sent to a model, frequently a model running on someone else's infrastructure under someone else's retention policy. You have now created a copy you do not control.

Hop 5 — Act. The model's answer causes a write: a CRM field updates, a ticket routes, an email sends, a payment queues. This hop is reversible only to the extent that you recorded enough to reverse it, and only where the target system supports reversal. An email does not.

Hop 6 — Retain. Everything produced along the way lands in warehouses, backups, log aggregators and SIEM. Reversibility here is schedule and discipline, not physics.

Read that last column again. Those six questions are the entire content of AI pipeline lineage. A pipeline that answers all six for an arbitrary record is governed; one that answers none is not ungoverned in a vague way, it is ungoverned in six specific, individually fixable ways. Hops 1, 2 and 6 are where the engineering effort usually goes, because classic data engineering already had tooling for them. Hops 3 through 5 are the ones AI added, and the ones with the weakest instrumentation in most stacks.

Why your run log will not answer the auditor's question

A run log records what the pipeline did. An auditor asks what happened to a person. Those are different queries against different keys, and the second one cannot be reconstructed from the first unless someone deliberately wrote the join.

Here is the shape of the mismatch. Your orchestrator emits a run record: job enrich_tickets, run 9f2a, 02:14, 41,802 rows, succeeded. Your model gateway emits a call record: model, tokens, latency, cost, service account. Both are useful. Neither contains the string that identifies the customer. Answering "what did you do with Ms. Okafor's data" needs a third record keyed on the record identifier, and that is the one almost nobody writes.

Regulators have already legislated in the direction of the third record without quite naming it. The EU AI Act's Article 10 requires providers of high-risk systems to document "data collection processes and the origin of data, and in the case of personal data, the original purpose of the data collection," per the text of Article 10, origin, per dataset, in writing. Article 12 goes further into runtime: high-risk systems "shall technically allow for the automatic recording of events (logs) over the lifetime of the system," according to Article 12. Automatic. Not "the team keeps a spreadsheet."

Retention is specified too. Article 19 requires providers to keep those automatically generated logs "for a period appropriate to the intended purpose of the high-risk AI system, of at least six months," and Article 26(6) places the same at-least-six-months obligation on deployers for logs under their control. If you operate a high-risk system, six months of automatic event records is a floor written into law, not a nice-to-have that competes with your storage budget.

The US framework arrives at the same place through a different door. NIST's AI Risk Management Framework states that "maintaining the provenance of training data and supporting attribution of the AI system's decisions to subsets of training data can assist with both transparency and accountability," in NIST AI 100-1. Attribution of decisions to subsets of data is exactly the backward trace. NIST is describing training data; the same mechanic applies with more force to retrieval, where the subset that shaped an answer is small, identifiable, and changes every query.

The security community converged independently. The OWASP Top 10 for LLM Applications lists "track data origins and transformations using tools like OWASP CycloneDX or ML-BOM" as the first prevention strategy for LLM04:2025, Data and Model Poisoning, alongside data version control to "detect manipulation." Three institutions with different mandates — a regulator, a standards body and a security consortium, all landed on provenance as the control. That is unusual enough to be worth noticing.

The practical translation: with only a run log and a call log, you are defensible in no specific case. The third record is cheap at build time and expensive to reconstruct later — the classic profile of a control that gets skipped.

The Recall Test: one record, both directions, in an afternoon

Here is a diagnostic you can run this week without buying anything. We call it the Recall Test, and it has two halves that must both pass.

Pick one real record. Not a synthetic one, not a class of records. One customer, one invoice, one ticket, one employee, chosen at random from the last 30 days by someone who is not the pipeline's author.

Half one, forward. Starting from that record's identifier in its system of origin, list every artifact that now exists because of it: landing-zone copies, normalised rows, chunks, embeddings, index entries, prompts sent to a vendor, model responses stored, downstream writes, warehouse rows, backups. Name the store and the identifier in each. Stop when you have an exhaustive list or when you cannot go further.

Half two — backward. Now pick one output the pipeline produced in the same window: a routed ticket, a generated summary, a scored lead. List every input that shaped it: which chunks were retrieved, which model version answered, which prompt template was in force, which upstream records those chunks came from.

Score it on time, not completeness. With enough determination and enough SQL you will usually get an answer eventually. The finding is how long it takes and how many people it needs.

The fourth row is the most common result we would expect in a stack assembled quickly during 2025, because retrieval tracing is the newest and least standardised of the six hops. The fifth row is the one that should stop a production launch.

The test is worth the afternoon even if you expect to pass. It is falsifiable in a way a governance questionnaire is not — a checklist asks whether you have lineage and everyone answers yes — and it produces an artifact, because the list itself is the start of the lineage ledger described below. The name is literal: it asks whether the pipeline can recall a record in both senses, remember where it went and call it back.

Forward trace: the blast radius of one bad record

The forward trace answers a question that arrives without warning: this record is wrong, revoked, or must be deleted — where else does it now live?

Four events trigger it, and every organisation running an AI pipeline will meet at least one:

A deletion request. Article 17 of the GDPR gives a data subject the right to erasure, and paragraph 2 pushes the obligation outward: a controller that made the data public must take reasonable steps to inform other controllers of a request to erase "any links to, or copy or replication of, those personal data," per the text of Article 17. Copy or replication. An AI pipeline is a copy-and-replication machine by design.

A correction. A source record was wrong for six weeks. Everything derived from it in those six weeks is also wrong, including any summary a model wrote and any decision that summary drove.

A poisoning or contamination finding. A document that should never have entered the corpus did. OWASP's LLM04 entry treats this as a security failure requiring provenance tracking, because the response to contamination is "find and remove everything downstream" — the forward trace, under a deadline.

A contract or licence change. A source's terms change, or a vendor relationship ends, and you have to demonstrate their data is out of your systems, including out of anything derived from it.

In all four cases the operational question is identical and the answer is a list. Without the list you choose between rebuilding the whole index and asserting something you cannot demonstrate. Rebuilding is not always unreasonable: for a few hundred thousand chunks a full re-embed is hours of compute and a known bill. For tens of millions, or a corpus that took six weeks to assemble because half the sources are rate-limited, "just rebuild it" becomes a project. Calculate that break-even before the incident, not during it.

What makes the forward trace tractable is a decision taken at hop 1: carry the source identifier as a first-class field through every subsequent hop. Not in a comment. Not inferrable from a filename. A field, indexed, that survives chunking. Every chunk knows its parent document; every document knows its source record; every model call records the chunk IDs it was given; every downstream write records the call that produced it. That chain is not sophisticated engineering. It is bookkeeping, and it is roughly free at build time.

It does not exist by default because most retrieval tooling makes it optional metadata rather than a required field, and defaults win. Our earlier analysis of governing the retrieval layer works through what a corpus record should carry; the forward trace is the reason those fields have to be mandatory rather than encouraged.

Backward trace: proving which inputs shaped one output

The backward trace answers the other question, the one that arrives from a customer, a regulator or a lawyer: why did the system say that?

For a retrieval-augmented AI pipeline, the honest answer has four components, and a pipeline that stores all four can reconstruct any answer it ever gave:

1. The prompt template version in force at the time, including system instructions.
2. The retrieved set — the identifiers of every chunk placed in context, in order, with their scores.
3. The model identity — provider, model name, and the specific version or snapshot, because "gpt-class model, some point in Q2" is not an answer.
4. The parameters — temperature, max tokens, tool definitions offered, and whether any tool was actually called.

Store those four and you can explain an output months later without re-running anything. Store none of them and the output is unattributable the moment the index changes, which is daily.

Notice what is not on the list: the model's internal reasoning. You cannot trace that and should not pretend to. What you can trace is the input surface, and that is what disputes are about. Nearly every real complaint about an AI-driven decision resolves to "the system used the wrong information," not "the transformer weighted attention badly."

Determinism is the caveat that catches people. Even with all four stored, replaying a call rarely reproduces the original text token for token: vendors change infrastructure, sampling introduces variance, snapshots get deprecated. So keep the record of what was sent and what came back, not a recipe for regenerating it. Teams planning to "just re-run it if anyone asks" find this out at the worst moment.

There is a standards answer forming here too, from two directions. OpenLineage, a Linux Foundation AI & Data graduate project, defines "an open framework for data lineage collection and analysis" built on three entities, dataset, job and run — each uniquely identified and extensible through facets. It is a real, adopted specification rather than a vendor's marketing diagram, and its facet mechanism is where AI-specific fields such as retrieved-chunk sets can live. From the security side, CycloneDX's ML-BOM "represents datasets, models, and configurations for AI and machine learning systems" and "documents provenance and ethical considerations for datasets," under the ECMA-424 standard. Neither is a complete answer on its own. Together they mean a team building lineage in 2026 is not inventing a format from nothing.

Embeddings do not forget

This is the section that changes how the rest of the article should be read, and it is the one the general AI pipeline literature almost universally skips.

At hop 3, text becomes a vector. Most vector stores implement deletion as a metadata flag: the vector stops being returned by queries, and the underlying data stays on disk until a compaction or rebuild happens — which may be never, on a store nobody is actively maintaining. Suppression from query results is not the same thing as erasure, and the difference is precisely the difference a data protection regulator cares about.

How much of the original survives is now measured rather than speculated about. In a June 2026 paper, Ghost Vectors: Soft-Deleted Embeddings Remain Reconstructible in HNSW Vector Databases, Chakraborttii and colleagues showed that deleted embeddings in HNSW indexes remain physically recoverable by reading the raw index files directly, bypassing the API. Using an off-the-shelf inversion method without domain tuning, they recovered "25.5% of exact person names and 46.4% of geographic locations" from a Wikipedia biography set, and on synthetic medical records reached "100% for both patient age and gender markers." Their proposed mitigation, epoch key rotation — encrypt vectors and discard the key on deletion, reduced observed recovery to zero and completed in 2.5 ms for 500 deleted vectors.

Two things follow, and they point in opposite directions.

The alarming one. If your compliance story is "we call delete() on the vector store," it is now weaker in a way someone can cite a paper about. A soft delete changes your product's behaviour. It does not necessarily remove anything from a disk image, a snapshot, or last Tuesday's backup.

The reassuring one. The mitigation is architectural rather than heroic: per-epoch encryption keys, hard compaction on a schedule, or rebuild-on-deletion for small corpora all close the gap. What it requires is knowing which vectors correspond to the record you were asked to erase — the forward trace again, arriving from a third direction.

The regulatory backdrop makes this less theoretical than it sounds. The European Data Protection Board's Opinion 28/2024 concluded that AI models trained with personal data "cannot, in all cases, be considered anonymous," and that for a model to be treated as anonymous, both the likelihood of extracting personal data about individuals in the training set and the likelihood of obtaining such data from queries "should be insignificant." The Opinion also holds that a deployer's own accountability includes assessing whether the model it is deploying was developed lawfully, taking into account "the source of the personal data." Provenance is not only an obligation you owe about your own data. It is a question you are expected to have asked about someone else's.

That last point is where a lot of 2026 procurement is quietly out of position. Buying a model is buying its history.

What the 2026 evidence says about pipeline transparency

The macro picture in 2026: deployment is accelerating and disclosure is going the other way.

Stanford HAI's 2026 AI Index reports organizational adoption at 88%. The same report's Responsible AI chapter records the AI Incident Database logging "362 in 2025, up from 233 in 2024", a rise of roughly 55% in a single year. On transparency, the Index's finding is blunt: after climbing from 37 to 58 between 2023 and 2024, the average Foundation Model Transparency Index score "dropped to 40 in 2025," and "major gaps persist in disclosure around training data, compute resources, and post-deployment impact."

Read those three together. Adoption is near-universal, incidents are climbing faster than adoption, and the upstream visibility you would need to investigate one is getting worse. If providers disclose less about their training data, the only provenance you can rely on is the provenance of the data you put in — the part of the AI pipeline you control.

The readiness picture is consistent. IBM's explainer cites a 2025 IBM Institute for Business Value survey of chief data officers in which "just 26% said they believed their data capabilities can power AI-enabled revenue streams." Three-quarters of the people who own the data layer do not think it is ready for what is being built on it.

A data-currency note, since it changes one number: the 2026 edition of the AI Index supersedes the 2025 edition, and the transparency score moved in the opposite direction between them, up through 2024, then down. Citing the older edition would have supported the opposite argument. It is worth checking which edition any transparency claim you read is drawing on.

Two sources a reader might reasonably expect here are missing. Gartner's and Forrester's primary research on data lineage adoption sits behind paywalls we did not purchase, and IBM's page cites a Gartner prediction we could not verify at its origin, so we have not repeated it. Vendor-published lineage benchmarks were excluded on the same principle.

A worked example: one support ticket, end to end

Abstractions get argued about; a single record does not. Here is one ticket through all six hops, with the lineage fields that make it traceable. The pipeline shape is a common one — support ticket triage with retrieval over an internal knowledge base — and the arithmetic below is illustrative, using stated assumptions, not measured from a live system.

The setup. A B2B software company receives about 900 support tickets a week. An AI pipeline reads each new ticket, retrieves relevant internal documentation, drafts a suggested reply and a priority, and either routes the ticket or posts the draft for an agent to approve.

Hop 1 — Ingest. Ticket TCK-88420 arrives in the helpdesk at 09:14. The pipeline pulls it via webhook. Lineage field written: source_system=helpdesk, source_id=TCK-88420, source_version=1, ingested_at=09:14:07.

Hop 2 — Normalise. The ticket body, the customer's account tier and the last three tickets from the same account are joined into one context object. Lineage field written: derived_from=[TCK-88420, ACC-3391, TCK-87102, TCK-86644, TCK-85901]. Five source identifiers, not one — this is where a naive forward trace loses records, because the ticket now carries other people's data.

Hop 3 — Enrich. The ticket text is embedded for similarity search against past resolutions. One vector is written to the index with parent_source_id=TCK-88420 as a required field. Separately, the retrieval step will pull documentation chunks; those already carry their own parent_doc_id.

Hop 4 — Infer. The prompt is assembled: system template triage-v7, the context object, and the eight retrieved chunks. Lineage record written for the call: template=triage-v7, retrieved=[KB-114#c3, KB-114#c4, KB-207#c1, ...] (all eight, in order, with scores), model=vendor/model-name@2026-05 snapshot, temperature=0.2, call_id=CALL-5f31.

Hop 5 — Act. The model returns priority P2 and a draft reply. The pipeline sets the helpdesk priority field and posts the draft as an internal note. Lineage record written: action=set_priority, target=TCK-88420.priority, previous_value=P3, new_value=P2, caused_by=CALL-5f31, reversible=true, undo=set priority to P3.

Hop 6 — Retain. All of the above lands in the warehouse nightly and in the log store immediately.

Now run the Recall Test on it.

Forward, from TCK-88420: one landing-zone copy, one normalised context object, one vector in the similarity index, one prompt sent to an external vendor at 09:14:31, one stored response, one priority write, one internal note, one warehouse row, and whatever backups covered 09:00–10:00. Nine artifacts, four systems, one of them a third party. Every one of those is findable by identifier because hop 1's field was carried forward.

Backward, from the priority change: caused_by=CALL-5f31 → template triage-v7, eight named chunks from three KB articles, a named model snapshot, and a context object naming five source records. The question "why was this escalated" has a complete answer that fits on one screen.

The illustrative arithmetic. Assume the lineage records cost roughly 2 KB per ticket across all hops. At 900 tickets a week that is about 1.8 MB weekly, under 100 MB a year before compression, a rounding error against the ticket data itself. Assume writing them costs one engineer three days, once. The alternative is unplanned reconstruction after an incident: reading orchestrator logs, correlating timestamps against gateway records, accepting gaps wherever timestamps are ambiguous. Days per investigation, repeated per investigation, producing an answer nobody trusts. Substitute your own volumes; the shape holds across two orders of magnitude, because lineage records scale with events while reconstruction cost scales with elapsed time and staff turnover.

The failure mode this example exposes. Delete the derived_from list at hop 2 and everything still works — pipeline runs, dashboards green, tickets triaged. You discover the loss when someone asks about ACC-3391, and by then the join is gone. Lineage failures are silent at write time and expensive at read time, the exact profile of a control that gets cut from a sprint.

The lineage ledger: five fields per hop

The Recall Test tells you where you stand. The lineage ledger is what you build so you never have to run it again in anger. It is one row per hop per record, with five fields, and it is deliberately small enough that "we didn't have time" is not a credible objection.

Four design rules make the ledger survive contact with production:

Write it in the same transaction as the work. A ledger written by a separate observability agent drifts, and drift is worse than absence because it produces confident wrong answers.

Make derived_from a list, always. Single-parent lineage is the most common modelling error, and joins are the most common shape in real pipelines.

Store reversal as text a human can execute. "set priority to P3" beats a boolean. The boolean tells you reversal is possible; the text tells you how, at 2am, six months later.

Give it the same retention as your longest obligation. If Article 19 sets a six-month floor and your contracts require two years, the ledger inherits two years. A lineage record that expires before the thing it describes is a filing system that eats its own index.

Where it lives matters less than that it exists in one place. A table in the operational database is enough at small scale; a dedicated append-only store with the same retention as your longest obligation is the shape at large scale. What does not work is spreading it across five systems, because then answering the Recall Test means five queries and a judgement call about which one is authoritative.

The ledger is not an alternative to run logs or model-call logs. It is the third record, keyed on the thing the questions are actually about, and it joins the other two together.

What per-record lineage costs, and what skipping it costs

There is no single price, but the cost structure is predictable enough to plan against. The figures below are illustrative estimates from stated assumptions, not benchmarks, we have not run a controlled build across multiple customers, and any vendor who quotes you a precise number without seeing your stack is guessing too.

The costs of skipping it are lumpier and land at bad moments:

• Per-investigation reconstruction. Every incident, complaint or deletion request becomes a bespoke engineering task rather than a query.
• Over-broad remediation. Without knowing which artifacts descend from a record, the safe response is to rebuild or purge more than you need to, which costs compute and downtime.
• Unprovable assertions. You can state that data was removed; you cannot demonstrate it. Under the erasure obligation in GDPR Article 17, the demonstration is the point.
• Slower approvals. Security and compliance reviewers respond to uncertainty by adding conditions. A pipeline that can answer provenance questions on demand clears review faster — which, for most teams, is the benefit that actually gets felt.

That last one is the honest commercial argument, and the compliance framing undersells it. Teams do not build lineage because a regulator asked. They build it because the third review cycle on the same pipeline costs more than the instrumentation would have.

Batch versus real time: what changes for lineage

Batch versus streaming is usually presented as a latency trade-off. For lineage the trade-off runs the other way.

Batch is easier to trace and easier to lose. A nightly job has a clean boundary: this run, this input set, this output set, so set-level lineage is nearly free. But batch encourages set-level thinking, and set-level lineage cannot answer questions about one person. The failure mode is a team that knows exactly which files last Tuesday's run consumed and cannot say whether one customer was in any of them.

Streaming is harder to trace and harder to lose. Every event is individually addressable, so lineage is naturally per-record; the difficulty is durability and ordering. Exactly-once ledger writes alongside the work is a genuine engineering problem, not bookkeeping. The standard answer is the transactional outbox: write the ledger row into the same database transaction as the work, and publish it from there, so a crash cannot commit one without the other.

A hybrid is the common real answer, and it has a specific lineage requirement: the batch and streaming paths must write into the same ledger with the same identifier scheme. Two ledgers that do not join are worse than one incomplete ledger, because they create the appearance of coverage. This is the version of the "both options" question worth costing before you pick a shape.

Six mistakes that break AI pipeline lineage

These are the recurring patterns, ordered roughly by how often they appear in architecture discussions rather than by severity.

1. Treating the run as the unit. The root of most of the others. The orchestrator's identifier is not the record's identifier, and no amount of dashboard sophistication converts one into the other.

2. Letting the source identifier die at chunking. Text gets split, chunks get IDs, and the parent reference sits in optional metadata that a later refactor drops. Make it required, with a not-null constraint: a five-minute decision at build time, an archaeology project later.

3. Modelling lineage as single-parent. Real hops join. A summary derives from twelve documents; a context object derives from five records; a fine-tune derives from a dataset that derives from four sources. A derived_from field that holds one value silently discards eleven-twelfths of the truth.

4. Trusting soft deletes. Ghost Vectors is the specific evidence; the general rule predates it. Where deletion is a flag, deletion is a behaviour change, not an erasure. Decide explicitly whether suppression meets your obligations, and write the reasoning down either way.

5. Logging the prompt but not the retrieved set. Extremely common, because the prompt looks like the complete input. If the retrieved chunk IDs are not stored separately, you cannot reconstruct which documents were in play once the index changes — and if the prompt itself was stored with the chunk text inlined, you now have a copy of that content in your log store, with its own retention and access questions.

6. Retaining the ledger for less time than the artifacts it describes. The pipeline keeps warehouse rows for seven years and lineage events for 30 days. Six months is the statutory floor for high-risk systems under the AI Act; your own longest obligation is the real number. Set them together or the ledger expires precisely when someone needs it.

A seventh, honourable mention: treating the vendor's console as your lineage. It is their record of their side of hop 4, on their retention policy, and it will not join to your identifiers. Useful, not sufficient.

When per-record lineage is not worth it

The argument in this article has a boundary, and pretending otherwise would be the kind of governance maximalism that gets ignored. Per-record lineage is not always the right investment.

When no personal or regulated data enters the pipeline at any hop. A pipeline over public technical documentation, generating internal engineering summaries, carries none of the erasure or provenance obligations that make lineage load-bearing. Set-level lineage is proportionate; per-record is overhead.

When every hop is genuinely reversible and cheap to rebuild. A small corpus you can re-embed in twenty minutes for a few dollars does not need a forward trace, because "rebuild it" is a complete answer to every question the forward trace would answer.

When the pipeline is read-only. If hop 5 does not exist and a human decides everything, the reversal question mostly evaporates. You still want the backward trace for disputes, but the blast radius is small.

When you are still deciding whether the pipeline should exist. A two-week prototype on sample data needs a decision, not a ledger. Instrumenting something you will throw away is waste. The honest sequence is prototype, decide, then instrument before the first real record arrives — and that moment arrives sooner than anyone plans for, which is how prototypes become production without a governance review.

One honest counter-position: strict data minimisation substitutes for lineage at the margin, and it is usually cheaper. If personal data never enters hops 3 through 5 because it was masked at ingest, most of this article's concerns do not apply to those hops. Masking at the boundary is the approach we described in our earlier analysis of inline data-loss prevention at the gateway.

Where LeapForce fits, and where it does not

LeapForce is a governance layer for the AI surface of the pipeline — hops 4 and 5 — not a data engineering platform. Our AI Gateway sits in the path of every model call and records it against the user, team, tool and agent that made it, and our Observability and Audit layer keeps the action record above those calls: which records an agent read, what it changed, and what it was refused, held tamper-evident on your retention schedule. The gateway rollout model we publish is Observe first. Enforce second. Optimize third., point one team's traffic at the gateway in observe mode before any rule exists, which is also the fastest honest way to discover what your AI pipeline is really touching. What we do not do is build hops 1 through 3: we are not an ETL tool, a vector store or a lineage catalogue, and the derived_from chain through your ingestion and enrichment layers has to be written by whoever owns those systems. Leapforce is in active development and per-capability build status is disclosed honestly; treat the paragraph above as a description of the control points, not a claim that every capability named across our site ships today.

Where this analysis is still uncertain

Four open questions, stated plainly.

We have not measured the retrofit cost. The build-versus-retrofit multiple in the cost section is a shape, not a number. We have not run instrumented rebuilds across a set of customer pipelines and published the distribution, and anyone who tells you the retrofit is "about 3x" without seeing your code is estimating from the same absence of data we are.

The vector-erasure position may move. The Ghost Vectors paper is recent, single-team work on HNSW indexes specifically. Its recovery rates are strong evidence that soft deletion is insufficient in the systems tested; they are not yet a settled finding across every index type and configuration, and the vendors will respond. Expect the practical guidance here to sharpen within a year.

Regulatory interpretation is unsettled where it matters most. The EU AI Act's Article 12 logging requirement and the GDPR's erasure obligation have not yet been tested against a retrieval index in an enforcement action we are aware of. Whether suppressing a vector satisfies erasure is currently a reasoned argument, not a decided question, and reasonable counsel disagree.

Lineage standards for retrieval do not exist yet. OpenLineage models datasets, jobs and runs well and can carry AI-specific data through facets; there is no widely adopted facet for "the chunks that shaped this answer." Teams are inventing their own field names, which means portability between tools is currently poor. If you build the ledger described above, expect to migrate it once.

Finally, the boundary of our own vantage point. We see hops 4 and 5 clearly because that is where our platform sits. Our reading of hops 1 through 3 comes from published research, standards documents and practitioner accounts rather than from operating those systems ourselves, and readers who run large data platforms will know things about hop 3 that this article does not.

Ready to Govern Your AI?

Talk to LeapForce — one controlled layer for every AI tool, connector, model, and agent.