Where Did That Number Come From? Provenance and Lineage for Rig Data

Building a defensible chain of custody from sensor to decision.

Part of the WillCo drilling-data-architecture series.

Figure 1. The same well, the same moment, two different numbers on two screens. The question is not which is wrong, but which one can explain itself. Illustrative.

🧭 Two numbers, one well, no defensible answer

Two systems on the same rig display different depths for the same well at the same moment. One reads a few feet deeper than the other. The instinct is to ask which one is wrong.

That is the wrong question. Either number may be internally correct for its own measurement definition, depth reference, and update state. The problem is that neither display makes those differences visible. One may update on a different cycle than the other. One may already carry a correction that the other has not received yet. The disagreement is not necessarily an error. It is the absence of context.

The real failure is quieter and more serious: neither number, on its own, explains what it represents, when it was captured, or how it was produced. Strip a value out of its origin and its history and you are left with a figure that looks precise and cannot be defended. It cannot survive an audit. It cannot be reconciled at handover. It cannot settle a dispute. And it should never, on its own, authorize an automated decision.

Earlier articles in this series argued for doing the correctness work once and publishing a single curated product that carries quality and uncertainty with every value. This piece takes the next step. Cleaning a value establishes that it is plausible. It does not establish where it came from. A value becomes trustworthy only when you can trace it back to its source, its timing, every transformation applied to it, the exact rule or model version responsible, each human intervention, and every revision, all the way to the preserved raw record.

That traceable history is provenance and lineage. It is the difference between a number you believe and a number you can prove.

🔍 Provenance and lineage: two related ideas

The two terms are close relatives, and vendors and standards use them in overlapping ways. The framing below is a practical one for drilling data, not the only accepted definition.

Provenance is the origin and history of a value: where it came from, what it represents, when it was captured, and what has happened to it since. Provenance answers, "What is this, where did it come from, and what has it been through?"

Lineage is the dependency path: the chain of inputs and transformations that produced the value you are looking at. Lineage answers, "If I pull this thread, what inputs, transformations, and decisions are tied to it upstream?" A derived figure such as equivalent circulating density has lineage running back through flow rate, mud properties, geometry, and depth, each with its own provenance.

Put simply, provenance is the story of one value, and lineage is the map of everything that value depends on. You need both. A value with provenance but no lineage tells you its own history but hides what it was computed from. A value with lineage but no provenance shows the wiring but cannot tell you whether any input was trustworthy in the first place.

A note on the phrase "chain of custody." It is used here as an engineering analogy for an unbroken, attributed trail from source to decision. Making that trail legal-grade evidence takes more than good metadata: verified identity, access control, integrity protection, and signed, audited records. The discipline in this article is the foundation that a legal-grade claim would be built on, not a substitute for those controls.

🪜 Follow one value from sensor to decision

Definitions are easy to nod along to and hard to apply. So take the bit-depth value from the opening and follow it, one value, the whole way from the sensor to the moment someone acts on it.

Source. In many surface systems, bit depth is calculated rather than measured directly. It begins with block position from a draw-works encoder, a pipe tally, and a selected depth reference. The provenance record has to name every one of those sources, because the value is only as defensible as the weakest input behind it.

Acquisition. The reading is captured at an instant and timestamped. Which clock stamped it, and when, matters more than it looks, as the next section shows.

Validation. The value is checked against range, rate of change, and physical plausibility. The outcome of that check, passed, flagged, or missing context, is part of the record, not a gate that silently drops the reading.

Derivation. Block position becomes bit depth through a calculation that applies the pipe tally and the depth reference. This is where two systems quietly diverge: a different datum or zero reference produces a different, and still valid, bit depth. Lineage has to capture which reference and which calculation version were used.

Human intervention. A driller re-zeroes the string, or corrects a tally entry. That action is a first-class event in the record, with who did it, when, and why, not an invisible nudge to the number.

Publication. The curated value is published, carrying its quality state, its uncertainty, and compact identifiers that point back to the source, rule, and revision records behind it.

Decision. Someone, or something, acts: picks up off bottom, correlates to an offset, triggers an alert. For that action to be defensible later, every step above has to be reconstructable.

Figure 2. A defensible value carries evidence from its source through acquisition, validation, derivation, annotation, publication, and use in a decision.

Read that chain in reverse and you have lineage: from the decision back through publication, annotation, derivation, validation, acquisition, and source, to the preserved raw reading. A value you can walk backward is a value you can defend. A value you cannot is a guess with good production values.

✏️ When the record changes

Rig data is not written once and left alone. Corrections arrive. A late pipe tally, a re-interpreted interval, a clock that turns out to have been off. The question is what happens to the earlier answer when the new one lands.

The wrong model overwrites it. The right model appends. A correction never erases the previous result; it records a new one alongside it, with the reason, the responsible actor, the rule or model version involved, and the time the change takes effect. The earlier answer stays in the record, marked as superseded, because six months later the only way to explain a decision is to see the number as it stood when the decision was made.

This is also why time needs more than a single stamp. A timestamp alone is not enough. Depending on the use, you may need to distinguish source time, acquisition time, processing time, publication time, and effective revision time. They are not the same, and collapsing them hides exactly the gaps that cause disputes.

That is not a theoretical worry. In a case documented by Matt Isbell of Hess Corp in Drilling Contractor (2017), systems on a rig each kept their own clock, and those clocks disagreed with the source system by about two minutes on one system and roughly fifteen minutes on another. When two feeds are a quarter hour apart and nobody has recorded which clock is which, "what happened first" becomes unanswerable, and any event reconstruction built on top of it inherits the error.

Figure 3. A governed correction preserves the earlier state and appends the reason, responsible actor, version, and effective time of the revision.

Append-only lineage has one hard dependency worth naming now: it can only explain a transformation if the original evidence is still there to replay. A preserved, replayable raw store is the foundation the whole chain rests on. That is an architecture topic in its own right, and a later article will take it on. Here the principle is enough: you cannot reconstruct what you did not keep.

🛠️ Where lineage earns its keep

Lineage is not a filing exercise. It pays for itself in the moments when someone has to explain, reconcile, or defend what the data said. Five of those moments recur on nearly every rig.

Late BHA information re-interprets an earlier interval

Bottom-hole assembly details often arrive after the activity they describe. When they land, they can change how an earlier interval should be read. With lineage, the earlier interpretation stays intact and the re-interpretation is recorded against it, with its own timing and reason. Without it, the record silently changes and nobody can see that it ever said something else.

A driller remark carries human context

A driller's note is often the only explanation for why a number looks the way it does. That remark has to retain its authorship, its timestamp, and its source identity. It is evidence, and evidence with no attribution is just hearsay in the log.

A missing note stays visible as a gap

Sometimes the context never arrives, or arrives late. The defensible move is to leave the gap visible as a gap, not to infer a plausible value and present it as fact. A recorded "no context available" is more trustworthy than a confident guess, because it tells the next reader exactly where the evidence stops.

A training set distinguishes what is real from what was repaired

Machine-learning models inherit the integrity of their training data. A defensible dataset keeps raw, repaired, human-labeled, and machine-labeled records distinguishable from one another. Blur those categories together and a model may unknowingly learn from repaired or machine-generated labels as though they were original observations, and its confidence becomes untraceable by construction.

Automated component provenance, briefly

Provenance is not only about readings. RFID tagging of downhole components is a small, concrete example: rugged tags already survive downhole conditions and can be read as drill-string components pass the rig floor, so a tool joint carries its own identity and the record of what was in the hole, and when, is captured automatically rather than reconstructed from memory. It is a specialized capability today, but it is the same principle applied to hardware: identity travels with the thing.

There is a thread running through all five. On a rig, data quality cannot be owned by one system at the end of the line. It has to be controlled by every entity that touches the data along the way. That is precisely the principle behind the Operators Group for Data Quality, formed in 2014: quality and context are a shared chain of custody, not a cleanup step bolted on at the end.

Figure 4. Event reconstruction is where lineage stops being paperwork and becomes the answer. Illustrative.

✅ What a defensible record contains

Translate all of that into something a team can check against. A defensible rig data record does not need every field physically stapled to every high-frequency sample. It needs compact identifiers that resolve to governed source, rule, model, and revision records. What follows is the set of questions each value must be able to answer, directly or by reference.

Required element	What it must answer
Source identity	Sensor, system, service, asset, channel, or human source.
Measurement definition	What the value actually represents, including the reference system.
Time context	Source time, acquisition time, processing time, and effective revision time as needed.
Unit and reference	Engineering unit, datum, depth reference, sign convention, and coordinate context.
Quality state	Validity result, exception, missing context, or quality classification.
Transformation	Normalization, filtering, repair, calculation, aggregation, or contextualization applied.
Rule or model version	Exact governed version that produced or modified the result.
Authorship	Human-sourced, machine-generated, or system-derived, with responsible identity where applicable.
Confidence or uncertainty	The measure used, its definition, and the operating conditions under which it applies.
Revision history	Previous result, replacement result, reason, approval, and effective time.
Raw evidence link	Resolvable path back to preserved source data and supporting context.

Not every value needs the deepest version of every row. How much provenance is worth carrying depends on what the value is used for. A reading that only flips a pumps-on indicator needs less than one that feeds an automated set-point. The depth of the record should scale with the consequence of the decision and the authority being granted.

Figure 5. The evidence required should deepen as operational consequence and decision authority increase.

❓ Five questions for any rig data platform

The checklist is for the data. These five questions are for the platform that holds it. They work as acceptance-test or procurement questions, and the honest answer to each is either a demonstration or a no.

1. Can this value be traced to its original source and measurement definition?

2. Can every transformation, correction, and contextualization step be identified?

3. Can we determine which rule, algorithm, or model version produced the result?

4. Are human corrections distinguishable from machine-generated or automatically repaired values?

5. Can the previous answer and its supporting evidence be reconstructed after a revision?

A platform that can demonstrate all five is one you can build automation on. A platform that cannot is one where every automated decision carries a liability nobody has priced.

Figure 6. An anonymous edit cannot be defended. A governed revision preserves the original and records who changed what, when, why, and under which version.

🏁 From traceable data to trustworthy automation

Cleaning answered one question: is this value plausible. Provenance and lineage answer the harder one: can this value be defended. The first makes data usable. The second makes it accountable, and accountability is the thing automation cannot do without.

Every argument in this series has been building toward the same place. Clean the data once. Carry quality and uncertainty with every value. Preserve the raw and keep it replayable. Record the full history, append-only, so a correction never erases what came before. Do all of that and you have more than clean data. You have data that can stand up when someone asks where a number came from, which is exactly the standard an automated decision has to meet before it earns the authority to act.

That is the bridge to the next and final step in this series: turning traceable, trustworthy data into a closed loop that can act on its own. Because the rule underneath all of it is simple.

You cannot automate a decision you cannot trace.

References and further reading

1. WillCo Drilling Intelligence, "Clean Once, Trust Everywhere: Rig Data Cleansing as a Single Source of Truth." willcodrilling.com

2. WillCo Drilling Intelligence, "Reevaluating the RTOC: From Watching Screens to Intelligent Anomaly Detection." willcodrilling.com

3. WillCo Drilling Intelligence, "Security at the OT/IT Boundary: Protecting the Rig's Network as Models and Data Cross IT." willcodrilling.com

4. WillCo Drilling Intelligence, "Beyond WITSML: Turning High-Frequency Rig Data Into Edge Decisions." willcodrilling.com

5. Matt Isbell (Hess Corp), "Unsynced time measurements can lead to data aggregation challenges," Drilling Contractor, November 7, 2017. drillingcontractor.org

6. Nathan Zenero and Michael Behounek, "Drilling to the limit: Operators Group for Data Quality aims to create common specifications," Drilling Contractor, April 27, 2016. drillingcontractor.org

7. W3C, "PROV-Overview: An Overview of the PROV Family of Documents," W3C Working Group Note, April 30, 2013. w3.org/TR/prov-overview

8. Thomas Redlinger, "RFID drill pipe heads offshore," Offshore, November 1, 2011. offshore-mag.com