Clean Once, Trust Everywhere: Rig Data Cleansing as a Single Source of Truth

Rig data should be cleaned once, governed properly, and trusted everywhere, not re-cleaned privately by every tool that reads it.

Across a rig, the same raw stream gets cleaned again and again, each consumer to its own standard. There is a better architecture: do the correctness work once in a governed service, keep the raw data preserved, and serve one curated product that every process can read and condition for its own use.

🧩 Everyone cleans the same data, separately

The raw stream coming off a rig is not ready to use. Sensors drop out and freeze, values spike out of physical range, the clocks on different systems drift apart, and units are not always consistent from one source to the next. Before any process can calculate, classify, or learn from that stream, someone has to condition it and check that it is fit to use. That work is data cleansing, and on most rigs the surprising part is not that it happens. It is that it happens many times over.

Today the "someone" who cleans the data is everyone, separately. The rig-state classifier runs its own cleansing. The torque-and-drag model runs its own. The hydraulics calculation, the machine-learning pipeline, the WITSML publisher: each takes the same raw stream and conditions it to its own needs, with its own rules for what counts as a good record and what to do with a bad one. There is no shared, curated copy. There are as many private versions of the data as there are consumers.

Two examples make the fragmentation concrete, and they come from the same measurement: flow rate.

Consider rig state. To decide whether the well is being circulated, the logic only needs to know whether validated flow is clearly above a configured threshold, read together with pump status and operating context. For that question the precise value does not matter. Whether the flow reads, say, 400 gpm or 800 gpm, the answer is the same: the pumps are running and the well is being circulated. The reading only has to be positive and clearly past the line. It is a threshold test, hardened with hysteresis and contextual checks so that sensor noise or an ambiguous channel does not flip the state. A coarse, well-guarded reading is enough.

Now consider real-time hydraulics. Here those same two numbers are not interchangeable at all: 400 gpm and 800 gpm drive very different equivalent-circulating-density and pressure-loss results, so the accuracy of the value is exactly what the trust in the calculation rests on. The model needs the measured flow rate at its available resolution, along with mud density and rheology, wellbore and drillstring geometry, depth, rotation, and other operating inputs. A value that is only approximately right pushes the result off, so anything that rounds or smooths the measurement has degraded the input. (Surface standpipe pressure, by contrast, is best used to check the modeled pressure response and flag abnormal behavior. ECD is not computed directly from SPP alone, though pressure measurements can still help calibrate and constrain the hydraulics model.)

Same sensor, same instant, two completely different definitions of "clean enough." Multiply that across every consumer on the rig and the result is the status quo: no shared curated record, just a crowd of processes each starting from the same raw stream and each ending somewhere slightly different.

A trustworthy shared layer retains immutable raw records, applies versioned quality rules, and publishes curated values with quality, uncertainty, and provenance metadata. Consumers may condition the data for use without rewriting the shared record.

💸 The hidden cost of cleaning it N times

Letting every consumer clean its own copy feels natural, because each one knows its own needs best. But it carries three costs that grow with every process added to the rig.

The first is duplicated compute. The same validity checks, the same gap handling, the same conditioning run over and over on the same stream, once per consumer. On a rig where compute is limited and the stack of real-time processes keeps growing, that is the same work paid for many times.

The second is repeated latency and resource contention inside every consumer path. Independent consumers often run in parallel, so their delays do not add up into one system-wide total. The problem is subtler: each answer carries its own avoidable preprocessing delay, each process competes for scarce edge resources, and small differences in filtering or timing can make outputs that should be simultaneous disagree with one another.

The third cost is the worst, because it is not waste but disagreement. When two consumers clean independently, they can reach different conclusions about what is valid. One model discards a record that the next model keeps. The rig-state view and the hydraulics view are then built on subtly different versions of the data, and when their outputs conflict there is no easy way to say which cleansing was right, because there was never one shared answer to begin with. Fragmented cleansing does not just cost cycles and milliseconds. It quietly costs agreement.

In Anaconda's 2020 cross-industry survey, loading and cleansing together represented 45 percent of reported work time. The result illustrates the burden of preparation, but it is not a drilling-specific measure of duplicated cleansing.

The lesson is not that cleansing is unimportant. It is that cleansing is being treated as each process's private chore when it should be treated as infrastructure: done once, governed deliberately, and shared.

🧼 What one governed quality service does

Picture a single quality service that sits between the raw stream and everything that uses it. Its job is narrow and well defined: take the raw data in, run the shared correctness work, and publish one curated product that every downstream process can read, while the original measurements stay preserved.

It starts by preserving the raw. The acquired values are written to an immutable, replayable raw layer and are never overwritten. Everything the service publishes is a derived product that sits beside that raw layer, not a replacement for it, so a bad rule can always be corrected and the data reprocessed from the original evidence.

On top of the raw, it does the universal conditioning: it normalizes identity, time, and units; it validates completeness, physical range, rate of change, stuck sensors, and cross-sensor consistency; and it checks each record against defined rules to judge whether it is fit for governed reuse. Validity here is fitness for purpose measured against those rules, not a claim that a number is absolutely true.

Then it handles gaps by an explicit, channel-specific policy rather than a blanket "fill a few samples" habit. The policy is written per channel and defines a maximum gap in time, not in sample count, because the same number of samples spans very different durations at different rates. Within that bounded window, and only where the channel physics is smooth and no process-state transition is crossed, the service may interpolate, and it marks the result with its method and an uncertainty flag. Outside that window, where the gap is long, the signal is fast-moving, or the record fails validity, the service does not guess: it preserves the gap, suppresses or down-rates the affected value, and leaves it for redundant validation or human confirmation.

What comes out is not a replacement for the raw record. It is a governed curated layer published beside the immutable raw. Each curated value carries its source identity, timestamps, original unit, quality status, transformation method, a confidence or uncertainty measure where applicable (as defined by the channel contract), and the version of the rule that produced it. That one shared product is what feeds data analytics, real-time engineering calculations, machine-learning and deep-learning models, and automation, while the original evidence remains available for audit, correction, and reprocessing.

Every record passes through identification, alignment, validation, policy-controlled repair, labeling, and operational monitoring before publication.

Whether a gap may be filled depends on signal dynamics and the consequence of a wrong value. High-consequence, fast-changing channels should not be silently interpolated.

Built like infrastructure, not a script

Centralizing the correctness work concentrates its value and its risk in one place, so the service has to be engineered like the shared dependency it is. A single faulty rule could otherwise degrade every consumer at once. That means high availability and buffering so the rig is never left without a quality feed, replay from the raw layer, canary rollout and rollback for rule changes, monitoring and audit of what the rules did, and clear ownership of each rule version. The service is maintained, not written once and forgotten: sensors drift, rigs get re-instrumented, and new consumers arrive with new expectations, and the governance around the rules is what keeps the shared product dependable as all of that changes.

🎯 One record, many needs: preserve fidelity, then label

There is an obvious objection, and the two examples from the start of this article are what raise it. If rig state needs only a thresholded reading and hydraulics needs the measured value, how can one shared product serve both without collapsing to the lowest common denominator? A record that was reduced to "above or below threshold" would satisfy rig state and ruin hydraulics.

The answer is the heart of this architecture. The service preserves the available fidelity of the measurement and retains the raw value underneath it. It never reduces a measurement to a verdict. It publishes the value at its available resolution, and alongside it the labels that tell a consumer how to use it: quality status, method, uncertainty where a value was derived, and provenance down to which sensor produced it and whether it was measured or interpolated.

Interpretation then stays with the consumer, where it belongs. The rig-state classifier reads the validated value and applies its own threshold and context rules. The hydraulics model reads the same value and uses the number directly in its physics. Same record, same shared product, two different interpretations. Neither consumer had to run its own cleansing, and neither was shortchanged.

That draws a clean line between two kinds of work that fragmented cleansing had tangled together. The universal correctness work belongs in the governed service: unit normalization, time alignment, validity against defined rules, bounded and policy-controlled gap handling, quality labeling, and provenance. This is work about whether the data is fit for shared reuse. The consumer-specific work stays downstream: model-specific smoothing, resampling to a model's cadence, feature engineering. This is a modeling choice about how a particular use case treats fit data, not a correctness fact about the measurement. Do the correctness work once; condition locally for use.

🏗️ The clean base feeds everything

Once the shared curated product exists, its value compounds, because every process now reads from the same governed base. Analytics, real-time engineering calculations, model training and inference, and automation all start from the same validated records, so they agree on what is valid before they ever disagree about what it means. The arguments that remain are about engineering and interpretation, which are worth having, rather than about whose cleaning was right, which never was.

For machine learning the shared base helps, but it is not the whole answer. Training and inference should use the same versioned definitions and transformations wherever practical. A shared curated layer removes one source of inconsistency between what a model learned and what it is served, but it does not by itself eliminate training-serving skew: the feature logic has to be versioned and reused as well, through a feature pipeline or feature store, and production data has to be monitored for drift, missingness, and changes in quality status. What the governed layer adds on top is the audit trail that automation needs before anyone will let it act: not just the value that drove a decision, but its quality, its uncertainty, and the rule version and provenance behind it. That traceability is the thread this series has followed from decision-grade data through to the control-authority boundary.

A shared quality service becomes operational infrastructure. Its rules, failures, and outputs must be monitored at the rig, not treated as a script that runs unattended. Illustrative.

🗺️ Where should it run: edge, central, or governed hybrid?

If the quality work should be one governed service, the next question is where that service should run. The honest options are at the edge, next to the sensors on the rig, centrally, in a data center or the cloud after ingest, or a governed hybrid that runs common, versioned rules in both places. Each solves a different part of the problem.

Running at the edge means the data is conditioned and validated at the source, before it travels. The advantages are latency, bandwidth, and resilience. On latency, real-time consumers on the rig get a ready curated stream with no round trip. On bandwidth, the edge can publish curated and contextualized data by default, retaining or transferring selected raw windows only when replay, investigation, model development, or policy requires them, rather than shipping a continuous flood of raw samples. On resilience, the rig keeps a usable feed even when the link to shore is slow or down. The costs are limited edge compute and the genuinely hard part, keeping one rule set consistent and current across many rigs and sensor configurations.

Running centrally means ingesting the raw first and conditioning it in the data center. The advantages are abundant compute, one codebase that is easier to govern and keep consistent fleet-wide, and the room for historical replay and late-data reconciliation. The costs are the mirror image, and they run in both directions: latency, because the curated product is only ready after the raw data has traveled to the center and the finished product has traveled back to the rig; bandwidth, because the raw stream has to be shipped up and the curated stream shipped back down for the edge to act on it; and a dependence on connectivity, because if the link is down the rig has nothing curated to read or run on.

Edge processing supports low latency and link independence; central processing supports fleet consistency and replay. A governed hybrid requires one governed rule lineage, declared versions, visible lineage, and explicit authority.

The practical answer is a governed hybrid, and the word that makes it work is authority. Edge and central services operate under one governed rule lineage, with two clearly assigned roles. The edge publishes the low-latency curated stream that is authoritative for real-time rig decisions and keeps running through a link outage. The central environment receives curated records along with the raw windows it needs, reconciles late-arriving data, supports historical replay, and governs fleet-wide rule releases, and its reconciled product is authoritative for retrospective and fleet work. Release timing may temporarily place the two environments on different declared versions of the same lineage, but every output carries the applied rule version and authority context, so differences from late data, updated calibration, or a revised rule can be reconciled rather than hidden. That is the same two-loop instinct this series described in the rig data fabric, applied now to the single question of data quality.

🩹 Data fixing is not data cleansing

Cleansing asks a binary question: is this record trustworthy enough to keep, or should it be flagged and set aside? That question matters, but stopping there wastes recoverable information. A short dropout in a channel is not automatically garbage. When the gap is brief and the surrounding operation is a known, regular state, the missing points can often be cured rather than discarded.

The distinction is deliberate. Cleansing removes or quarantines bad data. Fixing restores good data that was briefly absent. A sensor that drops a handful of samples during steady rotary drilling has not produced garbage; it has produced a short, bounded hole in an otherwise well-behaved signal, and that hole can be reconstructed by interpolation with far more confidence than the raw dropout alone would suggest.

Two conditions have to hold together before a gap is a candidate for fixing. First, the gap must be short in time, small enough that the underlying physical quantity cannot have changed in a way the surrounding samples fail to capture. Second, the operation during the gap must be identifiable, from the same sensors that recorded the surrounding data, as a regular, steady state and not a transition. A gap that straddles a connection, a reaming pass, a trip, or any process-state change is not a fixing candidate, because interpolation across a real event invents an operation that did not happen. When either condition fails, the record stays flagged, not fixed.

Every fix carries its own provenance. An interpolated value is never written back as if a sensor had measured it. It is labeled with the method used and an uncertainty flag, and the raw record, gap and all, remains preserved underneath. A consumer can always see which points were measured and which were reconstructed, and can use or ignore the reconstruction based on its own tolerance. This is the same discipline the governed service already applies to validation and repair, extended from flagging bad data to recovering good data.

The payoff is that quality control stops being purely subtractive. Instead of only removing what is unreliable, the pipeline can also recover what is recoverable, under explicit rules, and hand downstream engineering, machine learning, and automation a cleaner and more complete record than a flag-and-discard approach would ever leave behind.

✅ Clean once, govern everywhere

The shift this article argues for is small to state and large to build. Stop treating data cleansing as each process's private chore, and start treating it as shared, governed infrastructure: one maintained service that preserves the raw, does the correctness work once, and publishes one curated product that every consumer reads and conditions for itself.

The payoffs follow directly. Consistency, because every process reads the same validated base. Less duplicated compute, because the correctness work is done once rather than once per consumer. Less avoidable preprocessing latency inside each path. And the provenance, quality, and uncertainty labels that machine learning and automation need before anyone will let them act. The size of the efficiency gain depends on how many consumers a rig runs and how much each one currently repeats, so it is a WillCo Drilling Consulting analysis of the design, not a measured benchmark.

Across this series the picture has filled in one layer at a time: standards moved rig data into a shared shape, high-frequency analytics moved decisions to the edge, a two-loop fabric carried refined information between rig and office, the operations center was rethought around intelligent monitoring, and the boundary between the two worlds was secured. This article is about the quality of the water in all of those pipes. Do the correctness work once, govern it everywhere, and keep the original evidence so the shared product can always be checked and rebuilt.

A trusted shared data layer is built through inventory, contracts, versioned rules, proof, resilient deployment, and continuous assurance.

A trustworthy record is observed, validated, curated under policy, and then reused. The raw record remains preserved and every repair remains visible.

📚 References and further reading

WillCo Drilling Consulting series

From Rig Data to Drilling Autonomy: What the Drilling Data Standards WITS, WITSML, ETP and OSDU Actually Do

Beyond WITSML: Turning High-Frequency Rig Data into Edge Decisions

One Rig Data Fabric: Connecting Edge, WITSML and Federated Learning

Reevaluating the RTOC: From Watching Screens to Intelligent Anomaly Detection

Security at the OT/IT Boundary: Protecting the Rig's Network as Models and Data Cross It

External technical sources

SLB Oilfield Glossary, "Equivalent Circulating Density" - ECD as mud density plus annular pressure-loss contribution at true vertical depth. glossary.slb.com

Shahri, M., Kutlu, B., Thetford, T., et al. "Adopting Physical Models in Real-Time Drilling Application: Wellbore Hydraulics." SPE-191797-MS, SPE Liquids-Rich Basins Conference, 2018. Flow rate, RPM, mud properties, BHA and wellbore geometry as real-time model inputs; pump pressure as a calculated and monitored output. doi.org/10.2118/191797-MS

NIST Special Publication 1500-18r2, Research Data Framework - treats data quality as a lifecycle concern, including accuracy, completeness, relevance, consistency, and reliability, and addresses metadata and provenance as related lifecycle disciplines. Supports the "fit for governed reuse" framing over "single truth." doi.org/10.6028/NIST.SP.1500-18r2

Energistics, WITSML 2.0 Data Point Metadata - point-level metadata can describe quality, confidence, and edits; names and datatypes were not standardized in WITSML 2.0. docs.energistics.org

AWS Well-Architected Machine Learning Lens, MLREL03-BP02 - shared, versioned feature definitions reduce training-serving skew; monitoring remains necessary. docs.aws.amazon.com

Anaconda, "2020 State of Data Science" - task-allocation chart (n=1,099 of 2,360 respondents): cleansing 26 percent plus loading 19 percent equals 45 percent of reported work time. Cross-industry, not a drilling-specific measure. anaconda.com

🔤 Glossary of acronyms

BHA - Bottomhole assembly: the lower portion of the drillstring, including the bit and near-bit tools.

ECD - Equivalent circulating density: the effective mud density acting on the wellbore while circulating, from mud density plus annular pressure losses at depth.

HA - High availability: designing the service to keep running through component or link failures.

ML / DL - Machine learning / deep learning: data-driven models that learn patterns from historical and streaming data.

OT / IT - Operational technology (the rig's control and sensing systems) and information technology (office and cloud systems).

Provenance - The recorded origin of a value: which sensor produced it, and whether it was measured directly or derived.

RPM - Revolutions per minute: drillstring or top-drive rotation speed, an input to hydraulics and mechanics models.

SPP - Standpipe pressure: surface circulating pressure, useful for validating a hydraulics model and detecting anomalies.

TVD - True vertical depth: vertical depth of a point in the well, used in pressure and density calculations.

WITSML - Wellsite Information Transfer Standard Markup Language: the standard for exchanging contextualized well and drilling data.