Which Model Made That Call, and Is It Still Right?

Figure 1. A drilling engineer reviews a model's recommendation alongside the validation evidence behind it, keeping the decision with the person while the model accounts for itself.

The model has been right for six months

Consider this hypothetical scenario. A model on the rig recommends a parameter. It has been right for six months, across dozens of wells, so no one questions it anymore. That is exactly the problem that model governance for drilling AI exists to catch. On this well, in a formation the model has not adequately seen or validated, the recommendation is quietly wrong. It still arrives with the same confident number and the same clean interface. Nothing on the screen says the ground has changed underneath it.

Figure 2. A model can appear reliable even as operating conditions change, so governance makes the engineer ask which version made the recommendation and whether its validation still applies.

Ask the obvious questions and the room goes quiet. Which version of the model produced this recommendation? What was it trained on, and does this well look anything like that? When was it last validated, and against which wells? How would anyone know if it had drifted? Nobody can answer, because the model is treated as a fixed tool that was installed once and trusted ever since.

The previous article in this series gave a rig number a chain of custody: where it came from, what happened to it, every transformation back to the raw record. This piece asks the same of the thing that acts on that number. A recommendation is only as defensible as the model behind it, and a model earns trust the same way a value does: by being able to explain itself, and by being checked against the possibility that it has gone stale.

Two questions carry the whole article. 🔎 The first is backward-looking: which model made that call, and can you reconstruct why? 🧭 The second is forward-looking: is it still right, here, now, on this well? A model governance practice that cannot answer both is running on faith.

A model is a transformation, not a fixture

The instinct is to treat a deployed model as a finished object, like a wrench: built once, correct by construction, used until it breaks visibly. That instinct is wrong, and it is the source of most of the trouble.

A model in production is a data transformation. It takes rig inputs and produces new values: a classification, a probability, a recommended set-point. By the logic of the prior article, anything that transforms a value sits in that value's lineage and has to be identifiable, versioned, and reconstructable. A model is simply the most consequential transformation in the chain, because its output can authorize an action.

Seeing the model as a transformation has two consequences, and they map to the two questions from the opening. Because it is a transformation, it has provenance: an origin and a history you must be able to reconstruct. And because it was fit to a particular slice of the world, it has an expiration condition: it is valid only while the world still resembles the data it learned from. Reproducibility and drift are not two separate programs. They are the past tense and the future tense of the same question: can this model be trusted to act.

Model provenance: reconstructing why it made that call

Start with the backward-looking question. A recommendation lands, someone acts on it, and weeks later it has to be explained. To explain it you need the model's chain of custody, the same way the last article needed the number's. For a model that record has to contain, at minimum, the following.

The training data, with its own provenance. Which dataset, which wells, which time range, and in what state. This is where model governance inherits everything from the previous article: a model trained on data you cannot trace is untraceable by construction. It also has to keep raw, repaired, human-labeled, and machine-labeled records distinguishable, or the model may quietly learn from repairs and machine labels as if they were ground truth.

The industry has already made this case for drilling data specifically. The Operators Group for Data Quality, formed in 2014 by more than twenty operating companies and described by ConocoPhillips and Shell engineers in Drilling Contractor, argues that a derived value such as rate of penetration cannot be fairly compared across wells unless the method used to compute it is understood, and proposes contract language requiring that the computation be disclosed. A model that cannot tell a measured channel from a derived one is already on soft ground.

The features and their definitions. Exactly which inputs the model consumes, and how each is computed. A feature such as a smoothed rate of penetration or an equivalent circulating density carries its own lineage; if that definition changes, the model is being fed something it was not trained on, even when the label stays the same.

The algorithm, hyperparameters, and version. The governed version identifier that produced this specific result, with the algorithm and the settings behind it. Without a version, "the model said" names nothing you can go back to.

The validation record. How the model was tested before it was trusted: against which offset wells, under which conditions, and with what measured performance. This record is what the model's authority rests on, and the next sections are about why it has a shelf life.

The approval and the authority granted. Who signed off, and what the model was permitted to do: advise a human, or act on its own within limits. Authority is a governance decision, not a property of the code.

The decisions it influenced. A link from the model forward to the recommendations it made and the actions taken on them, so an outcome can be traced back to the exact version responsible.

Figure 3. A model's chain of custody runs from its training data through features, version, and validation to the decisions it influenced, and must be walkable backward from any one of them.

Treat that record the way the last article treated corrections: append-only. You never silently swap a deployed model for a new one. You deploy a new version, record what changed and why, retain the old version and its validation record, and keep the ability to say which version was live when a given decision was made. A model quietly replaced overnight is the automation equivalent of an anonymous edit to a depth value: it may even be an improvement, but it cannot be defended, because no one can reconstruct what was actually running.

None of this is unique to drilling, and none of it is new. The idea that a released model should travel with a short, documented record of its intended use, its evaluation conditions, and its performance characteristics has a name: the model card, proposed by Mitchell and colleagues in 2019 precisely so that models are not used in settings they were never meant for. Its companion on the data side, the datasheet for datasets from Gebru and colleagues, asks the same of a training set: where it came from, how it was collected, what it is and is not suitable for.

The mechanics are not exotic either. A model registry such as MLflow's already versions models and links each version to the run that produced it, carrying the aliases, tags, descriptions, and lineage a governed record needs. The pieces exist. The governance question is whether a drilling stack requires and maintains them consistently.

The envelope: a model is valid only where it was validated

Now the forward-looking question. The validation record from the last section is not a certificate of general correctness. It is a statement about a specific envelope: the model performed this well, against these wells, in these formations, within this operating range, with the sensors reading the way they read during training.

Step outside that envelope and the performance number no longer applies. A new basin, a different formation sequence, a recalibrated sensor, a changed mud program, a rig with a different drawworks response: any of these can put the current well outside the conditions the model was validated in.

The model will still produce an answer. A well-built model can carry a confidence estimate or an out-of-distribution flag, and those help, but a flag is not governance: it warns, it does not decide, and a set-point that lands just inside a numeric range can still be wrong for a formation the model never trained on. Left unwatched, the model simply extrapolates, confidently, into ground it has never seen.

This is not a theoretical worry in drilling. The Journal of Petroleum Technology, describing work on rate-of-penetration prediction, notes that earlier ROP models were built around a particular field or tool, which hurt their accuracy outside the field they were trained on, enough to motivate the effort to build a single model that would generalize.

Peer-reviewed work in Processes makes the same point more bluntly: geological variability across wells hinders the transferability of conventional intelligent models, which is why transfer-learning methods exist to adapt a model to a new well with a small amount of local data. Evidence from the geology a model was tested on does not establish how it will perform in a new geology without additional validation.

The upside, when the match holds, is real. A cloud-based machine-learning system for ROP, reported to the SPE/IADC International Drilling Conference and covered by the Oil & Gas Journal, posted meaningful gains over control footage in Permian field trials. The number worth noticing is the condition attached to it.

This is why "the model that was right last well" is a trap rather than a reassurance. Performance measured inside the envelope does not carry forward to conditions outside it without new evidence. The honest version of a model's status is not "validated," full stop. It is "validated for these conditions, and here is whether the current well is inside them."

Figure 4. A model's validation evidence applies only inside the envelope of conditions it was tested against, and this well sits outside it, where that evidence no longer carries without new validation.

Drift: the silent expiration

The envelope explains a model that is wrong on arrival for a new well. Drift explains a model that was right and slowly stops being right while nothing obvious changes. The world the model learned from keeps moving: formations evolve across a pad, sensors age and get recalibrated, programs and crews change, the mix of wells shifts. The model stays frozen at the moment it was trained. The gap between the two widens quietly.

The field has names for the pieces. Data drift is when the production inputs start looking different from the training distribution. Concept drift, in the definition set out by Gama and colleagues in their 2014 survey of the problem, is when the relationship between the inputs and the target the model predicts changes over time. Neither observation by itself proves the model has degraded. Each is a signal that the model's assumptions may no longer hold, and a prompt to check performance against outcomes wherever labels can be captured.

That check matters because the interface never changes: a model that has decayed looks exactly like one that has not, so the decline is invisible until someone goes looking for it. Keep all of this separate from sensor drift, the physical wander of an instrument that the Operators Group warns about. Sensor drift corrupts the input; concept drift means the world the model learned has moved. A governed program watches for each, and they are not the same failure.

Governance is the answer to decay, and it is continuous, not a one-time gate. A governed model is monitored in production on three things at least: whether its inputs still resemble the training distribution, whether its outputs are shifting, and, wherever outcomes can be captured, how its recommendations compare to what actually happened. Each of those has a threshold, and crossing a threshold triggers something specific: a flag, a review, a revalidation, or a retrain. Set-and-forget is not a lighter form of governance. It is the absence of governance.

This is the part the wider standards now take as given. The NIST AI Risk Management Framework organizes the whole lifecycle around four continuous functions, Govern, Map, Measure, and Manage, rather than a one-time approval.

ISO/IEC 42001, published in 2023 as the first international AI management system standard, is built on the same premise of establishing, maintaining, and continually improving the system. A drilling operator does not have to adopt either to borrow the posture behind both: a model in production is something you keep governing, not something you finished.

Retraining then becomes its own governed event, and it closes the loop back to provenance. A retrain is a new model version, with new training data that has its own provenance, a new validation record, a new approval, and an effective time. The old version does not vanish; it is retained and marked superseded, so the decisions it made while it was live can still be explained. Drift management and reproducibility are the same discipline seen from two ends.

Figure 5. Without monitoring, a model's decay is invisible until an incident; with monitoring, a crossed threshold triggers review or retraining before the wrong recommendation is acted on.

What a governed model record contains

Translate all of that into something a team can check against, the same way the last article turned provenance into a defensible-record checklist. A governed model does not need every artifact bolted to every recommendation. It needs compact identifiers that resolve to governed training-data, version, validation, and monitoring records. These are the questions each deployed model must be able to answer, directly or by reference.

As with the data record, depth should scale with consequence. A model that only colors an advisory indicator needs less than one permitted to move an automated set-point. The more authority a model holds, the deeper the training-data provenance, the validation record, and the live monitoring behind it have to be.

Figure 6. Governance depth should increase as both model authority and operational consequence increase.

Five questions for any model in your drilling stack

✅ The checklist is for the model record. These five questions are for the platform, the vendor, or the team that operates the model. They work as acceptance-test or procurement questions, and the honest answer to each is either a demonstration or a no.

1. Which version of the model produced this recommendation, and can you reproduce the result from the recorded inputs?

2. What was it trained on, and does the current well fall inside that envelope of conditions?

3. When was it last validated, against which offset wells, and with what measured performance?

4. How would you know it has drifted: what is monitored in production, and what triggers a review or retrain?

5. Who approved its level of authority, and what evidence does that authority rest on?

A platform that can demonstrate all five is a candidate for governed decision support, subject to validation, controls, and operating context. A platform that cannot is one where every automated recommendation carries an unquantified operational and governance risk, and every incident review starts with a question no one can answer.

Figure 7. Production state determines the governance action, from continued monitoring to pausing authority, revalidation, quarantine, or retirement.

From a traceable model to trustworthy automation

The last article made a value defensible. This one makes the model that acts on the value defensible too. Those are the two halves of the same requirement: before a machine is allowed to act, both the number it reads and the model that interprets it have to be able to explain themselves, and both have to be checked against the possibility that they have gone stale.

That is also the authority principle restated for models. The depth of evidence behind a model should match the authority it is granted. An advisory model whose output a human weighs can stand on a lighter record than one permitted to move a set-point on its own. Hand real authority to a model you cannot reproduce or are no longer watching, and you have not automated a decision. You have removed the human who used to catch it being wrong.

This is the last prerequisite before the series reaches its final step: taking traceable data and a governed model and letting them close the loop, acting on the rig under human authority. The rule underneath the whole series still holds: you cannot automate a decision you cannot trace. This article adds its twin.

References and further reading

1. WillCo Drilling, "Where Did That Number Come From? Provenance and Lineage for Rig Data." willcodrilling.com

2. WillCo Drilling, "Clean Once, Trust Everywhere: Rig Data Cleansing as a Single Source of Truth." willcodrilling.com

3. WillCo Drilling, "Reevaluating the RTOC: From Watching Screens to Intelligent Anomaly Detection." willcodrilling.com

4. WillCo Drilling, "Security at the OT/IT Boundary." willcodrilling.com

5. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., and Gebru, T. "Model Cards for Model Reporting." Proceedings of FAT* '19, pp. 220-229, 2019. arxiv.org/abs/1810.03993

6. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., and Crawford, K. "Datasheets for Datasets." Communications of the ACM, 64(12), pp. 86-92, 2021. arxiv.org/abs/1803.09010

7. Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., and Bouchachia, A. "A Survey on Concept Drift Adaptation." ACM Computing Surveys, 46(4), Article 44, 2014. doi.org/10.1145/2523813

8. National Institute of Standards and Technology. "Artificial Intelligence Risk Management Framework (AI RMF 1.0)," NIST AI 100-1, January 2023. doi.org/10.6028/NIST.AI.100-1

9. ISO/IEC 42001:2023, "Information technology - Artificial intelligence - Management system." iso.org/standard/81230.html

10. Nguyen, D. (ConocoPhillips) and Wiemers, T. (Shell Global Solutions). "Operators group proposes data contract addendum to effectively unlock and utilize drilling data." Drilling Contractor, January 30, 2017. drillingcontractor.org

11. SPE/IADC-204043-MS, "From Science to Practice: Improving ROP by Utilizing a Cloud-Based Machine-Learning Solution in Real-Time Drilling Operations." SPE/IADC International Drilling Conference, 2021. onepetro.org

12. Procyk, A. "Machine-learning AI enhances Permian basin drilling" (coverage of SPE/IADC-204043-MS). Oil & Gas Journal, July 5, 2021. ogj.com

13. "Machine-Learning Model Developed for Rate-of-Penetration Optimization." Journal of Petroleum Technology (SPE), February 1, 2023. jpt.spe.org

14. Huang et al. "Intelligent Prediction of Rate of Penetration Using Mechanism-Data Fusion and Transfer Learning." Processes, Vol. 12, No. 10, Article 2133, 2024. doi.org/10.3390/pr12102133

15. MLflow. "Model Registry" (official documentation), Linux Foundation. mlflow.org