TL;DR: MLOps budgets are inverted. CTOs allocate the majority of effort to model engineering. But data plumbing consumes the bulk of production hours across enterprise deployments. The fix is a reallocation framework built around a dedicated digital plumber role, three-stream versioning, and a 90-day audit that reveals the real cost split.

Key Takeaways: - Most ML project delays trace to under-resourced data plumbing, not model code - MLOps must version three connected streams: data, models, and application code - A dedicated digital plumber role sits between data scientist and cloud architect - A 90-day audit exposes the real cost split and accelerates restructuring - Proper plumbing shortens deployment timelines

Every CTO budgets MLOps like a software project. The standard budget treats model development as the dominant line item. Data orchestration, dataset versioning, and drift monitoring are afterthoughts. The assumption is clear: the model is the hard part. Everything else is plumbing.

Post-mortems tell a different story. Data orchestration, dataset versioning, and drift monitoring consume most production hours. The mismatch between budgeted effort and actual effort drives most ML project delays. Teams ship a working notebook, then spend months wiring it to live data.

The pattern repeats across banks, insurers, hospitals, and manufacturers. Deployment timelines expand when teams underestimate the data side. Timelines shrink when plumbing is prioritized from day one.

For context on how machine learning pipelines diverge from standard CI/CD, the gap is structural. Code is deterministic. Data is not.

The model is the visible artifact. The data plumbing is the invisible factory. And factories are expensive. But if headcount alone won't fix it, what is actually missing?

The reflex is to hire. Add two more data scientists. Add a DevOps team. The backlog shrinks for a few weeks, then balloons again. The reason is structural.

DevOps engineers lack the data-specific skills MLOps demands. Feature stores, dataset versioning, embedding pipelines, and retrieval-augmented generation indexing all need specialized knowledge.

A standard CI/CD pipeline tracks code commits. An MLOps pipeline must track data lineage, feature drift, and model decay. Those are different disciplines.

Data scientists aren't trained to build production-grade pipelines either. They prototype in notebooks. They don't orchestrate at scale. The notebook-to-production gap is where most projects stall. Generalist hiring does not close it. Specialized roles do.

The plumbing must stop breaking before retention improves. Teams that fight pipeline fires every week burn out. Teams that ship reliable systems stay. The difference is the data layer, not the application code.

For background on how ml training workloads differ from traditional software, the distinction matters. Training is batch. Serving is real-time. Both must share the same data layer.

Unlike standard CI/CD, MLOps tracks three connected streams. Not one. Three. Any drift in one stream invalidates the others. - Data stream: raw inputs, features, transformations, embeddings, labels - Model stream: trained artifacts, hyperparameters, evaluation metrics, GPU plans - Code stream: serving logic, API routes, infrastructure definitions

A change in any one stream can invalidate the others. Retrain a model on new data, and your serving code must handle the new feature set. Update a feature store schema, and your training pipeline breaks. Refactor serving code, and your latency SLAs shift.

Data versioning is a multi-dimensional problem. Schema changes, label drift, feature store mutations, and embedding index updates all need to be tracked and replayed. Most teams version their Git repo and call it done. That covers one stream out of three.

Model decay is the silent killer. Production data diverges from training distributions. Your fraud model trained on older patterns sees newer fraud schemes. Without continuous AI drift detection infrastructure, you find out from a customer complaint, not an alert.

For details on how model training pipelines must be treated as first-class artifacts, the discipline is closer to data engineering than software engineering. The 70% spans four layers.

Four layers. Each one eats hours that the budget never accounted for.

Source data orchestration: ingestion from disparate systems, schema reconciliation, freshness SLAs, and backfill logic for historical replay. A typical enterprise has dozens of source systems. Each has its own schema, its own latency, its own failure modes. Wiring them into a consistent feature pipeline takes weeks or months.

Feature pipelines: transformation logic, feature store population, online/offline consistency, and embedding regeneration. Vector databases that back these pipelines carry their own compliance cost curves that compound the data plumbing overhead.

Online and offline feature parity is a notorious source of silent bugs. The model trains on offline features computed in batch. It serves against online features computed at request time. If the logic drifts, the model degrades without any code change.

Model artifact management: versioning large neural network weights, GPU memory planning, caching strategies for inference, and model retirement workflows. Storage requirements grow with model size, versioning history, and rollback capacity. Storage becomes a planning exercise, not a line item.

Monitoring layer: data drift detection, concept drift alerts, prediction quality tracking, and feedback loops that trigger retraining pipelines. Most teams underinvest here. They deploy the model and assume metrics will stay green. They won't.

The data layer is the durable asset. The model is replaceable. The plumbing is what lasts. The team structure must look very different from a standard engineering org chart. The missing role is the digital plumber.

The term comes from enterprise architecture circles. The function is specific to MLOps. The digital plumber moves, combines, and serves data across systems: APIs, ETL platforms, CLI orchestration, and network security for data flows.

This role is distinct from the data scientist, who builds models. It is also distinct from the cloud architect, who provisions infrastructure. The plumber wires them together. Without this role, the data scientist writes a notebook, the architect spins up a cluster, and nothing talks to anything.

Required skills: - API development for model serving - Data pipeline tooling and data versioning systems - Mastery of ETL or shell scripting for data movement - GPU resource awareness for gpu inference workloads

A strong shell scripter is often more valuable than another framework specialist. CLI fluency matters because pipelines break in production. The fix is usually a one-line command at 2 AM.

With the role defined, the next step is a 90-day rebalancing plan that does not blow up the org chart.

You cannot rebalance what you have not measured. The first step is honesty about where the hours actually go.

Weeks 1-2: Time-track your current ML team. Categorize every hour into model development, data plumbing, and serving infrastructure. Most teams are shocked by the split. They assumed most hours go to model work. The actual number is much lower.

Weeks 3-6: Map your three streams. Identify which stream has the weakest versioning, monitoring, and reproducibility. For most teams, it is the data stream. Feature definitions live in notebooks. There is no single source of truth for what "customer tenure" means across training and serving.

Weeks 7-10: Pilot a feature store or [embedding](/ai-ml-training) pipeline. Pick your highest-volume use case. Measure latency, consistency, and retraining time before and after. The retraining window often drops from days to hours. Inference latency stabilizes because online and offline features finally agree.

Weeks 11-13: Restructure team allocation. Move toward a 70/30 split. Establish data plumbing SLAs alongside model performance metrics. The right plumbing does not just cut costs. It accelerates everything.

Without a clear scope, the same audit stretches to six months. With focused effort and clear measurement, it is 90 days. The difference is scope discipline. The 'everything' is a rebalancing of engineering time that unlocks faster deployment.

Teams that rebalance to 70% data plumbing compress deployment timelines. The rework elimination is where the savings come from.

Most ML project delays trace back to data issues found late. A model passes evaluation. Then it fails in production because the feature pipeline drifted. Retraining fixes it for a week. Then the drift returns. The team spends months patching symptoms. Proper plumbing catches drift at the source. The model becomes a replaceable component, not a fragile artifact.

The inference layer gets the attention. The data underneath is what survives. Replace the model every quarter. Keep the plumbing. The system runs.

Retention improves for a different reason. Teams stop firefighting data drift. They start iterating on model improvements. Engineers stay because the work is intellectually engaging, not because they are on call for pipeline outages at 2 AM. The 30/70 split is not a ratio. It is a survival strategy.

What is data plumbing in MLOps?

Data plumbing in MLOps refers to the infrastructure and pipelines that move, transform, version, and serve data across the ML lifecycle. It includes data ingestion, feature stores, embedding pipelines, dataset versioning, and drift monitoring. It is the connective tissue between raw data and production model serving.

Why is MLOps considered 70% data engineering?

Across enterprise deployments, most production hours go to data work. This includes orchestrating data flows, managing model artifacts, monitoring for drift, and maintaining feature consistency. Not to writing model code. The model itself is often a small fraction of total system effort.

What is a digital plumber in MLOps?

A digital plumber wires data between disparate sources. They build APIs for model serving and maintain ETL pipelines that feed model training and inference. This role bridges data scientists and cloud architects.

How long does MLOps deployment typically take?

Deployment timelines depend on the strength of the data plumbing layer. Teams with weak data plumbing often face rework across model, pipeline, and infrastructure layers. The rework cycle stretches timelines well beyond initial estimates.

What are the three streams MLOps must version?

MLOps requires versioning three connected streams: data, models, and application code. A change in any one stream can invalidate the others. Reproducibility in MLOps is harder than in standard software delivery.

Start a 90-day time-tracking audit on your own team to see your real 30/70 split.