The Future of Data Engineering: How AI and ML Are Transforming Workflows

Written by Gowtham Raj, Director at TartLabs, who leads custom software and AI engagements for enterprise, logistics, and education clients.

The Short Answer

AI has genuinely changed data engineering, though rarely in the direction the headlines describe. Pipelines have not been automated away. What got automated is the first draft: the SQL, the transformation model, the schema mapping. The consequence is a bottleneck that slid downstream, landing squarely on review.

Three pieces of evidence point the same way, and none of them is ambiguous. BIRD, the largest realistic text-to-SQL benchmark, has its leading system at 82.39% execution accuracy where humans reach 92.96%. The headline AI features in the transformation tool more teams use than any other went out as alpha, preview and beta rather than generally available. And DORA research run by Google itself records AI adoption lifting throughput while it degrades delivery stability.

Put those together and a single movement emerges: producing things became cheap, checking them did not, and the work drifted toward the checking.

The title pairs two technologies that deserve to be pulled apart, too. Large language models absorb the attention. Meanwhile the classical machine learning that drives anomaly detection inside data observability is the older and considerably steadier half of this story.

Our readers here are CTOs, founders and engineering leads who have to decide what to adopt, what to staff, and what to believe. Each figure that follows came from its primary source on 15 September 2026, and carries that date. A number of much-quoted data engineering statistics collapsed under a source check; rather than repeat them, we name them and leave them out.

Key Takeaways

• The accuracy gap is real and measured. Best-in-class text-to-SQL hits 82.39% on BIRD; humans hit 92.96%. That ~10-point gap is the whole argument for keeping a reviewer.
• The benchmarks themselves are shakier than the leaderboards suggest. A January 2026 paper found annotation error rates of 52.8% in BIRD Mini-Dev and 62.8% in Spider 2.0-Snow — and rankings on corrected data correlate only weakly with published ones.
• Nothing flagship is GA. dbt Core v2.0 (the Fusion engine) shipped alpha, dbt State preview, dbt Wizard beta — at the merger that created a 100,000-team vendor.
• AI helps throughput and hurts stability. DORA's ~5,000-respondent study: 90% use AI, >80% report productivity gains, 30% report little or no trust in its code — and stability gets worse, not better.
• The "19% slower" study has been complicated by its own authors. METR's February 2026 follow-up found a speedup, then said its own data was "only very weak evidence." Anyone quoting either number as settled is overreaching.
• The most dependable ML in the stack is not an LLM. Unsupervised anomaly detection in data observability learns each table's baseline instead of relying on hand-written thresholds — a longer-established category than LLM authoring, and a far cheaper one to get wrong.
• The labour market is splitting, not shrinking. BLS 2025–35: database administrators 0% growth, database architects 9%, data scientists 35%.
• Two famous numbers do not survive scrutiny: the "80% of time cleaning data" claim and Gartner's "$12.9M a year" data-quality cost. Both are explained below rather than quoted.

Every figure above is sourced inline in the sections that follow, with the date each page was read.

What "AI Is Transforming Data Engineering" Actually Means

Set the marketing aside and you find AI has arrived at four separate points in the data stack. Their maturity varies sharply, and treating them as one thing is precisely how teams talk themselves into disappointment.

Compare where the attention goes with where the money is. Demos come from the bottom row. Returns, for most teams right now, come from the two rows in the middle, optimisation and observability, which happen to be the two nobody bothers writing about.

Text-to-SQL: The Capability Everyone Assumes Is Solved

Text-to-SQL remains unsolved. Against a competent human, the best-measured system misses something like one query in every ten that the human handles correctly, and it hands back those misses as confident, neatly formatted result sets.

Natural-language-to-SQL sells the whole category. It also happens to be the capability with the most rigorous public measurement behind it, which makes it our best available proxy for how far any of this has genuinely travelled.

BIRD is the most realistic large benchmark: 12,751+ question-SQL pairs across 95 databases totalling 33.4 GB in 37+ professional domains, deliberately including dirty values and questions that need outside knowledge. Its leaderboard, read on 15 September 2026 and last updated 7 September 2026, puts the best system — DataGallery-Text2SQL, from Huawei's 2012 Labs — at 82.39% against a human baseline of 92.96%.

A gap of 10.57 percentage points reads as modest until you translate it into practice. Around one query in ten that a capable human would answer correctly comes back wrong from the best system on offer: wrong, and dressed in a tidy, confident result set. Faulty SQL raises no exception. It hands over a number, and that number ends up in somebody's board deck.

It gets harder when the database looks like yours

If BIRD is realistic, Spider 2.0, read the same day, is closer to punishing: 632 genuine enterprise text-to-SQL workflow problems, databases carrying over 1,000 columns and sometimes past 3,000, with answers that routinely run beyond 100 lines of SQL.

The difficulty jump is best shown by one model measured on both. GPT-4o scores 86.6% on the original Spider 1.0 and 10.1% on Spider 2.0. o1-preview manages 17.1%. Same model, same task description. The only thing that changed is that the schema started to look like a real company's warehouse. Both are earlier-generation models and current ones score higher; the point is the size of the drop when schema complexity rises, which has not gone away.

For anyone scoping a project, that collapse is the most useful figure anywhere in this article. Tidy schemas are what demos run against. Your warehouse is not tidy.

The benchmarks themselves are contested

Here is the part the leaderboards do not advertise. In January 2026, Tengjun Jin, Yoojin Choi, Yuxuan Zhu and Daniel Kang published Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards, which audited the reference answers themselves.

Their audit put annotation error rates at 52.8% for BIRD Mini-Dev and 62.8% for Spider 2.0-Snow. Re-running agents against corrected data moved performance by -7% to +31% in relative terms, with individual systems sliding -9 to +9 places in the rankings. Most damaging of all, the published rankings track the full uncorrected dev set closely (Spearman's r_s = 0.85, p = 3.26e-5), yet track the corrected subset only loosely (r_s = 0.32, p = 0.23, which is not statistically significant).

What dbt and Fivetran Actually Shipped

At the merger that created the category's dominant vendor, every flagship AI feature shipped as alpha, preview or beta. None of it shipped generally available — and that is the most honest maturity signal available anywhere in this market.

On 1 June 2026, Fivetran and dbt Labs completed their merger, announced the previous 13 October 2025. George Fraser continued as CEO, Tristan Handy became President, and the combined company reported serving "more than 100,000 data teams globally." This is the centre of gravity of the modern transformation layer.

Now look at what went out alongside the merger, paying attention to the release status in particular:

All four rows, including the release statuses, are as stated in Fivetran and dbt Labs' own merger announcement, read 15 September 2026.

Alpha, preview, beta. On merger day, at the category leader's flagship launch, with every commercial reason to call things finished.

None of this counts against dbt. Labelling early software honestly is the right behaviour, and putting the Fusion engine under Apache 2.0 is a good outcome for everyone downstream. What those release statuses correct is the ambient sense that AI-driven data engineering has become a solved thing you can simply buy. The organisations best positioned to ship it are calling it beta.

Quietly, Agents Schema is the most interesting entry on that list. Nominating one schema as the shared context layer for AI agents concedes something important: the hard part of agentic data work was never the reasoning. It was handing the agent a trustworthy, bounded account of what the warehouse contains. That is data modelling, the same problem data engineers were already holding.

The Machine Learning That Already Works

Nearly everything above concerns large language models. Yet the machine learning that earns its keep most reliably in a modern data stack is not an LLM at all, and it has been working away for years: unsupervised anomaly detection inside data observability.

Commercially the distinction matters, because these two are purchased, priced and trusted on different terms, and blurring them is how budgets end up in the wrong place.

What matters here is the mechanism. A rules-based check requires somebody to write, and then maintain, a threshold for every table, which is exactly the work that does not get done as a warehouse grows. Unsupervised models instead learn the baseline from the table's own history. As Anomalo's engineering team describes it (April 2024, read 15 September 2026), the appeal is that models "can learn from historical data and adapt to changing patterns," where traditional approaches "typically require manual tuning and adjustment of parameters, which can be time-consuming and may not adapt well to evolving patterns."

A caveat on our own standard: that is a vendor describing its own category, and we said above that we ignore vendor accuracy claims. We are citing it for the mechanism, which is uncontroversial and checkable, not for any performance number. The judgement that this category is more mature than LLM authoring is ours, based on our own engagements and on the simple fact that it has been sold and deployed for years longer. Treat it as reasoning, not as a measurement.

That is a data engineering workflow genuinely transformed, and it arrives with less marketing attached than any other. A chart that failed to break makes for a poor demo.

There is a neat symmetry here worth naming. LLMs raise how fast your pipelines change; classical ML raises how fast you catch the changes that went wrong. Survivability comes from the second, which is the argument for buying the unglamorous one first. (If you are weighing a bought platform against building detection in-house, our comparison of custom AI development versus off-the-shelf AI covers the trade-off.)

What AI Does to Engineering Work, Measured

Because data engineering is software engineering, the wider evidence on AI-assisted development carries over, and that evidence is far more equivocal than either the enthusiasts or the sceptics let on.

The largest recent study is Google's DORA State of AI-assisted Software Development, published 24 September 2025 from nearly 5,000 technology professionals worldwide. Its headline numbers:

• 90% report using AI at work
• more than 80% believe it has increased their productivity
• 30% report little or no trust in the code AI generates

Then comes the finding that bears hardest on pipelines. DORA reports a positive relationship between AI adoption and both delivery throughput and product performance. But in their words, "AI adoption does continue to have a negative relationship with software delivery stability."

Their explanation is precise: "AI accelerates software development, but that acceleration can expose weaknesses downstream. Without robust control systems, like strong automated testing, mature version control practices, and fast feedback loops, an increase in change volume leads to instability."

To a data team, "instability" is not an abstraction. It is a dashboard that is quietly wrong.

The verification tax

DORA's follow-up, ROI of AI-Assisted Software Development (2026.01), published 11 May 2026 and read 15 September 2026, gives the effect a name. They describe a J-curve of value realisation: an initial productivity dip before longer-term gains, caused by three things: the learning curve, downstream process adaptation, and what they call "the verification tax imposed by reviewing AI-generated code."

No phrase in this article is more useful. Producing things became cheap; checking them did not. Rather than vanishing, the work changed shape.

The Stack Overflow Developer Survey 2025 shows the tax being paid, from 33,412 responses on developer sentiment:

46% actively distrust AI accuracy against 33% who trust it, with just 3% saying they "highly trust" it. The top frustration, at 66%, is "AI solutions that are almost right, but not quite" — and 45.2% say debugging AI-generated code takes longer than writing it themselves.

For application code, "almost right, but not quite" is a survivable failure mode: a test goes red and somebody repairs it. For a data pipeline it is the worst possible failure mode, because almost-right output slides quietly onward into a report.

A cautionary note about the study everyone cites

Anyone who has read a single study on AI and developer productivity has probably read METR's randomised controlled trial of 10 July 2025: 16 seasoned open-source developers working 246 real tasks across repositories averaging 22,000+ stars and upward of a million lines of code. Beforehand they predicted a 24% speedup; afterwards they judged themselves 20% faster; the measurement put them 19% slower.

The fame is deserved, since genuine RCTs are scarce in a field saturated with vendor surveys. Quoting it in 2026 without the sequel, though, misleads, and that is exactly where most articles stop.

An update followed on 24 February 2026. Late-2025 data put the original developers at an 18% speedup and new recruits at 4%. The sign had flipped. METR declined to claim the result, noting that because developers increasingly refuse the no-AI condition, "our data is only very weak evidence." A redesign of the experiment is now underway.

The study also stated its own limits plainly from the start: "We do not claim that our developers or repositories represent a majority or plurality of software development work."

The Labour Market Is Splitting, Not Shrinking

One question sits underneath this topic more than any other: will AI eliminate data engineering jobs? No direct answer comes from the US Bureau of Labor Statistics, since no "data engineer" SOC code exists. But the neighbouring classifications say plenty, and they have just moved onto a new 2025–35 projection cycle, last modified 27 August 2026.

That detail matters more than it looks. A fresh ten-year cycle arrives from BLS roughly every two years, which means the 2024–34 numbers still circulating through most articles have been superseded.

The headline for database administrators and architects combined is an unremarkable 4%. The split underneath it is the actual finding:

• Database administrators: 0% growth. 75,000 jobs, median $104,620.
• Database architects: 9% growth. 69,500 jobs, median $139,500.
• Data scientists: 35% growth, from 275,600 to 371,000 jobs, median $120,230, about 24,800 openings a year.

Two jobs that sound adjacent, a nine-point divergence, and a $34,880 median pay gap. The distinction between them is almost exactly the distinction this article has been drawing: administering an existing system versus designing one. The work that can be specified as a routine is flat. The work that consists of deciding what the routine should be is growing.

As predictions go, that beats "AI will or won't take your job" for precision, and it arrives from the statistical agency rather than a vendor. If you are choosing where to invest, whether in your own skills or in the shape of your team, put it at the design end. (We have written separately on how data engineering and data science roles divide, which is the hiring-order version of this same question.)

The Numbers We Are Not Going to Repeat

Writing on this topic is dominated by two statistics. Neither withstands a source check, and the explanation of why is worth more than either number.

"Data scientists spend 80% of their time cleaning data." This traces to 2016-era coverage of a CrowdFlower survey, repeated across hundreds of secondary posts with no consistent primary citation. It is best treated as folklore. The defensible alternative is Anaconda's 2020 State of Data Science survey, with 2,360 respondents, which put data preparation at roughly 45% of practitioner time — and even that is now six years old, which we state rather than hide.

"Poor data quality costs organisations $12.9 million a year." This is Gartner's, and it is real in the sense that Gartner published it — in 2020, in its Magic Quadrant for Data Quality Solutions. Published analyses of that report describe the figure as drawn from 154 reference customers across 16 data-quality vendors, who were asked what they believed poor data quality was costing them. We flag that provenance as second-hand because the Magic Quadrant itself sits behind Gartner's paywall and cannot be independently inspected — which is part of the point. A six-year-old, self-reported estimate, from a sample composed entirely of companies already shopping for data-quality software, is not a population average; it is a rough survey of motivated buyers.

Anaconda's 8th Annual State of Data Science & AI Report offers a more current signal, though with its own caveat — only 214 engineers and data scientists were surveyed, which is small. It found 78% of organisations lack a formal AI deployment strategy, that data quality problems obstruct 45% of organisations trying to scale AI — a different measure from the 45% of practitioner time above, and an unfortunate coincidence of numbers — and that over half have no AI governance framework.

What to Actually Do About It

Read across all of that and the practical implications come out narrower and duller than the subject promises, which is usually a sign that they are real.

Adopt at the optimisation and observability layers first. Incremental build caching and automated anomaly detection have verifiable mechanisms and measurable savings. They are also the layers where being wrong is cheap. Start where a failure costs you a rebuild, not a quarter's reporting.

Treat generated SQL as a pull request, not an answer. A ~10-point accuracy gap against humans, on a benchmark whose own reference answers are contested, is a strong argument for review. The right mental model is a fast junior engineer whose work is good and who has never once said "I'm not sure about this one."

Spend the savings on testing, not on headcount reduction. This follows directly from DORA: throughput improves, stability degrades. If you take the speed gain and cut the review capacity that absorbs it, you have bought yourself instability. Data contracts, freshness and volume tests, and CI for transformations are what convert AI's speed into something safe.

Do not restructure your team around a beta. The most capable vendor in the category shipped its flagship AI features as alpha, preview and beta in June 2026. Pilot them. Do not build an organisational chart on them.

Invest at the design end. The BLS split — 0% for administrators, 9% for architects — is the clearest available signal about which skills compound. Modelling, contracts, and knowing what the business actually needs counted are not the parts being automated. (For teams weighing whether to build this capability in-house or bring it in, our note on dedicated developers versus an in-house team covers the trade-offs.)

The Bottom Line

Data engineering is being transformed by AI. The mechanism is that first drafts became nearly free — which sounds like it ought to shrink the job, and has instead moved it somewhere else.

The measurements are considerably more consistent with one another than the marketing is. Text-to-SQL reaches 82.39% against a human 92.96% — and when researchers audited the reference answers in two widely-used subsets of these benchmarks, more than half of them turned out to be wrong. The category leader shipped its flagship AI features in alpha, preview and beta. DORA finds throughput up and stability down, and names "the verification tax" as the reason the gains arrive late. Stack Overflow finds 84% adoption alongside 46% active distrust, with "almost right, but not quite" the single commonest complaint. And the BLS projects zero growth for the people who maintain data systems against 9% for the people who design them.

None of which makes the technology a disappointment. It locates the value, which is real, in specific places: drafting, optimisation, detection, and not, so far, unsupervised operation. DORA's finding points the same way: what separates teams that stay stable under AI-accelerated change is the presence of strong testing, version control and fast feedback. Absorbing a faster rate of change without absorbing a faster rate of error is what those control systems buy you.

Sequence is what we would push hardest on, and our advice inverts the usual counsel. The agent that writes your pipelines is the wrong starting point. Begin instead with the tests that would catch it being wrong. Then hand it the pipelines.

If you are weighing up how much of this to adopt, or want a second opinion on where your pipeline would break first, get in touch. We build and run this kind of infrastructure for a living, and we are happy to tell you when the answer is "fix your tests before buying anything."

Get in touch:

Thinking through a new idea or stuck with a challenge? Drop us a message—we'll listen, brainstorm, and help move things forward.