Some semiconductor fabs now collect hundreds of petabytes of manufacturing data per year, and shift-left strategies built on real-time equipment sensor data and AI/ML inference are the main driver. The payoff is faster yield learning. The cost is a data management problem that pits per-module optimization against cross-fab traceability.

Over the last five years, equipment data growth has been exponential, driven by fabs moving detection into in-situ, real-time process control. "Over the last five years, the equipment data segment has grown exponentially," said Ashish Gupta, senior data engineering manager for automation at Intel Foundry. "This explosion is primarily the result of fabs engaging in shift-left efforts into in-situ, real-time process control."

How much data are we talking about?

Equipment data has expanded along three dimensions, according to Jae Yong Park, vice president of enterprise software business at Onto Innovation. Suppliers have added hundreds of parameters over the past few years, sampling rates have climbed from roughly once per second to every tens of milliseconds, and engineers increasingly work with raw trace data instead of summaries.

The result: real-time use of data within a process module now requires sub-second processing, and hundreds of equipment sensors have increased data variety by at least 10× from a decade ago.

Fabs see only a fraction of what tools generate. Kartik Venkataraman, senior director of product management at Synopsys, points to RF plasma sources in etch and CVD chambers operating at megahertz levels, with pulsing at kilohertz — each pulse carrying a waveform with transient response and droop. "You can't possibly monitor every pulse. You're talking about megabytes per second of data," he said.

OEMs also withhold some data as proprietary IP. The remainder often exceeds what fab infrastructure can ingest.

What does shift left actually change?

Traditionally, engineers waited days, weeks, or months for yield excursion signals from downstream screening such as electrical test. With AI/ML inference models consuming real-time sensor data, fabs can now identify and act on excursions inside the process module itself — adjusting tool settings or aborting a process mid-operation, then triggering root-cause analysis and maintenance.

Test has shifted left for decades, and chiplet proliferation added a singulated-die test step to secure known good die (KGD). Wafer fabs are applying the same logic: purpose-built virtual metrology models let process modules self-identify yield-impacting shifts without waiting for downstream screens.

Gupta frames the data engineering consequence directly. In-phase shift left requires continuous, high-resolution surveillance, in-situ analytics, anomaly detection, virtual metrology, and adaptive test — activities he says contribute to a "4Vs (volume, velocity, variety, veracity) data tsunami."

The same logic extends below the tool. Water temperature, water pressure, and power fluctuations affect yield, and PDF Solutions' Steve Zamek notes that sub-fab data analysis is already feasible with existing standards and system technologies, with minimal barriers to adoption. PDF's software can interdict equipment before producing more scrap, provided robust virtual metrology models predict in-situ quality metrics from real-time parameters.

Where does the data strategy break down?

The core tension: teams optimize data management per process module and lose the context needed to connect steps. "The local optimization trap results in a situation you can describe as 'data rich but context poor,'" Gupta said. "Each process module optimizes fiercely for its specific, time-constraint challenges. While enterprise data integration, cross-module traceability, and global correlation are more difficult to prioritize."

That context is not optional — connecting data across steps is what trains and maintains the very in-situ inference models each module depends on. Traceability standards are lacking, which compounds the problem.

New entrants such as Rapidus and Terafab have an opportunity to start from scratch, treating fabs as information businesses, in the words of Siemens EDA's Jim Schiely, building "this huge graph of data. These graphs are exactly the kind of fuel that AI can cook with." Established fabs instead must remake decades-old modeling practices, leveraging modern Python libraries and automated neural network fitting that did not exist 25 years ago.

Digital twins help, but ground truth still comes from multi-physics simulation and real data, not LLMs. "Core engines still have to generate the ground truth, and this cannot be done by LLMs," Venkataraman said. "You need the actual data to anchor that in reality."

Can optimization scale beyond one fab?

Lam Research's Russell Dover describes fleet-level correction models that commingle data from geographically separate fabs, treating them as one virtual fleet — matching chambers across locations to improve Cpk and yield. Plasma processing is a strong virtual metrology platform: residue on chamber walls shifts gas-phase chemistry and etch rate as chambers age, and machine learning can detect patterns across RF impedance and gas flow that humans miss, estimating film thickness or etch depth before conventional measurements arrive, according to SandboxAQ's Shalini Sharma.

Shift left is accelerating yield learning across advanced nodes — gate-all-around logic, 3D DRAM, stacked NAND, and HBM/SiP multi-die packages. The unresolved question is whether fabs can preserve the context and traceability that fab-wide analytics requires while each module optimizes for its own time constraints.