Building a Data Lake That Earns Its Keep: Lessons from Billions of IoT Rows

Most data lakes are where data goes to be forgotten. Ours had a different job: catch faulty devices and faulty production components before downtime cost real money — across billions of rows of IoT data from a manufacturing operation.

Here’s what building that system taught me about data engineering that actually earns its keep.

Start with the decision, not the data

The classic mistake is ingesting everything first and asking questions later. We did the opposite: the lake existed to answer one question — what is about to fail, and what should we do about it? Every ingestion choice, every schema decision, every pipeline stage served that question.

A data lake without a decision to support is just expensive storage. Define the decision first, and the architecture mostly designs itself.

Ingestion at billion-row scale is a different sport

When you’re ingesting billions of IoT rows, the problems stop being about query performance and start being about survival:

• Throughput over elegance. The pipeline has to absorb bursts without falling over. Backpressure handling and idempotent writes matter more than clever transformations.
• Schema drift is constant. Devices get firmware updates, sensors get replaced, fields appear and disappear. If your pipeline breaks every time a device sneezes, you don’t have a pipeline — you have a pager generator. Build for evolution: permissive ingestion, strict validation downstream.
• Late and out-of-order data is normal. IoT networks are unreliable by nature. Your windowing and watermarking strategy isn’t an optimization; it’s the difference between correct and misleading.

The pipeline is the product

The core of the system wasn’t the lake itself — it was the data engineering pipeline running continuously over it, checking for faulty IoT devices and faulty manufacturing components and reporting them. This is the part most “data lake” projects underinvest in: they build the storage and declare victory.

Storage is table stakes. The value is in the continuous compute — the jobs that turn raw rows into signals. If nobody is running queries that change decisions, you don’t have a data platform. You have a data attic.

Then put AI on top — not underneath

Once the pipeline reliably surfaced faults, we connected an LLM system using RAG over the pipeline’s findings. Instead of just an alert saying “component X looks wrong,” engineers got a diagnosis of the fault plus possible solutions.

The ordering matters. AI on top of a broken pipeline just generates confident-sounding nonsense faster. The RAG layer worked because the retrieval corpus — the fault findings — was trustworthy. Data quality wasn’t a separate initiative; it was the foundation the AI stood on.

This is the pattern I keep seeing: teams want the AI layer first and the data engineering “later.” Later never comes, and the AI disappoints. Build the pipeline, earn trust in the data, then add intelligence.

From reactive to predictive is an organizational shift, not a technical one

The technical outcome: faults surface early, with AI-generated guidance on what to do about them. But the real shift was operational — the team moved from reactive firefighting to predictive maintenance.

That shift didn’t happen when the pipeline went live. It happened when the engineers trusted it enough to act on its findings without double-checking everything manually. Trust took time and a track record. There’s no shortcut: the system has to be right, repeatedly, in public, before people change how they work.

The checklist

If you’re building a data lake that needs to earn its keep:

1. Define the decision the lake serves before ingesting a single row.
2. Design ingestion for bursts, schema drift, and late data — not the happy path.
3. Invest in the continuous pipeline, not just the storage.
4. Get the data trustworthy before adding AI on top.
5. Measure success by changed decisions, not by rows stored.

A data lake is not a strategy. It’s infrastructure in service of decisions. Build it that way, and it pays for itself. Build it any other way, and it’s just a very expensive attic.

— Jugal