Category: Data Engineering

  • Dual-Purpose Data: Designing Foundations for Humans and Agents

    Every data foundation built in the last decade was designed for one consumer: people. Analysts, executives, data scientists — humans who look at dashboards, ask questions, and apply judgment. That foundation is about to get a second consumer that behaves nothing like the first: AI agents.

    Same data. Two consumers. Radically different needs. The architects who design for both, deliberately, will build the foundations that last. The ones who bolt agents onto human-shaped data will spend the next three years firefighting.

    The two consumers

    Humans and agents consume data in fundamentally different ways.

    A human looks at a dashboard, notices something odd, drills down, asks a colleague, and applies context the data doesn’t contain. Humans are fault-tolerant. They squint.

    An agent does none of that. It reads structured output, follows its instructions, and acts — no squinting, no “this doesn’t look right,” no hallway conversation to resolve ambiguity. It’s fast, tireless, and completely literal. Every gap a human would bridge with judgment becomes a gap the agent falls into.

    This isn’t an argument against agents. It’s the design constraint that changes everything: data good enough for humans is not automatically good enough for machines.

    What agents need that dashboards don’t

    Designing for agents surfaces requirements that human-centric foundations never had to meet:

    Freshness with guarantees, not vibes. A human tolerates “data refreshed nightly, mostly.” An agent acting on a stale customer record needs to know exactly how stale, and what it can and can’t do about it. Freshness becomes a contract with SLAs, not a footnote.

    Structure over presentation. Dashboards are forgiving — a human reads around a null, interprets a weird format, ignores the extra column. Agents need clean schemas, typed fields, explicit nulls, and documented enumerations. The pretty chart hides sins the API can’t.

    Permissions at the row, not the dashboard. Human access control often lives at the report or dashboard level. Agents need row-level, attribute-level enforcement — because the agent will happily read the salary column the dashboard hid, and then mention it in a summary. If the permission isn’t in the data layer, it doesn’t exist for agents.

    Grounding, not just retrieval. Humans bring world knowledge to ambiguous data. Agents need explicit grounding: definitions, lineage, quality verdicts, and business rules attached to the data itself. An agent that can’t see why a number is what it is will invent a why.

    If your access control lives in the dashboard, your agents are already reading the columns you hid.

    Design once, serve twice

    The good news: you don’t need two foundations. You need one foundation designed with both consumers in mind — and that’s a better foundation for humans too.

    The pattern is a layered architecture with a clean contract between layers. Raw ingestion at the bottom, governed and modeled in the middle, and serving layers on top — one serving humans through BI, one serving machines through APIs and agent tooling. Same governed core, different doors.

    The discipline is in the middle layer. Every dataset that serves both consumers needs: a schema agents can parse without guessing, quality verdicts a machine can read programmatically, lineage an agent can traverse, access policies enforced at the data layer, and semantic definitions shared with the human side. Build this once and both consumers benefit. Skip it and you’ll build it twice — once in panic, once properly.

    Governance for both

    Governance was already hard with one consumer. Two consumers with different failure modes makes it harder — and more important.

    The shift is from governance-as-documentation to governance-as-execution. Policies can’t live in PDFs when the consumer is a machine; they have to be enforced in the platform. Access rules, quality gates, retention policies, consent flags — all machine-readable, all enforced at query time, all auditable. The audit trail matters doubly now: when an agent acts on data, you need to prove what it saw, what rules applied, and why it was allowed to see it.

    Governance that lives in a PDF is governance for humans. Agents need governance in the platform.

    This is also where stewardship evolves. Data stewards used to answer human questions about data. Now they curate the machine-readable context agents depend on: definitions, quality rules, escalation paths for ambiguous cases. The steward’s audience doubled.

    The operating model

    Dual-purpose data needs a dual-purpose operating model. A few things change in practice:

    Quality becomes a production concern, not an analytics concern. When agents act on data in real time, a quality failure is an operational incident, not a reporting footnote. Data quality SLAs, monitoring, and incident response move from “nice to have” to “the thing that keeps agents from embarrassing you.”

    The catalog becomes an agent’s first stop. Data discovery for humans is browsing. For agents, the catalog is infrastructure — machine-readable metadata, searchable by capability, with quality and access signals attached. If your catalog is a website humans click through, agents can’t use it. If it’s an API with rich metadata, both can.

    Versioning gets serious. Humans adapt to schema changes with a grumble and a rewritten query. Agents break. Every schema change, every definition change, every deprecation needs versioning and notice — the same discipline APIs have had for years, now applied to data.

    The unfair advantage

    Here’s the thing most organizations miss: designing for agents makes the human experience better too. Cleaner schemas, enforced quality, real access control, versioned definitions — every one of these is something analysts have wanted for years. The agent is just the forcing function that finally makes it non-negotiable.

    The enterprises that build dual-purpose foundations now will have something rare: data their people trust and their machines can act on. Everyone else will discover — usually via an incident — that their foundation was built for exactly one consumer, and the second one doesn’t squint.

    Design for both. The humans will thank you. The agents won’t — but they won’t embarrass you either, and right now, that’s the higher bar.

    — Jugal

  • The Semantic Layer: The Most Underrated Architecture in Enterprise Data

    Ask three teams in your company to define “active customer” and you’ll get four answers. Marketing counts anyone who opened an email this quarter. Finance counts paying accounts. Product counts monthly active users. And the executive dashboard shows a fifth number that nobody can explain but everyone quotes.

    This is the metrics mess, and it’s the most expensive problem in enterprise data that nobody’s job title mentions. Billions get spent on lakes, warehouses, and pipelines, and then the business argues about what the numbers mean. The technology worked. The meaning didn’t.

    The fix has a name, and it’s the most underrated architecture in the data world: the semantic layer.

    What a semantic layer actually is

    A semantic layer is a single, governed definition of every business metric and dimension, sitting between the raw data and everyone who consumes it. “Revenue” is defined once — the logic, the filters, the grain, the edge cases — and every dashboard, report, API, and AI agent uses that definition.

    Not a wiki page describing revenue. Not a Confluence doc everyone ignores. The actual logic, executable, versioned, and shared. When someone asks “how is revenue calculated?”, the answer is the semantic layer, and it’s the same answer everywhere.

    If your metrics are defined in forty dashboards instead of one place, you don’t have metrics. You have opinions with charts.

    Why the old answers failed

    We’ve tried to solve this before. The data warehouse was supposed to be the single version of the truth — but the logic leaked into BI tools, dbt models, spreadsheets, and one-off SQL. The “metrics layer” wave of a few years ago got the idea right but often landed as yet another tool to adopt rather than a discipline to enforce.

    The failure pattern is always the same: the definitions live wherever it’s convenient in the moment, and convenience compounds into chaos. A semantic layer only works when it’s the only place metric logic is allowed to live. That’s a governance decision before it’s a technology decision.

    Conceptual, logical, physical: the modeling discipline underneath

    Under every good semantic layer is old-fashioned data modeling discipline — the kind that fell out of fashion and is now quietly saving AI projects.

    Conceptual modeling answers “what are the things we care about and how do they relate?” — customers, orders, products, the entities and relationships, in business language. Logical modeling adds structure: attributes, keys, grain, the rules that hold regardless of technology. Physical modeling is the implementation: tables, partitions, indexes, the performance reality.

    Most organizations skipped straight to physical — built the tables, worried about meaning later. Later never came. The semantic layer forces the discipline back in: you can’t define “revenue” once, governably, without agreeing on what a customer is and what counts as an order. The modeling conversation is the alignment conversation.

    The business glossary as a contract

    The glossary is the semantic layer’s public face: every metric and dimension, defined in plain language, with an owner, a lineage trail, and a version history. But the key word is contract. A glossary that nobody is obligated to use is documentation. A glossary wired into the platform — where the BI tool pulls definitions from it, where data contracts reference it, where new pipelines must map to it — is infrastructure.

    The test is simple: can a new analyst find the definition of a metric, see who owns it, trace it to source, and trust it — in an afternoon? If yes, you have a contract. If it takes a quarter and five Slack threads, you have a wiki.

    The semantic layer and AI

    Here’s what makes this urgent instead of merely important: AI agents need shared meaning more desperately than humans do.

    A human analyst who sees a weird revenue number will squint, ask around, and figure out the definition drift. An agent won’t. It will take the number, act on it, and compound the error into decisions. Worse, agents from different teams will each invent their own interpretation of the same metric — automated definition drift, at machine speed.

    The semantic layer is what grounds AI in shared business meaning. Retrieval gives agents facts; the semantic layer gives them definitions. An agent that queries “revenue” through the semantic layer gets the governed answer. One that writes its own SQL gets whatever it feels like.

    In the agentic era, the semantic layer isn’t a BI convenience. It’s the difference between grounded AI and confident fiction.

    Where to start

    Don’t boil the ocean. Pick the ten metrics the business argues about most — revenue, active customers, churn, cost, whatever sparks the weekly debate — and define them once, in one place, with owners and lineage. Wire one BI tool to consume them. Show the before and after: same question, one answer.

    Then expand. The goal isn’t a perfect enterprise ontology on day one; it’s breaking the pattern where every new dashboard invents its own definitions. Ten governed metrics, actually used, beat a hundred documented ones nobody trusts.

    The enterprises that get this right will discover something their competitors won’t: when the AI era demands grounded, consistent, explainable answers, they’ll already have the layer that provides them. Everyone else will be retrofitting meaning onto chaos — while their agents confidently report five different revenues.

    The work is unglamorous. The payoff isn’t. Define it once.

    — Jugal

  • The Identity Problem: Why Your “Single Customer View” Is Five Different People

    Every enterprise claims to have a single customer view. Almost none of them do. What they have is five versions of the customer — the CRM version, the billing version, the support version, the marketing version, and the one in the data warehouse that was stitched together three years ago and hasn’t been questioned since.

    This isn’t a cosmetic problem. Identity is the join key of the entire enterprise. When it’s wrong, every report is subtly wrong, every personalization is subtly off, and every AI agent built on top of it acts on fiction with total confidence.

    Why identity breaks

    Identity breaks because every source system has its own opinion about who a customer is, and none of them are wrong — they’re just incomplete. The CRM knows “Robert Smith, rob.smith@email.com.” Billing knows “Bob Smith, rsmith@work.com, account 88412.” Support knows a phone number and a first name. The mobile app knows a device ID. None of these records agree, and all of them are real.

    The naive fix is “just match on email.” It works until it doesn’t: shared family emails, work emails that change when people change jobs, customers with three accounts across three business units. Email is an attribute, not an identity.

    Match and merge: the mechanics

    Identity resolution comes down to two questions: which records refer to the same person, and what do you do about it?

    Matching is the first question, and it has two flavors. Deterministic matching — same email, same phone, same government ID — is precise but misses most real-world messiness. Probabilistic matching scores the likelihood that two records are the same person based on name similarity, address proximity, shared identifiers, and behavioral signals. Good systems use both: deterministic rules for the obvious cases, probabilistic scoring for the rest, and a human review queue for the gray zone in between.

    If your matching is fully automatic with no review queue, you don’t have identity resolution. You have identity roulette.

    Merging is the second question, and it’s where most projects quietly fail. Merging doesn’t mean deleting records. It means linking them: every source record survives, and a golden record sits above them as the resolved view. Delete the sources and you lose lineage, auditability, and the ability to un-merge when you get it wrong. And you will get it wrong sometimes.

    Survivorship: who wins

    When five records disagree about a customer’s address, which one is right? That’s survivorship, and it needs explicit rules — not vibes.

    The honest answer is “it depends on the attribute.” The most recent address usually wins. The legal name from the verified onboarding system beats the nickname someone typed into a support ticket. The phone number the customer used last week beats the one from 2019. Survivorship rules should be attribute-level, source-ranked, and visible — anyone should be able to ask “why does the golden record say this?” and get an answer, not a shrug.

    The golden record and the 360 view

    A golden record is the resolved, survivable, lineage-tracked representation of an entity — customer, product, location, supplier. The 360 view is what you build on top of it: every interaction, transaction, and touchpoint attached to one identity.

    Here’s what separates real 360 views from slideware: the golden record is operational, not analytical. If it only lives in the warehouse for reporting, it’s a museum piece. The real thing feeds the call center screen, the personalization engine, the fraud check, and the agent — in real time, through APIs, with the same identity everywhere.

    Identity in the agentic era

    This is where identity goes from important to existential. A dashboard showing a duplicated customer is an embarrassment someone might notice. An autonomous agent acting on a duplicated customer is a machine that emails the wrong person, applies the wrong discount, or discloses one customer’s data to another — at speed, at scale, without asking.

    Agents don’t have the human instinct to say “wait, this doesn’t look right.” They act on whatever identity you hand them. Every identity defect your organization tolerated in the BI era becomes an incident in the agentic era. The math changed; the tolerance can’t stay the same.

    Your agents will be exactly as trustworthy as your identity resolution. No more.

    What good looks like

    Good identity resolution has a shape you can recognize: source records preserved with lineage, match rules versioned and explainable, a review queue for uncertain matches, attribute-level survivorship rules, golden records served operationally through APIs, and metrics — match rate, merge rate, false-positive rate — that someone actually watches.

    It’s unglamorous work. Nobody demos identity resolution at a conference. But it’s the foundation everything else stands on: the 360 view, the personalization, the analytics, and now the agents. Get identity right and the rest gets easier. Get it wrong and the rest gets fictional.

    The enterprises that treat identity as infrastructure will have an almost unfair advantage as agents take over more of the work. The ones that don’t will have very fast, very confident systems acting on five different versions of the truth.

    — Jugal

  • What “AI-Ready Data” Actually Means: A Data Leader’s Field Guide

    Every board in America is asking about AI strategy right now. And somewhere in every one of those companies, a data leader is sitting in that meeting thinking: “You want AI? Have you seen our data?”

    That gap — between AI ambition and data reality — is where data leadership careers will be made and broken over the next three years. Not in model selection. Not in prompt engineering. In the unglamorous work of making enterprise data connected, trusted, and discoverable enough for intelligence to stand on.

    This is the field guide for closing it.

    Single source of truth — or a very expensive mirror

    Every enterprise has data. Most have five versions of every fact: the finance version, the marketing version, the operations version, and two more nobody will admit to. Ask an AI to answer a question on top of that, and it won’t resolve the contradictions. It will confidently pick one.

    A unified data platform isn’t a technology project. It’s a trust project. The technology — the lakehouse, the catalog, the pipelines — is the easy part. The hard part is getting every domain to agree on what a customer is, what revenue means, and which system wins when they disagree.

    If the business doesn’t trust the data, your single source of truth is a very expensive mirror.

    Three tests for whether you’re actually there: connected (can you trace a metric from dashboard to source system?), trusted (do business leaders quote your numbers without adding “but let me check”?), and discoverable (can a new analyst find the right dataset in an afternoon, not a quarter?). If any answer is no, you don’t have a platform yet. You have infrastructure.

    AI-readiness is a data problem wearing an AI costume

    Here’s the uncomfortable truth behind every stalled AI pilot I’ve seen: the model was fine. The data wasn’t. Wrong, stale, contradictory, or legally unusable — pick your poison.

    AI-readiness has a concrete definition, and it’s worth writing down: every dataset an AI system touches has a known owner, a documented lineage, a quality verdict, and a clear access policy. That’s it. Not a vision statement — an inventory.

    The leaders getting this right aren’t running “AI readiness programs” as a separate track. They’re folding readiness into the platform work they should have been doing all along: data contracts at ingestion, quality gates in pipelines, lineage as a first-class artifact. When the AI use case arrives, the data is already standing at attention.

    Governance that ships

    Governance has a branding problem, and data leaders created it. For a decade, governance meant committees, tickets, and six-week waits for access approvals. So the business learned to route around it — shadow pipelines, spreadsheet empires, “temporary” extracts that are now seven years old.

    Governance that slows teams down gets routed around. Every time.

    The fix isn’t less governance. It’s governance that ships: policies enforced in the platform itself, access granted by rules instead of meetings, quality checks running in the pipeline instead of in a quarterly review. The NIST AI Risk Management Framework gets this right — Govern, Map, Measure, Manage as a continuous cycle, not a stage gate. Embed it, automate it, and measure whether teams move faster with it than without it. That’s the only metric that matters.

    Data mesh, honestly

    Data mesh is the right answer to a real problem: centralized data teams don’t scale to hundreds of domains. Federated ownership, domain-oriented data products, self-service infrastructure — the principles are sound.

    But mesh without a platform is just distributed chaos with a trendy name. I’ve watched organizations declare “data mesh” and get sixteen incompatible definitions of a customer, each owned by a domain team with no incentive to reconcile them. The mesh needs a strong center: shared standards, common tooling, and a platform team whose job is making domain ownership easy, not just making it someone else’s problem.

    Be honest about where you are. If your domains can’t yet produce trustworthy data products on their own, you don’t need a mesh — you need a platform team and about eighteen months of discipline. Mesh is a maturity destination, not a starting position.

    Master data: the unsexy foundation

    Nobody gets promoted for fixing duplicate customer records. But here’s what changes the math: agentic AI.

    A bad customer record in a dashboard is a wrong number someone might catch. A bad customer record fed to an autonomous agent is an agent that emails the wrong person, bills the wrong account, or acts on fiction at machine speed. AI doesn’t just surface master data problems — it multiplies them.

    Your AI is only as smart as your customer record.

    Master data excellence — golden records for customer, product, location, the entities your business actually runs on — is the highest-leverage, lowest-glamour work in the data portfolio. Do it before the agents arrive, not after the incident.

    Forward deployment: adoption is the job

    The best data platform in the world is worthless if nobody uses it. And “build it and they will come” has a perfect failure record in enterprise data.

    The model that works is forward deployment: data engineers and governance specialists embedded with business units, operationalizing standards inside real workflows instead of publishing them from headquarters. You don’t drive adoption with town halls. You drive it by sitting with the finance team until their close process runs on your platform — and then letting them tell the story.

    Measure adoption, not deployment. Not “pipelines built” but “decisions made on governed data.” Not “datasets cataloged” but “analysts who found what they needed without filing a ticket.” The platform serves the business, and the metric should prove it.

    The leadership part

    None of this is a technology problem at its core. It’s an alignment problem, which makes it a leadership problem.

    Three disciplines matter most. First, executive fluency: translate everything into business outcomes. Nobody funds “data mesh.” They fund “cutting month-end close from nine days to three.” Second, organization building: hire for judgment, not just tooling — tools change every eighteen months, and the team that thrived through three platform migrations is worth more than the team that knows this year’s stack. Third, patience with urgency: this work takes years, but you must show value in quarters. Ship the thin slice that earns the next round of trust.

    The enterprises that get data right in the next three years will have an almost unfair advantage in AI — not because their models are better, but because their models stand on ground instead of fog. The ones that don’t will have very impressive pilot graveyards.

    The work is unglamorous. The payoff isn’t. That’s the job.

    — Jugal

  • Building a Data Lake That Earns Its Keep: Lessons from Billions of IoT Rows

    Most data lakes are where data goes to be forgotten. Ours had a different job: catch faulty devices and faulty production components before downtime cost real money — across billions of rows of IoT data from a manufacturing operation.

    Here’s what building that system taught me about data engineering that actually earns its keep.

    Start with the decision, not the data

    The classic mistake is ingesting everything first and asking questions later. We did the opposite: the lake existed to answer one question — what is about to fail, and what should we do about it? Every ingestion choice, every schema decision, every pipeline stage served that question.

    A data lake without a decision to support is just expensive storage. Define the decision first, and the architecture mostly designs itself.

    Ingestion at billion-row scale is a different sport

    When you’re ingesting billions of IoT rows, the problems stop being about query performance and start being about survival:

    • Throughput over elegance. The pipeline has to absorb bursts without falling over. Backpressure handling and idempotent writes matter more than clever transformations.
    • Schema drift is constant. Devices get firmware updates, sensors get replaced, fields appear and disappear. If your pipeline breaks every time a device sneezes, you don’t have a pipeline — you have a pager generator. Build for evolution: permissive ingestion, strict validation downstream.
    • Late and out-of-order data is normal. IoT networks are unreliable by nature. Your windowing and watermarking strategy isn’t an optimization; it’s the difference between correct and misleading.

    The pipeline is the product

    The core of the system wasn’t the lake itself — it was the data engineering pipeline running continuously over it, checking for faulty IoT devices and faulty manufacturing components and reporting them. This is the part most “data lake” projects underinvest in: they build the storage and declare victory.

    Storage is table stakes. The value is in the continuous compute — the jobs that turn raw rows into signals. If nobody is running queries that change decisions, you don’t have a data platform. You have a data attic.

    Then put AI on top — not underneath

    Once the pipeline reliably surfaced faults, we connected an LLM system using RAG over the pipeline’s findings. Instead of just an alert saying “component X looks wrong,” engineers got a diagnosis of the fault plus possible solutions.

    The ordering matters. AI on top of a broken pipeline just generates confident-sounding nonsense faster. The RAG layer worked because the retrieval corpus — the fault findings — was trustworthy. Data quality wasn’t a separate initiative; it was the foundation the AI stood on.

    This is the pattern I keep seeing: teams want the AI layer first and the data engineering “later.” Later never comes, and the AI disappoints. Build the pipeline, earn trust in the data, then add intelligence.

    From reactive to predictive is an organizational shift, not a technical one

    The technical outcome: faults surface early, with AI-generated guidance on what to do about them. But the real shift was operational — the team moved from reactive firefighting to predictive maintenance.

    That shift didn’t happen when the pipeline went live. It happened when the engineers trusted it enough to act on its findings without double-checking everything manually. Trust took time and a track record. There’s no shortcut: the system has to be right, repeatedly, in public, before people change how they work.

    The checklist

    If you’re building a data lake that needs to earn its keep:

    1. Define the decision the lake serves before ingesting a single row.
    2. Design ingestion for bursts, schema drift, and late data — not the happy path.
    3. Invest in the continuous pipeline, not just the storage.
    4. Get the data trustworthy before adding AI on top.
    5. Measure success by changed decisions, not by rows stored.

    A data lake is not a strategy. It’s infrastructure in service of decisions. Build it that way, and it pays for itself. Build it any other way, and it’s just a very expensive attic.

    — Jugal