Stale Data, Hidden Dependencies and a 24-Hour Hunt: A DataOps Story (Part 1)

A leadership dashboard showed old data and nobody could say why. How implicit dependencies turned one simple question into a 24-hour hunt.

By the G4X team · · 4 min read

Illustrative · fictional figures

About ten years ago, one of our clients had the kind of problem most companies would love to have. Their data was growing so fast that nobody could keep up with it.

New products, new teams, new pipelines. Every week someone built something that read from something else. Data engineering teams worked in silos, each owning its own corner and doing a perfectly good job inside it. Nobody was doing anything wrong. That’s the uncomfortable part.

Because the dependencies between those corners weren’t written down anywhere. One team produced a table. Another built a pipeline on top of it. A third built something on top of that. Upstream and downstream were less of an architecture and more of a rumour. If you had asked for the data lineage, you would have got a shrug.

The dashboard that looked fine

Then one day the leadership dashboard showed old data.

Stale data is the sneaky kind of failure. A broken dashboard screams. A stale one just sits there, looking confident.

To be clear, nobody made a decision based on those numbers. This isn’t that kind of horror story. But someone had to answer a very simple question: why is this data old?

Simple question. Not a simple answer.

Table archaeology

With no map, the only way to find out was by hand. Open the table. Check when it last refreshed. Work out what feeds it. Go one level up. Repeat. Find out who owns that one, ask them, wait for an answer, go up again.

One hunt, hop by hopDigging Waiting for a reply

  1. leadership_kpis

    Last refreshed two days ago. What feeds it?

  2. sales_summary

    Stale too. Owner unclear. Ask around.

  3. Waiting · 3 h

  4. orders_clean

    “It’s built from this.” Also stale. Message its owner.

  5. Waiting · 5 h

  6. orders_enriched

    One more level up. Different team, different time zone.

  7. Waiting overnight · 12 h

  8. payments_export

    The source. Its owner: “I didn’t know anyone used this.”

~23 hours to answer “why is this data old?” · about 3 of them spent digging

Illustrative · fictional table names and times

That kind of investigation could take 24 hours. A full day of engineers doing archaeology instead of engineering.

And at the end of one of those trails was an upstream owner who had no idea anyone downstream depended on their data. Not negligence. From where they sat, nobody did. Their pipeline was their pipeline. The fact that a leadership dashboard several hops away was quietly leaning on it was news to them.

That’s what implicit dependencies look like in real life. Not a diagram with a red arrow. A surprised person on the other end of a message.

The stakes: healthy data, on time, every time

Here’s what made it serious. The requirement wasn’t “most of the time” or “during business hours”. Data had to land healthy and on time, 24/7/365.

Not “the job ran”. Landed. Healthy. On time.

What the scheduler says

  • Job orders_dailySucceeded
  • Exit code0
  • Duration4 min 12 s

“The job ran.”

What leadership is paying for

  • LandedToday’s data missing
  • HealthyBuilt on two-day-old input
  • On timeDue 06:00 · still waiting at 09:12

Not landed.

Illustrative · a job can succeed while the data it promised never arrives

Data freshness sounds like a technical metric. In practice it’s a promise. Leadership pays for data that landed, not for a well-written explanation of why it didn’t land again. Every 24-hour investigation was a day where the honest answer to “is this right?” was “we’re checking”.

One hunt like that is an anecdote. Multiply it by an organisation growing this fast and it turns into a job description. Heroics don’t scale, and neither does a human being clicking through tables at 3 a.m.

What good would look like

Today there’s a whole vocabulary for this: data observability, data lineage, freshness SLAs. Back then there was a dashboard, a lot of tables, and whoever was on shift.

The goal, as we came to see it, was almost paradoxical. If the operations team was doing its job properly, nobody in the company would know it existed. Data would just be there. Healthy, on time, every day, and boring in the best possible way.

So how do you get from 24-hour table archaeology to a team so quiet that nobody knows it’s there?

It took years, a team spread across two hemispheres, and a fairly uncomfortable rethink of what “operations” should even mean.

To be continued.

Running a fabric, a NOC or a rollout?

Tell us what you are working on.

Contact us

Keep reading

Talk to our team

Name, email and a line about the work is all we need.

Topic optional

We use your details to respond to your enquiry.