This is the third of six part blog series on Ceph for data lakehouse. You can find part 2 here.

For many storage practitioners, the term data lakehouse can feel vague or overmarketed. It’s often described as something between a data lake and a data warehouse, without clearly explaining what changes at the storage layer—or why storage suddenly matters more than ever.

A simpler and more useful way to understand the lakehouse is through a familiar analogy: A data lakehouse is a refinery built on object storage. Raw data goes in. Refined, trusted data comes out—ready to power analytics and AI.

Raw Data Is Crude Oil

Enterprises generate massive volumes of data: logs, application events, files, images, transactions, sensor streams, etc. From a storage perspective, this data is large, relatively inexpensive to retain (stored on inexpensive media), and rarely accessed in its raw form.

That makes it crude data—valuable, but not directly usable.

Object storage platforms like IBM Ceph are ideal crude reservoirs:

• Massive horizontal scale
• Low cost per terabyte
• Durable, immutable objects
• No assumptions about data format

But simply storing crude oil doesn’t create value. Value requires refinement.

Why Files Alone Aren’t Enough

Early data lakes treated object storage as a dumping ground for files. Multiple tools wrote data directly into shared buckets, each application with its own conventions.

From a storage point of view, this leads to familiar problems:

• Partial writes and failed jobs leaving debris
• No clear indication which files were “good”
• No protection from concurrent writers
• No reliable way to reproduce historical states

This is crude oil in tanks with no refinery controls. You may have volume, but you don’t have value.

The Lakehouse Adds Refinery Controls—Not New Storage

A data lakehouse does not replace object storage. The data still lives as files—typically Parquet—stored in buckets.

What changes is that a control layer is added above those files. This layer introduces table abstractions over files, see the appendix below for more details.

Table Formats = Refinery Control Systems

Modern refineries rely on instrumentation to control pressure, temperature, and flow at every step.

Lakehouse table formats provide the same control layer:

• ACID transactions → guarantee data validity and reliability
• Time travel & versioning → ability to look at past “data set” versions
• Schema enforcement → predictable format for multiple consumers
• Metadata statistics → Know exactly what’s in every data set

In refinery terms, these formats ensure that refining happens with precision, not guesswork, you know exactly what data was used to produce your results.

Table Abstractions: Control Without Changing Files

At the storage level, nothing magical happens. Files remain files. Object storage behaves the same way. The table abstraction adds metadata that acts like a refinery control system.

Instead of saying:

“Read everything in this directory,”

a table says:

“This table consists of these specific files, at this specific version.”

Metadata tracks:

• Which files belong to the table
• Which snapshot is current
• What schema applies
• Which older files should be ignored

Readers never guess. Writers don’t overwrite. Files become immutable once committed.

Example – A version of a data set as defined by the “table” includes version 21 of file ‘foo’. A process writes a new ‘foo’ file. The new foo is version 22. The table and processes accessing data via the table definition use version 21 of the foo file, until the table is updated which may not happen until other files in the data set are updated. 

Transactions Without a Database

Data lakehouses deliver database‑like correctness while still using object storage. When new data arrives, refinement happens in stages:

1. New data files are written to object storage
2. A new metadata snapshot is created
3. That snapshot is atomically committed

Consumers either see the old snapshot (version) or the new one—never a partial state.

Versions, Time Travel, and Long‑Lived Data

Table metadata keeps prior snapshots. That enables:

• Reproducing yesterday’s analytics
• Auditing what data trained a model
• Rolling back bad refinement jobs

From a storage perspective, this means data is retained longer and reused more often. Silent corruption is no longer acceptable—because models won’t crash, they’ll just get less valuable.

Lakehouse Compute Engines = Refinery Machinery

Refineries use different machines for cracking, blending, and filtering.

Lakehouses do the same with decoupled compute:

• SQL engines → reporting and inspection
• Spark / distributed compute → large‑scale refinement
• ML workloads → feature extraction and training

Because compute and storage are decoupled:

• Refinement scales independently of storage
• Multiple teams can work on the same “crude” simultaneously
• Storage costs stay controlled while throughput stays high

Governance & Catalogs = Refinery Safety and Compliance Systems

No refinery operates without strict safety and compliance controls. In a lakehouse, governance layers act as:

• Safety seals – table row/column‑level access controls
• Quality checkpoints – data validation rules
• Chain‑of‑custody records – lineage and audit trails

This ensures:

• Sensitive data doesn’t reach unauthorized consumers
• Contaminated batches are isolated immediately
• Every data point can be traced back to its source

Trust is built into the refinery—not added later.

Streaming + Batch = Continuous Refining

Modern refineries don’t shut down to refine new oil nor do data lakehouses.

Lakehouses support:

• Batch refinement for historical analysis
• Streaming refinement for real‑time data

This allows continuous blending—data is always fresh, and AI systems respond instantly to changing conditions.

Serving & Consumption = Fuel Delivery System

Finally, fuel must reach the race car at the exact moment it’s needed.

Lakehouse outputs feed:

• BI dashboards
• Model training pipelines
• Real‑time inference services

Delivery is:

• Fast
• Consistent
• Purpose‑built

No jerry cans. No delays. No contamination at the last mile.

Where Ceph Fits: The Physical Refinery

Ceph:

• stores objects
• ensures durability
• verifies integrity (checksums, scrubbing)
• scales cost‑effectively
• executes self‑healing rebuilds
• tiers to lower‑cost media, including tape

Ceph doesn’t decide how data is structured or queried. It ensures the refinery never leaks, never silently degrades fuel quality, and can scale without redesign. Storage is no longer just keeping data safe—it is protecting the fuel supply for analytics and AI.

Summary

AI performance doesn’t come from collecting more crude, it comes from running a better refinery. Lakehouses provide the structure, discipline, and scalability required to turn raw data into trusted, task‑specific, race‑ready fuel.

The engines will keep evolving. The teams that win are the ones that perfect the refinery. A data lakehouse doesn’t replace object storage. It elevates it. Seen through a storage lens:

• Files are crude oil
• Ceph is the refinery foundation
• Table abstractions are the control systems
• Analytics and AI are the engines

When storage is treated as a dumping ground, refinement fails. When storage is engineered like a refinery, data becomes fuel. This is why storage architecture sits at the center of the modern data lakehouse.

Appendix: “Table Abstractions Over Files”

Here is a simple example of how tables organize files.

At the lowest level, nothing magical happens to the data files themselves. The files are organized such that the data contained in the files are thread-safe without having to deal with complicated locking schemes.

The data is still:

• Files (typically Parquet)
• Stored in object storage (S3)
• Written and read via standard APIs

What changes is how those files are organized, tracked, and accessed.

A table abstraction is a metadata layer that tells systems:

• Which files belong together
• How they should be interpreted
• Which versions are valid
• Which files are safe to read right now

The Storage‑Native View (Files Only)

From a storage perspective, this is what raw object storage looks like:

bucket/

data/

file_001.parque

file_002.parquet

file_003.parquet

Problems:

• No indication if files are complete
• No schema enforcement
• No idea which files are “current”
• If a job fails mid‑write, partial files may exist
• Readers can see inconsistent states

This is crude oil in tanks with no refinery controls.

The Table Abstraction Adds a Control Layer (Metadata)

A lakehouse table format (e.g., Apache Iceberg) adds metadata files alongside the data files.

Now storage looks like this:

bucket/

              table/

                            data/

                                          file_001.parquet

                                          file_002.parquet

                             metadata/

metadata.json

snapshot_001.avro

snapshot_002.avro

From storage point of view:

• Data files are unchanged
• Metadata files describe how to use them

What the Metadata Actually Does

Defines “Which Files Are in the Table”

Instead of “read everything in this folder,” the table metadata says:

“This table consists of these specific files, and no others.”

That means:

• Old files can exist but be ignored
• New files are invisible until the table is updated
• Readers never guess

Makes File Updates Transactional (ACID‑like)

When new data is written:

1. New Parquet files are written first
2. A new metadata snapshot is created
3. The snapshot is atomically committed

Readers:

• Either see the old snapshot
• Or see the new snapshot
• Never see half‑written data

This is huge from a data correctness standpoint.

Refinery analogy:
Fuel isn’t released until the entire batch passes inspection.

Tracks Versions (Time Travel)

Old metadata snapshots are kept.

This means:

• You can query “yesterday’s table”
• Reproduce model training inputs
• Roll back bad writes

Storage teams will recognize this as versioned data, but implemented above the object storage semantics.

Enforces Structure Without Changing Storage

The table abstraction defines:

• Column names and types
• Partitioning rules
• Evolution rules (what can change safely)

But:

• Parquet files remain standard
• Object storage remains generic
• No proprietary storage format is introduced

This is control without lock‑in to a proprietary solution.

Why This Matters

Without Table Abstractions

• Object storage behaves like a shared filesystem (without effective locks)
• Multiple writers cause collisions
• Readers see inconsistent states
• Silent corruption or partial writes propagate upstream

With Table Abstractions

• Object storage becomes safe for concurrent access
• Files are immutable once committed
• Metadata controls visibility and correctness
• Storage durability and integrity finally matter upstream

AI and analytics workloads assume:

“If I can read it, it must be correct.”

Table abstractions make that assumption valid.

How Ceph Fits

The table abstraction:

• Lives above Ceph
• Uses Ceph as reliable object storage
• Depends on Ceph’s consistency and integrity guarantees

Ceph makes table abstractions trustworthy.

You can access part four here.