This is the fifth of six part blog series on Ceph for data lakehouse.  The fourth part can be here.

This section describes key S3‑compatible object storage features valuable in a data lakehouse deployment.

Table Buckets (Logical Organization of Tables)

How it works

An S3 Table Bucket is a specialized type of S3 bucket organized to store and manage tabular data as tables rather than just as independent objects. It’s part of the evolution of object storage to better support data lakehouse use cases, where data is organized, versioned, and accessed like a table instead of directories of files.

Value to the data lakehouse

• Clear separation between tables and other objects
• Enables transactional table behavior on object storage
• Simplifies governance by managing data access at the table, not bucket or object level.
• Lifecycle policies align to Table versions (snapshots), not object versions or tags/prefixes.

Simple example

A table bucket organizes data by table, not simply replicating a directory type structure.

s3://lakehouse/sales_table/ (finance team access)

data/

part-0001.parquet

part-0002.parquet

part-0003.parquet

metadata/

snapshot-123.avro

snapshot-124.avro

s3://lakehouse/campaign_table/ (marketing team access)

              data/ …

              metadata…

This provides a predictable, standardized structure for the data lakehouse consumers simplifying model / application development and access control.

S3 Select (Predicate Pushdown on Objects)

How it works

S3 Select allows applications to query a subset of data within an object using SQL‑like expressions, returning only matching rows or columns. S3 Select leverages the compute resources in the Ceph cluster to execute the query, rather than reading the entire object into the data lakehouse compute layer to execute the query.

Value to the lakehouse

• Reduces data transferred across the network
• Improves performance for exploratory queries
• Lowers compute cost for simple queries

Simple example

SELECT * FROM s3object WHERE region = 'US'

Only matching rows are returned instead of the full object reducing network bandwidth and server resource consumption. 

S3A Connector (Hadoop & Spark Integration)

How it works

The S3A connector is a Hadoop-compatible file system interface that allows engines like Spark, Hive, and Hadoop to read and write data directly to Ceph S3 storage. It lets distributed data processing engines treat S3 as if it were a filesystem—but optimized for object storage semantics, supporting legacy analytics applications to leverage data lakehouse data.

Value to the lakehouse

Enables legacy Spark‑based analytics to run against data lakehouse object storage

Adjusts operations to align to S3 object constraints

No in-place updates

Object immutability

Directory = prefix, not real folder

Separates compute from storage to enable independent scaling.

Simple example

spark.read.parquet("s3a://lakehouse/sales_table/data/")

Spark processes data directly from object storage.

Tiering (Hot, Warm, Cold Data Separation)

How it works

Tiering policies automatically move objects between storage classes based on access patterns or age.

Value to the lakehouse

• Keeps frequently accessed data on fast media
• Moves historical data to low‑cost tiers
• Enables long‑term retention without cost explosion

Simple example

30 days → warm tier

180 days → archive tier

Old training data is retained cheaply but remains accessible.

Lifecycle Policies (Automated Data Management)

How it works

Lifecycle policies automate object transitions, expiration, and deletion based on rules.

Value to the lakehouse

• Enforces data retention policies
• Prevents storage sprawl
• Reduces operational overhead

Simple example

Delete raw logs after 2 years

Ensures compliance without manual cleanup.

Object Tagging (Metadata for Governance)

How it works

S3-compatible object storage provides two complementary mechanisms for organizing and governing data:

Value to the lakehouse

• Enables fine‑grained governance
• Supports cost allocation
• Simplifies compliance enforcement

Prefixes: Logical Data Domains

Prefixes act like virtual folders within a bucket and are commonly used to organize data by purpose, lifecycle stage, business domain, or sensitivity.

Example:

s3://lakehouse/

  raw/

  refined/

  curated/

  ai-training/

  finance/

  healthcare/

In a data lakehouse, prefixes often align to:

• Medallion layers (Bronze, Silver, Gold)
• Business domains
• Data products
• Table locations
• Retention classes

Lifecycle policies, access controls, and analytics workflows can be applied at the prefix level.

Object Tags: Dataset-Level Metadata

Object tags provide key-value metadata that is maintained with object metadata.

Examples:

classification=pii

retention=7years

owner=finance

data_product=customer360

region=us

Tags enable policies based on the characteristics of the data rather than solely its location.

For example:

classification=pii

can trigger:

• Additional encryption requirements
• Longer retention periods
• Restricted access controls
• Governance workflows

Prefixes provide organizational structure, while tags provide business context.

Think of prefixes as answering:

"Where does this data belong?"

And tags as answering:

“What is this data?”

Together they provide a powerful governance framework.

Governance platforms such as watsonx.governance can use both structures to classify, catalog, and apply policies consistently.

Additionally, the effective use of prefixes and tags enables:

• Policy Based Lifecycle management
• Chargebacks and cost allocation
• Fine-grained capacity and growth reporting
• Improved data discovery
• More efficient event management

Event Notifications (Event‑Driven Pipelines)

How it works

S3 can emit events when objects are created, updated, or deleted, triggering downstream processing.

Value to the lakehouse

• Enables automated refinement pipelines
• Supports near‑real‑time ingestion
• Reduces manual orchestration

Simple example

New object → trigger Spark job → update Iceberg table

Tags and prefixes enable refined event processing based on metadata.

Data refinement happens automatically on arrival in the bucket.

Strong Consistency & Object Immutability

Modern S3 implementations provide read‑after‑write consistency and immutable object semantics.

Value to the lakehouse

• Safe concurrent access by multiple engines
• Required for transactional table formats
• Prevents partial or inconsistent reads

Simple example

Write completes → object immediately visible

Readers never see half‑written data.

Versioning (Historical State Preservation)

How it works

Object versioning preserves older versions of objects when they are updated or deleted.

Value to the lakehouse

• Protects against accidental deletes
• Supports audit and recovery
• Complements table‑level time travel

Simple example

metadata.json (v1, v2, v3)

Previous table states can be restored if needed

KV Cache Storage (Accelerating AI Inference and RAG Workloads)

How It Works

Large Language Models (LLMs) generate and consume Key-Value (KV) caches during inference. These caches store intermediate attention states that can be reused across requests rather than recomputed.

Traditionally, KV caches reside in GPU memory because of performance requirements. However, as models, context windows, and user sessions grow, GPU memory becomes an expensive and limited resource.

Modern AI architectures increasingly tier KV caches across:

GPU memory (hot cache)

System memory (warm cache)

High-performance object storage (cold cache)

In these architectures, S3-compatible storage provides durable persistence for cached context and inference state.

Value to the lakehouse

• Reading from S3 KV cache significantly reduces expensive GPU computation
• Persisting KV caches enables long-running interactions without forcing repeated model computation

For more information on KV Cache with Ceph, take a look at the article by Kyle Bader and Tushar Gohad 

Under Development: Vector Buckets (Embeddings and AI Features)

How it works

Vector buckets store high‑dimensional numeric arrays (embeddings) as objects, typically in Parquet or binary formats. These vectors are generated by AI models and reused for search, retrieval, and inference.

Value to the lakehouse

• Enables reuse of expensive AI feature generation
• Keeps embeddings close to source data
• Supports RAG and similarity search workflows

Simple example

s3://lakehouse/embeddings/

customer_vectors.parquet

AI pipelines read vectors directly without recomputation.

Summary: Why S3 Is the Right Foundation

S3‑compatible object storage provides more than cheap capacity. Its native features—when combined with lakehouse table formats and analytics engines—form the operational backbone of the data refinery:

• Files become tables
• Buckets become reservoirs
• Events become pipelines
• Policies become governance

Together, these capabilities allow a data lakehouse to operate at scale with correctness, efficiency, and long‑term cost control.

You can find part 6 here.