Introduction:
Organizations with established Cloudera deployments often face a modernization challenge: how to adopt lakehouse capabilities such as open table formats, decoupled storage and compute, and multi-engine analytics without disrupting existing workloads.
Rather than choosing between maintaining Cloudera or migrating entirely to a new platform, organizations can adopt a coexistence approach. By deploying watsonx.data on IBM Fusion alongside Cloudera and using IBM Storage Ceph as a shared object storage layer, both platforms can access the same Apache Iceberg data.
-
Cloudera continues to run existing workloads using the S3A connector.
-
IBM Storage Ceph provides the shared S3-compatible storage layer.
-
watsonx.data on IBM Fusion accesses the same Iceberg tables stored in IBM Storage Ceph bucket through its Iceberg catalog and query engines.
-
This approach allows organizations to introduce lakehouse capabilities incrementally while preserving existing data pipelines and operational processes.
What you will learn:
In this blog, you'll learn how to:
-
Connect Cloudera to IBM Storage Ceph using the S3A connector.
-
Promote Apache Iceberg tables from HDFS to IBM Storage Ceph object storage.
-
Register those tables in watsonx.data on IBM Fusion.
-
Enable simultaneous access from both Cloudera and watsonx.data without maintaining duplicate datasets.
By the end of this guide, you'll have a shared lakehouse architecture where Cloudera and watsonx.data can access the same Iceberg tables stored in IBM Storage Ceph.
IBM Fusion and watsonx.data - What They Are:
IBM Fusion is IBM's integrated infrastructure software platform designed to simplify the deployment and management of data-intensive workloads on premises and in hybrid cloud environments. Its native storage services are Fusion Data Foundation (container-native Ceph, for Kubernetes workload persistence) and IBM Storage Scale (high-performance file and object storage for AI/HPC). IBM Storage Ceph is a separate enterprise object store — deployed independently on its own bare-metal nodes — that serves as the shared data lake storage tier accessible by both watsonx.data and Cloudera over the S3 API.
watsonx.data is IBM's open lakehouse platform. It provides multi-engine query access (Presto, Spark, and others) to data stored in open formats such as Apache Iceberg and Apache Parquet — without locking data into a proprietary store. watsonx.data connects to IBM Storage Ceph as an external S3 object store for data lake storage — which is precisely what makes the coexistence architecture possible: IBM Storage Ceph is not just a generic S3-compatible store that both platforms happen to support — it is the *shared foundation* of the whole stack.
Why IBM Storage Ceph as the bridge?
- S3-Compatible: Both Cloudera (via S3A) and watsonx.data access the same `s3a://` URIs — no translation layer needed.
- Independently deployed: IBM Storage Ceph runs on its own bare-metal nodes outside OCP, so its lifecycle, scaling, and data are fully decoupled from both IBM Fusion and Cloudera.
- On-Premises & Hybrid: Runs in your own data center to meet data residency and compliance requirements.
- Cost-effective: Built on commodity hardware and open-source software, reducing upfront infrastructure costs compared to proprietary alternatives.
- Scalable: Scales out by adding nodes; suitable for petabyte-scale data lakes.
- Highly Available: Replicates data across nodes and failure domains for resilience without manual intervention.
- High Performance: Enables parallel data access through native integration with Apache Hadoop and Apache Spark. Provides a cache accelerator (D3N) and server-side query pushdown (S3 Select) to reduce data movement and accelerate analytical workloads.
- Data Governance Ready: Manages metadata at scale to enforce governance policies, track data lineage, and monitor data usage across everything stored in the lake.
Architecture Overview:
The diagram below shows how the three components fit together. IBM Storage Ceph — deployed independently on dedicated bare-metal nodes alongside IBM Fusion — is the central shared storage tier. Cloudera accesses it via the S3A protocol. watsonx.data on IBM Fusion accesses the same storage via its registered S3 catalog. An Iceberg table promoted to IBM Storage Ceph is immediately visible to both platforms — no synchronization, no replication, no duplicate copies.
Note: After promoting a table to Ceph, Cloudera and watsonx.data are reading the same physical Parquet and metadata files. There is no ETL pipeline between them, no scheduled sync, and no second copy of the data.
Prerequisites:
Before you begin, make sure you have the following ready:
- IBM Storage Ceph Endpoint : The URL of your IBM Storage Ceph Object Gateway (RGW), e.g., https://ceph.example.com:443
- Access Key : Your S3-compatible access key ID for authentication
- Secret Key : Your S3-compatible secret access key for authentication
- Bucket Name : The name of the bucket in IBM Storage Ceph where you will store or promote your Iceberg data
- Cloudera Cluster : A running Cloudera CDP or CDH cluster with the HDFS and Spark services available
- Admin Access : Cloudera Manager admin access (for CDP) or SSH access to cluster nodes (for direct `core-site.xml` edits)
- watsonx.data Instance : A running watsonx.data instance on IBM Fusion with an Iceberg catalog configured and associated to the target Ceph bucket
Note: IBM Storage Ceph credentials are created and managed through the Ceph Object Gateway (RGW) dashboard or CLI. If you do not have credentials, contact your Ceph administrator.
Step 1: Configure Cloudera to Connect to IBM Storage Ceph
Cloudera uses the S3A filesystem connector (part of Apache Hadoop) to communicate with S3-compatible object stores. IBM Storage Ceph exposes an S3-compatible API through its Object Gateway (RGW), making it a drop-in target for the S3A connector.
The configuration is done through a set of Hadoop properties, either in `core-site.xml` directly or via Cloudera Manager. Once set, any service on the cluster — Spark, Hive, HDFS shell commands — can access IBM Storage Ceph using the `s3a://` URI scheme.
Option 1. Configure via Cloudera Manager
If you are running Cloudera Data Platform (CDP) with Cloudera Manager, follow these steps to apply the configuration without manually editing files on each node:
1. Open Cloudera Manager and navigate to the HDFS service.
2. Go to the Configuration tab.
3. In the search box, type "Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml".
4. Click the field and paste in the five `<property>` blocks from the Required S3A Properties section below.
5. Click Save Changes.
6. Restart the HDFS service (and any other affected services such as Spark, Hive, or YARN) when prompted.
Option 2. Configure via Manual core-site.xml Editing
If you are managing a generic CDH cluster without Cloudera Manager, you will need to edit `core-site.xml` directly on each node:
1. SSH into a cluster node (repeat for all nodes)
2. Open the Hadoop configuration file:
```bash
sudo vi /etc/hadoop/conf/core-site.xml
```
3. Add the five `<property>` blocks from the Required S3A Properties section inside the existing `<configuration>` element.
4. Save the file and repeat on all cluster nodes (name nodes, data nodes, edge nodes).
5. Restart HDFS services or any other affected services:
```bash
# systemd-managed clusters
sudo systemctl restart hadoop-hdfs-namenode hadoop-hdfs-datanode
# init.d-managed clusters
sudo service hadoop-hdfs-namenode restart && sudo service hadoop-hdfs-datanode restart
```
Also restart Spark, Hive, or YARN if they are running on the same nodes.
Required S3A properties
The following properties must be configured. Replace each placeholder with the values from your IBM Storage Ceph environment.
<!-- Host and port of your IBM Storage Ceph Object Gateway (RGW). -->
<property>
<name>fs.s3a.endpoint</name>
<value><YOUR_CEPH_HOST>:<PORT></value>
</property>
<!-- Set to true if your RGW endpoint uses HTTPS. -->
<property>
<name>fs.s3a.connection.ssl.enabled</name>
<value>true</value>
</property>
<!-- Your S3-compatible access key ID -->
<property>
<name>fs.s3a.access.key</name>
<value><YOUR_ACCESS_KEY></value>
</property>
<!-- Your S3-compatible secret access key -->
<property>
<name>fs.s3a.secret.key</name>
<value><YOUR_SECRET_KEY></value>
</property>
<!-- Use path-style URLs (ceph-host/bucket/key). Required unless Ceph RGW is configured with rgw_dns_name and a wildcard DNS record for virtual-hosted-style access. -->
<property>
<name>fs.s3a.path.style.access</name>
<value>true</value>
</property>
<!-- The S3A filesystem implementation class.-->
<property>
<name>fs.s3a.impl</name>
<value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
</property>
<!-- Only needed when ALL of the following apply:
- fs.s3a.connection.ssl.enabled=true (no effect over plain HTTP)
- Java 8 (default_jsse strips GCM ciphers for performance, HADOOP-15669)
- Ceph RGW offers only GCM ciphers (e.g. ECDHE-RSA-AES256-GCM-SHA384),
causing a silent TLS handshake failure
Safe to omit on Java 9+, HTTP endpoints, or when RGW offers non-GCM suites. -->
<property>
<name>fs.s3a.ssl.channel.mode</name>
<value>default_jsse_with_gcm</value>
</property>
Step 2: Validate the Connection to IBM Ceph
Once the configuration is saved and services are restarted, run a quick connectivity check to confirm Cloudera can read from and write to IBM Ceph before moving on.
Test 1: List bucket contents with the Hadoop shell
From any node, run:
```bash
hadoop fs -ls s3a://<YOUR_BUCKET_NAME>/
```
A successful response will list the contents of your bucket (or show an empty directory if the bucket is empty). Any authentication or connectivity error will appear here immediately, making this the fastest way to confirm the S3A configuration is working.
Test 2: Read and Write a small file with Spark
From a PySpark shell on any cluster node:
spark.createDataFrame([(1, "hello"), (2, "ceph")], ["id", "message"]) \
.write.mode("overwrite").parquet("s3a://<YOUR_BUCKET_NAME>/connectivity-test/")
spark.read.parquet("s3a://<YOUR_BUCKET_NAME>/connectivity-test/").show()
If `show()` prints the two rows, the Spark → S3A → IBM Ceph pipeline is fully working.
Note: This relies on Spark inheriting the S3A settings from `core-site.xml` configured in Step 1. If you are running Spark in client mode outside the cluster, pass the credentials explicitly via `spark.conf.set("spark.hadoop.fs.s3a.*", ...)` before running the above.
Step 3: Promote Apache Iceberg Tables from HDFS to Shared Ceph Storage
With Cloudera connected to IBM Storage Ceph, the next step is to promote existing Iceberg tables from HDFS onto the shared Ceph storage tier. After this step, the table lives on IBM Storage Ceph and is accessible from *both* Cloudera and watsonx.data simultaneously. Cloudera's existing jobs continue running unchanged — they simply read and write to `s3a://` instead of `hdfs://`. watsonx.data gains immediate access to the same table through its own query engines, with no data duplication and no cutover window required.
Step 3.1: Copy the Table Directory to IBM Storage Ceph
The promotion follows sub-steps before anything is uploaded: pull from HDFS to a staging location, rewrite `.metadata.json` paths, rewrite `.avro` manifest paths, push to Ceph, then clean up local staging. This order ensures only clean, `s3a://`-correct files ever land on object storage and that time travel to all historical snapshots works from day one.
Set below shell variables before you start. Every command below uses them — you will not need to edit individual commands.
```bash
# --- HDFS source ---
HDFS_NAMENODE="namenode.example.com" # hostname of the HDFS NameNode
HDFS_TABLE_PATH="/warehouse/mydb/orders" # path to the table root in HDFS (no trailing slash)
# --- Ceph destination ---
CEPH_ENDPOINT="https://s3.example.com" # your Ceph S3 endpoint URL
CEPH_BUCKET="my-ceph-bucket" # destination bucket name
DEST_PATH="warehouse/mydb" # key prefix inside the bucket (no leading/trailing slash)
# --- Staging ---
TABLE_NAME="orders" # table name — also used as the local folder name
STAGING_DIR="/tmp/iceberg-staging" # writable directory with enough free space
```
Set `STAGING_DIR` to wherever suits your environment — `/tmp/iceberg-staging` works on most Linux nodes and macOS laptops alike.
Step 3.1.a: Pull from HDFS to local staging
Run this from any node with the `hdfs` CLI:
```bash
mkdir -p "$STAGING_DIR/$TABLE_NAME"
hdfs dfs -get "hdfs://$HDFS_NAMENODE$HDFS_TABLE_PATH/data" "$STAGING_DIR/$TABLE_NAME/data"
hdfs dfs -get "hdfs://$HDFS_NAMENODE$HDFS_TABLE_PATH/metadata" "$STAGING_DIR/$TABLE_NAME/metadata"
```
Verify both directories are present:
```bash
ls "$STAGING_DIR/$TABLE_NAME/"
# Expected: data/ metadata/
```
**Optional — copy the staging directory to your local machine:**
```bash
# Run this on your local machine
mkdir -p "$STAGING_DIR"
scp -r user@node:"$STAGING_DIR/$TABLE_NAME" "$STAGING_DIR/"
```
Then set the same `STAGING_DIR` variable in your local shell and continue with steps 2.1b–2.1d there.
Step 3.1.b: Rewrite .metadata.json paths
The `.metadata.json` files embed absolute HDFS paths in multiple fields — `"location"`, `"manifest-list"` (per snapshot), and `"metadata-file"` (in the metadata log). All must be updated to `s3a://` before upload. A single `sed` global replacement covers all fields in one pass.
1. First, preview what will be replaced:
```bash
grep -r "hdfs://" "$STAGING_DIR/$TABLE_NAME/metadata/"*.metadata.json
```
2. Rewrite all `.metadata.json` files in place:
```bash
# Linux (GNU sed)
sed -i \
"s|hdfs://$HDFS_NAMENODE$HDFS_TABLE_PATH|s3a://$CEPH_BUCKET/$DEST_PATH/$TABLE_NAME|g" \
"$STAGING_DIR/$TABLE_NAME/metadata/"*.metadata.json
# macOS (BSD sed) — note the empty string '' after -i
sed -i '' \ "s|hdfs://$HDFS_NAMENODE$HDFS_TABLE_PATH|s3a://$CEPH_BUCKET/$DEST_PATH/$TABLE_NAME|g" \
"$STAGING_DIR/$TABLE_NAME/metadata/"*.metadata.json
```
3. Verify no `hdfs://` references remain:
```bash
grep -r "hdfs://" "$STAGING_DIR/$TABLE_NAME/metadata/"*.metadata.json
# Expected: no output
```
Important: Do not proceed until this check returns no output. If any `hdfs://` paths remain, Iceberg will attempt to read data from HDFS instead of Ceph.
Tip: Take note of the most recent`.metadata.json` file (the one with the highest sequence number, e.g., `00002-def456.metadata.json`). You will need this file path in Step 2.3.
Step 3.1.c: Rewrite .avro manifest paths locally
The `.avro` manifest files also embed `hdfs://` paths in their binary Avro records — both the per-data-file paths in manifest files (`*-m0.avro`) and the per-manifest paths in manifest list files (`snap-*.avro`). These cannot be fixed with `sed` because:
- Avro strings are length-prefixed in binary — a raw byte replacement with a different-length string corrupts the encoding
- The `manifest_length` field in `snap-*.avro` records stores the byte size of the corresponding `*-m0.avro` file — this must also be updated after reserializing
Use the provided `patch_iceberg_avro.py` (download it from https://github.com/IBM/watsonx-data/blob/main/cloudera_augmentation/patch_iceberg_avro.py) script which handles both correctly:
```bash
pip install fastavro # one-time
python3 patch_iceberg_avro.py \
--table-dir "$STAGING_DIR/$TABLE_NAME" \
--old-prefix "hdfs://$HDFS_NAMENODE$HDFS_TABLE_PATH" \
--new-prefix "s3a://$CEPH_BUCKET/$DEST_PATH/$TABLE_NAME"
```
The script processes files in the correct order — manifest files (`*-m0.avro`) first, then manifest list files (`snap-*.avro`) — so that updated byte sizes are always known before `manifest_length` is written.
Step 3.1.d: Push from local staging to IBM Ceph
```bash
aws --endpoint-url "$CEPH_ENDPOINT" s3 cp \
"$STAGING_DIR/$TABLE_NAME/" \
"s3://$CEPH_BUCKET/$DEST_PATH/$TABLE_NAME/" \
--recursive \
--no-guess-mime-type
```
Verify the copy landed correctly on Ceph:
```bash
aws --endpoint-url "$CEPH_ENDPOINT" s3 ls "s3://$CEPH_BUCKET/$DEST_PATH/$TABLE_NAME/"
# Expected:
# PRE data/
# PRE metadata/
```
Step 3.1.e: Clean up local staging
```bash
rm -rf "$STAGING_DIR/$TABLE_NAME/"
```
Summary: what gets rewritten and why
*.metadata.json`, `*-m0.avro`, `snap-*.avro All three must be rewritten locally before upload. Skipping the `.avro` rewrite means Iceberg will attempt to read historical snapshot data from HDFS — reads succeed only while HDFS is reachable, and fail entirely when accessed from watsonx.data or after HDFS is decommissioned.
Step 3.2: Register the Table in Spark’s Iceberg Catalog
Use the Iceberg `register_table` procedure in a PySpark session to register the table in your target catalog. This tells Iceberg where to find the table's metadata on IBM Storage Ceph — no data is moved or duplicated in this step.
spark.sql(f"""
CALL {target_catalog}.system.register_table(
table => '{table_name}',
metadata_file => '{metadata_file_path}'
)
""")
Where:
- `target_catalog` — name of your target Iceberg catalog (configured in Spark/Hive Metastore)
- `table_name` — fully qualified table name in the form `schema.table`
- `metadata_file_path` — full `s3a://` path to the most recent `.metadata.json` file in the table's `metadata/` directory (the rewritten file with `s3a://` paths, not the original HDFS version)
Key points about `register_table`:
- It registers an existing Iceberg table from its metadata — it does not create new data or copy anything.
- The `metadata_file` parameter must point to the most recent `.metadata.json` file, not a directory.
- After registration, the table is fully managed by the target catalog and can be queried and written to like any other Iceberg table.
Step 3.3: Verify the Promotion from Cloudera
After registration, run a few quick checks to confirm the table is healthy and data is accessible from IBM Ceph via Cloudera:
# Check the table is visible in the catalog
spark.sql(f"SHOW TABLES IN {target_catalog}.<YOUR_SCHEMA_NAME>").show()
# Verify row count matches the original HDFS table
spark.sql(f"SELECT COUNT(*) FROM {target_catalog}.<YOUR_SCHEMA_NAME>.<YOUR_TABLE_NAME>").show()
# Sample a few rows to verify data integrity
spark.sql(f"SELECT * FROM {target_catalog}.<YOUR_SCHEMA_NAME>.<YOUR_TABLE_NAME> LIMIT 10").show()
If the row count matches your original table and the sample rows look correct, the table is now live on shared Ceph storage and Cloudera can query it at full speed. Leave the original HDFS copy in place until you are confident to remove.
Step 3.4: Rename the Table (Optional)
If you want existing applications to continue working without any code changes, you can perform a rename swap: rename the old table to a backup name, then rename the new Iceberg table to the original name.
spark.sql("ALTER TABLE <previous_table_name> RENAME TO <previous_table_name_backup>")
spark.sql("ALTER TABLE <table_name> RENAME TO <previous_table_name>")
Keep the backup table in place until you are fully confident the new table is stable. Note that dropping the backup table will also delete its underlying HDFS data files.
Step 4: Register the Table in watsonx.data
This is the step that completes the modernization story. The Iceberg table is now on IBM Storage Ceph. Registering the table in the watsonx.data Iceberg catalog makes it immediately queryable from watsonx.data's Presto, Spark engines, with no data movement and no impact on the Cloudera workloads already running against the same files.
Prerequisite: Your watsonx.data instance must have an Iceberg catalog configured and associated to the same IBM Ceph bucket where the table was promoted in Step 2. Contact your IBM Fusion administrator if this is not yet set up.
Appraoch 1: watsonx.data Catalog Sync
The simplest way to make a table visible in watsonx.data is to use the Sync Metadata option on the Iceberg catalog associated with the target Ceph bucket. watsonx.data scans the bucket, discovers the Iceberg metadata, and registers all tables automatically — no code required.
1. In the watsonx.data UI, navigate to Infrastructure Manager → Catalogs.
2. Locate the Iceberg catalog connected to your target Ceph bucket.
3. Click the catalog's action menu and select Sync Metadata.
4. Wait for the sync to complete, then browse to Data Manager to confirm the table appears under the correct schema.
This approach is recommended when you are promoting multiple tables at once, as a single sync operation discovers and registers all of them.
Approach 2: Register via MDS Iceberg REST API
For programmatic or automated registration, use the watsonx.data Metadata Service (MDS) Iceberg REST API. This is the right choice for CI/CD pipelines or scripted promotion workflows.
```bash
curl --request POST \
--url "https://<MDS_REST_ENDPOINT>/mds/iceberg/v1/<TARGET_CATALOG_NAME>/namespaces/<SCHEMA_NAME>/register" \
--header "Authorization: Bearer <YOUR_TOKEN>" \
--header "Content-Type: application/json" \
--header "accept: application/json" \
--data '{
"name": "<TABLE_NAME>",
"metadata-location": "s3a://<YOUR_BUCKET_NAME>/<path_to_table>/metadata/<latest_metadata_file>.metadata.json"
}'
```
Replace the placeholders:
- `<MDS_REST_ENDPOINT>` — the MDS endpoint of your watsonx.data instance
- `<TARGET_CATALOG_NAME>` — the name of the Iceberg catalog in watsonx.data associated with your Ceph bucket
- `<SCHEMA_NAME>` — the schema (namespace) where the table should appear
- `<TABLE_NAME>` — the name to register the table under
- `<latest_metadata_file>` — the most recent `.metadata.json` file identified in Step 2.1b
Note: The schema must already exist in the target catalog before calling this endpoint. Create it in the watsonx.data UI or via the catalog API if needed.
Approach 3: Register via Spark in watsonx.data
If you prefer to stay within a PySpark workflow, you can register the table using the Iceberg `register_table` procedure executed against the watsonx.data Spark engine. This is the same procedure used in Step 2.3, but now targeting the watsonx.data catalog instead of the Cloudera catalog.
spark.sql(f"""
CALL {target_catalog}.system.register_table(
table => '{table_name}',
metadata_file => '{metadata_file_path}'
)
""")
Verify Dual Access
Once registered, confirm the table is accessible from watsonx.data by running a query in the watsonx.data query editor:
```sql
-- In watsonx.data query editor (Presto/Spark engine)
SELECT COUNT(*) FROM <YOUR_WATSONX_ICEBERG_CATALOG_NAME>.<YOUR_SCHEMA_NAME>.<YOUR_TABLE_NAME>;
```
The row count should match what you saw in Step 2.4 from Cloudera. At this point, both platforms are reading the same physical files on IBM Ceph — the coexistence architecture is fully operational.
Note: this exercise has been verified on the following software versions:
Cloudera Runtime (CDP Private Cloud Base)
-
Runtime: 7.1.9
- Parcel version: CDH-7.1.9-1.cdh7.1.9.p1042.65851740
Watsonx.data on IBM Cloud Pak for Data
Conclusion
In this guide, you configured Cloudera to access IBM Storage Ceph object storage using the S3A connector, promoted existing Apache Iceberg tables from HDFS to IBM Storage Ceph by updating Iceberg metadata and manifest references, and registered those tables in both Cloudera and watsonx.data catalogs. The result is a shared storage architecture where both platforms can access the same Iceberg tables stored on IBM Storage Ceph through standard S3 interfaces.
By using IBM Storage Ceph as a common object storage foundation, organizations can introduce watsonx.data on IBM Fusion alongside existing Cloudera environments without requiring a wholesale migration of data or workloads. Existing data pipelines and operational processes can continue to run in Cloudera, while analytics, AI, and lakehouse workloads gain access to the same datasets through watsonx.data's open query engines.
This coexistence approach enables a phased modernization strategy that reduces migration risk, avoids maintaining separate copies of data, and leverages open standards such as Apache Iceberg and S3-compatible object storage. As additional tables are promoted to IBM Storage Ceph, the shared lakehouse can expand incrementally, providing a scalable foundation for analytics, data engineering, and AI initiatives while preserving the value of existing Cloudera investments.
#watsonx.data
#Cloudera
#StorageCeph
#Bucket