Connect with HPC subject matter experts and discuss how hybrid cloud HPC Solutions from IBM meet today's business needs.
Originally posted by: TingXue
Delta Lake is an open source storage layer that brings reliability to data lakes. It’s not included with IBM Spectrum Conductor 2.4.0; however, you can integrate the two. Here's is an example of adding Delta Lake 0.4.0 to an existing IBM Spectrum Conductor 2.4 instance group called SIG243.
data = spark.range(0, 5)
data.write.format("delta").save("<shared location>/delta-table")
df = spark.read.format("delta").load("<shared location>//delta-table")
df.show()
This code creates a Delta Lake table, reads from the table, and displays the table. If it run successfully, it displays an empty table without any errors, like this:
And it's that simple! Enjoy the integration and example.
As always, let us know what you think using our Slack channel!
Copy