Earlier this year, I collaborated with IBM to migrate 20 petabyes of Cloud Object Storage data from one bucket to another and from COS Gen 1 (IaaS) to COS Gen 2 (PaaS).  To my knowledge no one else has moved this much data at one time before.  My team expected this to take 6-8 months, but those were just guesses based on a previous experience in a very different environment.  After moving 20 PB of data in under 2 months between 2 COS buckets, IBM invited me to present this project with IBM's @Pierre Carlson at the IBM TechXchange 2023 conference.  We presented and had a blast at the conference.  Because not everyone was able to make the session, I wrote a LinkedIn Article summarizing my presentation.  I invite you to take a read through and come up with some questions and comments.

In the future, I intend to blog about some of the specifics of the rclone parameters we used and sharing parts of my script that managed the migration.  Automation and prefixes are the keys to success here.