Open Source Offerings

 View Only

Announcing IBM Big Replicate LiveData Migrator 1.8.3

By Charlie Park posted Tue December 15, 2020 05:05 PM



Announcing IBM Big Replicate LiveData Migrator 1.8.3


I am very excited to announce the release of IBM Big Replicate LiveData Migrator 1.8.3


Big Replicate LiveData Migrator migrates HDFS data to other Hadoop-compatible file systems and cloud storage, including ongoing changes made to those data before, throughout, and after migration. This data solution is purpose-built to migrate data lake content easily and simply for architects, administrators, and operators. It is immediate, live, and scalable. Organizations can manage their migrations through a robust and intuitive command-line interface or from the self-documenting REST API, track and monitor migration progress, and take advantage of optional management capabilities in the browser-based user interface.


Differentiating factors of Big Replicate LiveData Migrator:

  • Hadoop and object storage. Works across a variety of big data source and target environments, including all major Hadoop and object storage technologies.
  • Petabyte scale. Migrates big data sets at scale to cloud storage and eliminates the need to halt changes made to the data sets during migration.
  • Selective migration. Provides flexible options that enable users to select only those data sets that they want to migrate. Administrators can define replication policies that identify and control the data that is replicated between clusters, selectively exclude data from migration to specific clusters in the new environment, or move the data to an archive.
  • One pass. Migrates existing data sets with a single pass through the source storage system, eliminating the overhead of repeated scans.
  • Consensus-based consistency. Incorporates changes that are made to the data sets under migration with a strongly consistent distributed consensus engine, enabling applications to continue to modify the donor system's data without causing divergence between donor and beneficiary.
  • Strengthened consistency. As changes occur anywhere in the donor system, LiveData Migrator creates and helps ensure beneficiary data consistency.


Bandwidth Management

Users now have the ability to control the amount of bandwidth consumed by LiveData Migrator during operation by defining a bandwidth policy that is applied across all migrations. This allows users to control and manage the bandwidth usage for LiveData Migrator when sharing the bandwidth with other applications with limited network capacity.


ADLSGen2 Service principal support

Service principals are now supported as an additional authentication mechanism for ADLSgen2 storages along with previously supported Access/Secret keys. This allows for more secure options to connect and migrate your data into ADLSgen2 storages.


Stop and Resume

This release now allows users to stop and resume their individual migrations. The user will be able to resume a migration if the events are within the buffer limit. This is configurable and details can be found in the LiveData Migrator Documentation.


  • The Stop and Resume functionality is only available via CLI


[Preview] Hive Migration

Hive migration is now available for preview in this release of LiveData Migrator. Hive metadata migration is now possible with limited platform support and functionality. The deployment and operation is available via the new ‘Remote CLI’ that is included with your download of LiveData Migrator. Limited migration functionality via the UI is also available. Please refer to the documentation for a detailed overview of supported operation and functionality.

  • IMPORTANT: This feature is not recommended for production.


Supported Hive Migrator Metastores

  • HDFS Apache Hive
  • Azure HDI 3.6/4.0 Hive Internal Metastore
  • Azure HDI 3.6/4.0 Hive External Metastore
    • Azure SQL DataBase

[Preview] Remote CLI

A new CLI is now available which includes all the familiar functionality of the embedded CLI along with operation of various new preview features like Hive migrations.

Detailed release notes can be found in the Wandisco community page
Technical documentation can be found in Wandisco Docs page
This new release of Big Replicate offerings is available for download from Passport Advantage and Passport Advantage Express website.
Learn more by visiting US.