Hi

I am looking for performance/be-awares etc for anyone who may have moved a significant volume (+ 100 million docs) from NetApps ONTAP storage to ECS, however any source storage to ECS comments would be ideal.

Any help notes on throughput/approach would be grateful.

Hi,

What is ECS?

My team has done a move like this several times. The sweeps work well, it's just a little hard to track where you are at in the process. Be methodical and create sweeps for definable chunks of the docs you need to move so that you can track progress. We have also found that running multiple sweeps simultaneously that select portions of what need to be moved is better performing than running one sweep that selects all of it.

I would be happy to have more conversation if you like.

Hi. ECS is Dell Elastic Cloud Storage (was Centera), it is like WORM storage in a box, where you can lock your content away for x years.

With the Sweep moves that you performed, what was the volume/av doc size/throughput speed and what was the source/destination storage devices?

We have the same device. We have done moves from: NetApp to NetApp, NetApp to Isilon, Isilon to ECS, ECS to Azure and Isilon to Azure. We have moved 80TB during one project. I would need to go back and look at some notes to grab stats

Honestly, the type of device is less of an issue, many other factors come into play. We are currently moving 175 mill docs from File Storage to ICOS on-premise and are 80% through.

How are you attaching the ECS (FCD, S3 Advanced Storage Area, generic S3)?

If you take retention serious, IMHO FCD in aligned mode is your only choice... and that is what I insist with all my customers.

If FCD you have intermediate steps like writing to the staging area, inserting a record in the content queue table, then you have the content queue processor that is difficult to control without knowing internals, the the documents gets shuffled around in the staging area, then written to the ECS, deleted from staging area, then the record is deleted.

Given that, expect 30 docs/sec with larger documents (MBs) to 60 docs/sec smaller documents with initial parameters without tuning.

With tuning you can go up to 80-120 docs/sec up to a point where production is impacted. The content queue processor is a single host component and therefore is subject to limited scaling.

In theory (and sometimes in practice) you could temporarily add CPEs until the migration is done, but you cannot control where the queue processor runs (albeit it seems to have an affinity to the CPE where the sweep definition was done).

Hope this helps,

/Gerold

Hi

I had copied content before via Bulk content sweeps. This new customer still has content in File Net IS using (NetApps) attached and wants to copy to CPE using ECS bucket. I had estimated 3mill for a 20hr day (doc size 40kb) so your useful info here gives me confidence that I am in that correct ballpark, but it could perhaps increase by maybe 50% with some tuning/multiple CE's.

Appreciate your feedback.

You should be fine with you estimate., IS for sure won't be the bottleneck..

BTW: I was wrong in my reply, the host where the sweep job runs cannot be controlled , not the queue processor (that is even more mystery)

You can tune with threads and batch sizes and some other, but adding CPEs might help you with production performance, but not with the migration. As I said theese are bound to a single host...

If by 'ECS bucket' you mean a connection via generic S3 device or S3 Advanced Storage (which I would then recommend) your peformance will be better, as there is no staging area mumbling and fumbling... and the writes will be synchronous.

But before you cannect like that that make sure the customer is aware of the limitations concerning retention lateron.

/Gerold