Dear Sir/Madam,

We would like to use DataPower to transfer a very large file (about 60GB) from a Window machine folder to a Linux machine folder. We currently create a multiple gateway service with settings below

1. FTP Poller Handler that use ftps connection to poll file from Window landing zone folder

2. The processing rule is just transfer file using sftp connection to Linux landing zone folder

3. Use streaming mode by setting Flow control to on and setting Stream Output to Back and Stream Output to Front as Stream Messages

We tested a 62GB file in our UAT env with 100GB installed memory. It could finish in 1 hour. But I noticed maximum memory usage 33%.

We would like to know what is the best way to do this big file (about 60GB) transfer.

Thanks,

Tang

Hi Tang,

if the memory usage is the issue, you can minimize it by making sure, that your processing policy doesn't actually "process" the payload. Use of passthrough request and response types is an easy way to achieve this. Also, is you are forced to actually implement a processing policy use NULL and PIPE contexts whenever it is possible to minimize the memory consumption. For example, we have a MPGW that moves files as large as 10 GBs and the memory consumption is measured in megabytes not gigabytes.

--HP

Hi HP,

Thanks a lot for your answering my question.

Yesterday I changed to use NULL as input context for transform step to get backend url. But in restults step I use INPUT as input context. I tested 7 times with same 63GB big file. Only 1 times was successful. All other 5 tests failed with following errors.

20220414T024305.216Z [0x80e006c1][network][error] mpgw(PhoneRecordPoller): tid(603061) gtid(85e646e362577774000933b5): Error in stream. Descriptor number unavailable. Additional information: Internal error (4)

20220414T024305.216Z [0x80c00008][multistep][error] mpgw(PhoneRecordPoller): tid(603061)[request] gtid(85e646e362577774000933b5): rule (PhoneRecord-Route_rule_0): Implied action Passing input through without processing. failed: Read error

20220414T024305.216Z [0x00530002][mpgw][error] mpgw(PhoneRecordPoller): tid(603061)[error] gtid(85e646e362577774000933b5): Read error

I don't understand what what these errors mean.

You are right. When I watched memory use. Its maximum memory usage is just 19%

Actually before I changed to use NULL as input context in transform to get backend url. I also did several times tests. Only one was successful although I restarted device. All other tests are failed with same errors as above.

Thanks,

Tang

The best way to handle extremely large files (multi-gigabyte request/response) is to stream it. It really means you can't do any processing other than just passing from front to back in DataPower.

See https://www.ibm.com/docs/en/datapower-gateway/10.0.1?topic=streaming-messages-through-datapower-services

Challenge is it means any dp extension function calls are out the window. All processing actions are susceptible to 'breaking' your streaming.

You can also use a log target with a event subscription of memory-report category at debug level to track a transactions memory usage in a processing policy as it passes through actions.

Hi HP,

Finally we found the issue. It is the windows ftp server that our MPGW polls. When I changed to use other Linux machine ftp server, we could transfer 60GB file in about 1 hour and no problem.

Thanks a lot for your help.

Tang

Np. Good to hear you solved your issue.

--HP