Originally posted by: rarw

Hi,

We have an 879 MB file that fails on the ftp with "FTP Transfer Aborted Due to Timeout & socket error 32 occurred (Broken pipe)" error messages (zero lines sent) (~3 minutes). (FTP trace is included below, edited for server and file location...)

We are ftp'ing from a Sun, Solaris 10 box to a mainframe. If I run the same maps on the Sun box, and ftp to itself, there is no problem, the ftp is successful (takes about 5 minutes).

The mainframe has a time out of 600 seconds when there is no activity. Transfers themselves can take longer.

WTx 8.1.0.2 (build 38), Command server, FTP adapter

Thanks in advance for your help - Becky

FTP Trace:

FTP Adapter, Version 8.1(38)
Copyright © 2004, IBM Corporation. All rights reserved
Built for SUN on Jul 16 2007 at 11:50:05.
Retry Count is 0. Retry Interval is 0. OnFailure is Rollback.
Fetch Unit is 0. Card Mode is Integral.
Will process 922096092 bytes of data.
Run Started at 02:31:36.858 on 02/28/09.
Options used: trace,name,acct,url
URL FTP://<account>:*****@<server><file location>/BDCH01S;type=ASCII will be used.
WSAStart: Entering
WSAStart: bWSStarted = 0, nWSClients = 0
Sockets Client ID 1 enabled.
WSAStart: Exiting (rc = 1)
Using ftp: protocol.
Server host is '<server>'.
Using port '21' for the connection.
Connecting to '<server>'.
Socket Opened.
Connected.
>220 Service Ready For New User, Please Log In. (S)
<USER <account>

>331 Please Enter 'Password'. (S)
<PASS
>332 Please Enter 'Account/SCC'. (S)
<ACCT <account>

>230 User Logged In; Proceed. (S)
<SYST

>215 GCOS 8 FTP Server version 8UA4.0FT update 5.3 Aug 6 2008 09:49:14. (S)
No system-specific parameters set.
<CWD <location>

>250 Current Working Directory is '<location>'. (S)
Using <location>/ftpBAA8daWVt as tempfile.
Wrote 922096092 bytes to <location>/ftpBAA8daWVt.
Rollback specified; returning to map engine.
Cleanup call from prior Rollback; re-initializing instance storage.
Map return is 0.
<TYPE A

>200 Type set to A; form set to N. (S)
<PASV

>227 Entering Passive Mode (<IP>). (S)
Using port '61362' for the connection.
Connecting to '<IP>'.
Socket Opened.
Connected.
<STOR BDCH01S

>150 ASCII Data Connection for '<location>/BDCH01S'. (S)
WSASend: Socket error 32 occurred (Broken pipe).
(0 lines sent)
Socket Closed.
Connection closed.
<ABOR

>426 Transfer Aborted Due to Timeout, 0 Bytes Transferred. (F)
Data in <location>/ftpBAA8daWVt deleted.
Run Terminated at 02:33:58.017 on 02/28/09.
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender
#DataExchange

Originally posted by: phil2030

I am not a mainframe person but, have you tried FTPing without staging the file? Looking at your trace it looks like it wrote all the bytes to the staging area. I take it you've checked permissions, disk space, character sets(?), etc.
#DataExchange
#IBMSterlingTransformationExtender
#IBM-Websphere-Transformation-Extender

Originally posted by: rarw

Is there a methood of turning off the creation of the temp file in the ftp adapter so that it is not created? The failure appears to occur after the connection to the remote server is made and the temp file has been created (~1.5 min).

Also, can the ftp adapter be run manually? Do you know where the executable is located? We looked - could not find it - even doing a ps -ef while the map was running.

Thanks,
Becky
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: phil2030

The FTP Client is embedded in the adapter (and/or managed by the launcher), so I doubt you would see an external process if that is what you mean.
The temporary file is created by the stage command, if you aren't using it, then I guess the FTP server is doing it of its own accord and you will probably need to investigate its settings.
Have you tried another FTP client to send the same file? Does the same thing happen?
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: rarw

Hi, Thanks for your response.

We are NOT using the "staging" command, so the ftp server must be creating the temp file.

Recap (WTx 8.1.0.2, build 38 - Command Server):
ftp - Solaris to Solaris works - ftp takes 5 min
ftp - manual from Solaris to mainframe works ~8 min.
ftp - WTx Solaris to mainframe - fails (brokern pipe, time out (~1.5 minute connection is made to mainframe, temp file is created on Solaris, connection is terminated by mainframe (broken pipe))

File is ~ 1 GB.

We will do research to determine if the ftp temp file can be eliminated on the Solaris ftp side.

Thank you,
Becky
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: repanzer

Possibly try changing the TYPE=A. Maybe to TYPE=I. Or try switching between port and passive.
Wait a minute, you're going to an MVS mainframe??

If so, you need the site commands; LREL, block=FB or VB,
#IBMSterlingTransformationExtender
#DataExchange
#IBM-Websphere-Transformation-Extender

Originally posted by: repanzer

and block size if I remember correctly. The file also has to be created already. Empty, but created.
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: rarw

Hi,

Thanks for your responses.

The server we are ftping to is a BULL mainframe (Sun to BULL). We have someone looking at the ftp timeout for the command stream on the BULL side, and will do some testing once we get a test area set up.

Thanks,
Becky
#IBM-Websphere-Transformation-Extender
#DataExchange
#IBMSterlingTransformationExtender

Originally posted by: rarw

An addtiional thought:

Even though we do NOT use the STAGING command for the FTP adapter, it seems like the temporary file is being created by the WTx FTP adapter. I think this because the location of the temporary file is from a path that is set in the envsetup.

DTX_TMP_DIR="${DTX_HOME_DIR}/tmp"

If the temp ftp file was being created by the FTP server, I would expect the path to contain /var/something (UNIX box) fro the temporary file.

Does anyone know more about the WTx FTP adapter to prove this wrong?

Thanks,
Becky
#DataExchange
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender

Originally posted by: rarw

Hi,

This problem has been resolved; I am not a WTx expert, but this is what I have learned from resolving the problem.

In WTx (8.1.0.2 build 38) the ftp adapter uses an internal ftp client to ftp the file; the operating system ftp executable was not used in the ftp. If you analyze an ftp trace, you will see that the ftp connection is made to the receiving server and then the WTx process makes a copy of the file to ftp on the sending server before actually ftping the file. Thus, the ftp data timeout on the receiving server is ticking away while the temp file is being created on the sending server.

With the help of IBM, we found that there are two specific points that needed to be addressed in order to successfully ftp our ~1 GB file in WTx from the user acceptance testing (UAT) SUN Solaris 10 server to a BULL mainframe. First of all, there must be sufficient memory on the sending server to create the temporary ftp file. Secondly, the time it takes, starting from connecting to the receiving server (BULL) and making the temp file on the sending server (SUN), to actually sending data in the ftp must be less that the ftp data timeout on the receiving server.

Here are the specifics relative to our circumstances:

(1) On the UAT SUN server, which has 2 GB of memory, all available resources were being consumed on the SUN box when ftping the ~1 GB file. I freed up as much memory as possible. It is worth repeating that in production, with 8 GB memory, and no adjustments to the WTx process or the BULL mainframe, the WTx process successfully ftp’d the ~1 GB file.
a. On UAT, I also edited the map to use “file not (!) create” for the transaction on the out cards (instead of using “sink” – memory). This may be helpful, but having enough resources was the critical factor.
(2) On the BULL mainframe, the ftp data timeout was changed from 60 sec (1 minute) to 600 seconds (5 minutes). This may need to be adjusted for files larger than 1 GB; we have yet to test a 2 GB file (WTx limit for processing a file in non-burst mode) – currently insufficient memory in UAT to do the test.
a. The BULL ftp command (control) timeout parameter was not set on the server, and not altered for this test.

It must be noted that when the ftp failed due to an abort or timeout, and the file was not ftp’d, the map did NOT fail; a successful map return was received.

Thanks for all of your help and participation in trying to resolve this issue.

Becky
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange

Originally posted by: rarw

Thanks!
#IBM-Websphere-Transformation-Extender
#IBMSterlingTransformationExtender
#DataExchange