We are using the OpenSSH package downloaded from Advantage to sftp files from a remote server to the filesystem where the Integration Server is running. We are beginning to frequently experience an issue where the sftp commands will fail with the following error:

com.wm.app.b2b.server.ServiceException: I/O Exception running command: java.io.IOException: Not enough space

After some research, we determined that the error was being caused by a lack of physical memory on the server where the IS is running. Killing some processes that were consuming a large amount of memory allowed the sftp commands to execute successfully.

In researching the issue, however, we observed that the IS executing the sftp commands was consuming a large (3GB) of memory. We have java startup parameters limiting the memory of the JVM to 512MB (-mx512M). It appears that the OpenSSH is forking new processes that are allowing the Integration Server to exceed the memory limitations we have tried to impose with jvm startup parameters.

Has anyone seen memory issues such as this associated with OpenSSH before?

Are you sending files from or to IS? Which version of IS? How large are the files that you are attempting to send via SFTP? If the SFTP services work like the normal FTP services do, files received are loaded into memory until fully transfered and then the file contents are passed to the processing service.

Mark

We are sending files to the IS. The largest file we have tried sending is 9MB, so I’m not sure where a 3GB memory footprint would come from.

Agree. Files that small should not consume your IS memory. Not sure whether the OpenSSH example package is supported, but you might see if Support will give you a hand.

Mark

I think, WM OpenSSH pacakge executes shell commands using Java Runtime.exec (WM CommandRunner). This would cause IS to fork off child/ren with same resources as the parent (IS).
Not enough space error is probably result of running out of file descriptors due to forks from IS.

~tS

I downloaded the package from advantage, installed it.

what are the next steps, please reply as early as possible.

if try to pass the host name of my ftp server as input to remote host,
user userID password in batchscript it is throwing this error

Name or service not known
Couldn’t read packet: Connection reset by peer

We also have a problem in which sftp using openssh is not happening at times. Can someone tell where the error message “I/O Exception running command” will be written. We have the nohup.out file and the IS server log file being written. But we could not find the entry there.

regards

Rangarajan

Hi ,

We are also facing the same issue, but we didnot have this issue in Dev and UAT. We are facing this in Production. can you please let me know if there is a solution or work around for this issue.

Thanks

Hi ,

We are receiving error on the Integration Server error log, Since they have throw exception in the service provided by web methos PS.

Can some on help if there is any way we can check what is the memory used by this service.

thanks.

Sudheer

Hi,

I did an implementation with OpenSSH “hostkey” based authentication and everything went fine, I was looking for a way to have SFTP with UserID/Password instead of host key,
and I got this java code form [url]http://www.jcraft.com/jsch/[/url] it works fine for me based on hostkey; ( and I manage to get it working with pwd too by calling shell script since it was killing the server-and I still working on calling directly from IS-).

So as an alternative you can use this package to implement SFTP hostkey based authentication instead of the OpenSHH.

Thanks,
Mohammad

Hi,

Here’s is what I need to do:
SFTP Get a 10-15GB large file in webMethods 6.1, convert it from ASCII to Binary (and vice versa) and SFTP Put to another system.

Pls advise what would be the best way to do this? I’m thinking of using OpenSSH pkg but worried of overloading Pipeline Memory if IS tries to keep the file in memory while picking it. Pls help!

Rgds,
Sandeep

Hi Sandeep,

The OpenSSH utility spawns a new thread with memory equal the the JAVA_MIN_MEM (-Xms) parameter set for you jvm. As long as that value is enough to run the sftp from an OS standpoint, you’ll be fine. The contents of the file are not loaded into the pipeline. The memory requirements to sftp the file should be much less than the 10GB file size.

Does that answer your question?

Thanks,
Brad

Hi Sandeep,

I think it is very large file, so it is better to have customization of service to read in chunk and right a way writ e to diff location.

thx
sudheer

Thanks a ton, Brad. But do you think it can handle a 10GB file even if the JVM is set to, say 1GB?

Secondly, once I get the file and place it in a folder on IS I need to invoke some Unix commands to act on these files - e.g. dos2unix. I was thinking of using a wrapper to invoke a Unix script which does this. Is there any other way to play around with a 10GB file?

Appreciate a quick response.

Rgds,
Sandeep

Thanks Sudheer. But chunking would involve a lot of exception handling needed. If I make use of wrapper to invoke the Unix commands directly, wont it be good?

Rgds,
Sandeep

Our requirement is that we should have different RSA key pairs for different SFTP partners, so that if we want to change one of the keys the other partners are not affected.

I didnt find any reference of creating multiple key pairs for multiple hosts in the OpenSSH PDF. Can someone throw some light on this please?

Rgds,
Sandeep

Would suggest using SCP instead of SFTP from the same package. You may timeout (which is configurable) but I believe you avoid memory issues.