I’m trying to connect to a new partner via sftp. With the local sftp logic you can select password or you can select key. They require both. Is this possible?
webMethods 9.8

Hi Mark,

what do you mean by “they require both”?

There is a host key to identify the server which can be retrieved by the “get host key” button.

On the other side you can either choose to login with user/password or by rsa-key.
Why is logging in with both methods needed?

Regards,
Holger

Thanks for the reply Holger,
They are requiring both auth types. In the user alias settings it’s a radial button for auth type, so I’d assume using the built in SFTP isn’t going to be possible. This is the first time I’ve even heard of someone requiring both for login. I know you can require both, but I’ve never had to with a webMethods partner.
I have manually logged in and can see that it first verifies the ssh key then moves on and verifies the password (not to be confused with passphrase). Only with both can I login, which the partner verified is their requirement.

Taking it that no one knows a way to accomplish this
Support was pretty sure it’s not possible with the built in SFTP, hopefully I can find an out of the box solution…

Hi Mark,

in this case you might want to consider opening a feature request in Brainstorm (you will find a link in Empower for this).

Let´s see what Development Team thinks about this.

Regards,
Holger

Will do, Thanks again.

Have you verified webMethods can’t support password if choose key authentication? Or just read it in document?

I have tried and support verified that you can select only one type of authentication. You select it with the radial button which saves that selection to the sftpUserAliases.cnf file. I even tried modifying the cnf to contain both, but that didn’t do anything.

Hi,

currently webMethods is only able to use only one of the authentication types.
That is the reason why I have asked to open a feature request in Empower.
If you have already an Incident open with SAG Support provide this number in the feature request for reference.

to sum up the config:
server alias has the host key of the SFTP server.
user alias should know about the private key and its passphrase as well as of the user name and its password.

Can you elaborate, why your partner is requiring both authentication methods in sequence?

Regards,
Holger

was just answering Xiaowei. Opened a brainstorm ticket on Empower (05508).
I’m not sure why they believe both authentications should be used, but that is what they are requiring. Which they say plenty of there other partners had no issues complying with… I have connected to them, but only through filezilla or the cmd line and cmd works because interactively, I’m sitting in the directory with the private key.

Have you tried creating two SFTP user alias for same SFTP server alias. One with authentication type “Public Key” and another with “Password”.

Your service which connects to SFTP server should do the following.

When connecting to SFTP server first try with “Public Key” user alias. if it fails then use the “Password” user alias.

Hi Naga,

as far as I understood Mark, he needs to first authenticate with key and passphrase and than with user and password additionally in the same session. This would require the second login attempt to have an input of session-id returned by the first authentication attempt which will not be returned by the first attempt due to incomplete authentication.

Interesting attempt anway, might be useful for partners, which can handle either of the authentication types depending on which one succeeds. “If one fails, try the other. If both fail → bad luck”.

But in Marks case it is “If first succeeds, try the second too” → “If both succeed, provide session-id” or “If either one of them fail → bad luck”.

Regards,
Holger

It is exactly as Holger put it. Process will pass the key, then passes the password check. Either fail then the login fails. I did however try to create 2 exact aliases, but that isn’t possible. Are not allowed 2 of the same named aliases, no surprise there.

This thread might help, [url]http://tech.forums.softwareag.com/techjforum/posts/list/37418.page[/url], I’m still verifying the solution.

I’m not seeing from that thread where you would input password. Looks like the same fields get passed as what you would set up in the alias. Wonder if anyone using it knows… Takes a lot of resources to get none official downloads installed, but if it works, would be worth it.

The source code attached in that thread seems a standalone java service which could be used to create a SFTP session, and password seems to be hardcoded as “password”.
Just verified, the class MyUserInfo is never been used, so you’re right, this sample code doesn’t work.

I took a look more about the webMethods native SFTP function, and the conclusion is “The support team tells the truth, it doesn’t support both key and password”.

There are 2 reasons:

1. The object to store connection information (com.wm.app.b2b.server.sftp.client.SFTPClientManager.ConnectionDetails) only has one field to store password. So password and passphrase share the same field.

private class ConnectionDetails
{
String host;
String user;
String authType;
String privateFileLocation;
String proxyAlias;
String hostKeyAlias;
int port;
int connTimeout;
byte[] password;
Map<String, String> config;

private ConnectionDetails() {}
}

2. webMethods uses library JSCH to connect to SFTP, and there are serval user auth method like UserAuthGSSAPIWithMIC, UserAuthNone, UserAuthPassword, UserAuthPublicKey, but no both key and password.

Looking for solution? There are 3 steps.

1. Optionally modify the SFTP related DSP to allow user input more data to provide both password and passphrase. It’s ok if you don’t do that, because once you set the password and passphrase for user alias, the password value will be stored in outbound password separately, no matter what auth key you choose.
2. Hack webMethods class and add one more field to store passphrase, and also use that password when create session.
3. Implement a new JSCH user auth method, and somehow register it to JSCH. It’s the most difficult one, and what I’m working on now.

Finally I made it. Although it seems not possible by analyzing JSCH user auth methods, somehow the code below work really great.

And it’s much easier than I thought before, no need 3 steps any more, just one java service.

import com.wm.data.*;
import com.wm.util.Values;
import com.wm.app.b2b.server.Service;
import com.wm.app.b2b.server.ServiceException;
import com.wm.app.b2b.server.Resources;
import com.wm.app.b2b.server.Server;
import com.wm.app.b2b.server.sftp.client.*;
import com.jcraft.jsch.*;
import java.util.Properties;
import java.io.FileReader;
import com.wm.util.JournalLogger;
import java.io.*;
import java.lang.reflect.Field;
import pub.CommonUtils;
import com.wm.passman.PasswordManager;
import com.wm.passman.PasswordManagerException;
import com.wm.security.OutboundPasswordStore;
import com.wm.util.security.WmSecureString;

public static final void login(IData pipeline) throws ServiceException {
IDataCursor dataCursor = pipeline.getCursor();
boolean bReuseSession = false;
String sessionKey = null;

try {
String userAlias = CommonUtils.getRequiredStrParam(dataCursor, "userAlias");
String reuseSession = IDataUtil.getString(dataCursor, "reuseSession");
bReuseSession = (reuseSession != null) && (reuseSession.equalsIgnoreCase("true"));

//Return cached sessionKey
if (bReuseSession) {
sessionKey = sftpSessionManager.getSessionForAlias(userAlias);
}

if (sessionKey == null){
//Retrieve user alias info
IData getUserAliasInput = IDataFactory.create();
IDataCursor getUserAliasInputCursor = getUserAliasInput.getCursor();
IDataUtil.put(getUserAliasInputCursor, "alias", userAlias);
getUserAliasInputCursor.destroy();
SFTPUserAlias sftpUserAlias = sftpManager.getUserAlias(getUserAliasInput);

//Retrieve server alias info
IData getServerAliasInput = IDataFactory.create();
IDataCursor getServerAliasInputCursor = getServerAliasInput.getCursor();
IDataUtil.put(getServerAliasInputCursor, "alias", sftpUserAlias.getSftpServerAlias());
getServerAliasInputCursor.destroy();
SFTPServerAlias sftpServerAlias = sftpManager.getServerAliasInfo(getServerAliasInput);

//Configure session
JSch jsch = new JSch();
//jsch.setHostKeyRepository(getHostKeyRepository());
jsch.addIdentity(getPrivateKeyFile(sftpUserAlias.getKeyFileLocation()), retrivePasswordAsString("wm.is.admin.sftpclient.pass.phrase." + userAlias));
Session session = jsch.getSession(sftpUserAlias.getUserName(),
sftpServerAlias.getHostName(), sftpServerAlias.getPort());
session.setPassword(retrivePasswordAsString("wm.is.admin.sftpclient.password." + userAlias));

//Init session config map
java.util.Properties config = new java.util.Properties();
config.put("kex", sftpUserAlias.getPreferredKeyExchangeAlgo());
String compression = "none";
if ("zlib".equals(sftpUserAlias.getCompression())) {
compression = "zlib,none";
}
config.put("compression.s2c", compression);
config.put("compression.c2s", compression);
config.put("compression_level", String.valueOf(sftpUserAlias.getCompressionLevel()));
config.put("MaxAuthTries", String.valueOf(sftpUserAlias.getNoOfRetries()));
//config.put("StrictHostKeyChecking", "yes");
config.put("StrictHostKeyChecking", "no");
config.put("PreferredAuthentications", "publickey,password");
session.setConfig(config);

//Connect to SFTP server
session.connect();

//Cache sessionKey for reuse
sessionKey = sftpSessionManager.addSession(session, sftpUserAlias.getSessionTimeout(), userAlias);

//Return sessionKey
CommonUtils.mergeOutput(dataCursor, "sessionKey", sessionKey);
}
} catch (Exception e) {
throw new ServiceException(e);
}finally{
dataCursor.destroy();
}
}

private static SFTPSessionManager sftpSessionManager = SFTPSessionManager.getInstance();
private static SFTPClientManager sftpManager = SFTPClientManager.getInstance();
private static Resources resources = new Resources(Server.getHomeDir(), true);
private static File identitiesDir = resources.getDir(resources.getSFTPDir(), "identities");

private static String retrivePasswordAsString(String passHandle) throws PasswordManagerException {
String password = null;
if (passHandle != null){
PasswordManager passman = OutboundPasswordStore.getStore();
WmSecureString secureString = passman.retrievePassword(passHandle);
password = secureString.toString();
}
return password;
}

private static String getPrivateKeyFile(String keyFileName){
if (keyFileName == null) {
return keyFileName;
}
return new File(identitiesDir, keyFileName).getAbsolutePath();
}

private static HostKeyRepository getHostKeyRepository() throws Exception{
Field field = sftpManager.getClass().getDeclaredField("sftpSvrAliasManager");
field.setAccessible(true);
return (HostKeyRepository)field.get(sftpManager);
}	

Service inputs:
userAlias : String
reuseSession : String (Optional)
Service outpus:
sessionKey : String

Before invoking this service, you have to config server alias and user alias the same as how you use native SFTP, just remember you MUST config user alias as password first, then change to key.
Basically you could replace build-in login service (pub.client.sftp:login) with this one, then you could use build-in SFTP services (like pub.client.sftp:get) with the output sessionKey.

The code is tested on my 9.9, so let me know if you have any issues on 9.8.

This is the final version, including 2 fixes.

1. Set “StrictHostKeyChecking” to “yes”, previous version is “no”, so it behaves the same as native.
2. Add service outputs “returnCode” and “returnMsg”, so it could fully replace the build-in login service.

The package hold this service has to be dependent on WmPublic.

import com.wm.data.*;
import com.wm.util.Values;
import com.wm.app.b2b.server.Service;
import com.wm.app.b2b.server.ServiceException;
import com.wm.util.JournalLogger;
import com.wm.util.security.WmSecureString;
import com.wm.util.i18n.MessageFormatter;
import com.wm.app.b2b.server.Resources;
import com.wm.app.b2b.server.Server;
import com.wm.app.b2b.server.sftp.client.*;
import com.wm.passman.PasswordManager;
import com.wm.passman.PasswordManagerException;
import com.wm.security.OutboundPasswordStore;
import com.wm.resources.WmPublicMsgBundle;
import com.jcraft.jsch.*;
import pub.CommonUtils;
import java.util.*;
import java.io.*;
import java.lang.reflect.Field;

public static final void login(IData pipeline) throws ServiceException {
IDataCursor dataCursor = pipeline.getCursor();
boolean bReuseSession = false;
String sessionKey = null;

try {
String userAlias = CommonUtils.getRequiredStrParam(dataCursor, "userAlias");
String reuseSession = IDataUtil.getString(dataCursor, "reuseSession");
bReuseSession = (reuseSession != null) && (reuseSession.equalsIgnoreCase("true"));

//Return cached sessionKey
if (bReuseSession) {
sessionKey = sftpSessionManager.getSessionForAlias(userAlias);
}

if (sessionKey == null){
//Retrieve user alias info
IData getUserAliasInput = IDataFactory.create();
IDataCursor getUserAliasInputCursor = getUserAliasInput.getCursor();
IDataUtil.put(getUserAliasInputCursor, "alias", userAlias);
getUserAliasInputCursor.destroy();
SFTPUserAlias sftpUserAlias = sftpManager.getUserAlias(getUserAliasInput);

//Retrieve server alias info
IData getServerAliasInput = IDataFactory.create();
IDataCursor getServerAliasInputCursor = getServerAliasInput.getCursor();
IDataUtil.put(getServerAliasInputCursor, "alias", sftpUserAlias.getSftpServerAlias());
getServerAliasInputCursor.destroy();
SFTPServerAlias sftpServerAlias = sftpManager.getServerAliasInfo(getServerAliasInput);

//Configure session
JSch jsch = new JSch();
jsch.setHostKeyRepository(getHostKeyRepository());
jsch.addIdentity(getPrivateKeyFile(sftpUserAlias.getKeyFileLocation()), retrivePasswordAsString("wm.is.admin.sftpclient.pass.phrase." + userAlias));
Session session = jsch.getSession(sftpUserAlias.getUserName(),
sftpServerAlias.getHostName(), sftpServerAlias.getPort());
session.setPassword(retrivePasswordAsString("wm.is.admin.sftpclient.password." + userAlias));
session.setHostKeyAlias(sftpUserAlias.getSftpServerAlias());

//Init session config map
java.util.Properties config = new java.util.Properties();
config.put("kex", sftpUserAlias.getPreferredKeyExchangeAlgo());
String compression = "none";
if ("zlib".equals(sftpUserAlias.getCompression())) {
compression = "zlib,none";
}
config.put("compression.s2c", compression);
config.put("compression.c2s", compression);
config.put("compression_level", String.valueOf(sftpUserAlias.getCompressionLevel()));
config.put("MaxAuthTries", String.valueOf(sftpUserAlias.getNoOfRetries()));
config.put("StrictHostKeyChecking", "yes");
config.put("PreferredAuthentications", "publickey,password");
session.setConfig(config);

//Connect to SFTP server
session.connect();

//Cache sessionKey for reuse
sessionKey = sftpSessionManager.addSession(session, sftpUserAlias.getSessionTimeout(), userAlias);

//Return sessionKey
CommonUtils.mergeOutput(dataCursor, "sessionKey", sessionKey);

populateResult(dataCursor, null, 2);
}
} catch (Throwable th) {
CommonUtils.throwAsServiceException(th);
}finally{
dataCursor.destroy();
}
}

private static SFTPSessionManager sftpSessionManager = SFTPSessionManager.getInstance();
private static SFTPClientManager sftpManager = SFTPClientManager.getInstance();
private static Resources resources = new Resources(Server.getHomeDir(), true);
private static File identitiesDir = resources.getDir(resources.getSFTPDir(), "identities");
private static ResourceBundle msgBundle = ResourceBundle.getBundle(WmPublicMsgBundle.class.getName());

private static String retrivePasswordAsString(String passHandle) throws PasswordManagerException {
String password = null;
if (passHandle != null){
PasswordManager passman = OutboundPasswordStore.getStore();
WmSecureString secureString = passman.retrievePassword(passHandle);
password = secureString.toString();
}
return password;
}

private static String getPrivateKeyFile(String keyFileName){
if (keyFileName == null) {
return keyFileName;
}
return new File(identitiesDir, keyFileName).getAbsolutePath();
}

private static HostKeyRepository getHostKeyRepository() throws Exception{
Field field = sftpManager.getClass().getDeclaredField("sftpSvrAliasManager");
field.setAccessible(true);
return (HostKeyRepository)field.get(sftpManager);
}	

private static void populateResult(IDataCursor dataCursor, String command, int msgId){
MessageFormatter formatter = new MessageFormatter(msgBundle);
String message = null;
if (command != null) {
message = formatter.format(147, msgId, new Object[] { command });
} else {
message = formatter.format(147, msgId, new Object[0]);
}
CommonUtils.mergeOutput(dataCursor, "returnCode", "0");
CommonUtils.mergeOutput(dataCursor, "returnMsg", message);
}

Thanks you very much. Will get this installed hopefully right after the holidays. Some of the key people that would implement this aren’t available. I’m not entirely sure how to implement what you suggest, so I believe I just need to copy your code, yes?

Hi Mark,

create a new java service in one of your custom packages and give it the signature as described in Xiaoweis first post.
After that fill in the code in the appropriate sections of the service.
Do not forget to save afterwards.

After that you can invoke the new service instead of the built-in service in your code.

Regards,
Holger

Yes

not sure if it’s because I’m on 9.8 but can’t import either com.jcraft.jsch.*; or pub.CommonUtils;

To import pub.CommonUtils, you need set the package which hold this service to be depended on WmPublic.

I checked my 9.8, it has jsch.jar under <SAG_DIR>/common/lib/ext. Maybe you see a warning in Designer, but it won’t affect the compile, just ignore it and save the service.

Authentication Type
The type of authentication that Integration Server uses to authenticate itself to the SFTP server.
Client authentication can be either by password or by public and private keys.

Hi Shabaz,

this is the standard behaviour that you can choose between the two types.

But the requirement in this thread is that the SFTP-Server requires both types in sequence which is currently not possible with the built-in implementation.

Regards,
Holger

Unfortunately I’m not proficient enough in JAVA and can not get this code to run on v9.8. I’ve used the jcraft in other Java services, but not the pub.CommonUtils. Running the copied code just errors out on me I believe I built the java service correct, but it doesn’t seem to work. Thanks for all your hard work. I’ll keep hacking at it, maybe I have code missed up.
Thanks,
Mark

Hi Mark,

did you add the package dependency to WmPublic package to your package as was requested earlier in this thread?

Can you provide the error message so we can have a look at it.

Please check for the SCG_TPS and/or SCG_TPL Fixes for 9.8 to see if there is an updated version of the jsch.jar included.

Regards,
Holger

Is there a way to make the package a dependency without adding it into the package?
I see no errors, but I also see no session key, message or code returned.
Will double check the updates, pretty sure we are up to date on the jar fixes

I do have the test package I’m working in set to have the wmPublic as a dependency. Sorry, I misunderstood.

Hi Mark,

Still have problems? If so I could send you a package.

That would be great. Thanks

Here it comes.
EnhancedSFTPClient.zip (12.6 KB)

Thanks everyone, think this will work for me.
So only difference was I set package dependency version to the version I’m using and you set to ..
Besides that, I had switched to a sftp login which I knew was working. This causes a null point error.
Must have only the duel login scenario for this code to work. Also must save the alias with the password, the update and change to key and save. Only works in this order, any other creates a null point error.
And to boot the client I was trying to connect to locked my account and unfortunately error messages doesn’t return from this java for me unless it was successful. So I’ve been chasing nothing…

I got “com.jcraft.jsch.JSchException: Auth fail” when trying to connect SFTP with a disabled account, not sure if this is the “locked account” you mentioned. Please provide more detail about your error.

And I also found a bug that might cause failed to retrieve reused sessionKey. The “return sessionKey” code is placed in wrong position.
Please use attached new package or adjust the code as below.

...
//Cache sessionKey for reuse
sessionKey = sftpSessionManager.addSession(session, sftpUserAlias.getSessionTimeout(), userAlias); 	
}

//Return sessionKey
CommonUtils.mergeOutput(dataCursor, "sessionKey", sessionKey);

populateResult(dataCursor, null, 2);
} catch (Throwable th) {
CommonUtils.throwAsServiceException(th);
...

EnhancedSFTPClient.zip (12.6 KB)

That is exactly what the locked out issue was. those errors are not coming back as failed returnCode/Msg. Can only find that in the results as long as you stop right after trying to login.
Updated the code.
Thanks again

The native login service will throw exception either in this case.

The flow exits on the java errors. I’ll just wrap everything in a try/catch and grab the last error on fail. Thanks

Seems there is a slight hiccup. I tried creating a new user alias. Saved with password, then saved with key. NullPointerException.
Seems the code works the first time, but any past that don’t work. I even deleted my original working alias, and now nothing works.
Seems error is somewhere in the retrivePassword
“Caused by: java.lang.NullPointerException
at kit.enhancedSFTPClient.retrivePasswordAsString(enhancedSFTPClient.java:141)
at kit.enhancedSFTPClient.login(enhancedSFTPClient.java:87)”

Is there a way to flush whatever it’s looking for and start new?

If you have problem to retrieve password value, you could use service pub.security.outboundPasswords:setPassword to force adding a new one or use service pub.security.outboundPasswords:updatePassword to update an existing one. For one sftp user alias, make sure both password below exist.

The key input follow the pattern below:
For sftp client user password, it’s “wm.is.admin.sftpclient.password.”
For sftp client key phrase, it’s “wm.is.admin.sftpclient.pass.phrase.”
Example my user alias is “test”, then corresponding key is “wm.is.admin.sftpclient.password.test” and “wm.is.admin.sftpclient.pass.phrase.test”.

You need invoke service pub.security.util:createSecureString to convert a String to SecureString, and pass it the these services I mentioned above.

And if you have security error to invoke these services, set extended setting watt.security.ope.AllowInternalPasswordAccess=true, then restart IS.

Thanks again.
Will try and let you know results. Waiting on IS reboot.

password returns the password
pass.phrase is null (not using a passphrase on the key)
Added new alias with password
updated alias to use key
Ran enhancedSFTPClient:login
Could not run ‘login’
com.wm.app.b2b.server.ServiceException: java.lang.NullPointerException
I have 2 identical (minus the alias name) aliases. First one I created works. Second fails. removed the first one, now none of the alias work. Even adding back exact doesn’t work.

this "You need invoke service pub.security.util:createSecureString to convert a String to SecureString, and pass it the these services I mentioned above. " I didn’t understand

Because pass.phrase is null, so you need to use service pub.security.outboundPasswords:updatePassword to set it.
And this service accepts SecureString Object as input, not regular String, so you need to use service pub.security.util:createSecureString to convert a String value to SecureString Object then pass to updatePassword service.

since there is no pass phrase being used, am I supposed to pass the value from the wm.is.admin.sftpclient.password. to the wm.is.admin.sftpclient.pass.phrase.?
Also isInternal should be true, correct?

So if I’m reading correctly:
get password, convert password to secureString, update pass.phrase password value with string.

isInternal should be true, yes.
The wm.is.admin.sftpclient.pass.phrase value should be the same as PassPhrase field when you setup a public key authentication type. And it’s determined when the private key is generated. It’s probably not the same with wm.is.admin.sftpclient.password.

there is no pass.phrase, this is why its null.

Then you could use service pub.security.outboundPasswords:setPassword to force adding one

Am I missing something? password and pass phrase match what was added to the alias. there was no pass phrase used, so that null should be correct.

Ok. So I recreated everything in a new instance.
1st alias, code works
2nd alias, code fails

I create both Alias’s the exact same way.(save password, save key). The first Alias has the password from the first save in the pass phrase of the second save (pass.phrase) The second Alias I create does not save anything in the pass phrase.
This is the message I received when trying to manually update password. wm.is.admin.sftpclient.pass.phrase.FTB. Key does not exist.

Got it.
You have to have the password in the passphrase, which can only be added with the setPassword function (not sure how it works the first time only).
So you can getPassword of the password., convertSecureString, createSecureString, setPassword for pass.phrase. then updatePassword for pass.phrase.

Yes, the passphrase can’t be empty. We actually have the same problem when configure keystore.

So, we’re good now?

I believe so. Not sure what to do if we used pass phrases, but we don’t so I’m not going to go down that path
Again thanks a lot for the help. it’s appreciated.

Not leveraging SAG implementation using third-party (https://www.sshtools.com/en/) SSH/SFTP library instead, offering support for different sFTP flavors, different authentication methods (key, or password) and much more.
Build small package with wrapper Java-flow-services (login, logout, cd, put, get …) allowing our developers to concentrate on implementing (business) logic.

Hi Xiaowei/Mark,

We are on 9.10 and have a mandated requirement from our partner to use Multi Factor Auth (Both Password and SSH Key) for SFTP. Tried following all the steps and waiting for our admins to update the extended setting watt.security.ope.AllowInternalPasswordAccess=true.

Meanwhile, had quick question.
While migrating to higher env. and eventually to Prod, assume we need to follow the same steps to set-up the alias, updated IS setting and manually set the pass phrase. Could you please confirm.

Could you please let me know if there are any other known issues based on your experience with this approach?

Regards,
Hari