answer from IBM Support (thanks for great support!)
RC4-HMAC has been disabled in that new JDK (because it's vulnerable since many years..) so in the keytab you just have RC4 keys, you could be experiencing the problem.When creating our Keytab Files from ActiveDirectory, the ktpass command used the RC4-HMAC algorithmus. We've similiar issues in other applications where with the current java release, Kerberos is getting into trouble 'cause of old algorithmus which should be replaced.
to verify your keytab:
java com.ibm.security.krb5.internal.tools.Klist -e -k keytabfileExample.E:\IBM\WebSphere\AppServer\java\jre\bin>java com.ibm.security.krb5.internal.tools.Klist -e -k E:\IBM\krb\mykeytab.keytab
in my case the result is
[1] Principal: HTTP/hostname.subdomain.domain@ABC.NETKVNO: 1Verschlüsselungstyp (skey, tkt): RC4 with HMAC, {1}------------------------------
Klaus Schmidinger
Teamlead IT4IT
Julius Blum GmbH
------------------------------