This is a “feature” of StringTokenizer. It doesn’t correctly handle empty tokens. As of JDK 1.4, String.split() does what you are looking for.
If you must use an older JDK and you want to stay with StringTokenizer, you can use the alternate constructor:
StringTokenizer(String str, String delim, boolean returnDelims)
This will return the tokens along with the delimiters. You will have to write your own logic to strip the delimiters and recognize the delimiters-only case that signals an empty field.
#webMethods#Flow-and-Java-services#Integration-Server-and-ESB