I’m invoking a current service by putting my desired input file into the /ns/… directory and it works if I do the following steps with the inboundFile object:
StreamToBytes, bytesToStream. I then begin reading pieces out of the stream. My program
is written to make use of the pub.io.read() with a bufferArray, etc.

The problem is how do I get this to work WITHOUT loading the whole file into memory? I have tried casting the inboundFile (via FTP) as an InputStream, BufferedInputStream and nothing seems to work. Does anyone know a workaround,
or a different cast I need to try? If the answer is in documentation, please tell me which doc, I have been searching for quite some time.

Below is the code I was using to cast where inStream is what is mapped in from the inboundFile object that the FTP sends in: Am I doing this wrong?

IDataCursor pipelineCursor = pipeline.getCursor();

InputStream iStream = (InputStream) IDataUtil.get(pipelineCursor, “inStream”);

//BufferedInputStream bis = new BufferedInputStream(iStream);

IDataUtil.put( pipelineCursor, “outStream”, iStream);

pipelineCursor.destroy();

Why don’t you use FTP port to receive file which will invoke the service that process that input stream?

Hi, If I wasn’t clear, that is what I am currently doing. I have an FTP port setup and I am sending the file into my service and the service executes. Currently, I am having to use
the StreamToBytes, bytesToStream flows. This loads the entire file into my pipeline
and exposes it as a stream. What I want is the ability to use the pub.io.read() directily
with the inbound data file without having to load the entire file into the pipeline.

Does this make sense?

When I take the inboundFile object from the FTP port and pass it into the pub.io.read()
it fails and doesn’t recognize it as a valid stream object.

_brett.

Ok could you specify what error you are getting exactly? what kind of file are you receiving i.e. XML, flat file etc and is there a special need to call pub.io.read() ?

Via ftp, the normal input var is ffdata, which is an InputStream from which you can read. What IS version are you using? I’m not sure where inboundFile nor inStream would come from. Do you have custom content handlers or some other processing before your service is invoked?

I’m receiving a binary file. It’s not XML, nor is it a flat-file. The reason I want to use
the pub.io.read() is so that I can consume chunks of the file over the stream and process,
without having to consume the entire file at once. I don’t want to use up too much memory.
So, I read 4 bytes, process and go on to the next 4, or n-number of bytes.

I need my service to be able to consume the file n-bytes at a time. I had a test case working, when I used the pub.file.getFile and I would load it as a stream. I then passed that stream along and pulled the bytes as needed. I’m assuming the FTP service dropped off a stream object, but I could be wrong. Do you know what type of Object it is?

I’m not getting any errors at the moment. The pub.io.read just doesn’t recognize the stream.

I will try the service input as ffdata.

_brett.

I posted my questions before I saw the exchange with TK. What content type are you using when FTP’ing the file? That will determine which content handler gets used which in turn determines the state of the pipeline, variable names and their types, when your service is invoked.

Running IS version 6.5. The input to my service has been “inboundFile” which is an object,
that when input into the StreamToBytes, it works and processes my data. However,
when I take that same object (inboundFile) and pass into pub.io.read() it says “Stream Required”. This is what lead me to believe that I needed to convert the inboundFile
object to InputStream or something like it.

I have read several posts that say to use different input variable names such as: inboundFile, *inFile, and now ffdata. So far, the inboundFile works only as described above.
Couldn’t get ffdata to work.

_brett.

right now, i’m transfering in Binary mode. The only data I see from my FTP client is:
Entering Passive Mode:
STOR
150 BINARY mode data connection for
226 Binary Transfer

That’s all I know on how I’m transfering.

What is the content type used by the client when put’ing the file? As mentioned before, that will determine which content handler IS uses which in turn determines what parameter(s) will be placed into the pipeline. I’m not sure which content hander creates an “inboundFile” variable.

If pub.io:streamToBytes works, but pub.io:read does not (they both take InputStream inputs named stream), then something is definitely amiss and probably time to engage wM tech support to remote troubleshoot. Lacking that, can you post the pertinent details of your service or the service itself?

Not mode. Content type. Sounds like you’re probably not specifying a content type, so it’s likely that the default content handler is being used.

To specify a content type on the client side:

put sourceFile.ext destFile.ext;content-type

Example

put foo.xml bar.xml;text/xml

This will cause the XML content handler to be invoked, parsing the content into a node object for use within your service.

If you’re not specifying a content type, then it is determined from the filename extension. What is that name?

I’m hoping we can zero in on what content hander is involved here so we can figure out what exactly the “inboundFile” var really is.

The content of the file is a mix of binary data. There are tiff-encoded images along
with EBCDIC encoded data. There isn’t really a file extension. I’m all ears if there’s a
content type I could specify to get the stream working without having to load the entire file into the pipeline memory prior to processing it.

_brett.

Is there a specific doc you are referencing that maps the content-type to the pipeline data generated? If so, I can download it from Advantage if you have the doc name.

My original post about ffdata was inaccurate. It will only be that when the right content type is specified.

To make sure I have this straight: you’re not specifying a content type and not using an extension (hopefully that’s yes or no, not a “not really”–it needs to be explicit or your service may not get invoked as expected). That means the default content handler is being used, which means the data is in an InputStream object named contentStream.

But…

If there really is a inboundFile var in the pipeline at the start of your service, the question is where did it come from?

If you want to try another approach, in the FTP client specify:

put srcFile destfile;application:x-wmflatfile

This will cause the flat file content handler to be invoked–which doesn’t do much of anything other than put a var named ffdata into the pipeline. So it doesn’t matter what the file contents really are, they are accessible via the ffdata var unchanged.

For files put to your service, you should by contract specify a consistent extension or a consistent content type (which will permit any arbitrary filename extension).

Unfortunately, no. There isn’t a summary of the content types to the pipeline var AFAIK.

Thanks Reamon,

It looks like this is moving forward. I changed to the contentStream and now the flow
service I have is processing only part of the file. I’m not sure why. I don’t know if
the contentStream adds extra bytes to the front of the stream or if the encoding
has changed during transmission, but thanks for the help.

_brett.

Finally got this working so far. Here’s what was changed. The input to my service needed to be contentStream. The pipeline revealed I was getting a wm.server.net.FTPInputStream.
This doesn’t work with pub.io.read() when trying to ready n-bytes at a time when n < total number of bytes in the stream. So I casted the FTPInputStream to a plain InputStream then wrapped it in a BufferedInputStream, then when I FTP’d the file into the service it worked as desired.

The only fallback I’ve noticed is that I can typically only send one file per connection, otherwise strange things start happening down the line with the other files on that same connections. If I re-connect for each inbound file, things seem to work just fine.

Thanks for the help.

_brett.

Thanks for the follow-up on how you resolved the issue. Most folks don’t do that but it is quite helpful for future readers.

Do you close the stream on the IS side? You might also drop everything in the pipeline at the end. That may help (or may not) with the multiple files per session.

Yes, I do close the stream and drop all items at the end of the service.

I did find some documentation on the FTP inbound stuff. It’s in
the Flat File Schema Developer’s Guide. Not a lot of detail but enought to point in the right direction with regard to the content-types, etc.

_brett.

I did a test and it worked (uses ffdata since my file is has .txt extension)
Find attached the sample service SandBox.Nancy:testForum (You will need to create the package first since this is an export from developer)

Also find enclosed the pipeline file showing the output. Let me know if this doesn’t help.

Hi Brett,

Not completely sure what the case is you are trying to solve, but I think you may want to consider your architecture. If you are receiving binary files, and (as you said) you would like to receive multiple files concurrently, I think you should try to write the file to disk asap once received, to ensure the release of the thread. Once on disk (even in a temp dir) it will be much easier and more efficient to process the file. Additional benefit: includes archiving.

To write a file to disk from the FTP input, you need a java service that takes any sub class of Inputstream (aha, such the FTPInputStream), and writes it to a file location.

Something like this:

// Get params
IDataCursor pc = pipeline.getCursor();
InputStream s = (InputStream)IDataUtil.get(pc, “contentStream”);
String f = IDataUtil.getString(pc, “filename”);

// TODO: check whether file is writable, etc

// Try to write stream to file, fiddle with the buffer size for performance
try {
FileOutputStream fout = new FileOutputStream(f, true);
int pos = 0;
int read = 0;
byte buffer = new byte[8192];
while ((read = s.read(buffer, 0, buffer.length)) > 0) {
fout.write(buffer, 0, read);
pos += read;
}
s.close();
} catch (Exception e) {
throw new ServiceException(e.toString());
}

// Output nr of bytes written to file
IDataUtil.put(pc, “bytesWritten”, String.valueOf(pos));
pc.destroy();

After running this, the stream is closed and can be dropped. If you would use a broker to submit a notification to a trigger, the current thread can return a confirmation to the FTP client, and a separate thread can start the processing of the file. Setting the trigger to concurrent easily allow parallel processing.

Hope this helps!

Chris

I’m not sure when it was added, but 6.5 has the ability to write an incoming file to disk and publish an event without needing to write a service to store the contents.

In the IS Administrator’s Guide, look at watt.server.userFtpRootDir in Appendix B. Server Configuration Parameters. It describes how IS can behave more like a “real” FTP server, storing the put file to disk and publishing a “putCompletedNotification” rather than invoking a service.

True, but as I understood from brettp’s posts, he really wanted to manage the byte stream, doing 4 bytes at a time, for whatever reason, I assumed he would be past that station.
If that is not the case, separating the native file handing for FTP as standard funcs would be the best option indeed.

Chris