When there is a option available to choose between CSV and XML file for huge volume of data communication from a 3rd party application s/w over HTTP communication to a webMethods infrastructure, which one is best recommended and what are the issues with either approach.
Any input with regard to this will be highly appreciated.

Sushil,

I prefer to use XML exchange via HTTP communication rather than using CSV (flatfile).

Since webMethods can handle huge volume of data with huge xml’s like available(PIP,CIDX,OAGIS,ebXML,Custom Cannonicals etc…standard documents)and using TN LargerFile Handling capabilities or IS NodeIterator logic we can process the inbound XML documents in a robust manner.

Just my thoughts,

RMG,
I am with the same opinion as you. I need to justify my stand. I have the following points in favor of CSV where as I don’t have any points in favor of XML over CSV. I can use wMFlatFile to handle the CSV. Is there any technical issues of handling flat file with related to TN, as I am not use to flat file APIs so extensively.
1>
CSV does have one advantage over XML for specification in new designs. CSV has much lower overhead, thereby using much less bandwidth and storage than XML. For new designs where overhead issues have high importance or criticality, CSV may be the better choice.
2>
The absolute theoretical best advantage for XML is one-letter element names in the XML and every field quoted in the CSV. Even with this unrealistic advantage, XML’s overhead will be slightly greater than 200% of CSV’s. Typically though, XML’s overhead will be 900% to over 9000% greater than CSV’s.

Thanks for your feedback.
Sushil

Sushil,

Regarding handling of CSV flatfile (inbound)i dont have extensive experience on this ,but i have done CSV Outbound and i didnt seen any issues as of now.

And regarding CSV vs XML,i go with XML world, since it is easy and userfriendly for webMethods tools like parsing,validation etc…One more advantage is developer will have a deep understanding about xml structure,definitions…

And i might agree with your comments that “CSV has much lower overhead, thereby using much less bandwidth and storage than XML”. since xml documents has lot of parent/child relations,recursive elements etc…

When comes to performance any way webMethods handles any large docs and so here the storage,preformance issues can be eliminated.

Anyone has more comments for this thread?

HTH,
RMG

RMG,
Appreciate your input to this thread.

Thanks
Sushil

Hi Sushil,
Instead of looking into network overhead/bandwidth as the most important factor for choosing your design approach, I suggest you also consider Maintainability as one of your key priorities.

This is more of a consultant point of view, but I consider one must look into this aspect for any design solution.

The solution needs to stand the test of time. It should not be unnecessarily complex, if that complexity can be traded for simplicity to change and maintain overtime.

I think one way to arrive at this decision of using CSV vs. XML will be to prototype both solutions and test it for large inputs (ones which you except in your production). Compare the results; if the difference is quite large, you may have your answer. If it isn’t, then I guess, XML should be the way to go.

Hope this helps you arrive at your decision. I think RMG and you covered most of the issues either approach has.

• Saurabh.