IBM webMethods Hybrid Integration

IBM webMethods Hybrid Integration

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.



#Automation


#Applicationintegration
#webMethods
#Integration
 View Only
  • 1.  Website page scraping through HTTP Post

    Posted 11/13/08 07:19 AM

    Hi All,

    We have a requirement to develop a interface to log into a client website and pass inputs in a particular page and get the output from it. Please find the process below.

    1. Login into a website using User ID and Password.
    2. navigate to a particular page.
    3. Provide inputs in that page and submit.
    4. Confirm the provided inputs.
    5. Read a particular data from the output html page.

    Note: website is not implemented for m/c to m/c communication, but requirement is to implement an interface using webMethods.
    We have enabled the Cookies handler in the extended settings and tried to do a HTTP POST to the website passing all the required inputs. We are always being redirected to the login page.

    Please help us in providing a solution to this.

    Thanks.


    #webmethods-Protocol-and-Transport
    #webMethods
    #Integration-Server-and-ESB


  • 2.  RE: Website page scraping through HTTP Post

    Posted 11/13/08 06:11 PM

    I don’t know of any current webMethods products that support website screen scraping. That integration approach was always very unreliable since the most minor change in the source website will cause it to fail.

    Mark


    #webmethods-Protocol-and-Transport
    #webMethods
    #Integration-Server-and-ESB


  • 3.  RE: Website page scraping through HTTP Post

    Posted 11/15/08 07:29 AM

    Hi Mark,

    Thanks for the response.
    We are aware that this approach is unreliable, but this is the only was for the business to full fill the requirements at present, and i got a suggestion that if we enable the cookies handling we can achieve the website scraping. So can you plese help me in this.

    Thanks,
    Ram.


    #webmethods-Protocol-and-Transport
    #webMethods
    #Integration-Server-and-ESB


  • 4.  RE: Website page scraping through HTTP Post

    Posted 11/15/08 01:41 PM

    I don’t think that would help, but feel free to try it. I do not believe webMethods IS is the right tool for this job.


    #Integration-Server-and-ESB
    #webMethods
    #webmethods-Protocol-and-Transport