IBM Z and LinuxONE IBM Z

  • 1.  zeroMQ in z/OS

    Posted Thu September 10, 2020 03:29 AM
    Hello, everyone,

    We are looking for a way to send stuff to an ElasticSearch repository from traditional z/OS applications (basically, IMS transactions written in PL/I). We could use the ELK stack RESTful services, but we don't want to tightly couple our business transactions with the ELK backends (we had a bad experience some months ago). The "canonical" way to do it in z/OS would be using MQSeries as "buffer" and a MQ reader to send the actual stuff to ELK, but we'd like to find a leaner/lighter way to do it.

    Enter zeroMQ. It is a lightweight, zero configuration messaging system written in C++ that _theoretically_ can be compiled under z/OS USS. It can use TCP sockets, IPC endpoints and memory buffers as communication channels and on paper its throughput is more than adequate for our needs. What we would do is to have a bunch of zeroMQ consumers as started tasks which would use the RESTful API to send stuff to ELK, while the PL/1 transactions would send asynchronously to the 0MQ queues.

    My question to the community is: have anyone got any experience running zeromq in the mainframe? Is there any known gotcha that could make this solution unworkable?

    Mixing up traditional z/OS code and UNIX solutions seems like a interesting (and maybe fun) thing to explore!

    Have a nice day, and take care,

    Jordi


    ------------------------------
    Jordi Guillaumes Pons
    ------------------------------


  • 2.  RE: zeroMQ in z/OS

    Posted Fri September 11, 2020 08:13 AM
    Why not do an ELK installation on an IFL?  What am I missing?

    ------------------------------
    STEVEN DICKENS
    ------------------------------



  • 3.  RE: zeroMQ in z/OS

    IBM Champion
    Posted Fri September 11, 2020 09:35 AM
    If I "read between the lines" here ...

    We could use the ELK stack RESTful services, but we don't want to tightly couple our business transactions with the ELK backends (we had a bad experience some months ago).

    I take this to mean that they've experienced blocking or performance issues when putting data "on the wire" within the transaction process.

    Enter zeroMQ. It is a lightweight, zero configuration messaging system ...

    Let's use something that purports to be light on the sending side to avoid bogging down the transactions and protect us from transmission or backend delays via buffering.

    What we would do is to have a bunch of zeroMQ consumers as started tasks which would use the RESTful API to send stuff to ELK, while the PL/1 transactions would send asynchronously to the 0MQ queues.

    And this is how we would configure to for availability/capacity.

    Given these concerns and how you are looking to address them, the location of the ELK backend (IFL, x86 servers, someone's cloud) does not appear to be interesting at the moment.

    I have no idea whether the C++ zeroMQ implementation would successfully compile on z/OS nor any concept of whether or not it would perform to your needs.  I would suggest considering Java (JeroMQ) implementation to "offload" this processing to a zIIP specialty engine (if you have any). 

    For the C++ implementation, the additional consumption of general purpose processor capacity will likely be proportional to message rate; because of this, the C++ version may wind up consuming more general purpose (MLC chargeable) capacity than you might want to pay for.  Depending on your MLC scheme, this could drive up IMS (and other MLC product) costs.

    If zeroMQ doesn't work out, there are a number of other open source MQ implementations that offer Java implementations (e.g. ActiveMQ), as well.


  • 4.  RE: zeroMQ in z/OS

    Posted Fri September 11, 2020 03:04 PM
    Hi Jordi,
    Sounds like a good use case for the Common Data Provider.  Here is a link :-

    https://www.ibm.com/uk-en/marketplace/common-data-provider-for-z-systems

    You could get the IMS transactions to write log information of what you want transferred and get CDP to send the log data to the ELK repository.

    ------------------------------
    Sean Catlow
    ------------------------------



  • 5.  RE: zeroMQ in z/OS

    IBM Champion
    Posted Mon September 14, 2020 04:19 PM
    With IMS, our tools team has created a variety of demos using open source to present IMS data in a variety of different formats. We've done with to non-z/OS linux systems via ELK with output in Kibana and Grafana and to splunk. We've also done this successfully with ELK and Grafana within z/OS Connect Extensions. Here's a bunch of links for demo's and how to's: 
    See a demonstration
    Work with the dashboard using data we provide – no connection to your mainframe needed
      Docker container:
      https://
    hub.docker.com/r/fundisoftware/cexca20-splunk
    How to configure the live feed on your system
      Recipe:
      https://
    developer.ibm.com/recipes/tutorials/forwarding-a-live-feed-of-ims-connect-events-to-splunk/
    Sample Splunk dashboards to get started with your data
     
    Sample dashboards:
      https://
    splunkbase.splunk.com/app/4320/


    ------------------------------
    James Porell
    ------------------------------