IBM Z and LinuxONE IBM Z

Expand all | Collapse all

Free product to UNTERSE SMF and VB data on distributed platforms

  • 1.  Free product to UNTERSE SMF and VB data on distributed platforms

    Posted Tue March 09, 2021 03:54 PM

    I hope this isn't considered advertising if it's for a free product, but we'd like to announce our new WWUNTERSE (no-charge) product.

     

    Here at Watson & Walker we love SMF data.  And judging by the volumes of it that most z/OS sites generate, we are not the only ones.  SMF can tell you just about anything you could want to know about what is happening on your systems.  The downside of having that vast sea of information is that it takes up lots of space – we have clients that generate over 2.5TB of SMF data per day.  And if you are like nearly every other z/OS customer, you send at least some of that data to distributed systems for postprocessing.  Sending TBs of SMF data over the network can take a long time, especially if the target system is remote.

     

    The good news is that SMF data tends to compress very well.  Compression ratios of around 7:1 are normal, and even 10:1 is not unusual, depending on the mix of record types.  If you are transferring gigabytes or even terabytes of data every day, those good compression ratios are vital.

     

    The not-so-good news is that the most common tools for compressing SMF data on z/OS do not have a distributed equivalent:

     

        AMATERSE achieves excellent compression with SMF data, but there is no tool on Windows or Linux to unterse the tersed version of VBS files.

     

        zEDC is another popular tool for compressing SMF data on z/OS.  However, the compressed sequential data sets created by BSAM or QSAM are in a proprietary format.  So even if you could get them over to a distributed system without them being uncompressed, there is no tool for Windows or Linux that can read those compressed files.

     

    To address this need, and because we spent a lot of our own time moving SMF data around, our colleague Mario Bezzi created a program called WWUNTERSE.  WWUNTERSE runs on Windows and Linux.  It can process tersed files that were created from VBS files.  It also supports tersed versions of F, FB, V, VB, and VBS files.  And it provides the option to include the Block Descriptor Word (BDW) in the output file (as required by MXG/SAS on distributed systems).  And the best news of all is that there is no charge for WWUNTERSE.  It is available for download now, from the Free Tools page of the Watson & Walker public website at https://watsonwalker.com/software/free-tools/

     

    Please take it for a test drive and let us know what you think.

     

    Stay Tuned!

     

    Cheryl Watson

     





  • 2.  RE: Free product to UNTERSE SMF and VB data on distributed platforms

    Posted Tue March 09, 2021 04:16 PM
    Cheryl:

    I downloaded your wwunterse after it was mentioned at SHARE last week. Works great, just needs a GUI interface to it. :-)

    ------------------------------
    Jim Elliott
    Senior IT Consultant
    GlassHouse Systems Inc.
    ------------------------------



  • 3.  RE: Free product to UNTERSE SMF and VB data on distributed platforms

    Posted Mon August 02, 2021 02:20 PM
    Hi Cheryl

    It's great to hear that there is now a supported program that will UNPACK a TERSE file from z/OS on a PC platform.

    Did you know that that there is a program called WINTERSE that has been around for many years that does the same thing as WWUNTERSE and also handles the PACK/SPACK process so that you can PACK/SPACK a PC file to be uploaded to z/OS and then use AMATERSE to UNPACK it?

    We use WINTERSE to support our PC based analytics process for z/OS SMF data, including our "Monitor for SAS" SMF records.

    However and sadly, WINTERSE is not supported and I have tried searching in vain for contacts at intervals for years, but unfortunately all my leads fails to reveal an signs of an Owner, Supporter nor source code for WINTERSE.

    So .. are you considering, or would you consider, extending WWUNTERSE to handle the PACK/SPACK process as well?

    Cheers
    Steve
    CEO and Founder, ITMetrics Ltd

    ------------------------------
    Steve Bagshaw
    ------------------------------



  • 4.  RE: Free product to UNTERSE SMF and VB data on distributed platforms

    Posted Mon August 02, 2021 02:37 PM

    Hi Steve,

     

    I'm in touch with our developer, but because of the upcoming SHARE conference, he won't be able to consider this until September. We'll post something then or get back to you directly.

     

    P.S. I'm fairly new to the IBM Community forums. It seems like there is a difference between doing a 'reply all' from your email server and 'reply to group' in the body of the message. Can anyone provide advice on that?

     

    Best regards,

    Cheryl Watson

    Watson & Walker, Inc.

     






  • 5.  RE: Free product to UNTERSE SMF and VB data on distributed platforms

    Posted Wed August 04, 2021 04:20 PM
    I have a free open source Java program that will uncompress gzip data into z/OS datasets. You can compress on a PC using gzip, upload to a FB z/OS dataset and uncompress the data into a VBS etc dataset. You need to know the output RECFM and LRECL and specify them in the JCL, they are not stored like they are using TERSE, but if you are compressing on a PC TERSE wouldn't know the attributes anyway.

    I use it to transfer SMF data from PC to mainframe.

    Gzip performance also seems better than TERSE - particularly if you have zEDC available.

    The program is available here:
    https://github.com/BlackHillSoftware/z-java/tree/master/java/compress

    ------------------------------
    Andrew Rowley
    ------------------------------



  • 6.  RE: Free product to UNTERSE SMF and VB data on distributed platforms

    Posted Wed August 04, 2021 04:20 PM
    There is another program. I used it several times. It is written in Java and, theoretically, can work anywhere: https://github.com/openmainframeproject/tersedecompress
    It's good when there is a choice :)


    ------------------------------
    Nikolay Kultashev
    ------------------------------



  • 7.  RE: Free product to UNTERSE SMF and VB data on distributed platforms

    Posted Wed August 04, 2021 07:22 PM
    Mario and I added the VB binary support to tersedecompress, so I'm glad you find it useful. :-)

    The differences between TERSE and the Java compression utility I wrote are:

    TERSE
    • Mainframe -> mainframe or mainframe -> PC using these utilities
    • Part of the OS, installed on every system, supported on the mainframe by IBM 
    • Stores and recreates dataset attributes on the mainframe

    z/OS Java compress program
    • Mainframe -> mainframe, mainframe -> PC or PC -> mainframe
    • Gzip format: better performance and compression than TERSE, particularly if zEDC is available
    • Does not store dataset attributes, you need to know them and specify them in JCL when uncompressing
    • Gzip format can be created and read on most platforms, z/OS needs some help (this program) to read and write from datasets


    ------------------------------
    Andrew Rowley
    ------------------------------



  • 8.  RE: Free product to UNTERSE SMF and VB data on distributed platforms

    Posted Wed December 22, 2021 11:51 AM
    Hi @Cheryl Watson​, thank you for posting this. Extremely timely for me. 

    I grabbed the current tools (Unix & Windows) but i am wondering if by any change you'd have a Mac OS X version.... (i run on Mac and it would avoid me having to look for a way to run one of the other versions). 

    Cheers and Happy Holidays!

    Cesar

    ------------------------------
    Cesar Kastoun
    ------------------------------



  • 9.  RE: Free product to UNTERSE SMF and VB data on distributed platforms

    Posted Thu December 23, 2021 09:45 AM
    Hello Cheryl, i downloaded the zip file but my anti-virus is telling me it has malware in it (below). A quick research says it's probably a false positive as the file has an autorun and many anti-virus software associates that with malware. 

    Any way around this? 

    Many thanks for your help

    Cesar


    This message was created automatically by mail delivery software. Your email message was not delivered to the intended recipients because malware was detected.

    --- Additional Information ---:

    Subject: Re: Your recommendation - Need access to robust Linux or Window for a couple of weeks
    Sender: xxx@DELOITTE.com

    Time received: 12/22/2021 3:54:24 PM
    Message ID:<SJ0PR85MB0510A603BC05B2E3D951752AA17D9@SJ0PR85MB0510.NAMPRD85.PROD.OUTLOOK.COM>
    Detections found:
    WWUNTERSE-V1R3M0.zip W64/Autorun.EO.gen!Eldorado

    ------------------------------
    Cesar Kastoun
    ------------------------------



  • 10.  RE: Free product to UNTERSE SMF and VB data on distributed platforms

    Posted Mon December 27, 2021 04:37 PM
    Hi Cesar, Mario here, replying on behalf of Cheryl as I am the maintainer of WWUNTERSE.

    Would you please provide more information about the issue? We have hundreds of customers who downloaded the tool, and nobody reported a similar issue so far.

    We virus checked the file before making it available in April, and we didn't change it so far. I scanned it again today under Windows using  the latest version of Norton-360 and again, I don't see any problem. This is the scan log.

    -----------------
    Scan Statistics:
    Scan Start:
    Local: 27/12/2021 11:46
    UTC: 27/12/2021 10:46
    Scan Time: 0 seconds
    Scan Targets: C:\Users\mario\Desktop\WWUNTERSE-V1R3M0.zip
    Counts:
    Total items scanned: 5
    - Files & Directories: 5
    - Registry Entries: 0
    - Processes & Startup Items: 0
    - Network & Browser Items: 0
    - Other: 0
    - Trusted Files: 0
    - Skipped Files: 0

    Total security risks detected: 0
    Total items resolved: 0
    Total items that require attention: 0

    Resolved Threats:
    No risks have been resolved

    Unresolved Threats:
    No unresolved risks
    ----------------------

    Thank you,
    mario

    ------------------------------
    Mario Bezzi
    ------------------------------