IBM Security Z Security

 View Only

  • 1.  Understanding the behavioral differences between PROFLIST and LIKELIST

    Posted Fri May 05, 2023 09:38 AM

    We have a batch job step which processes a CARLa run containing 5 separate NEWLIST queries against SMF. I am looking to make this as efficient as possible because the run averages 60 minutes of CPU time to process. To that end I am looking to pre-select SMF records that qualify for any of the 5 NEWLIST statements and only read through the entirety of the SMF records a single time.

    My understanding/assumption is that the CKRCARLA program is currently reading all the input SMF records 5 separate times for the five separate NEWLISTs.

    Reading the CARLa Command Reference v2.5.0, it seems that-

    ·       LIKELIST refers to, and reuses, SELECT/EXCLUDE criteria specified in another NEWLIST and appends any SELECT/EXCLUDE criteria specified in the current NEWLIST to that. It will read through the entirety of the input data provided using that combined selection criteria.

    ·       PROFLIST takes the results from a specified NEWLIST (stored in memory) and filters that down further using the SELECT/EXCLUDE criteria in the current NEWLIST.

    Is my assumption correct?

    Am I interpreting the manual correctly?



    ------------------------------
    Michael Nielsen
    ------------------------------


  • 2.  RE: Understanding the behavioral differences between PROFLIST and LIKELIST

    Posted Fri May 05, 2023 10:31 AM

    Hi Michael,

    The program should read the SMF records only once regardless.

    The purpose of LIKELIST is to save CPU by not having to evaluate an equivalent selection multiple times.

    The special property of PROFLIST is that it selects a particular RACF profile as an entity as opposed a segment. In other words, if you select BASE segments in a first NEWLIST, you can apply that profile selection (based on fields in the BASE segment) to a different NEWLIST in which you are selecting, say, TSO segments. This keyword does not apply to TYPE=SMF [for which the records are not profile segments].

    If you want to show the SELECT / EXCLUDE statements for your five NEWLISTs, I could take a look if I have any recommendations for optimizing them.

    [I take it that you use the batch processing only once against a particular set of input SMF files; otherwise you might reduce the time needed by creating an input source that does not contain records that do not get selected by any of them.]

    Regards,



    ------------------------------
    Jeroen Tiggelman
    IBM - Software Development and Level 3 Support Manager IBM Security zSecure Suite
    Delft
    ------------------------------

    Original Message


  • 3.  RE: Understanding the behavioral differences between PROFLIST and LIKELIST

    IBM Champion
    Posted Mon May 08, 2023 04:55 AM
    Edited by Rob van Hoboken Mon May 08, 2023 04:59 AM

    Hi Michael
    If you select SMF records like this:

    newlist type=smf name=access
    select type=80 user=ibmuser event=access
    newlist type=smf name=commands
    select type=80 user=ibmuser event=allcmds

    CKRCARLA will do (approximately):

    read SMF record
    process selection for newlist access
    find field type, compare with 80; find field user, (translate to uppercase), compare with IBMUSER; find field event, compare decimal value with equivalent of ACCESS
    keep record for ACCESS processing if all three match
    process selection for newlist commands
    find field type, compare with 80; find field user, (translate to uppercase), compare with IBMUSER; find field event, compare decimal value with equivalent of any of the command events
    keep record for COMMANDS processing if all three match

    So, the record is only read once for (any number of) newlists, but fields are found and evaluated for each newlist (again and again).

    You could also optimize the selection by putting identical field criteria into a dummy newlist that does not keep the record, but only finds and evaluates the fields:

    newlist type=smf name=smfsel outlim=0
    select type=80 user=ibmuser 
    newlist type=smf name=access
    select likelist=smfsel event=access
    newlist type=smf name=commands
    select likelist=smfsel event=allcmds

    This time, CKRCARLA will do (approximately):

    read SMF record
    process selection for newlist smfsel
    find field type, compare with 80; find field user, (translate to uppercase), compare with IBMUSER; find field event
    set TRUE value for LIKELIST processing if all three match, set FALSE value if any fail
    do NOT keep record for this newlist, due to OUTLIM=0
    process selection for newlist access
    retrieve LIKELIST result from SMFSEL; find field event, compare decimal value with equivalent of ACCESS
    keep record for ACCESS processing if match
    process selection for newlist commands
    retrieve LIKELIST result from SMFSEL; find field event, compare decimal value with equivalent of any of the command events
    keep record for COMMANDS processing if match

    You see that common selection criteria are only processed once.  This approach is used extensively in the ISPF panels and reports, where a newlist name=smfsel is generated (or specified in SYSIN DD *) and referenced in subsequent newlists, see e.g. SCKRCARL(CKALFR80)

    You can think of the SMFSEL newlist as a globally defined boolean that contains the result of all select and exclude commands in the dummy newlist.  You reference the result by a LIKELIST=SMFSEL, like you would reference a boolean.

    ------------------------------
    Rob van Hoboken
    ------------------------------



  • 4.  RE: Understanding the behavioral differences between PROFLIST and LIKELIST

    IBM Champion
    Posted Mon May 08, 2023 05:28 AM
    Edited by Rob van Hoboken Mon May 08, 2023 05:28 AM

    By the way, the translate to uppercase that I mentioned is performed using a UNICODE service routine that is processor intensive.  So if you have a gazillion records to process, and you know that the field value is always an uppercase character value, so can prevent conversion by using an exact match compare, like so:

    select user='IBMUSER'c

    the c modifier states you are sure the left-hand field is already uppercase.  Do not (try to) use this on convenience field values for fields that aren't really character, like ACCESS or ALLCMDS on the EVENT field: EVENT is really a decimal that has selection and output in terms that people understand instead of numbers.

    ------------------------------
    Rob van Hoboken
    ------------------------------



  • 5.  RE: Understanding the behavioral differences between PROFLIST and LIKELIST

    Posted Mon May 08, 2023 10:42 AM
    Thanks both for the explanations. This has clarified a lot for me.

    This is the CARLa run in its current state.

    mergelist
    newlist type=smf dd=ckreport nopage retain
    def RcdDat(3,char,'RcdDat')      as smf_field(11,3)
    select  class=user, RcdDat='xxxxxF'x,
            EVENT=RACINIT, EVENT<>RACINIT(0,8,13,12,32,40)
    newlist type=smf dd=ckreport nopage retain
    select  class=user, RcdDat='xxxxxF'X, EVENT=RACINIT(8,13)
    newlist type=smf dd=ckreport nopage retain
    select  type=80 class=dataset, RcdDat='xxxxxF'X
    newlist type=smf dd=ckreport nopage retain
    select  type=80 class<>(user,dataset), RcdDat='xxxxxF'X
    newlist type=smf dd=ckreport nopage retain
    select  type=80, RcdDat='xxxxxF'X, class=(FSOBJ,PROCESS,FSSEC,FSACCESS,FSEXEC,UNIXPRIV,FACILITY,SURROGAT)
    exclude unix_function=(chdir,dub,undub_exit)
    endmerge

    Using the information and examples you've provided, this could be split into two groups of newlists- Type=RACINIT and Type=80. If I create dummy newlists it would look similar to this.
     
    mergelist
    newlist type=smf name=racinit outlim=0
    def RcdDat(3,char,'RcdDat')      as smf_field(11,3)
    select  class=user, EVENT=RACINIT, RcdDat='xxxxxF'x
    newlist type=smf dd=ckreport nopage retain
    select  likelist=racinit, EVENT<>RACINIT(0,8,13,12,32,40)
    newlist type=smf dd=ckreport nopage retain
    select  likelist=racinit, EVENT=RACINIT(8,13)
    newlist type=smf name=type80 outlim=0
    select  type=80, RcdDat='xxxxxF'x
    newlist type=smf dd=ckreport nopage retain
    select  likelist=type80, class=dataset
    newlist type=smf dd=ckreport nopage retain
    select  likelist=type80, class<>(user,dataset)
    newlist type=smf dd=ckreport nopage retain
    select  likelist=type80, class=(FSOBJ,PROCESS,FSSEC,FSACCESS,FSEXEC,UNIXPRIV,FACILITY,SURROGAT)
    exclude unix_function=(chdir,dub,undub_exit)
    endmerge

    Additional cleanup could be done as well.
    -the report is always for yesterday and the SMF file is for yesterday, so the RcdDat selection criteria could be dropped
    -I believe the class=user selection could be dropped from the RACINIT searches
    -class<>(user,dataset) was intended to capture general resources and can (should?) be written as class=general

    Does this seem like an appropriate approach? Is there anything else you would suggest?


    ------------------------------
    Michael Nielsen
    ------------------------------

    Original Message


  • 6.  RE: Understanding the behavioral differences between PROFLIST and LIKELIST

    Posted Mon May 08, 2023 11:15 AM

    Yes, that looks good. :-)



    ------------------------------
    Jeroen Tiggelman
    IBM - Software Development and Level 3 Support Manager IBM Security zSecure Suite
    Delft
    ------------------------------

    Original Message