We have a batch job step which processes a CARLa run containing 5 separate NEWLIST queries against SMF. I am looking to make this as efficient as possible because the run averages 60 minutes of CPU time to process. To that end I am looking to pre-select SMF records that qualify for any of the 5 NEWLIST statements and only read through the entirety of the SMF records a single time.

My understanding/assumption is that the CKRCARLA program is currently reading all the input SMF records 5 separate times for the five separate NEWLISTs.

Reading the CARLa Command Reference v2.5.0, it seems that-

·       LIKELIST refers to, and reuses, SELECT/EXCLUDE criteria specified in another NEWLIST and appends any SELECT/EXCLUDE criteria specified in the current NEWLIST to that. It will read through the entirety of the input data provided using that combined selection criteria.

·       PROFLIST takes the results from a specified NEWLIST (stored in memory) and filters that down further using the SELECT/EXCLUDE criteria in the current NEWLIST.

Is my assumption correct?

Am I interpreting the manual correctly?

------------------------------
Michael Nielsen
------------------------------

Hi Michael,

The program should read the SMF records only once regardless.

The purpose of LIKELIST is to save CPU by not having to evaluate an equivalent selection multiple times.

The special property of PROFLIST is that it selects a particular RACF profile as an entity as opposed a segment. In other words, if you select BASE segments in a first NEWLIST, you can apply that profile selection (based on fields in the BASE segment) to a different NEWLIST in which you are selecting, say, TSO segments. This keyword does not apply to TYPE=SMF [for which the records are not profile segments].

If you want to show the SELECT / EXCLUDE statements for your five NEWLISTs, I could take a look if I have any recommendations for optimizing them.

[I take it that you use the batch processing only once against a particular set of input SMF files; otherwise you might reduce the time needed by creating an input source that does not contain records that do not get selected by any of them.]

Regards,

------------------------------
Jeroen Tiggelman
IBM - Software Development and Level 3 Support Manager IBM Security zSecure Suite
Delft
------------------------------

Hi Michael
If you select SMF records like this:

newlist type=smf name=access
select type=80 user=ibmuser event=access
newlist type=smf name=commands
select type=80 user=ibmuser event=allcmds

CKRCARLA will do (approximately):

read SMF record
process selection for newlist access
find field type, compare with 80; find field user, (translate to uppercase), compare with IBMUSER; find field event, compare decimal value with equivalent of ACCESS
keep record for ACCESS processing if all three match
process selection for newlist commands
find field type, compare with 80; find field user, (translate to uppercase), compare with IBMUSER; find field event, compare decimal value with equivalent of any of the command events
keep record for COMMANDS processing if all three match

So, the record is only read once for (any number of) newlists, but fields are found and evaluated for each newlist (again and again).

You could also optimize the selection by putting identical field criteria into a dummy newlist that does not keep the record, but only finds and evaluates the fields:

newlist type=smf name=smfsel outlim=0
select type=80 user=ibmuser 
newlist type=smf name=access
select likelist=smfsel event=access
newlist type=smf name=commands
select likelist=smfsel event=allcmds

This time, CKRCARLA will do (approximately):

read SMF record
process selection for newlist smfsel
find field type, compare with 80; find field user, (translate to uppercase), compare with IBMUSER; find field event
set TRUE value for LIKELIST processing if all three match, set FALSE value if any fail
do NOT keep record for this newlist, due to OUTLIM=0
process selection for newlist access
retrieve LIKELIST result from SMFSEL; find field event, compare decimal value with equivalent of ACCESS
keep record for ACCESS processing if match
process selection for newlist commands
retrieve LIKELIST result from SMFSEL; find field event, compare decimal value with equivalent of any of the command events
keep record for COMMANDS processing if match

You see that common selection criteria are only processed once.  This approach is used extensively in the ISPF panels and reports, where a newlist name=smfsel is generated (or specified in SYSIN DD *) and referenced in subsequent newlists, see e.g. SCKRCARL(CKALFR80)

You can think of the SMFSEL newlist as a globally defined boolean that contains the result of all select and exclude commands in the dummy newlist.  You reference the result by a LIKELIST=SMFSEL, like you would reference a boolean.

------------------------------
Rob van Hoboken
------------------------------

By the way, the translate to uppercase that I mentioned is performed using a UNICODE service routine that is processor intensive.  So if you have a gazillion records to process, and you know that the field value is always an uppercase character value, so can prevent conversion by using an exact match compare, like so:

select user='IBMUSER'c

the c modifier states you are sure the left-hand field is already uppercase.  Do not (try to) use this on convenience field values for fields that aren't really character, like ACCESS or ALLCMDS on the EVENT field: EVENT is really a decimal that has selection and output in terms that people understand instead of numbers.

------------------------------
Rob van Hoboken
------------------------------

------------------------------
Michael Nielsen
------------------------------

Yes, that looks good. :-)

------------------------------
Jeroen Tiggelman
IBM - Software Development and Level 3 Support Manager IBM Security zSecure Suite
Delft
------------------------------

Original Message:
Sent: Mon May 08, 2023 10:41 AM
From: Michael Nielsen
Subject: Understanding the behavioral differences between PROFLIST and LIKELIST

Thanks both for the explanations. This has clarified a lot for me.

This is the CARLa run in its current state.

mergelist

newlist type=smf dd=ckreport nopage retain
def RcdDat(3,char,'RcdDat')      as smf_field(11,3)
select  class=user, RcdDat='xxxxxF'x,

        EVENT=RACINIT, EVENT<>RACINIT(0,8,13,12,32,40)

newlist type=smf dd=ckreport nopage retain

select  class=user, RcdDat='xxxxxF'X, EVENT=RACINIT(8,13)

newlist type=smf dd=ckreport nopage retain

select  type=80 class=dataset, RcdDat='xxxxxF'X

newlist type=smf dd=ckreport nopage retain

select  type=80 class<>(user,dataset), RcdDat='xxxxxF'X

newlist type=smf dd=ckreport nopage retain

select  type=80, RcdDat='xxxxxF'X, class=(FSOBJ,PROCESS,FSSEC,FSACCESS,FSEXEC,UNIXPRIV,FACILITY,SURROGAT)

exclude unix_function=(chdir,dub,undub_exit)

endmerge

Using the information and examples you've provided, this could be split into two groups of newlists- Type=RACINIT and Type=80. If I create dummy newlists it would look similar to this.

 

mergelist

newlist type=smf name=racinit outlim=0
def RcdDat(3,char,'RcdDat')      as smf_field(11,3)

select  class=user, EVENT=RACINIT, RcdDat='xxxxxF'x

newlist type=smf dd=ckreport nopage retain

select  likelist=racinit, EVENT<>RACINIT(0,8,13,12,32,40)

newlist type=smf dd=ckreport nopage retain

select  likelist=racinit, EVENT=RACINIT(8,13)

newlist type=smf name=type80 outlim=0

select  type=80, RcdDat='xxxxxF'x

newlist type=smf dd=ckreport nopage retain

select  likelist=type80, class=dataset

newlist type=smf dd=ckreport nopage retain

select  likelist=type80, class<>(user,dataset)

newlist type=smf dd=ckreport nopage retain

select  likelist=type80, class=(FSOBJ,PROCESS,FSSEC,FSACCESS,FSEXEC,UNIXPRIV,FACILITY,SURROGAT)

exclude unix_function=(chdir,dub,undub_exit)

endmerge

Additional cleanup could be done as well.
-the report is always for yesterday and the SMF file is for yesterday, so the RcdDat selection criteria could be dropped
-I believe the class=user selection could be dropped from the RACINIT searches
-class<>(user,dataset) was intended to capture general resources and can (should?) be written as class=general

Does this seem like an appropriate approach? Is there anything else you would suggest?

------------------------------
Michael Nielsen
------------------------------