IBM Security Z Security

Hi,

I'm reading a selection of records from the SMF logstreams, and need to be able to output the date and time of the latest SMF record in this format:
FROM=(2022/193,23:30:30:00)'

I can easily get the date and date-time of the latest SMF record with defines:
Define type=smf EndDT("     End",15) max(datetime)
Define type=smf EndD("    EndD",8,juliandate) max(date)

However I can determine the time of the latest SMF record this way, because:
Define type=smf EndT("    EndT",8,smftime) max(time)
results in the maximum time of 23:59:59 when the SMF extract spans midnight.

I've attempted to get the information out of EndDT (the max datetime) using another Define, but this reliably gives me CKR0425 12 Field "ENDDT" to be processed not valid for NEWLIST TYPE=SMF

How can I obtain the time value I need?

------------------------------
Peter Buckley
------------------------------

Hi Peter.
If you want to find the first or the last timestamp in an SMF data set, you have to use the combined date+time value.  That is what you would do with a command

newlist type=smf
define #first min(datetime)
define #last max(datetime)
summary #first #last 

If you want to split up the resulting #first and #last in your output, you must use (output) formats in the DEFINE or SUMMARY command, as described in section "Format names" of the LIST command syntax.  CONVERT in the DEFINE command will do you no good because this works during input processing, so before the SUMMARY takes hold.

I do not remember but you can try if SMFTIME works on the defined fields, like so

summary "FROM=(" | #last(juldate,8) | "," | #last(smftime,5) | ")"

Maybe those format routines are smart, maybe they're picky, just try it out, and try out some others from the syntax manual.
​​​​​​
------------------------------
Rob van Hoboken
------------------------------

Hi Rob,

Sadly this doesn't work.
  Summary  #last(smftime,5)    consistently gives me a time of 52:48

I tried using the format as part of a define, but this gives the same result.
I couldn't find any other format in the manual which would give a meaningful value.

​In this particular job, I'm extracting all SMF up to the current time, so I solved the problem by adding
  Select datetime=today
to ​the newlist which produces the summary, and reverted to using max(time).

I'd still be interested in a general solution to this problem though.

Thanks,

------------------------------
Peter Buckley
------------------------------

Original Message:
Sent: Thu July 14, 2022 05:10 AM
From: Rob van Hoboken
Subject: SMF Time of most recent record

Hi Peter.
If you want to find the first or the last timestamp in an SMF data set, you have to use the combined date+time value.  That is what you would do with a command

newlist type=smf
define #first min(datetime)
define #last max(datetime)
summary #first #last 

If you want to split up the resulting #first and #last in your output, you must use (output) formats in the DEFINE or SUMMARY command, as described in section "Format names" of the LIST command syntax.  CONVERT in the DEFINE command will do you no good because this works during input processing, so before the SUMMARY takes hold.

I do not remember but you can try if SMFTIME works on the defined fields, like so

summary "FROM=(" | #last(juldate,8) | "," | #last(smftime,5) | ")"

Maybe those format routines are smart, maybe they're picky, just try it out, and try out some others from the syntax manual.
​​​​​​
------------------------------
Rob van Hoboken
------------------------------

Hi Peter
I do not know of another format name either.  So you've proven that SMFTIME only works with a 4 byte field containing (only) the time value.  It doesn't know how to pick the time from a full 8 byte SMF timestamp.
You could do, of course, write the full date and time value to a temporary data set and add an extra step that converts the date + time into the desired FROM= value, like so

// EXEC C2RC
//HIGHEST DD DISP=(,PASS),DSN=&&HIGHEST
//SYSIN DD *
newlist type=smf nopage dd=highest
define #last max(datetime)
summary #last(juldate,8) #last(18)
// EXEC C2RC
//HIGHEST DD DISP=(OLD,PASS),DSN=&&HIGHEST
//CKR2PASS DD SYSOUT=*
//SYSIN DD *
deftype type=$highest
alloc type=$highest dd=highest
newlist type=$highest nopage dd=ckr2pass
  define #juldate as word(record,1)
  define #time as word(record,3)
  sortlist record(0) / "FROM=(" | #juldate(0) | "," | #time(0) | ")"
​​
------------------------------
Rob van Hoboken
------------------------------

Hi Peter, As you have found out max(datetime) clearly must be used to be able to accomodate input SMF data that spans over a midnight .  The time field can be used to find earliest/latest time of all days in input data, but not time from earliest/latest day (unless by coincidence).

I think you probably should submit an RFE to get and smf data in that exact format you desire.

In the meantime, How critical is it to generate that exact same format ?  The datetime format is acceptable to a subsequent carla run in a SELECT statement (this is what the SIEM adaptors do by excluding any input smf records up to that high water mark from the previous run.)

Could you perhaps (temporarily) make do with generating such a statement with just date, no time (or a hardcoded 00:00:00.00), along with a CARLA EXCLUDE statement that will exclude any records read prior to the previous max(datetime)? 
  See bottom of SCKRCARL member CKQLEEF for how they do it (dd=SMFHWOUT)

It would mean that you may be reading in SMF data you don't need to , but you won't be reporting on it. 

A minor cosmetic question:  Did you really mean to have three semicolons, or perhaps FROM=(2022/193,23:30:30.00)'

------------------------------
Simon Dodge
------------------------------

Original Message:
Sent: Wed July 13, 2022 06:38 AM
From: Peter Buckley
Subject: SMF Time of most recent record

Hi,

I'm reading a selection of records from the SMF logstreams, and need to be able to output the date and time of the latest SMF record in this format:
FROM=(2022/193,23:30:30:00)'

I can easily get the date and date-time of the latest SMF record with defines:
Define type=smf EndDT("     End",15) max(datetime)
Define type=smf EndD("    EndD",8,juliandate) max(date)

However I can determine the time of the latest SMF record this way, because:
Define type=smf EndT("    EndT",8,smftime) max(time)
results in the maximum time of 23:59:59 when the SMF extract spans midnight.

I've attempted to get the information out of EndDT (the max datetime) using another Define, but this reliably gives me CKR0425 12 Field "ENDDT" to be processed not valid for NEWLIST TYPE=SMF

How can I obtain the time value I need?

------------------------------
Peter Buckley
------------------------------

Hi Simon,

Thanks for your reply. I should explain what I'm doing more clearly.

I need to run a job to analyse SMF records from the logstream, at regular intervals. I need to be sure that I pick up all records since the last run (but only those records).
I could read the entire logstream and then Select the relevant timespan in Carla, but that's very inefficient. So I use LOGR subsystem parameters instead.
E.g. Alloc type=SMF dsn=IFASMF.stream subsys=(LOGR,IFASEXIT,'FROM=(2022/206,23:30:00),LOCAL','sid(SYSx)')

The format of date and time is determined by IFASEXIT, so we cannot change this.

I already have a workaround as mentioned above (Generate the MAX(TIME) after Selecting date=TODAY), so I think that's good enough. I doubt this will be a common use case.
So I'm happy to close this topic. Many thanks to yourself and Rob.

------------------------------
Peter Buckley
------------------------------

Original Message:
Sent: Tue July 19, 2022 10:06 AM
From: Simon Dodge
Subject: SMF Time of most recent record

Hi Peter, As you have found out max(datetime) clearly must be used to be able to accomodate input SMF data that spans over a midnight .  The time field can be used to find earliest/latest time of all days in input data, but not time from earliest/latest day (unless by coincidence).

I think you probably should submit an RFE to get and smf data in that exact format you desire.

In the meantime, How critical is it to generate that exact same format ?  The datetime format is acceptable to a subsequent carla run in a SELECT statement (this is what the SIEM adaptors do by excluding any input smf records up to that high water mark from the previous run.)

Could you perhaps (temporarily) make do with generating such a statement with just date, no time (or a hardcoded 00:00:00.00), along with a CARLA EXCLUDE statement that will exclude any records read prior to the previous max(datetime)? 
  See bottom of SCKRCARL member CKQLEEF for how they do it (dd=SMFHWOUT)

It would mean that you may be reading in SMF data you don't need to , but you won't be reporting on it. 

A minor cosmetic question:  Did you really mean to have three semicolons, or perhaps FROM=(2022/193,23:30:30.00)'

------------------------------
Simon Dodge

Original Message:
Sent: Wed July 13, 2022 06:38 AM
From: Peter Buckley
Subject: SMF Time of most recent record

Hi,

I'm reading a selection of records from the SMF logstreams, and need to be able to output the date and time of the latest SMF record in this format:
FROM=(2022/193,23:30:30:00)'

I can easily get the date and date-time of the latest SMF record with defines:
Define type=smf EndDT("     End",15) max(datetime)
Define type=smf EndD("    EndD",8,juliandate) max(date)

However I can determine the time of the latest SMF record this way, because:
Define type=smf EndT("    EndT",8,smftime) max(time)
results in the maximum time of 23:59:59 when the SMF extract spans midnight.

I've attempted to get the information out of EndDT (the max datetime) using another Define, but this reliably gives me CKR0425 12 Field "ENDDT" to be processed not valid for NEWLIST TYPE=SMF

How can I obtain the time value I need?

------------------------------
Peter Buckley
------------------------------

Hey Peter
Does my workaround to use a 2nd step that formats the normal date-time value into a FROM=() command work for you?

------------------------------
Rob van Hoboken
------------------------------

Original Message:
Sent: Wed July 27, 2022 05:46 AM
From: Peter Buckley
Subject: SMF Time of most recent record

Hi Simon,

Thanks for your reply. I should explain what I'm doing more clearly.

I need to run a job to analyse SMF records from the logstream, at regular intervals. I need to be sure that I pick up all records since the last run (but only those records).
I could read the entire logstream and then Select the relevant timespan in Carla, but that's very inefficient. So I use LOGR subsystem parameters instead.
E.g. Alloc type=SMF dsn=IFASMF.stream subsys=(LOGR,IFASEXIT,'FROM=(2022/206,23:30:00),LOCAL','sid(SYSx)')

The format of date and time is determined by IFASEXIT, so we cannot change this.

I already have a workaround as mentioned above (Generate the MAX(TIME) after Selecting date=TODAY), so I think that's good enough. I doubt this will be a common use case.
So I'm happy to close this topic. Many thanks to yourself and Rob.

------------------------------
Peter Buckley
------------------------------

Hi Rob,

That workaround produces the time in the correct format (once I corrected the typo:  define #time as word(record,2) )
However the date now comes out as 28Jul2022, rather than  2022/209.
Inevitably, trying to use juliandate in the first pass has no effect, and trying to use it to modify the #juldate output results in no value being output for that field.​

------------------------------
Peter Buckley
------------------------------

Original Message:
Sent: Wed July 27, 2022 05:59 AM
From: Rob van Hoboken
Subject: SMF Time of most recent record

Hey Peter
Does my workaround to use a 2nd step that formats the normal date-time value into a FROM=() command work for you?

------------------------------
Rob van Hoboken

Original Message:
Sent: Wed July 27, 2022 05:46 AM
From: Peter Buckley
Subject: SMF Time of most recent record

Hi Simon,

Thanks for your reply. I should explain what I'm doing more clearly.

I need to run a job to analyse SMF records from the logstream, at regular intervals. I need to be sure that I pick up all records since the last run (but only those records).
I could read the entire logstream and then Select the relevant timespan in Carla, but that's very inefficient. So I use LOGR subsystem parameters instead.
E.g. Alloc type=SMF dsn=IFASMF.stream subsys=(LOGR,IFASEXIT,'FROM=(2022/206,23:30:00),LOCAL','sid(SYSx)')

The format of date and time is determined by IFASEXIT, so we cannot change this.

I already have a workaround as mentioned above (Generate the MAX(TIME) after Selecting date=TODAY), so I think that's good enough. I doubt this will be a common use case.
So I'm happy to close this topic. Many thanks to yourself and Rob.

------------------------------
Peter Buckley
------------------------------

Ah, then your original DEFINE command comes in handy, using DATE instead of DATETIME to find the julian date:

// EXEC C2RC
//HIGHEST DD DISP=(,PASS),DSN=&&HIGHEST
//SYSIN DD *
newlist type=smf nopage dd=highest
define #lastdate max(date)
define #last max(datetime)
summary #lastdate(juldate,8) #last(18)
// EXEC C2RC
//HIGHEST DD DISP=(OLD,PASS),DSN=&&HIGHEST
//CKR2PASS DD SYSOUT=*
//SYSIN DD *
deftype type=$highest
alloc type=$highest dd=highest
newlist type=$highest nopage dd=ckr2pass
define #juldate as word(record,1)
define #time as word(record,2)
sortlist record(0) / "FROM=(" | #juldate(0) | "," | #time(0) | ")"

------------------------------
Rob van Hoboken
------------------------------

Original Message:
Sent: Thu July 28, 2022 05:26 AM
From: Peter Buckley
Subject: SMF Time of most recent record

Hi Rob,

That workaround produces the time in the correct format (once I corrected the typo:  define #time as word(record,2) )
However the date now comes out as 28Jul2022, rather than  2022/209.
Inevitably, trying to use juliandate in the first pass has no effect, and trying to use it to modify the #juldate output results in no value being output for that field.​

------------------------------
Peter Buckley
------------------------------

Hi Rob,

That's true enough.
In which case, there's no need for the 2-pass process.
I'm happy that:
  The date in max(datettime) will always be the same as max(date);
  The max(time) for smf records from the max(date) is always the same as the time in max(datetime).
So I have everything I need in the first pass.

------------------------------
Peter Buckley
------------------------------

Original Message:
Sent: Thu July 28, 2022 07:29 AM
From: Rob van Hoboken
Subject: SMF Time of most recent record

Ah, then your original DEFINE command comes in handy, using DATE instead of DATETIME to find the julian date:

// EXEC C2RC
//HIGHEST DD DISP=(,PASS),DSN=&&HIGHEST
//SYSIN DD *
newlist type=smf nopage dd=highest
define #lastdate max(date)
define #last max(datetime)
summary #lastdate(juldate,8) #last(18)
// EXEC C2RC
//HIGHEST DD DISP=(OLD,PASS),DSN=&&HIGHEST
//CKR2PASS DD SYSOUT=*
//SYSIN DD *
deftype type=$highest
alloc type=$highest dd=highest
newlist type=$highest nopage dd=ckr2pass
define #juldate as word(record,1)
define #time as word(record,2)
sortlist record(0) / "FROM=(" | #juldate(0) | "," | #time(0) | ")"

------------------------------
Rob van Hoboken

Original Message:
Sent: Thu July 28, 2022 05:26 AM
From: Peter Buckley
Subject: SMF Time of most recent record

Hi Rob,

That workaround produces the time in the correct format (once I corrected the typo:  define #time as word(record,2) )
However the date now comes out as 28Jul2022, rather than  2022/209.
Inevitably, trying to use juliandate in the first pass has no effect, and trying to use it to modify the #juldate output results in no value being output for that field.​

------------------------------
Peter Buckley
------------------------------