This post refers to entries in the SystemOut.log /

Maximo has a number of processes that execute in the background. One of the key background components is the crontask.

If one of these fails then there is often no error message visible in the front-end. One of the key background components is the crontask.

This is a scheduled piece of code that runs and performs various functions.

Each crontask has an instance which records the configuration and schedule for the crontask run.

Maximo will log details about the crontask instance runs every minute. These entries record the last time that the instance ran. The crontask name separates the crontask and instance with a . e.g. SLROUTECLEANUP.SLROUTE

Crontask Name = SLROUTECLEANUP

Instance = SLROUTE

[INFO] [MAXIMO] [] BMXAA6372I - Host name: 1.1.1.2. Server name: MAXIMO. Cron task name: SLROUTECLEANUP.SLROUTE. Last run: 2016-11-24 08:24:37.0

[INFO] [MAXIMO] [] BMXAA6372I - Host name: 1.1.1.2. Server name: MAXIMO. Cron task name: DepreciationFinTransCronTask.InstDepreciationCronTask. Last run: 2016-11-25 16:19:35.0

You may have multiple instances for a crontask where each instance handles a different siteID.

Understanding where a crontask has been running

Crontasks can run on different servers/JVMs and move around depending on how the system is configured and what is happening on the system. Configuring a system to control where the crontasks can be run is a challenge that is too big for a single blog posting and IBM have published a number of crontasks and blogs that cover different aspects of this challenge.

Identifying where these crontasks have run can be important if you suspect that a crontask has experienced an error during a run.

Each crontask has a history tab which can be used to record crontask runs.

Figure 1 Viewing the crontask history tab

Each ACTION entry is an entry where the crontask performed its run.

Be default the crontask runs are not logged so it isn’t possible to see in the front-end which host/JVM they ran on or when they ran.

Tip -> Change the history configuration so that it is enabled and has 1000 history records.

Tip -> Don’t worry if Maximo keeps more than 1000 records for short periods of time, the crontask manager code will eventually clean up the old history records.

7.5 introduced the attribute labelled Runtime Error attribute for storing error messages.

Unfortunately very few out of box crontasks write anything in this attribute when there is a problem. So if the crontask fails to perform as you expect then you need to look in the SystemOut.log file

In fairness to IBM some crontasks could generate a lot of output e.g. if a condition means that an error is generated for every record it processes. It would be useful if the crontask modified the attribute if there is a problem e.g. to indicate that there was a problem and details are in the logs.

Tip -> If someone is developing a crontask for you then check if they are using this attribute, even if it is just to indicate that there was a problem and the logs should be checked.

In some cases the crontask produces their own log file e.g. PMWOGen crontask or the Reorder log file.

An interesting point is that these special log files don’t use the same logging component as the main Maximo component when they write their entries. If they did then the entries would have the same log entry format e.g. datestamp.

This is sample PMwogen.log entries:

Generate work order from PMs on site BEDFORD finished 2/10/16 10:19 AM.

BMXAA3212E - Error while generating work order for PM PM-CART.

               BMXAA3186E - The asset or location associated with PM PM-CART has a status of NOT READY. Change the status to an Active status to generate work orders.

Generate work order from PMs on site BEDFORD finished 2/10/16 10:19 AM.

BMXAA3212E - Error while generating work order for PM PM-CONV1.

               BMXAA3186E - The asset or location associated with PM PM-CONV1 has a status of NOT READY. Change the status to an Active status to generate work orders.

Sometimes it would be useful to have the full datestamp when something is going wrong. I have had a situation where the PMWOGen crontask took 8 hours to run. The crontask writes an entry with the hour and minute when it completes a record but with three entries in a minute you can’t tell how long each took and if one was significantly slower than the others.

Finding crontask related log entries in 7.5 and later

7.5 introduced a new feature to help correlate cron entries.

Tip -> Enable this by setting the system property mxe.logging.CorrelationEnabled to 1. None of my customers have reported performance degradation after setting this value.

When the correlation flag has been enabled it prefixes log entries with an ID e.g. CID-CRON-2941 that identifies messages for the crontask run. It also writes messages that indicate the start and end of the crontask run and the length of time that it ran for.

This ID can be very useful when dealing with crontasks that run for several minutes or when there are lots of log messages being written by multiple pieces of unrelated code.

 [INFO] [MAXIMO] [CID-CRON-2941] Correlation started.

[INFO] [MAXIMO] [CID-CRON-2941] BMXAA6719I - USER = (MAXADMIN) SPID = (192) app (null) object (MAXUSER) : select * from maxuser where userid = 'MAXADMIN' and status in (select value from synonymdomain where domainid = 'MAXUSERSTATUS' and maxvalue = 'ACTIVE')

[INFO] [MAXIMO] [CID-CRON-2940] BMXAA6719I - USER = (MAXADMIN) SPID = (69) app (null) object (TASKSCHEDULER) : select * from taskscheduler where taskname='DepreciationFinTransCronTask.InstDepreciationCronTask'

[INFO] [MAXIMO] [CID-CRON-2941] BMXAA6719I - USER = (MAXADMIN) SPID = (71) app (null) object (PERSON) : select * from person where personid = 'MAXADMIN'

[INFO] [MAXIMO] [CID-CRON-2941] Correlated data: BEGIN TaskName:AsyncImmediateJobCron InstanceName:AsyncImmediate ElapsedTime:16 ms END

This sample indicates that two crontasks were running at the same time. This is indicated by the two different correlation IDs CID-CRON-2940 and CID-CRON-2941.

There is a similar setting for user related operations and this will be discussed in a different article.

7.6 crontask moves & what it means for the logs

At 7.6 the crontask manager code was modified so that each time a crontask runs it switches between JVMs. This was implemented to ensure that the crontask kept running in the event of a JVM failing.

Figure 2 Crontask history at 7.6 when an escalation runs every minute

If a crontask has a problem then it means that we have more work to investigate the cause of the problem. An example is if a record is entered via the front-end and an escalation (which uses a crontask) then updates the record. If the escalation fails to update the record then the logs need to be checked to see what error occurred.

Users often don’t know/remember the time that they performed an action so the time window for when the action occurred could be varied. In some cases the users won’t know when the action should occur e.g. if a crontask runs at a pre-set interval.

In a multi-JVM installation we need to check the crontask history to see which JVM the crontask was running on. Once we know which JVM it was running on then we can check the relevant logs.

For a crontask running once a day this is a small irritation and it means checking a different log every day.

If the crontask runs more often (e.g. a crontask that runs every minute) then it becomes necessary to check all the logs using automated searches.

I would be interested in people’s views on this innovation and how it has affected your support tasks.

Vetasi training/information

One of the limitations of crontasks is the need to stop and start the crontask when you change crontask parameters. The lifecycle of a crontask and why this occurs is explained in the Vetasi Java read only course.

There are tools available to quickly search through the logs and these may be discussed in a later topic if there is sufficient interest.

Vetasi recognises that controlling where crontasks run can be a serious issue and we have a document that we can discuss with customers to explain the different options and the implications.