Another idea is to forward errpt to syslog, and then log to a remote
syslog server in your syslog.conf (@remote).
You have to add the second stanza from here to the ODM which forwards errpt to
syslog by piping to /usr/bin/logger:
https://adamssystems.nl/posts/simple-error-reporting/Then you will get both syslogs and errpt at the remote syslog server.
Separately, I'd focus on how /var is going read only. Can you fsck it
online and see if it throws an error? Linux filesystems often have a
remount read only on error, but I don't recall that on AIX.
On Thu, May 19, 2022 at 09:11:37AM +0000, Jos? Pina Coelho via IBM Community wrote:
> The lack of messages in the errorlog can be caused by /var going read-only. Is it the only filesystem that goes read-only ?
>
>
>
>
>
> If it's not there already, temporarily add to syslog.conf:
>
>
> *.debug /dev/console
>
>
> (Note that this may be excessively noisy)
>
>
>
>
>
> As an alternative, create a 1GB ramdisk, format it as JFS2, mount it in /ram, touch /ram/log, add to syslog.conf:
>
>
> *.debug /ram/log rotate 1m files 512
>
>
>
>
>
> That way, even if /var goes read-only, there's a way to keep writing the logs.
>
>
>
>
> ------------------------------
> Jos? Pina Coelho
> IT Specialist at Kyndryl
> ------------------------------
> -------------------------------------------
> Original Message:
> Sent: Wed May 18, 2022 09:01 AM
> From: Yaacov Fried
> Subject: aix 7.2 strange behaviour
>
> Hi,
>
> Thanks for replying.
> - Console window is open but no messages (not also in conslog or any other system log).
> - rootvg is a local disk (no SAN at all)
> - The clock sets back to the correct time by itself after short period (minutes)
>
> Jacob
>
> ------------------------------
> Yaacov Fried
> ------------------------------
>
> Original Message:
> Sent: Wed May 18, 2022 08:46 AM
> From: Russell Adams
> Subject: aix 7.2 strange behaviour
>
> Jacob,
>
> Try leaving a console window open and watch for messages.
>
> Are you booting from SAN? Remounting /var and OS commands failing are
> often a sign rootvg is not available.
>
> Do you set the clock forward, or does it fix itself?
>
> Thanks.
>
> On Wed, May 18, 2022 at 12:35:44PM +0000, Yaacov Fried via IBM Community wrote:
> > Hi all,
> >
> > Since upgrading a server from 7200-03-03-1914 to 7200-03-04-1938, I have noticed strange things occurring:
> > every day, not always the same time of the day
> > - the filesystems (at least /var) are remount to read only mode
> > - the system clock goes back three hours!
> > - most os commands failed with permission denied (even as root)
> >
> > then after few minutes, everything goes back to normal.
> > Nothing shows on: errlog, conslog, syslog (all facilities) etc.
> >
> > I locked users out, stopped all users jobs and cron jobs but it didn't help.
> > I suspect the upgrade but downgrading is to expensive to do base on a hunch.
> >
> > Any advice?
> > Jacob
> >
> > ------------------------------
> > Yaacov Fried
> > ------------------------------
> >
> >
> > Reply to Sender :
https://community.ibm.com/community/user/eGroups/PostReply?GroupId=6049&MID=256257&SenderKey=95a8b517-d5d3-4b7c-89e9-f4a0984012e6 <https: community.ibm.com/community/user/egroups/postreply?groupid="6049&MID=256257&SenderKey=95a8b517-d5d3-4b7c-89e9-f4a0984012e6">
> >
> > Reply to Discussion :
https://community.ibm.com/community/user/eGroups/PostReply?GroupId=6049&MID=256257 <https: community.ibm.com/community/user/egroups/postreply?groupid="6049&MID=256257">
> >
> >
> >
> > You are subscribed to "AIX" as
Russell.Adams@AdamsSystems.nl <
russell.adams@adamssystems.nl>. To change your subscriptions, go to
http://community.ibm.com/community/user/preferences?section=Subscriptions. <http: community.ibm.com/community/user/preferences?section="Subscriptions."> To unsubscribe from this community discussion, go to
http://community.ibm.com/HigherLogic/eGroups/Unsubscribe.aspx?UserKey=c23dfccc-9910-40ae-beeb-fdcbced5bf1f&sKey=KeyRemoved&GroupKey=7b554d78-d4dc-417a-b4dc-017e309e5c91. <http: community.ibm.com/higherlogic/egroups/unsubscribe.aspx?userkey="c23dfccc-9910-40ae-beeb-fdcbced5bf1f&sKey=KeyRemoved&GroupKey=7b554d78-d4dc-417a-b4dc-017e309e5c91.">
>
>
> ------------------------------------------------------------------
> Russell Adams
Russell.Adams@AdamsSystems.nl <
russell.adams@adamssystems.nl>
> Principal Consultant Adams Systems Consultancy
>
https://adamssystems.nl/ <https: adamssystems.nl/="">
>
>
> Original Message:
> Sent: 5/18/2022 3:49:00 AM
> From: Yaacov Fried
> Subject: aix 7.2 strange behaviour
>
> Hi all,
>
> Since upgrading a server from 7200-03-03-1914 to 7200-03-04-1938, I have noticed strange things occurring:
> every day, not always the same time of the day
> - the filesystems (at least /var) are remount to read only mode
> - the system clock goes back three hours!
> - most os commands failed with permission denied (even as root)
>
> then after few minutes, everything goes back to normal.
> Nothing shows on: errlog, conslog, syslog (all facilities) etc.
>
> I locked users out, stopped all users jobs and cron jobs but it didn't help.
> I suspect the upgrade but downgrading is to expensive to do base on a hunch.
>
> Any advice?
> Jacob
>
> ------------------------------
> Yaacov Fried
> ------------------------------
>
>
>
> Reply to Sender :
https://community.ibm.com/community/user/eGroups/PostReply?GroupId=6049&MID=256394&SenderKey=61c6a8df-6e3d-4b5e-86a5-7ba96a6879a8>
> Reply to Discussion :
https://community.ibm.com/community/user/eGroups/PostReply?GroupId=6049&MID=256394>
>
>
> You are subscribed to "AIX" as
Russell.Adams@AdamsSystems.nl. To change your subscriptions, go to
http://community.ibm.com/community/user/preferences?section=Subscriptions. To unsubscribe from this community discussion, go to
http://community.ibm.com/HigherLogic/eGroups/Unsubscribe.aspx?UserKey=c23dfccc-9910-40ae-beeb-fdcbced5bf1f&sKey=KeyRemoved&GroupKey=7b554d78-d4dc-417a-b4dc-017e309e5c91.------------------------------------------------------------------
Russell Adams
Russell.Adams@AdamsSystems.nlPrincipal Consultant Adams Systems Consultancy
https://adamssystems.nl/
Original Message:
Sent: 5/19/2022 5:12:00 AM
From: José Pina Coelho
Subject: RE: aix 7.2 strange behaviour
The lack of messages in the errorlog can be caused by /var going read-only. Is it the only filesystem that goes read-only ?
If it's not there already, temporarily add to syslog.conf:
*.debug /dev/console
(Note that this may be excessively noisy)
As an alternative, create a 1GB ramdisk, format it as JFS2, mount it in /ram, touch /ram/log, add to syslog.conf:
*.debug /ram/log rotate 1m files 512
That way, even if /var goes read-only, there's a way to keep writing the logs.
------------------------------
José Pina Coelho
IT Specialist at Kyndryl
------------------------------
Original Message:
Sent: Wed May 18, 2022 09:01 AM
From: Yaacov Fried
Subject: aix 7.2 strange behaviour
Hi,
Thanks for replying.
- Console window is open but no messages (not also in conslog or any other system log).
- rootvg is a local disk (no SAN at all)
- The clock sets back to the correct time by itself after short period (minutes)
Jacob
------------------------------
Yaacov Fried
Original Message:
Sent: Wed May 18, 2022 08:46 AM
From: Russell Adams
Subject: aix 7.2 strange behaviour
Jacob,
Try leaving a console window open and watch for messages.
Are you booting from SAN? Remounting /var and OS commands failing are
often a sign rootvg is not available.
Do you set the clock forward, or does it fix itself?
Thanks.
On Wed, May 18, 2022 at 12:35:44PM +0000, Yaacov Fried via IBM Community wrote:
> Hi all,
>
> Since upgrading a server from 7200-03-03-1914 to 7200-03-04-1938, I have noticed strange things occurring:
> every day, not always the same time of the day
> - the filesystems (at least /var) are remount to read only mode
> - the system clock goes back three hours!
> - most os commands failed with permission denied (even as root)
>
> then after few minutes, everything goes back to normal.
> Nothing shows on: errlog, conslog, syslog (all facilities) etc.
>
> I locked users out, stopped all users jobs and cron jobs but it didn't help.
> I suspect the upgrade but downgrading is to expensive to do base on a hunch.
>
> Any advice?
> Jacob
>
> ------------------------------
> Yaacov Fried
> ------------------------------
>
>
> Reply to Sender : https://community.ibm.com/community/user/eGroups/PostReply?GroupId=6049&MID=256257&SenderKey=95a8b517-d5d3-4b7c-89e9-f4a0984012e6
>
> Reply to Discussion : https://community.ibm.com/community/user/eGroups/PostReply?GroupId=6049&MID=256257
>
>
>
> You are subscribed to "AIX" as Russell.Adams@AdamsSystems.nl. To change your subscriptions, go to http://community.ibm.com/community/user/preferences?section=Subscriptions. To unsubscribe from this community discussion, go to http://community.ibm.com/HigherLogic/eGroups/Unsubscribe.aspx?UserKey=c23dfccc-9910-40ae-beeb-fdcbced5bf1f&sKey=KeyRemoved&GroupKey=7b554d78-d4dc-417a-b4dc-017e309e5c91.
------------------------------------------------------------------
Russell Adams Russell.Adams@AdamsSystems.nl
Principal Consultant Adams Systems Consultancy
https://adamssystems.nl/
Original Message:
Sent: 5/18/2022 3:49:00 AM
From: Yaacov Fried
Subject: aix 7.2 strange behaviour
Hi all,
Since upgrading a server from 7200-03-03-1914 to 7200-03-04-1938, I have noticed strange things occurring:
every day, not always the same time of the day
- the filesystems (at least /var) are remount to read only mode
- the system clock goes back three hours!
- most os commands failed with permission denied (even as root)
then after few minutes, everything goes back to normal.
Nothing shows on: errlog, conslog, syslog (all facilities) etc.
I locked users out, stopped all users jobs and cron jobs but it didn't help.
I suspect the upgrade but downgrading is to expensive to do base on a hunch.
Any advice?
Jacob
------------------------------
Yaacov Fried
------------------------------
</https:></russell.adams@adamssystems.nl></http:></http:></russell.adams@adamssystems.nl></https:></https:>