AIX

The lack of messages in the errorlog can be caused by /var going read-only.  Is it the only filesystem that goes read-only ?

If it's not there already, temporarily add to syslog.conf:

*.debug /dev/console

(Note that this may be excessively noisy)

As an alternative, create a 1GB ramdisk, format it as JFS2, mount it in /ram, touch /ram/log, add to syslog.conf:

*.debug /ram/log rotate 1m files 512

That way, even if /var goes read-only, there's a way to keep writing the logs.

------------------------------
José Pina Coelho
IT Specialist at Kyndryl
------------------------------

Original Message:
Sent: Wed May 18, 2022 09:01 AM
From: Yaacov Fried
Subject: aix 7.2 strange behaviour

Hi,

Thanks for replying.
- Console window is open but no messages (not also in conslog or any other system log).
- rootvg is a local disk (no SAN at all)
- The clock sets back to the correct time by itself after short period (minutes)

Jacob

------------------------------
Yaacov Fried

Original Message:
Sent: Wed May 18, 2022 08:46 AM
From: Russell Adams
Subject: aix 7.2 strange behaviour

Jacob,

Try leaving a console window open and watch for messages.

Are you booting from SAN? Remounting /var and OS commands failing are
often a sign rootvg is not available.

Do you set the clock forward, or does it fix itself?

Thanks.

On Wed, May 18, 2022 at 12:35:44PM +0000, Yaacov Fried via IBM Community wrote:
> Hi all,
>
> Since upgrading a server from 7200-03-03-1914 to 7200-03-04-1938, I have noticed strange things occurring:
> every day, not always the same time of the day
> - the filesystems (at least /var) are remount to read only mode
> - the system clock goes back three hours!
> - most os commands failed with permission denied (even as root)
>
> then after few minutes, everything goes back to normal.
> Nothing shows on: errlog, conslog, syslog (all facilities) etc.
>
> I locked users out, stopped all users jobs and cron jobs but it didn't help.
> I suspect the upgrade but downgrading is to expensive to do base on a hunch.
>
> Any advice?
> Jacob
>
> ------------------------------
> Yaacov Fried
> ------------------------------
>
>
> Reply to Sender : https://community.ibm.com/community/user/eGroups/PostReply?GroupId=6049&MID=256257&SenderKey=95a8b517-d5d3-4b7c-89e9-f4a0984012e6
>
> Reply to Discussion : https://community.ibm.com/community/user/eGroups/PostReply?GroupId=6049&MID=256257
>
>
>
> You are subscribed to "AIX" as Russell.Adams@AdamsSystems.nl. To change your subscriptions, go to http://community.ibm.com/community/user/preferences?section=Subscriptions. To unsubscribe from this community discussion, go to http://community.ibm.com/HigherLogic/eGroups/Unsubscribe.aspx?UserKey=c23dfccc-9910-40ae-beeb-fdcbced5bf1f&sKey=KeyRemoved&GroupKey=7b554d78-d4dc-417a-b4dc-017e309e5c91.


------------------------------------------------------------------
Russell Adams Russell.Adams@AdamsSystems.nl
Principal Consultant Adams Systems Consultancy
https://adamssystems.nl/


Original Message:
Sent: 5/18/2022 3:49:00 AM
From: Yaacov Fried
Subject: aix 7.2 strange behaviour

Hi all,

Since upgrading a server from 7200-03-03-1914 to 7200-03-04-1938, I have noticed strange things occurring:
every day, not always the same time of the day
- the filesystems (at least /var) are remount to read only mode
- the system clock goes back three hours!
- most os commands failed with permission denied (even as root)

then after few minutes, everything goes back to normal.
Nothing shows on: errlog, conslog, syslog (all facilities) etc.

I locked users out, stopped all users jobs and cron jobs but it didn't help.
I suspect the upgrade but downgrading is to expensive to do base on a hunch.

Any advice?
Jacob

------------------------------
Yaacov Fried
------------------------------

The lack of messages in the errorlog can be caused by /var going read-only.  Is it the only filesystem that goes read-only ?

If it's not there already, temporarily add to syslog.conf:

*.debug /dev/console

(Note that this may be excessively noisy)

As an alternative, create a 1GB ramdisk, format it as JFS2, mount it in /ram, touch /ram/log, add to syslog.conf:

*.debug /ram/log rotate 1m files 512

That way, even if /var goes read-only, there's a way to keep writing the logs.

------------------------------
José Pina Coelho
IT Specialist at Kyndryl

Original Message:
Sent: Wed May 18, 2022 09:01 AM
From: Yaacov Fried
Subject: aix 7.2 strange behaviour

Hi,

Thanks for replying.
- Console window is open but no messages (not also in conslog or any other system log).
- rootvg is a local disk (no SAN at all)
- The clock sets back to the correct time by itself after short period (minutes)

Jacob

------------------------------
Yaacov Fried

Original Message:
Sent: Wed May 18, 2022 08:46 AM
From: Russell Adams
Subject: aix 7.2 strange behaviour

Jacob,

Try leaving a console window open and watch for messages.

Are you booting from SAN? Remounting /var and OS commands failing are
often a sign rootvg is not available.

Do you set the clock forward, or does it fix itself?

Thanks.

On Wed, May 18, 2022 at 12:35:44PM +0000, Yaacov Fried via IBM Community wrote:
> Hi all,
>
> Since upgrading a server from 7200-03-03-1914 to 7200-03-04-1938, I have noticed strange things occurring:
> every day, not always the same time of the day
> - the filesystems (at least /var) are remount to read only mode
> - the system clock goes back three hours!
> - most os commands failed with permission denied (even as root)
>
> then after few minutes, everything goes back to normal.
> Nothing shows on: errlog, conslog, syslog (all facilities) etc.
>
> I locked users out, stopped all users jobs and cron jobs but it didn't help.
> I suspect the upgrade but downgrading is to expensive to do base on a hunch.
>
> Any advice?
> Jacob
>
> ------------------------------
> Yaacov Fried
> ------------------------------
>
>
> Reply to Sender : https://community.ibm.com/community/user/eGroups/PostReply?GroupId=6049&MID=256257&SenderKey=95a8b517-d5d3-4b7c-89e9-f4a0984012e6
>
> Reply to Discussion : https://community.ibm.com/community/user/eGroups/PostReply?GroupId=6049&MID=256257
>
>
>
> You are subscribed to "AIX" as Russell.Adams@AdamsSystems.nl. To change your subscriptions, go to http://community.ibm.com/community/user/preferences?section=Subscriptions. To unsubscribe from this community discussion, go to http://community.ibm.com/HigherLogic/eGroups/Unsubscribe.aspx?UserKey=c23dfccc-9910-40ae-beeb-fdcbced5bf1f&sKey=KeyRemoved&GroupKey=7b554d78-d4dc-417a-b4dc-017e309e5c91.


------------------------------------------------------------------
Russell Adams Russell.Adams@AdamsSystems.nl
Principal Consultant Adams Systems Consultancy
https://adamssystems.nl/


Original Message:
Sent: 5/18/2022 3:49:00 AM
From: Yaacov Fried
Subject: aix 7.2 strange behaviour

Hi all,

Since upgrading a server from 7200-03-03-1914 to 7200-03-04-1938, I have noticed strange things occurring:
every day, not always the same time of the day
- the filesystems (at least /var) are remount to read only mode
- the system clock goes back three hours!
- most os commands failed with permission denied (even as root)

then after few minutes, everything goes back to normal.
Nothing shows on: errlog, conslog, syslog (all facilities) etc.

I locked users out, stopped all users jobs and cron jobs but it didn't help.
I suspect the upgrade but downgrading is to expensive to do base on a hunch.

Any advice?
Jacob

------------------------------
Yaacov Fried
------------------------------

It's sure is a strange behavior...

If it's not done yet, configure ntp.  Also check root's crontab to make sure you're not running ntpdate.
(That back&forth sounds like cron+ntpdate+round-robin to TWO ntp servers, one of which is off by three hours)

As to the read-only filesystem, if it was a filesystem integrity issue I'd expect a panic+dump, but try to follow the procedure for corrupted rootvg filesystems/logvolume (boot from NIM/DVD, varyon the rootvg, fsck all filesystems, reformat the log volume).  <=== mksysb first.

Also, even with internal disks, I've seen non-logging problems (non frequent), try to set the rootvg to critical, that way the machine will reboot if the rootvg disk goes offline.