I have a disturbing system, which every once in a while freezes. The
sceen on the monitor is some X scene (usually has Chrome running, but
they again that is often the case) but the keyboard,mouse, do nothing.
Trying to log on from the network fails with no response from the
machine. Alt-ctrl-del does nothing. The only way to recover is via the
power switch on the back.
Afterwards, looking at /var/log/syslog, or /var/log/messages shows
nothingsignificant that I can see just before the freeze.

Updated Mga8. Kernel 5.15.88-desktop-1.mga8.

The only thing in the journalctl logs from just before is

-------------------------------
Jul 24 23:22:34 tunnel.physics.ubc.ca dnf[2573748]: teams 8.9 kB/s | 1.5 kB 00:00
Jul 24 23:22:35 tunnel.physics.ubc.ca dnf[2573748]: Metadata cache created.

Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: dnf-makecache.service: Succeeded.
Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: Finished dnf makecache.
Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: dnf-makecache.service: Consumed 1.527s CPU time.
Jul 24 23:28:00 tunnel.physics.ubc.ca kernel: Shorewall:sshd-fw:DROP:IN=eno1 OUT= MAC=4c:ed:fb:c2:2a:f3:a0:ab:1b:88:6e:58:08:00 SRC=183.106.205.242 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=56 ID=64660 PROTO=TCP SPT=31651 DPT=22 WINDOW=16988 RES=0x00 SYN URGP=0
Jul 24 23:35:31 tunnel.physics.ubc.ca kernel: Shorewall:net-fw:DROP:IN=eno1 OUT= MAC=4c:ed:fb:c2:2a:f3:a0:ab:1b:88:6e:58:08:00 SRC=185.225.74.53 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=54321 PROTO=TCP SPT=33231 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0
-- Reboot --
---------------------------------------

I recently replaced the power supply thinking it might be the cause.
But although freezes seem to be occuring less frequently, it is still
occasionally freezing. (It seems to be occuring every two or three
months).

On Fri, 28 Jul 2023 01:16:54 -0000 (UTC), William Unruh wrote:
> I have a disturbing system, which every once in a while freezes. The
> sceen on the monitor is some X scene (usually has Chrome running, but
> they again that is often the case) but the keyboard,mouse, do nothing.
> Trying to log on from the network fails with no response from the
> machine. Alt-ctrl-del does nothing. The only way to recover is via the
> power switch on the back.
> Afterwards, looking at /var/log/syslog, or /var/log/messages shows
> nothingsignificant that I can see just before the freeze.
>
> Updated Mga8. Kernel 5.15.88-desktop-1.mga8.
>
> The only thing in the journalctl logs from just before is
>
> -------------------------------
> Jul 24 23:22:34 tunnel.physics.ubc.ca dnf[2573748]: teams 8.9 kB/s | 1.5 kB 00:00
> Jul 24 23:22:35 tunnel.physics.ubc.ca dnf[2573748]: Metadata cache created.
>
> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: dnf-makecache.service: Succeeded.
> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: Finished dnf makecache.
> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: dnf-makecache.service: Consumed 1.527s CPU time.
> Jul 24 23:28:00 tunnel.physics.ubc.ca kernel: Shorewall:sshd-fw:DROP:IN=eno1 OUT= MAC=4c:ed:fb:c2:2a:f3:a0:ab:1b:88:6e:58:08:00 SRC=183.106.205.242 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=56 ID=64660 PROTO=TCP SPT=31651 DPT=22 WINDOW=16988 RES=0x00 SYN URGP=0
> Jul 24 23:35:31 tunnel.physics.ubc.ca kernel: Shorewall:net-fw:DROP:IN=eno1 OUT= MAC=4c:ed:fb:c2:2a:f3:a0:ab:1b:88:6e:58:08:00 SRC=185.225.74.53 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=54321 PROTO=TCP SPT=33231 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0
> -- Reboot --
> ---------------------------------------
>
>
> I recently replaced the power supply thinking it might be the cause.
> But although freezes seem to be occuring less frequently, it is still
> occasionally freezing. (It seems to be occuring every two or three
> months).

Off hand it seems like the cpu gets into a tight loop and quits processing
interrupts.

I would install lm_sensors, configure/run lm_sensors, sensord and enable core dump.

I have also modified ~/.bash_profile to check for core dump files and
uses xmessage to provide a pop up if any are found.

William Unruh <unruh@invalid.ca> writes:
> I have a disturbing system, which every once in a while freezes. The
> sceen on the monitor is some X scene (usually has Chrome running, but
> they again that is often the case) but the keyboard,mouse, do nothing.
> Trying to log on from the network fails with no response from the
> machine. Alt-ctrl-del does nothing. The only way to recover is via the
> power switch on the back.
> Afterwards, looking at /var/log/syslog, or /var/log/messages shows
> nothingsignificant that I can see just before the freeze.
>
> Updated Mga8. Kernel 5.15.88-desktop-1.mga8.
>
> The only thing in the journalctl logs from just before is
>
> -------------------------------
> Jul 24 23:22:34 tunnel.physics.ubc.ca dnf[2573748]: teams 8.9 kB/s |
> 1.5 kB 00:00
> Jul 24 23:22:35 tunnel.physics.ubc.ca dnf[2573748]: Metadata cache created.
>
> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]:
> dnf-makecache.service: Succeeded.
> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: Finished dnf makecache.
> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]:
> dnf-makecache.service: Consumed 1.527s CPU time.
> Jul 24 23:28:00 tunnel.physics.ubc.ca kernel:
> Shorewall:sshd-fw:DROP:IN=eno1 OUT=
> MAC=4c:ed:fb:c2:2a:f3:a0:ab:1b:88:6e:58:08:00 SRC=183.106.205.242
> DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=56 ID=64660 PROTO=TCP
> SPT=31651 DPT=22 WINDOW=16988 RES=0x00 SYN URGP=0
> Jul 24 23:35:31 tunnel.physics.ubc.ca kernel:
> Shorewall:net-fw:DROP:IN=eno1 OUT=
> MAC=4c:ed:fb:c2:2a:f3:a0:ab:1b:88:6e:58:08:00 SRC=185.225.74.53
> DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=54321 PROTO=TCP
> SPT=33231 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0
> -- Reboot --
> ---------------------------------------

There’s a chance that the kernel had something to say that couldn’t be
written to the logs due to the crash; unfortunately that can be rather
hard to get hold of in this case. If the machine has a serial port then
you might be able to get it to write kernel logs to that. But you might
just be out of luck l-(

--
https://www.greenend.org.uk/rjk/

On 7/28/23 03:16, William Unruh wrote:
> I have a disturbing system, which every once in a while freezes. The
> sceen on the monitor is some X scene (usually has Chrome running, but
> they again that is often the case) but the keyboard,mouse, do nothing.
> Trying to log on from the network fails with no response from the
> machine. Alt-ctrl-del does nothing. The only way to recover is via the
> power switch on the back.
> Afterwards, looking at /var/log/syslog, or /var/log/messages shows
> nothingsignificant that I can see just before the freeze.

This reminds me of the issue I have had with running two plasma5
sessions, after a while they will just eat up the memory and when all is
gone, then everything freezes, you can't login from remote and local
login takes too long from entering username to entering password that
the login is canceled.

I would run a memory check once in a while, for example use crontab to
run this one liner:

date && ps -Ao user,uid,comm,pid,pcpu,pmem --sort=-pmem | head -n 11 &&
echo "----" >> /var/log/memusage.log

Then you can see if there is a process that grows.

--
//Aho

On Fri, 28 Jul 2023 10:27:47 +0200, J.O. Aho wrote:
> On 7/28/23 03:16, William Unruh wrote:
>> I have a disturbing system, which every once in a while freezes. The
>> sceen on the monitor is some X scene (usually has Chrome running, but
>> they again that is often the case) but the keyboard,mouse, do nothing.
>> Trying to log on from the network fails with no response from the
>> machine. Alt-ctrl-del does nothing. The only way to recover is via the
>> power switch on the back.
>> Afterwards, looking at /var/log/syslog, or /var/log/messages shows
>> nothingsignificant that I can see just before the freeze.
>
> This reminds me of the issue I have had with running two plasma5
> sessions, after a while they will just eat up the memory and when all is
> gone, then everything freezes, you can't login from remote and local
> login takes too long from entering username to entering password that
> the login is canceled.
>
> I would run a memory check once in a while, for example use crontab to
> run this one liner:
>
> date && ps -Ao user,uid,comm,pid,pcpu,pmem --sort=-pmem | head -n 11 &&
> echo "----" >> /var/log/memusage.log
>
> Then you can see if there is a process that grows.

Cute, and with just a little bit of scripting/coding you could automate
the check by flagging any line with value above some watermark for cpu
and memory percentage.

If it was me, I would print any greater than 1.0 for ether one.

Since values have a decimal I would guess that I would have to
use bc to test greater than watermark value if using bash.

Simple case statement to change watermark values based on command.
where needed.

If you want a journal entry it would be something like
echo 'msg_here' | systemd-cat -t app_name_here -p type_msg_here

message types emerg, alert, crit, err, warning, notice, info, debug

In a multi-node set up I send an email to LAN admin.

On Fri, 28 Jul 2023 01:16:54 +0000, William Unruh wrote:

> I have a disturbing system, which every once in a while freezes. The
> sceen on the monitor is some X scene (usually has Chrome running, but
> they again that is often the case) but the keyboard,mouse, do nothing.
> Trying to log on from the network fails with no response from the
> machine. Alt-ctrl-del does nothing. The only way to recover is via the
> power switch on the back.
> Afterwards, looking at /var/log/syslog, or /var/log/messages shows
> nothingsignificant that I can see just before the freeze.
>
> Updated Mga8. Kernel 5.15.88-desktop-1.mga8.
>
> The only thing in the journalctl logs from just before is
>
> -------------------------------
> Jul 24 23:22:34 tunnel.physics.ubc.ca dnf[2573748]: teams 8.9 kB/s | 1.5 kB 00:00
> Jul 24 23:22:35 tunnel.physics.ubc.ca dnf[2573748]: Metadata cache created.
>
> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: dnf-makecache.service: Succeeded.
> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: Finished dnf makecache.
> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: dnf-makecache.service: Consumed 1.527s CPU time.
> Jul 24 23:28:00 tunnel.physics.ubc.ca kernel: Shorewall:sshd-fw:DROP:IN=eno1 OUT= MAC=4c:ed:fb:c2:2a:f3:a0:ab:1b:88:6e:58:08:00 SRC=183.106.205.242 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=56 ID=64660 PROTO=TCP SPT=31651 DPT=22 WINDOW=16988 RES=0x00 SYN URGP=0
> Jul 24 23:35:31 tunnel.physics.ubc.ca kernel: Shorewall:net-fw:DROP:IN=eno1 OUT= MAC=4c:ed:fb:c2:2a:f3:a0:ab:1b:88:6e:58:08:00 SRC=185.225.74.53 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=54321 PROTO=TCP SPT=33231 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0
> -- Reboot --
> ---------------------------------------
>
>
> I recently replaced the power supply thinking it might be the cause.
> But although freezes seem to be occuring less frequently, it is still
> occasionally freezing. (It seems to be occuring every two or three
> months).

Pure speculation, but I think a update to the lib rpms months ago
introduced something that now and then causes the Gnome desktop
on Wayland to lock up.

My main machine and backup are both Intel 64-bit quad-core cpu, with
ram 32 GB for the former and 4 GB for the latter. Loads imposed are
trivial (mostly Opera browser, with firefox the alternative for any
siteopera does not work well with).

Now and then the desktop would lock up. When things were worst, the
backup would lock up every few days or maybe few weeks, and the main
machine less often.

Often, it would prove impossible to kill the browser or desktop.
Power switch restart is easier than shifting to the other
machine and trying to ssh in to reboot so that was my remedy.

Inability to come up with any hint of why, but rarity of lock-up,
has led me to just wait in expectation that someday (Mageia 9
perhaps) such problems will be ironed out.

Lockup has been less often in the past couple of months.

Cheers!

jim b.

--
UNIX is not user-unfriendly, it merely
expects users to be computer friendly.

On 2023-07-27 21:16, William Unruh wrote:

> Updated Mga8. Kernel 5.15.88-desktop-1.mga8.

Did you mean you updated FROM that kernel?

I hope so. Latest Mageia 8 kernel is 5.15.122-1.

TJ

On 7/27/2023 9:16 PM, William Unruh wrote:
> I have a disturbing system, which every once in a while freezes. The
> sceen on the monitor is some X scene (usually has Chrome running, but
> they again that is often the case) but the keyboard,mouse, do nothing.
> Trying to log on from the network fails with no response from the
> machine. Alt-ctrl-del does nothing. The only way to recover is via the
> power switch on the back.
> Afterwards, looking at /var/log/syslog, or /var/log/messages shows
> nothingsignificant that I can see just before the freeze.
>
> Updated Mga8. Kernel 5.15.88-desktop-1.mga8.
>
> The only thing in the journalctl logs from just before is
>
> -------------------------------
> Jul 24 23:22:34 tunnel.physics.ubc.ca dnf[2573748]: teams 8.9 kB/s | 1.5 kB 00:00
> Jul 24 23:22:35 tunnel.physics.ubc.ca dnf[2573748]: Metadata cache created.
>
> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: dnf-makecache.service: Succeeded.
> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: Finished dnf makecache.
> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: dnf-makecache.service: Consumed 1.527s CPU time.
> Jul 24 23:28:00 tunnel.physics.ubc.ca kernel: Shorewall:sshd-fw:DROP:IN=eno1 OUT= MAC=4c:ed:fb:c2:2a:f3:a0:ab:1b:88:6e:58:08:00 SRC=183.106.205.242 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=56 ID=64660 PROTO=TCP SPT=31651 DPT=22 WINDOW=16988 RES=0x00 SYN URGP=0
> Jul 24 23:35:31 tunnel.physics.ubc.ca kernel: Shorewall:net-fw:DROP:IN=eno1 OUT= MAC=4c:ed:fb:c2:2a:f3:a0:ab:1b:88:6e:58:08:00 SRC=185.225.74.53 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=54321 PROTO=TCP SPT=33231 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0
> -- Reboot --
> ---------------------------------------
>
>
> I recently replaced the power supply thinking it might be the cause.
> But although freezes seem to be occuring less frequently, it is still
> occasionally freezing. (It seems to be occuring every two or three
> months).

This bug does not yield to linear thinking, unfortunately.

It's not a hardware issue. There's something in the middle
of the graphics stack, which runs at frame rate (VSYNC), and
it has debounce for mouse built into it. (Like, your Razor mouse ?
Bad bad idea. You want a moldy old mouse with low DPI right now. Switch to PS/2 ports,
if ya gottem.) It uses a timer. It gets behind. And, it lunches the input subsystem.

We're seeing this elsewhere. Or, at least, log messages of the same type
as the bug. Have a look in your /var/log.

Try a radical change of DE, and see if stability returns.

As one developer described it "it's not your CPU which is too slow,
it is the Mutter architecture which is too slow".

Now, it's either that, or something has changed which has brought
an old bug back.

Paul

On 7/28/23 11:56, Bit Twister wrote:
> On Fri, 28 Jul 2023 10:27:47 +0200, J.O. Aho wrote:
>> On 7/28/23 03:16, William Unruh wrote:
>>> I have a disturbing system, which every once in a while freezes. The
>>> sceen on the monitor is some X scene (usually has Chrome running, but
>>> they again that is often the case) but the keyboard,mouse, do nothing.
>>> Trying to log on from the network fails with no response from the
>>> machine. Alt-ctrl-del does nothing. The only way to recover is via the
>>> power switch on the back.
>>> Afterwards, looking at /var/log/syslog, or /var/log/messages shows
>>> nothingsignificant that I can see just before the freeze.
>>
>> This reminds me of the issue I have had with running two plasma5
>> sessions, after a while they will just eat up the memory and when all is
>> gone, then everything freezes, you can't login from remote and local
>> login takes too long from entering username to entering password that
>> the login is canceled.
>>
>> I would run a memory check once in a while, for example use crontab to
>> run this one liner:
>>
>> date && ps -Ao user,uid,comm,pid,pcpu,pmem --sort=-pmem | head -n 11 &&
>> echo "----" >> /var/log/memusage.log
>>
>> Then you can see if there is a process that grows.
>
> Cute, and with just a little bit of scripting/coding you could automate
> the check by flagging any line with value above some watermark for cpu
> and memory percentage.
>
> If it was me, I would print any greater than 1.0 for ether one.
>
> Since values have a decimal I would guess that I would have to
> use bc to test greater than watermark value if using bash.

You could use awk, have to always point out this great tool as it bears
my name

date && ps -Ao user,uid,comm,pid,pcpu,pmem --sort=-pmem | awk -v
min="0.3" '$6 >= min || $6=="%MEM"' && echo "----"

Just change the -v min="0.3" to the value you think it has be or more to
be displayed, we also keep displaying the column names, sure there is
better ways of doing this, but my skills not that great.

> In a multi-node set up I send an email to LAN admin.

In a multi node setup I would use icinga or nagios to monitor things,
sure you can do the same with elk too.

--
//Aho

On 2023-07-28, Bit Twister <BitTwister@mouse-potato.com> wrote:
> On Fri, 28 Jul 2023 01:16:54 -0000 (UTC), William Unruh wrote:
>> I have a disturbing system, which every once in a while freezes. The
>> sceen on the monitor is some X scene (usually has Chrome running, but
>> they again that is often the case) but the keyboard,mouse, do nothing.
>> Trying to log on from the network fails with no response from the
>> machine. Alt-ctrl-del does nothing. The only way to recover is via the
>> power switch on the back.
>> Afterwards, looking at /var/log/syslog, or /var/log/messages shows
>> nothingsignificant that I can see just before the freeze.
>>
>> Updated Mga8. Kernel 5.15.88-desktop-1.mga8.
>>
>> The only thing in the journalctl logs from just before is
>>
>> -------------------------------
>> Jul 24 23:22:34 tunnel.physics.ubc.ca dnf[2573748]: teams 8.9 kB/s | 1.5 kB 00:00
>> Jul 24 23:22:35 tunnel.physics.ubc.ca dnf[2573748]: Metadata cache created.
>>
>> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: dnf-makecache.service: Succeeded.
>> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: Finished dnf makecache.
>> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: dnf-makecache.service: Consumed 1.527s CPU time.
>> Jul 24 23:28:00 tunnel.physics.ubc.ca kernel: Shorewall:sshd-fw:DROP:IN=eno1 OUT= MAC=4c:ed:fb:c2:2a:f3:a0:ab:1b:88:6e:58:08:00 SRC=183.106.205.242 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=56 ID=64660 PROTO=TCP SPT=31651 DPT=22 WINDOW=16988 RES=0x00 SYN URGP=0
>> Jul 24 23:35:31 tunnel.physics.ubc.ca kernel: Shorewall:net-fw:DROP:IN=eno1 OUT= MAC=4c:ed:fb:c2:2a:f3:a0:ab:1b:88:6e:58:08:00 SRC=185.225.74.53 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=54321 PROTO=TCP SPT=33231 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0
>> -- Reboot --
>> ---------------------------------------
>>
>>
>> I recently replaced the power supply thinking it might be the cause.
>> But although freezes seem to be occuring less frequently, it is still
>> occasionally freezing. (It seems to be occuring every two or three
>> months).
>
>
> Off hand it seems like the cpu gets into a tight loop and quits processing
> interrupts.
>
> I would install lm_sensors, configure/run lm_sensors, sensord and enable core dump.
>
> I have also modified ~/.bash_profile to check for core dump files and
> uses xmessage to provide a pop up if any are found.
>

OK, Here are the last two sensord reports just before the freeze
I do not see anything out of the ordinary here. The last one (23:28:55)
occured. The freeze ( as inferred from the last entry into /var/log/syslog
occured aroung 23:35:31

(
--------------------------------
Jul 24 23:35:31 tunnel kernel: [1181287.585321] Shorewall:net-fw:DROP:IN=eno1 OUT= MAC=4c:ed:fb:c2:2a:f3:a0:ab:1b:88:6e:58:08:00 SRC=185.225.74.53 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=54321 PROTO=TCP SPT=33231 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0
Jul 25 09:13:25 tunnel kernel: [ 0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12
--------------------------------------

----------------------------------
Jul 24 23:08:55 tunnel sensord: Chip: nvme-pci-0600
Jul 24 23:08:55 tunnel sensord: Adapter: PCI adapter
Jul 24 23:08:55 tunnel sensord: Composite: 24.9 C (min = -40.1 C, max = 83.8 C)
Jul 24 23:08:55 tunnel sensord: Sensor 2: 24.9 C (min = -40.1 C, max = 83.8 C)
Jul 24 23:08:55 tunnel sensord: Chip: coretemp-isa-0000
Jul 24 23:08:55 tunnel sensord: Adapter: ISA adapter
Jul 24 23:08:55 tunnel sensord: Package id 0: 29.0 C
Jul 24 23:08:55 tunnel sensord: Core 0: 28.0 C
Jul 24 23:08:55 tunnel sensord: Core 1: 27.0 C
Jul 24 23:08:55 tunnel sensord: Core 2: 28.0 C
Jul 24 23:08:55 tunnel sensord: Core 3: 27.0 C
Jul 24 23:08:55 tunnel sensord: Chip: acpitz-acpi-0
Jul 24 23:08:55 tunnel sensord: Adapter: ACPI interface
Jul 24 23:08:55 tunnel sensord: temp1: 27.8 C
Jul 24 23:28:55 tunnel sensord: Chip: nvme-pci-0600
Jul 24 23:28:55 tunnel sensord: Adapter: PCI adapter
Jul 24 23:28:55 tunnel sensord: Composite: 24.9 C (min = -40.1 C, max = 83.8 C)
Jul 24 23:28:55 tunnel sensord: Sensor 2: 24.9 C (min = -40.1 C, max = 83.8 C)
Jul 24 23:28:55 tunnel sensord: Chip: coretemp-isa-0000
Jul 24 23:28:55 tunnel sensord: Adapter: ISA adapter
Jul 24 23:28:55 tunnel sensord: Package id 0: 29.0 C
Jul 24 23:28:55 tunnel sensord: Core 0: 28.0 C
Jul 24 23:28:55 tunnel sensord: Core 1: 27.0 C
Jul 24 23:28:55 tunnel sensord: Core 2: 27.0 C
Jul 24 23:28:55 tunnel sensord: Core 3: 27.0 C
Jul 24 23:28:55 tunnel sensord: Chip: acpitz-acpi-0
Jul 24 23:28:55 tunnel sensord: Adapter: ACPI interface
Jul 24 23:28:55 tunnel sensord: temp1: 27.8 C
----------------------------------------------

I cannot find any core files, and I do not think I am suppressing them.

On 2023-07-28, J.O. Aho <user@example.net> wrote:
> On 7/28/23 03:16, William Unruh wrote:
>> I have a disturbing system, which every once in a while freezes. The
>> sceen on the monitor is some X scene (usually has Chrome running, but
>> they again that is often the case) but the keyboard,mouse, do nothing.
>> Trying to log on from the network fails with no response from the
>> machine. Alt-ctrl-del does nothing. The only way to recover is via the
>> power switch on the back.
>> Afterwards, looking at /var/log/syslog, or /var/log/messages shows
>> nothingsignificant that I can see just before the freeze.
>
> This reminds me of the issue I have had with running two plasma5
> sessions, after a while they will just eat up the memory and when all is
> gone, then everything freezes, you can't login from remote and local
> login takes too long from entering username to entering password that
> the login is canceled.

Except in my case, nothing I type does anything, the mouse cursor does
not move, Teh Alt-ctrl-F keys do nothing, alt-ctrl-det or alt-ctrl-bksp
do nothing, gkrellm stops updating. Usually when I have run out of
memory, somethings still work, and the machine slows down drastically as
it starts to swap. This is just a complete sudden freeze. (Unlike this
past time, sometimes this has happened while I am working on the
machine. This time it happened while I was asleep)
>
> I would run a memory check once in a while, for example use crontab to
> run this one liner:
>
> date && ps -Ao user,uid,comm,pid,pcpu,pmem --sort=-pmem | head -n 11 &&
> echo "----" >> /var/log/memusage.log
>
> Then you can see if there is a process that grows.
>

On Fri, 28 Jul 2023 22:14:16 -0000 (UTC), William Unruh wrote:
> On 2023-07-28, Bit Twister <BitTwister@mouse-potato.com> wrote:
>> On Fri, 28 Jul 2023 01:16:54 -0000 (UTC), William Unruh wrote:
>>> I have a disturbing system, which every once in a while freezes. The
>>> sceen on the monitor is some X scene (usually has Chrome running, but
>>> they again that is often the case) but the keyboard,mouse, do nothing.
>>> Trying to log on from the network fails with no response from the
>>> machine. Alt-ctrl-del does nothing. The only way to recover is via the
>>> power switch on the back.
>>> Afterwards, looking at /var/log/syslog, or /var/log/messages shows
>>> nothingsignificant that I can see just before the freeze.
>>>
>>> Updated Mga8. Kernel 5.15.88-desktop-1.mga8.
>>>
>>> The only thing in the journalctl logs from just before is
>>>
>>> -------------------------------
>>> Jul 24 23:22:34 tunnel.physics.ubc.ca dnf[2573748]: teams 8.9 kB/s | 1.5 kB 00:00
>>> Jul 24 23:22:35 tunnel.physics.ubc.ca dnf[2573748]: Metadata cache created.
>>>
>>> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: dnf-makecache.service: Succeeded.
>>> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: Finished dnf makecache.
>>> Jul 24 23:22:35 tunnel.physics.ubc.ca systemd[1]: dnf-makecache.service: Consumed 1.527s CPU time.
>>> Jul 24 23:28:00 tunnel.physics.ubc.ca kernel: Shorewall:sshd-fw:DROP:IN=eno1 OUT= MAC=4c:ed:fb:c2:2a:f3:a0:ab:1b:88:6e:58:08:00 SRC=183.106.205.242 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=56 ID=64660 PROTO=TCP SPT=31651 DPT=22 WINDOW=16988 RES=0x00 SYN URGP=0
>>> Jul 24 23:35:31 tunnel.physics.ubc.ca kernel: Shorewall:net-fw:DROP:IN=eno1 OUT= MAC=4c:ed:fb:c2:2a:f3:a0:ab:1b:88:6e:58:08:00 SRC=185.225.74.53 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=54321 PROTO=TCP SPT=33231 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0
>>> -- Reboot --
>>> ---------------------------------------
>>>
>>>
>>> I recently replaced the power supply thinking it might be the cause.
>>> But although freezes seem to be occuring less frequently, it is still
>>> occasionally freezing. (It seems to be occuring every two or three
>>> months).
>>
>>
>> Off hand it seems like the cpu gets into a tight loop and quits processing
>> interrupts.
>>
>> I would install lm_sensors, configure/run lm_sensors, sensord and enable core dump.
>>
>> I have also modified ~/.bash_profile to check for core dump files and
>> uses xmessage to provide a pop up if any are found.
>>
>
> OK, Here are the last two sensord reports just before the freeze
> I do not see anything out of the ordinary here. The last one (23:28:55)
> occured. The freeze ( as inferred from the last entry into /var/log/syslog
> occured aroung 23:35:31
>
> (
> --------------------------------
> Jul 24 23:35:31 tunnel kernel: [1181287.585321] Shorewall:net-fw:DROP:IN=eno1 OUT= MAC=4c:ed:fb:c2:2a:f3:a0:ab:1b:88:6e:58:08:00 SRC=185.225.74.53 DST=192.168.0.3 LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=54321 PROTO=TCP SPT=33231 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0
> Jul 25 09:13:25 tunnel kernel: [ 0.000000] microcode: microcode updated early to revision 0xf0, date = 2021-11-12
> --------------------------------------
>
> ----------------------------------
> Jul 24 23:08:55 tunnel sensord: Chip: nvme-pci-0600
> Jul 24 23:08:55 tunnel sensord: Adapter: PCI adapter
> Jul 24 23:08:55 tunnel sensord: Composite: 24.9 C (min = -40.1 C, max = 83.8 C)
> Jul 24 23:08:55 tunnel sensord: Sensor 2: 24.9 C (min = -40.1 C, max = 83.8 C)
> Jul 24 23:08:55 tunnel sensord: Chip: coretemp-isa-0000
> Jul 24 23:08:55 tunnel sensord: Adapter: ISA adapter
> Jul 24 23:08:55 tunnel sensord: Package id 0: 29.0 C
> Jul 24 23:08:55 tunnel sensord: Core 0: 28.0 C
> Jul 24 23:08:55 tunnel sensord: Core 1: 27.0 C
> Jul 24 23:08:55 tunnel sensord: Core 2: 28.0 C
> Jul 24 23:08:55 tunnel sensord: Core 3: 27.0 C
> Jul 24 23:08:55 tunnel sensord: Chip: acpitz-acpi-0
> Jul 24 23:08:55 tunnel sensord: Adapter: ACPI interface
> Jul 24 23:08:55 tunnel sensord: temp1: 27.8 C
> Jul 24 23:28:55 tunnel sensord: Chip: nvme-pci-0600
> Jul 24 23:28:55 tunnel sensord: Adapter: PCI adapter
> Jul 24 23:28:55 tunnel sensord: Composite: 24.9 C (min = -40.1 C, max = 83.8 C)
> Jul 24 23:28:55 tunnel sensord: Sensor 2: 24.9 C (min = -40.1 C, max = 83.8 C)
> Jul 24 23:28:55 tunnel sensord: Chip: coretemp-isa-0000
> Jul 24 23:28:55 tunnel sensord: Adapter: ISA adapter
> Jul 24 23:28:55 tunnel sensord: Package id 0: 29.0 C
> Jul 24 23:28:55 tunnel sensord: Core 0: 28.0 C
> Jul 24 23:28:55 tunnel sensord: Core 1: 27.0 C
> Jul 24 23:28:55 tunnel sensord: Core 2: 27.0 C
> Jul 24 23:28:55 tunnel sensord: Core 3: 27.0 C
> Jul 24 23:28:55 tunnel sensord: Chip: acpitz-acpi-0
> Jul 24 23:28:55 tunnel sensord: Adapter: ACPI interface
> Jul 24 23:28:55 tunnel sensord: temp1: 27.8 C
> ----------------------------------------------

I suggest that you should set max value allowed for temps.

>
> I cannot find any core files, and I do not think I am suppressing them.

Pretty sure they are not enabled by default.
You have to make configuration file changes.
Quick scan of my install/change scripts finds this custom
drop-in file and settings

# grep core /etc/sysctl.d/xx__sysctl.conf
# Enabling suid dump PID appending and set core location and name
kernel.core_uses_pid = 1
kernel.core_pattern = /var/tmp/%e_%p_%s.core
# net.core.rmem_max = 1048576

On Sat, 29 Jul 2023 00:02:20 +0200, J.O. Aho wrote:
> On 7/28/23 11:56, Bit Twister wrote:
>> On Fri, 28 Jul 2023 10:27:47 +0200, J.O. Aho wrote:
>>> On 7/28/23 03:16, William Unruh wrote:
>>>> I have a disturbing system, which every once in a while freezes. The
>>>> sceen on the monitor is some X scene (usually has Chrome running, but
>>>> they again that is often the case) but the keyboard,mouse, do nothing.
>>>> Trying to log on from the network fails with no response from the
>>>> machine. Alt-ctrl-del does nothing. The only way to recover is via the
>>>> power switch on the back.
>>>> Afterwards, looking at /var/log/syslog, or /var/log/messages shows
>>>> nothingsignificant that I can see just before the freeze.
>>>
>>> This reminds me of the issue I have had with running two plasma5
>>> sessions, after a while they will just eat up the memory and when all is
>>> gone, then everything freezes, you can't login from remote and local
>>> login takes too long from entering username to entering password that
>>> the login is canceled.
>>>
>>> I would run a memory check once in a while, for example use crontab to
>>> run this one liner:
>>>
>>> date && ps -Ao user,uid,comm,pid,pcpu,pmem --sort=-pmem | head -n 11 &&
>>> echo "----" >> /var/log/memusage.log
>>>
>>> Then you can see if there is a process that grows.
>>
>> Cute, and with just a little bit of scripting/coding you could automate
>> the check by flagging any line with value above some watermark for cpu
>> and memory percentage.
>>
>> If it was me, I would print any greater than 1.0 for ether one.
>>
>> Since values have a decimal I would guess that I would have to
>> use bc to test greater than watermark value if using bash.
>
> You could use awk, have to always point out this great tool as it bears
> my name
>
> date && ps -Ao user,uid,comm,pid,pcpu,pmem --sort=-pmem | awk -v
> min="0.3" '$6 >= min || $6=="%MEM"' && echo "----"
>
> Just change the -v min="0.3" to the value you think it has be or more to
> be displayed, we also keep displaying the column names, sure there is
> better ways of doing this, but my skills not that great.

Yeah, but different nodes can have different apps needing higher limits.
I'll be using bash and a case statement to change max values based on app.

for example on my myth node I have
# ps -Ao user,uid,comm,pid,pcpu,pmem --sort=-pmem | head -n 11
USER UID COMMAND PID %CPU %MEM
bittwis+ 1500 mythfrontend 4124 15.4 6.8
mysql 976 mysqld 780 1.1 3.2
mythtv 90 mythbackend 3804 1.5 3.0
bittwis+ 1500 net_applet 2223 0.2 1.3
bittwis+ 1500 xfwm4 2153 1.6 1.2

yet my normal web browsing node has
]$ ps -Ao user,uid,comm,pid,pcpu,pmem --sort=-pmem | head -n 11
USER UID COMMAND PID %CPU %MEM
root 0 Xorg 3147 0.4 0.9
bittwis+ 1500 xfwm4 4645 0.1 0.7
bittwis+ 1500 xfdesktop 4661 0.0 0.7
mysql 978 mysqld 3194 0.0 0.5
esept 1513 geany 9858 0.1 0.3
root 0 geany 9953 0.4 0.3

On 7/29/23 00:20, William Unruh wrote:
> On 2023-07-28, J.O. Aho <user@example.net> wrote:
>> On 7/28/23 03:16, William Unruh wrote:
>>> I have a disturbing system, which every once in a while freezes. The
>>> sceen on the monitor is some X scene (usually has Chrome running, but
>>> they again that is often the case) but the keyboard,mouse, do nothing.
>>> Trying to log on from the network fails with no response from the
>>> machine. Alt-ctrl-del does nothing. The only way to recover is via the
>>> power switch on the back.
>>> Afterwards, looking at /var/log/syslog, or /var/log/messages shows
>>> nothingsignificant that I can see just before the freeze.
>>
>> This reminds me of the issue I have had with running two plasma5
>> sessions, after a while they will just eat up the memory and when all is
>> gone, then everything freezes, you can't login from remote and local
>> login takes too long from entering username to entering password that
>> the login is canceled.
>
> Except in my case, nothing I type does anything, the mouse cursor does
> not move, Teh Alt-ctrl-F keys do nothing, alt-ctrl-det or alt-ctrl-bksp
> do nothing, gkrellm stops updating. Usually when I have run out of
> memory, somethings still work, and the machine slows down drastically as
> it starts to swap. This is just a complete sudden freeze. (Unlike this
> past time, sometimes this has happened while I am working on the
> machine. This time it happened while I was asleep)

This is the exact same and as you mention the the swap, it's exactly the
same and I can say even if you disable swap you will have the same behavior.

I think you should monitor the memory usage, you could add this to your
/etc/crontab

*/5 * * * * root (date && ps -Ao user,uid,comm,pid,pcpu,pmem
--sort=-pmem | awk -v min="10.0" '$6 >= min || $6=="%MEM"' && echo
"----" >> /var/log/mem.log)

Will run every 5 minutes and give you a lost of all processes that uses
10% or more of your memory. Then you take a look at /var/log/mem.log
from time to time or after a freeze and see which processes you had there.

What you can also do is to have an ssh connection from another machine
always running, this way you may be able to kill processes, you will not
be able to run ps, top, htop or other tools as they will be so slow that
they will block your shell and all you can do is just reset the machine
by pressing the button. If your issue is the desktop environment, I
would recommend to switch to another one. Enlightenment I think may have
the least of these issues, but I'm no fan of it, at the moment I'm using
lxqt.

--
//Aho

William Unruh <unruh@invalid.ca> writes:
> Except in my case, nothing I type does anything, the mouse cursor does
> not move, Teh Alt-ctrl-F keys do nothing, alt-ctrl-det or alt-ctrl-bksp
> do nothing, gkrellm stops updating. Usually when I have run out of
> memory, somethings still work, and the machine slows down drastically as
> it starts to swap. This is just a complete sudden freeze. (Unlike this
> past time, sometimes this has happened while I am working on the
> machine. This time it happened while I was asleep)

Yes. Normal behavior for running out of RAM is swapping, and for running
out of RAM+swap is to start killing user processes.

In this case:

| Trying to log on from the network fails with no response from the
| machine.

Does it respond to ping?

If so then the kernel’s still working, at least a bit (the lack of
kernel logs suggest everything above that is dead).

If it does not ping then the kernel has crashed, either due to a kernel
bug or a hardware fault.

--
https://www.greenend.org.uk/rjk/

On Sat, 29 Jul 2023 04:18:32 -0400, J.O. Aho <user@example.net> wrote:
> */5 * * * * root (date && ps -Ao user,uid,comm,pid,pcpu,pmem
> --sort=-pmem | awk -v min="10.0" '$6 >= min || $6=="%MEM"' && echo
> "----" >> /var/log/mem.log)

That is not reliable. /proc/$PID/comm may contain spaces so by time awk gets it
the column number is wrong.

[dave@x3 ~]$ ps -Ao user,uid,comm,pid,pcpu,pmem --sort=-pmem | awk -v min="3.0" '$6 >= min || $6=="%MEM"'
USER UID COMMAND PID %CPU %MEM
dave 500 opera 6969 2.8 4.7
ddclient 468 ddclient - slee 5691 0.0 0.0
[dave@x3 ~]$ cat /proc/5691/comm
ddclient - slee
[dave@x3 ~]$ cat /proc/5691/cmdline
ddclient - sleeping for 90 seconds[dave@x3 ~]$

I don't see a reliable field separator.

Regards, Dave Hodgins

On 7/29/23 15:09, David W. Hodgins wrote:
> On Sat, 29 Jul 2023 04:18:32 -0400, J.O. Aho <user@example.net> wrote:
>> */5  * * * * root (date && ps -Ao user,uid,comm,pid,pcpu,pmem
>> --sort=-pmem | awk -v min="10.0" '$6 >= min || $6=="%MEM"' && echo
>> "----" >> /var/log/mem.log)
>
> That is not reliable. /proc/$PID/comm may contain spaces so by time awk
> gets it
> the column number is wrong.
>
> [dave@x3 ~]$ ps -Ao user,uid,comm,pid,pcpu,pmem --sort=-pmem | awk -v
> min="3.0" '$6 >= min || $6=="%MEM"'
> USER       UID COMMAND             PID %CPU %MEM
> dave       500 opera              6969  2.8  4.7
> ddclient   468 ddclient - slee    5691  0.0  0.0
> [dave@x3 ~]$ cat /proc/5691/comm
> ddclient - slee
> [dave@x3 ~]$ cat /proc/5691/cmdline
> ddclient - sleeping for 90 seconds[dave@x3 ~]$
>
> I don't see a reliable field separator.
>
> Regards, Dave Hodgins

Easy fix:ps -Ao user,uid,pid,pcpu,pmem,comm --sort=-pmem | awk -v
> min="3.0" '$5 >= min || $5=="%MEM"'

--
//Aho

On 2023-07-29, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote:
> On Sat, 29 Jul 2023 04:18:32 -0400, J.O. Aho <user@example.net> wrote:
>> */5 * * * * root (date && ps -Ao user,uid,comm,pid,pcpu,pmem
>> --sort=-pmem | awk -v min="10.0" '$6 >= min || $6=="%MEM"' && echo
>> "----" >> /var/log/mem.log)
>
> That is not reliable. /proc/$PID/comm may contain spaces so by time awk gets it
> the column number is wrong.
>
> [dave@x3 ~]$ ps -Ao user,uid,comm,pid,pcpu,pmem --sort=-pmem | awk -v min="3.0" '$6 >= min || $6=="%MEM"'

How about $NF instead of $6?

> USER UID COMMAND PID %CPU %MEM
> dave 500 opera 6969 2.8 4.7
> ddclient 468 ddclient - slee 5691 0.0 0.0
> [dave@x3 ~]$ cat /proc/5691/comm
> ddclient - slee
> [dave@x3 ~]$ cat /proc/5691/cmdline
> ddclient - sleeping for 90 seconds[dave@x3 ~]$
>
> I don't see a reliable field separator.
>
> Regards, Dave Hodgins

On 2023-07-29, Richard Kettlewell <invalid@invalid.invalid> wrote:
> William Unruh <unruh@invalid.ca> writes:
>> Except in my case, nothing I type does anything, the mouse cursor does
>> not move, Teh Alt-ctrl-F keys do nothing, alt-ctrl-det or alt-ctrl-bksp
>> do nothing, gkrellm stops updating. Usually when I have run out of
>> memory, somethings still work, and the machine slows down drastically as
>> it starts to swap. This is just a complete sudden freeze. (Unlike this
>> past time, sometimes this has happened while I am working on the
>> machine. This time it happened while I was asleep)
>
> Yes. Normal behavior for running out of RAM is swapping, and for running
> out of RAM+swap is to start killing user processes.

Yes, and there is no evidence for that.

>
> In this case:
>
>| Trying to log on from the network fails with no response from the
>| machine.
>
> Does it respond to ping?

No.

>
> If so then the kernel’s still working, at least a bit (the lack of
> kernel logs suggest everything above that is dead).
>
> If it does not ping then the kernel has crashed, either due to a kernel
> bug or a hardware fault.

That is sure what it looks like. Weird thing is that the video card is
still sending out the last image, so it is running.

>

On Sat, 29 Jul 2023 09:09:32 -0400, David W. Hodgins wrote:
> On Sat, 29 Jul 2023 04:18:32 -0400, J.O. Aho <user@example.net> wrote:
>> */5 * * * * root (date && ps -Ao user,uid,comm,pid,pcpu,pmem
>> --sort=-pmem | awk -v min="10.0" '$6 >= min || $6=="%MEM"' && echo
>> "----" >> /var/log/mem.log)
>
> That is not reliable. /proc/$PID/comm may contain spaces so by time awk gets it
> the column number is wrong.
>
> [dave@x3 ~]$ ps -Ao user,uid,comm,pid,pcpu,pmem --sort=-pmem | awk -v min="3.0" '$6 >= min || $6=="%MEM"'
> USER UID COMMAND PID %CPU %MEM
> dave 500 opera 6969 2.8 4.7
> ddclient 468 ddclient - slee 5691 0.0 0.0
> [dave@x3 ~]$ cat /proc/5691/comm
> ddclient - slee
> [dave@x3 ~]$ cat /proc/5691/cmdline
> ddclient - sleeping for 90 seconds[dave@x3 ~]$
>
> I don't see a reliable field separator.

My bash solution
while read -r line ; do
set -- $line
_bin=$3
shift $(( $# - 2 ))
_cpu=$1
_mem=$2

On 29/07/2023 11:44, Richard Kettlewell wrote:
> William Unruh <unruh@invalid.ca> writes:
>> Except in my case, nothing I type does anything, the mouse cursor does
>> not move, Teh Alt-ctrl-F keys do nothing, alt-ctrl-det or alt-ctrl-bksp
>> do nothing, gkrellm stops updating. Usually when I have run out of
>> memory, somethings still work, and the machine slows down drastically as
>> it starts to swap. This is just a complete sudden freeze. (Unlike this
>> past time, sometimes this has happened while I am working on the
>> machine. This time it happened while I was asleep)
>
> Yes. Normal behavior for running out of RAM is swapping, and for running
> out of RAM+swap is to start killing user processes.

The problem is that the swapping in and out Xorg and the desktop
environment makes things extremely slow and the tradition in Linux is to
kill a random process (not sure if it's changed nowadays), the
likelihood that the right process is killed is slim and if not the right
one is killed, the swapping will not end and the swap will keep the
system slow and even if it kills the right process it will in reality
take hours before anything happens due of the swap in and out.

My experience is that things do not get better even if you disable swap
(seldom you really need swap when having 64GB RAM), for some reason it
seems to be the disk is as active as during the swap in / out. Could it
have to do with zram?

Something that makes a difference is setting a hard memory limit on the
process, then it will be killed when it tries to use more RAM than the
limit.

Of course at this point we don't know what the issues is for OP and he
don't use plasma5, so I doubt it's plasmashell that is his issue.

--
//Aho

On Sat, 29 Jul 2023 14:52:19 -0400, Bit Twister <BitTwister@mouse-potato.com> wrote:

> On Sat, 29 Jul 2023 09:09:32 -0400, David W. Hodgins wrote:
>> On Sat, 29 Jul 2023 04:18:32 -0400, J.O. Aho <user@example.net> wrote:
>>> */5 * * * * root (date && ps -Ao user,uid,comm,pid,pcpu,pmem
>>> --sort=-pmem | awk -v min="10.0" '$6 >= min || $6=="%MEM"' && echo
>>> "----" >> /var/log/mem.log)
>>
>> That is not reliable. /proc/$PID/comm may contain spaces so by time awk gets it
>> the column number is wrong.
>>
>> [dave@x3 ~]$ ps -Ao user,uid,comm,pid,pcpu,pmem --sort=-pmem | awk -v min="3.0" '$6 >= min || $6=="%MEM"'
>> USER UID COMMAND PID %CPU %MEM
>> dave 500 opera 6969 2.8 4.7
>> ddclient 468 ddclient - slee 5691 0.0 0.0
>> [dave@x3 ~]$ cat /proc/5691/comm
>> ddclient - slee
>> [dave@x3 ~]$ cat /proc/5691/cmdline
>> ddclient - sleeping for 90 seconds[dave@x3 ~]$
>>
>> I don't see a reliable field separator.
>
> My bash solution
> while read -r line ; do
> set -- $line
> _bin=$3
> shift $(( $# - 2 ))
> _cpu=$1
> _mem=$2

The solution of putting the comm field last works fine.

$ ps -Ao user,uid,pid,pcpu,pmem,comm --sort=-pmem | awk -v min="2.0" '$5 >= min || $5=="%MEM"'
USER UID PID %CPU %MEM COMMAND
dave 500 6969 3.7 4.9 opera
dave 500 33667 7.5 4.0 firefox
dave 500 6186 0.7 2.9 plasmashell
dave 500 34910 1.3 2.3 Isolated Web Co
dave 500 34975 0.8 2.3 Isolated Web Co
dave 500 53719 2.1 2.0 Isolated Web Co

The Isolated Web Content processes are all firefox.

Regards, Dave Hodgins

On 2023-07-29, J.O. Aho <user@example.net> wrote:
> On 29/07/2023 11:44, Richard Kettlewell wrote:
>> William Unruh <unruh@invalid.ca> writes:
>>> Except in my case, nothing I type does anything, the mouse cursor does
>>> not move, Teh Alt-ctrl-F keys do nothing, alt-ctrl-det or alt-ctrl-bksp
>>> do nothing, gkrellm stops updating. Usually when I have run out of
>>> memory, somethings still work, and the machine slows down drastically as
>>> it starts to swap. This is just a complete sudden freeze. (Unlike this
>>> past time, sometimes this has happened while I am working on the
>>> machine. This time it happened while I was asleep)
>>
>> Yes. Normal behavior for running out of RAM is swapping, and for running
>> out of RAM+swap is to start killing user processes.
>
> The problem is that the swapping in and out Xorg and the desktop
> environment makes things extremely slow and the tradition in Linux is to
> kill a random process (not sure if it's changed nowadays), the
> likelihood that the right process is killed is slim and if not the right
> one is killed, the swapping will not end and the swap will keep the
> system slow and even if it kills the right process it will in reality
> take hours before anything happens due of the swap in and out.
>
> My experience is that things do not get better even if you disable swap
> (seldom you really need swap when having 64GB RAM), for some reason it
> seems to be the disk is as active as during the swap in / out. Could it
> have to do with zram?
>
> Something that makes a difference is setting a hard memory limit on the
> process, then it will be killed when it tries to use more RAM than the
> limit.
>
> Of course at this point we don't know what the issues is for OP and he
> don't use plasma5, so I doubt it's plasmashell that is his issue.

Actually I do use Plasma and sddm.
But as I said there is little indication beforehand that is is swapping
that is the problem (in the times that it happened to me while I was
working on the system.-- as I said this last time it occured just before
bedtime and I discovered it next morning. )

>

On Sat, 29 Jul 2023 16:09:21 -0400, David W. Hodgins wrote:
> On Sat, 29 Jul 2023 14:52:19 -0400, Bit Twister <BitTwister@mouse-potato.com> wrote:
>
>> On Sat, 29 Jul 2023 09:09:32 -0400, David W. Hodgins wrote:
>>> On Sat, 29 Jul 2023 04:18:32 -0400, J.O. Aho <user@example.net> wrote:
>>>> */5 * * * * root (date && ps -Ao user,uid,comm,pid,pcpu,pmem
>>>> --sort=-pmem | awk -v min="10.0" '$6 >= min || $6=="%MEM"' && echo
>>>> "----" >> /var/log/mem.log)
>>>
>>> That is not reliable. /proc/$PID/comm may contain spaces so by time awk gets it
>>> the column number is wrong.
>>>
>>> [dave@x3 ~]$ ps -Ao user,uid,comm,pid,pcpu,pmem --sort=-pmem | awk -v min="3.0" '$6 >= min || $6=="%MEM"'
>>> USER UID COMMAND PID %CPU %MEM
>>> dave 500 opera 6969 2.8 4.7
>>> ddclient 468 ddclient - slee 5691 0.0 0.0
>>> [dave@x3 ~]$ cat /proc/5691/comm
>>> ddclient - slee
>>> [dave@x3 ~]$ cat /proc/5691/cmdline
>>> ddclient - sleeping for 90 seconds[dave@x3 ~]$
>>>
>>> I don't see a reliable field separator.
>>
>> My bash solution
>> while read -r line ; do
>> set -- $line
>> _bin=$3
>> shift $(( $# - 2 ))
>> _cpu=$1
>> _mem=$2
>
> The solution of putting the comm field last works fine.
>
> $ ps -Ao user,uid,pid,pcpu,pmem,comm --sort=-pmem | awk -v min="2.0" '$5 >= min || $5=="%MEM"'
> USER UID PID %CPU %MEM COMMAND
> dave 500 6969 3.7 4.9 opera
> dave 500 33667 7.5 4.0 firefox
> dave 500 6186 0.7 2.9 plasmashell
> dave 500 34910 1.3 2.3 Isolated Web Co
> dave 500 34975 0.8 2.3 Isolated Web Co
> dave 500 53719 2.1 2.0 Isolated Web Co

Well, damn, got to go get crowbar to get head out of derriere.
Nice solution.

On Sat, 29 Jul 2023 23:40:43 -0400, Bit Twister <BitTwister@mouse-potato.com> wrote:
> Nice solution.

It was J.O. Aho that posted that solution.

Regards, Dave Hodgins