Arch big update caused kernel panic -- help !!!!!

[abarbarian]

There was a huge Arch update a couple of days ago. When I shut down and rebooted I got a message saying " kernel panic - could not sync"  with no other real information that I could make sense of.

 

However I could boot from my fallback image and everything seems to work just fine.

 

Any ideas why my main image fails but my fallback works ???? An more importantly how can I track the fault in my main image and how can I  trouble shoot and fix it ???

 

Yes I do have full system backups but that would just revert me to an earlier time and I could apply the big upgrade and presumably I would end up with the same problem.

 

[securitybreach]

Are you using grub? If so, run the following after booting your falback:

 

pacman -S grub efibootmgr os-prober linux

grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=grub

grub-mkconfig -o /boot/grub/grub.cfg

 

 

[securitybreach]

If using legacy instead of UEFI, run this:

 

pacman -S grub linux

grub-install --recheck --target=i386-pc /dev/sda

grub-mkconfig -o /boot/grub/grub.cfg

[abarbarian]

1 hour ago, securitybreach said:

Are you using grub?

 

Really you think I would use some old tech on my ultra modern Arch set up. Nah I am using rEFind. It is working, at least it is letting me use the fallback image.

 

I did a

 

pacman -Qkk

 

this threw up a lot of errors,

 

warning: vlc: /usr/lib/vlc/plugins/plugins.dat (Modification time mismatch)
warning: vlc: /usr/lib/vlc/plugins/plugins.dat (Size mismatch)
warning: vlc: /usr/lib/vlc/plugins/plugins.dat (MD5 checksum mismatch)
warning: vlc: /usr/lib/vlc/plugins/plugins.dat (SHA256 checksum mismatch)
warning: systemd: /var/log/journal (GID mismatch)
warning: neofetch: /usr/bin/neofetch (Modification time mismatch)
warning: neofetch: /usr/bin/neofetch (Size mismatch)
warning: neofetch: /usr/bin/neofetch (MD5 checksum mismatch)
warning: neofetch: /usr/bin/neofetch (SHA256 checksum mismatch)
warning: miru-bin: /opt/Miru/chrome-sandbox (Permissions mismatch)
warning: libutempter: /usr/lib/utempter/utempter (GID mismatch)
warning: libutempter: /usr/lib/utempter/utempter (Permissions mismatch)
warning: intel-ucode: /boot/intel-ucode.img (Permissions mismatch)
warning: intel-ucode: /boot/intel-ucode.img (Modification time mismatch)
warning: cups: /etc/cups/subscriptions.conf (Permissions mismatch)
warning: cups: /etc/cups/printers.conf (Permissions mismatch)



backup file: util-linux: /etc/pam.d/login (Modification time mismatch)
backup file: util-linux: /etc/pam.d/login (Size mismatch)
backup file: util-linux: /etc/pam.d/login (MD5 checksum mismatch)
backup file: util-linux: /etc/pam.d/login (SHA256 checksum mismatch)
backup file: systemd-swap: /etc/systemd/swap.conf (Modification time mismatch)
backup file: systemd-swap: /etc/systemd/swap.conf (MD5 checksum mismatch)
backup file: systemd-swap: /etc/systemd/swap.conf (SHA256 checksum mismatch)
backup file: systemd: /etc/systemd/journald.conf (Modification time mismatch)
backup file: systemd: /etc/systemd/journald.conf (Size mismatch)
backup file: systemd: /etc/systemd/journald.conf (MD5 checksum mismatch)
backup file: systemd: /etc/systemd/journald.conf (SHA256 checksum mismatch)
backup file: systemd: /etc/systemd/system.conf (Modification time mismatch)
backup file: systemd: /etc/systemd/system.conf (MD5 checksum mismatch)
backup file: systemd: /etc/systemd/system.conf (SHA256 checksum mismatch)
backup file: systemd: /etc/systemd/user.conf (Modification time mismatch)
backup file: systemd: /etc/systemd/user.conf (Size mismatch)
backup file: systemd: /etc/systemd/user.conf (MD5 checksum mismatch)
backup file: systemd: /etc/systemd/user.conf (SHA256 checksum mismatch)
backup file: sudo: /etc/sudoers (Modification time mismatch)
backup file: sudo: /etc/sudoers (Size mismatch)
backup file: sudo: /etc/sudoers (MD5 checksum mismatch)
backup file: sudo: /etc/sudoers (SHA256 checksum mismatch)
backup file: reflector: /etc/xdg/reflector/reflector.conf (Modification time mismatch)
backup file: reflector: /etc/xdg/reflector/reflector.conf (Size mismatch)
backup file: reflector: /etc/xdg/reflector/reflector.conf (MD5 checksum mismatch)
backup file: reflector: /etc/xdg/reflector/reflector.conf (SHA256 checksum mismatch)
backup file: pam: /etc/environment (Modification time mismatch)
backup file: pam: /etc/environment (Size mismatch)
backup file: pam: /etc/environment (MD5 checksum mismatch)
backup file: pam: /etc/environment (SHA256 checksum mismatch)
backup file: pacman-mirrorlist: /etc/pacman.d/mirrorlist (Modification time mismatch)
backup file: pacman-mirrorlist: /etc/pacman.d/mirrorlist (Size mismatch)
backup file: pacman-mirrorlist: /etc/pacman.d/mirrorlist (MD5 checksum mismatch)
backup file: pacman-mirrorlist: /etc/pacman.d/mirrorlist (SHA256 checksum mismatch)
backup file: pacman: /etc/pacman.conf (Modification time mismatch)
backup file: pacman: /etc/pacman.conf (Size mismatch)
backup file: pacman: /etc/pacman.conf (MD5 checksum mismatch)
backup file: pacman: /etc/pacman.conf (SHA256 checksum mismatch)
backup file: mkinitcpio: /etc/mkinitcpio.conf (Modification time mismatch)
backup file: mkinitcpio: /etc/mkinitcpio.conf (Size mismatch)
backup file: mkinitcpio: /etc/mkinitcpio.conf (MD5 checksum mismatch)
backup file: mkinitcpio: /etc/mkinitcpio.conf (SHA256 checksum mismatch)
backup file: logrotate: /etc/logrotate.conf (Modification time mismatch)
backup file: logrotate: /etc/logrotate.conf (Size mismatch)
backup file: logrotate: /etc/logrotate.conf (MD5 checksum mismatch)
backup file: logrotate: /etc/logrotate.conf (SHA256 checksum mismatch)
backup file: glibc: /etc/locale.gen (Modification time mismatch)
backup file: glibc: /etc/locale.gen (Size mismatch)
backup file: glibc: /etc/locale.gen (MD5 checksum mismatch)
backup file: glibc: /etc/locale.gen (SHA256 checksum mismatch)
backup file: filesystem: /etc/fstab (Modification time mismatch)
backup file: filesystem: /etc/fstab (Size mismatch)
backup file: filesystem: /etc/fstab (MD5 checksum mismatch)
backup file: filesystem: /etc/fstab (SHA256 checksum mismatch)
backup file: filesystem: /etc/group (Modification time mismatch)
backup file: filesystem: /etc/group (Size mismatch)
backup file: filesystem: /etc/group (MD5 checksum mismatch)
backup file: filesystem: /etc/group (SHA256 checksum mismatch)
backup file: filesystem: /etc/gshadow (Modification time mismatch)
backup file: filesystem: /etc/gshadow (Size mismatch)
backup file: filesystem: /etc/gshadow (MD5 checksum mismatch)
backup file: filesystem: /etc/gshadow (SHA256 checksum mismatch)
backup file: filesystem: /etc/hosts (Modification time mismatch)
backup file: filesystem: /etc/hosts (Size mismatch)
backup file: filesystem: /etc/hosts (MD5 checksum mismatch)
backup file: filesystem: /etc/hosts (SHA256 checksum mismatch)
backup file: filesystem: /etc/issue (Modification time mismatch)
backup file: filesystem: /etc/issue (Size mismatch)
backup file: filesystem: /etc/issue (MD5 checksum mismatch)
backup file: filesystem: /etc/issue (SHA256 checksum mismatch)
backup file: filesystem: /etc/passwd (Modification time mismatch)
backup file: filesystem: /etc/passwd (Size mismatch)
backup file: filesystem: /etc/passwd (MD5 checksum mismatch)
backup file: filesystem: /etc/passwd (SHA256 checksum mismatch)
backup file: filesystem: /etc/resolv.conf (Modification time mismatch)
backup file: filesystem: /etc/resolv.conf (Size mismatch)
backup file: filesystem: /etc/resolv.conf (MD5 checksum mismatch)
backup file: filesystem: /etc/resolv.conf (SHA256 checksum mismatch)
backup file: filesystem: /etc/shadow (Modification time mismatch)
backup file: filesystem: /etc/shadow (Size mismatch)
backup file: filesystem: /etc/shadow (MD5 checksum mismatch)
backup file: filesystem: /etc/shadow (SHA256 checksum mismatch)
backup file: cups: /etc/cups/subscriptions.conf (Modification time mismatch)
backup file: cups: /etc/cups/subscriptions.conf (Size mismatch)
backup file: cups: /etc/cups/subscriptions.conf (MD5 checksum mismatch)
backup file: cups: /etc/cups/subscriptions.conf (SHA256 checksum mismatch)
backup file: cups: /etc/cups/printers.conf (Modification time mismatch)
backup file: cups: /etc/cups/printers.conf (Size mismatch)
backup file: cups: /etc/cups/printers.conf (MD5 checksum mismatch)
backup file: cups: /etc/cups/printers.conf (SHA256 checksum mismatch)
backup file: cpupower: /etc/default/cpupower (Modification time mismatch)
backup file: cpupower: /etc/default/cpupower (MD5 checksum mismatch)
backup file: cpupower: /etc/default/cpupower (SHA256 checksum mismatch)
backup file: bash: /etc/bash.bashrc (Modification time mismatch)
backup file: bash: /etc/bash.bashrc (Size mismatch)
backup file: bash: /etc/bash.bashrc (MD5 checksum mismatch)
backup file: bash: /etc/bash.bashrc (SHA256 checksum mismatch)

 

I tried re-installing several of the programs with warnings but it made no difference. Realised that just reinstalling would just use the same configs. Guess I would have to uninstall then reinstall. Dubious about doing that as I do not know if that will make the system unusable or possibly mess with the fallback image that is working.

 

My system has been kept up to date with almost daily updates to the system. It isn't as though I have left weeks between updates. Puzzled yet again.

 

Did you have any trouble with the big update ??

 

 

[V.T. Eric Layton]

Hmm... you've got some problems there.

 

My Arch daze are quite outdated. I can't help at all. Master SecurityBreach will have to pull your chestnuts out of the fire for you.

[abarbarian]

Trying to use my normal bootin I get

 

Quote

systemd [1] exited with irqs disabled

kernal panic - not syncing attemp to kill init

 

There are a few error codes that I have not shown but that is the main message.

 

I can not use Ctrl-Alt-Del or get a different tty by Alt-F1 F2 F3 etc just a locked screen . So I have to switch of power to shut down.

[securitybreach]

19 hours ago, abarbarian said:

Did you have any trouble with the big update ??

 

 

No issues on 5 of my arch machines. And I am not for sure what you mean as a "big update" as I didn't get anything out of the ordinary.

 

I have never used rEFind but I was thinking that maybe the kernel update or the bootloader entry got messed up when it updated, hence why I was getting you to reinstall the kernel and bootloader.

[securitybreach]

18 hours ago, abarbarian said:

Trying to use my normal bootin I get

 

 

There are a few error codes that I have not shown but that is the main message.

 

I can not use Ctrl-Alt-Del or get a different tty by Alt-F1 F2 F3 etc just a locked screen . So I have to switch of power to shut down.

 

Well Kernel Panic means that you can't do anything as it can't load the required kernel. All you can do is a hard power off.

 

Quote

Kernel panic refers to a computer error from which the system's operating system (OS) cannot quickly or easily recover. It occurs when there is a low-level fatal error and the operating system’s kernel is unable to fix it.

 

[abarbarian]

1 hour ago, securitybreach said:

No issues on 5 of my arch machines. And I am not for sure what you mean as a "big update" as I didn't get anything out of the ordinary.

 

I have never used rEFind but I was thinking that maybe the kernel update or the bootloader entry got messed up when it updated, hence why I was getting you to reinstall the kernel and bootloader.

 

The big update surprised me as I keep very up to date, normally update twice a week or so.  The update was almost a gigabyte.

 

My comments about grub were just warped Brit humour. I checked my boot configs they looked ok. Uninstalled the intel-unicode.image due to the warning mentioned above and reinstalled. Did a ,

 

For recursively removing orphans and their configuration files.

# pacman -Rns $(pacman -Qtdq)

 

To get rid of any crud , there was quite a lot. Then did a ,

 

To re-install all packages (those from the repo’s), in case of emergency:

 pacman -Sy $(pacman -Q | cut -d " " -f1 | grep -v "$(pacman -Qm | cut -d " " -f1)")

 

Also did a ,

 

mkinitcpio -p

and a 

mkinitcpio -P 

 

Shouldn't all of that sorted out replacing the kernel and stuff.

 

Non of the above helped at all. Still running on the fallback image.

 

Thanks for the assist by the way.

 

Edited May 7, 2023 by abarbarian

[wa4chq]

Have you tried Windoze?  Just sayin'....  

[abarbarian]

30 minutes ago, wa4chq said:

Have you tried Windoze?  Just sayin'....  

 

Grrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr

[abarbarian]

This is making no sense to me.

 

I deleted

 

initramfs-linux.img

initramfs-linux-fallback.img

vmlinuz-linux

intel-ucode.img

mkinitcpio.d - folder contains presets

 

Uninstalled  " linux " and reinstalled then did a " mkinitcpio -P ".

 

An I can not boot from the " initramfs-linux.img " only from the " initramfs-linux-fallback.img " which makes no sense at all.

 

Quote

Every time a kernel is installed or upgraded, a pacman hook automatically generates a .preset file saved in /etc/mkinitcpio.d/. For example linux.preset for the official stable linux kernel package. A preset is simply a list of information required to create initial ramdisk images, instead of manually specifying the various parameters and the location of the output files. By default, it contains the instructions to create two images:

1. the default ramdisk image created following the directives specified in the mkinitcpio #Configuration, and
2. the fallback ramdisk image, same as above except that the autodetect hook is skipped during creation, thus including a full range of modules which supports most systems.

After creating the preset, the pacman hook calls the mkinitcpio script which generates the two images, using the information provided in the preset.

 

Acording to the wiki both .images use the same programs and the same settings apart from the " autodetect" hook. So theoretically they should both work.

[V.T. Eric Layton]

WIPE THE DRIVE!

 

Reinstall a nice, fresh Arch. All will be well!

 

___

 

Install archinux!

You'll never need to install it again. I did and I'm really happy!

 

[crp]

there is no 'Safe Mode Boot' option?

 

[securitybreach]

I would probably reinstall as well. It doesn't make sense though as this normally wouldn't happen.

[securitybreach]

I have always cancelled whenever I ran pacman -Rns $(pacman -Qtdq) as it tries to uninstall a lot of required things as well. That alone may have screwed your install all together. My terminal, file manager and other things show up on the list.

 

 

If I let that run, my installation would basically be broken completely as it removed apps that I use on a regular basis

[sunrat]

Just restore your system backup that of course you make at least once a week, and always before doing major upgrades.

[securitybreach]

1 hour ago, sunrat said:

Just restore your system backup that of course you make at least once a week, and always before doing major upgrades.

I only do a clonezilla backup once or twice a year and havent had to restore in probably 10 years.

[sunrat]

3 hours ago, securitybreach said:

I only do a clonezilla backup once or twice a year and havent had to restore in probably 10 years.

 

Indeed. I got slack about doing system backups as I rarely need them. Well, one never needs backups until one does.

I use fsarchiver to back up partitions every month or two. I find it to be much less hassle than Clonezilla. Always backup before installing another OS in multiboot, or before major OS upgrades.

[securitybreach]

I only run one distro but good advice.

[abarbarian]

20 hours ago, V.T. Eric Layton said:

Reinstall a nice, fresh Arch. All will be well!

 

Too much trouble. Though probably the best advice.

16 hours ago, crp said:

there is no 'Safe Mode Boot' option?

 

I am running with the fallback image which is pretty much the same thing.

 

13 hours ago, securitybreach said:

I have always cancelled whenever I ran pacman -Rns $(pacman -Qtdq) as it tries to uninstall a lot of required things as well.

If I let that run, my installation would basically be broken completely as it removed apps that I use on a regular basis

 

I never checked what that command decided was crud, guess I should have. The system was borked already before I ran it though so I guess it is not to blame for this particular gremlin.

11 hours ago, sunrat said:

Just restore your system backup that of course you make at least once a week, and always before doing major upgrades.

 

I could and may yet do. I do a full backup every time I spot a kernel upgrade or large update then I make the upgrade and then reboot. I have been caught out before with the no backup scenario before. Yesum this player learns by his mistakes.

 

9 hours ago, securitybreach said:

I only do a clonezilla backup once or twice a year and havent had to restore in probably 10 years.

 

Show of.

 

13 hours ago, securitybreach said:

It doesn't make sense though as this normally wouldn't happen.

 

Yup for most folk but it seems like I have a pc that is home to a family of gremlins. Or possibly a Ghost in the Machine

 

Think I will head over to the Arch Forums and see what they can make of it all.

Edited May 8, 2023 by abarbarian

[V.T. Eric Layton]

16 hours ago, securitybreach said:

I would probably reinstall as well. It doesn't make sense though as this normally wouldn't happen.

 

I was really just pulling Erik's leg when I posted that above, but personally... if it got to be too much of a hassle, I would wipe it and start over. I did that many, many times in the ol' daze. Nowadays, though, I would just mirror my most recent (before the bugaboo issue) backup onto the primary drive.

[V.T. Eric Layton]

I rsync my Primary OS (/root & /home partitions + /storage) twice a month on Sundays, usually. I clone (using Clonezilla) my Windows drive maybe twice a year. Since the Windows is crippled (no Network access), I don't have to deal with updates or crud from the Internet on that OS.

[abarbarian]

So I decided to use my rsync backup with crossed fingers.

 

After a quick read here on the rsync threads I decide to use a Porteus Live USB as Eric said they had a grysnc program that might make life a little easier what with mount point etc.

Bloomin Porteus 5 don't not have grysnc they have not included it since V3. In searching for grysnc I googled " grysnc Porteus " and the fourth option was

 

https://forums.scotsnewsletter.com/index.php?/topic/60642-mirroring-partitions-w-rsync-grsync/

 

Movin on. I tested replacing my BOOT partition and it looked ok. So I modified my original backup script and renamed it "rev-rsync.sh" and am running it now from Porteus.

In the words of the song,

Will it work ? Will I be rich ? Here is what she said to me  ......

[V.T. Eric Layton]

Wow! I had completely forgotten about that post.

[V.T. Eric Layton]

You can download Porteus 3.2 from HERE. Easy-peasy.

[securitybreach]

I also do not really mess with my systems once I have them setup as I like. That and I am pretty decent at maintaining it.

[abarbarian]

On 5/8/2023 at 10:46 PM, securitybreach said:

I also do not really mess with my systems once I have them setup as I like. That and I am pretty decent at maintaining it.

 

Yeah me neither of late. Apart from trying out a few programs I have done no fiddling around at all. Which is why the catastrophe is such a surprise.

 

Things got even worse.

 

I decided to try and replace the borked system with me backup. I seem to remember that I had actually done a real life test that had worked out so it should work , right.

It might have gone pretty smoothly if only I had not had a bright idea.

My ESP and ROOT partitions needed some adjustments. I wanted the ESP larger and the ROOT smaller.So I made the changes with gparted. Now I only have one nvme drive and I use "/dev/nvme0np*"descriptors both in my refind.conf and fstab so I thought that making small changes to partition sizes would not be problematic for booting. I was right and wrong in my thinking.

Used the rsync script and only gave it a quick look as it seemed to work well.

Tried to boot and ended up with a black screen with a GRUB prompt. Hmmmmmmmmmmmm. After some deep thought it struck me that I may have transfered the stuff correctly but with the partition changes the initramfs-linux.image stuff would not work so I would need to make some changes.

A long time ago when I first tried to use chroot I found it very hard to understand and use. Years later as like now, using chroot seemed so easy to do. I was just about to do a "mkinitcpio -P" when I had the bright idea to do a "pacman -Syu" first. Why not, I would have to upgrade sometime so why not now. I should have waited.

On rebooting I had the same fail with my main image, exactly the same.

The fallback image did work. Sort of. Whilst I could get to a working login prompt I could no longer use "startx" or "/usr/bin/wmaker" or anything to get to my old graphical desktop.It took me quite a while to figure out what the heck was going on. Seem like my rsync script was not 100%. My boot part worked OK, my ROOT part worked OK, my HOME part worked sort of. It placed its contents in "/home" instead of "/home/bloodaxe". Once I found the problem a simple copy and paste brought me almost back to normal. My FireFox was messed up but I did manage to get my bookmarks back which was my main concern.

So here I am after a day of beating my head against a brick wall still in the same situation, only able to boot via a fallback image.

Thinking about it I should have not done a update whilst in chroot mode. I would have ended up with a fully working set up which would have helped me how ? After an update I would still have ended up as I am now with no clue as to why.

 

This information below still bothers me. If the main image and the fallback image are made from the same components apart from the "autodetect" feature then the problem must be something to do with "modules"  I am going to try running mkinitcpio with the autodetect left out of the build.

 

On 5/7/2023 at 2:35 PM, abarbarian said:

Every time a kernel is installed or upgraded, a pacman hook automatically generates a .preset file saved in /etc/mkinitcpio.d/. For example linux.preset for the official stable linux kernel package. A preset is simply a list of information required to create initial ramdisk images, instead of manually specifying the various parameters and the location of the output files. By default, it contains the instructions to create two images:

1. the default ramdisk image created following the directives specified in the mkinitcpio #Configuration, and
2. the fallback ramdisk image, same as above except that the autodetect hook is skipped during creation, thus including a full range of modules which supports most systems.

After creating the preset, the pacman hook calls the mkinitcpio script which generates the two images, using the information provided in the preset.

 

[wa4chq]

oh wait, you said pacman hook....sorry

[abarbarian]

Well I tried making an image with the autodetect feature left out. Never made a difference. An of course I did not as the same programs etc are being used.   I still have the same problem. Here is the screen I get if I try to boot up normally,  Why this is occurring is a total mystery and I simply can not understand why.

 

 

This whole incident is making me feel pretty miserable so I am going to just limp along with the fallback image and when it goes bums up I may just give up altogether.