Upgraded Endeavour OS

[raymac46]

It cannot be a hardware issue because 10 years ago I had a different desktop system and was running Windows 10 and still had the problem.

With Manjaro I use pamac and haven't had the issue in VBox. I am pretty sure I am unique on this forum in that I run Linux VMs in VBox on Windows. Saturnian may be another VBox user.

 I suppose I could reinstall EOS and give it a try with pacman -Syyu. I have been just using yay for EOS updates. Generally I use Reflector to rank the mirrors. There are some pretty good Canadian mirrors - mainly universities.

As far as ISP goes I get my service from Ottawa. I am just on the outskirts of Ottawa, not more than 20 miles from the ISP server.

Edited December 2, 2024 by raymac46

[raymac46]

OK I have reinstalled EndeavourOS. Parameters are:

Xfce desktop

VBoxVideo adapter 256MB

Windows 11 host

3 CPUs 6 GB Ram 20 GB HDD

Normal MBR install with GRUB

 

Booted and runs OK with Guest utils and full screen.

I'll have to wait a couple of days to update and I will NOT install anything from AUR.

 

[Hedon James]

2 hours ago, raymac46 said:

It cannot be a hardware issue because 10 years ago I had a different desktop system and was running Windows 10 and still had the problem.

With Manjaro I use pamac and haven't had the issue in VBox. I am pretty sure I am unique on this forum in that I run Linux VMs in VBox on Windows. Saturnian may be another VBox user.

 I suppose I could reinstall EOS and give it a try with pacman -Syyu. I have been just using yay for EOS updates. Generally I use Reflector to rank the mirrors. There are some pretty good Canadian mirrors - mainly universities.

As far as ISP goes I get my service from Ottawa. I am just on the outskirts of Ottawa, not more than 20 miles from the ISP server.

Correct me if I am wrong, but I thought yay was an AUR helper only?  Does yay "overlay" pacman, or just provide AUR functionality?

 

I've heard of pamac, but not used it in Manjaro.  I'm more familiar with Octopi, installed on Manjaro LXQT.  Octopi is also a GUI, which provides visual notification when updates are available (via little red or green "ghosts" like those characters in the 80s Pacman  video game), but I still use pacman via the CLI, for the reason cited with -Syyu flags.

[Hedon James]

52 minutes ago, raymac46 said:

OK I have reinstalled EndeavourOS. Parameters are:

Xfce desktop

VBoxVideo adapter 256MB

Windows 11 host

3 CPUs 6 GB Ram 20 GB HDD

Normal MBR install with GRUB

 

Booted and runs OK with Guest utils and full screen.

I'll have to wait a couple of days to update and I will NOT install anything from AUR.

 

it will be interesting to see if the -Syyu flag provides a different outcome than -Syu flags.  The other variable that needs to be addressed, IMO, is the kernel.  I use the stock LTS kernel in Endeavour, and the latest stock kernel in Arch VMs.  SB indicates Zen kernel is in the core repos and supported by Arch.

 

Inasmuch as Zen isn't "vanilla", I wonder if they bake VBox drivers into that kernel?  We know your system corrupts when the kernel updates and reboots.  We just don't know if the corruption occurs BEFORE update/upgrade (addressed by -Syyu flag), or as a result of the update/upgrade.  Zen kernel MAY be a solution also, IMO.

 

OTOH, changing kernels MAY be a way to force the corruption with a quicker reboot? 

Edited December 2, 2024 by Hedon James

[raymac46]

Yay will update both the official repos and build stuff as needed from the AUR. You can issue the command yay as user and it'll ask for your admin password then do its thing.

I forgot to mention I added the LTS kernel. However I got corruption in the shared libraries before a kernel update took place.

Pamac also handles both official repo and AUR updates.

Down the rabbit hole we go.

[Hedon James]

3 hours ago, raymac46 said:

Yay will update both the official repos and build stuff as needed from the AUR. You can issue the command yay as user and it'll ask for your admin password then do its thing.

I forgot to mention I added the LTS kernel. However I got corruption in the shared libraries before a kernel update took place.

Pamac also handles both official repo and AUR updates.

Down the rabbit hole we go.

That is actually helpful info, as it confirms it is NOT the kernel corrupting.  If your shared libraries are corrupting, I wonder if you're experiencing "partial update/upgrade"?  Can you get to a terminal prompt in VB and use the -Syyu flags for pacman update? That would be the textbook use for the double "yy" flags in pacman....rebuild package database for suspected issues with partial/failed/corrupt package on mirror.

[raymac46]

Just did a pacman -Syyu minor update from the terminal and everything was OK after a reboot.

Right now I have 3 distros in VBox - MX, Manjaro and EOS. All have Xfce desktops, all use VBoxSVGA video, all are MBR - GRUB boots. So the real variable here is distro. We'll see how it goes.

[raymac46]

Performed a somewhat larger upgrade and after a reboot everything seems to be OK. Using Firefox and posting from the EOS Virtual Machine. Used pacman -Syyu.

[Hedon James]

20 hours ago, raymac46 said:

Just did a pacman -Syyu minor update from the terminal and everything was OK after a reboot.

Right now I have 3 distros in VBox - MX, Manjaro and EOS. All have Xfce desktops, all use VBoxSVGA video, all are MBR - GRUB boots. So the real variable here is distro. We'll see how it goes.

 

Best news we've had so far!  Syncing the installed database with mirror(s) sure seems to check all the boxes with the symptoms you have observed and reported.  Fingers crossed!

[Hedon James]

56 minutes ago, raymac46 said:

Performed a somewhat larger upgrade and after a reboot everything seems to be OK. Using Firefox and posting from the EOS Virtual Machine. Used pacman -Syyu.

 

I'd like to see a few more updates, and maybe some more VM "corruption" that gets fixed from the terminal, before I believe that maybe we've solved the issue.  Right now, could just be a happy coincidence....but looking promising IMO!

[raymac46]

One of the problems with this lib corruption is that it usually crashes the VM to the point where I can't see the logs. Maybe when it happens next time I'll try an arch-chroot from the ISO and see if I can read the pacman log on the crashed system.

Thanks HJ for your second opinions and helpful guidance here. I realize that we are not dealing with anything life-threatening here but it is fun - after a fashion - to diagnose these problems.

[Hedon James]

3 hours ago, raymac46 said:

One of the problems with this lib corruption is that it usually crashes the VM to the point where I can't see the logs. Maybe when it happens next time I'll try an arch-chroot from the ISO and see if I can read the pacman log on the crashed system.

Thanks HJ for your second opinions and helpful guidance here. I realize that we are not dealing with anything life-threatening here but it is fun - after a fashion - to diagnose these problems.

that's the best time to diagnose a problem, IMO.  when it's not "critical" yet.  nothing more frustrating than troubleshooting a mission-critical system, when you're on a deadline, and keep hitting roadblocks.

 

tinker when it doesn't matter (as much), and acquire the knowledge for later, when it DOES matter.  JMO...

[raymac46]

One more medium sized upgrade mostly audio plug-ins but it seems that pacman -Syyu is working better than anything else we have tried so far. After reboot all is well and I can run Firefox and post from it. I'll keep on keeping on.

[Hedon James]

31 minutes ago, raymac46 said:

One more medium sized upgrade mostly audio plug-ins but it seems that pacman -Syyu is working better than anything else we have tried so far. After reboot all is well and I can run Firefox and post from it. I'll keep on keeping on.

love hearing this!

 

This is the beauty of this forum....multiple minds coming at a problem from different angles.  Not sure either of us would have resolved this situation on our own, but once we started sharing thoughts, stacking the evidence, and eliminating variables, everything started to point back to your INITIAL ASSESSMENT that it seemed pacman was getting corrupted.

 

so it made perfect sense (to me, at least) that a VM which is only fired up periodically would experience "mirror" issues that a bare-metal installation does not experience, because it is on 24/7 and works out the mirror issues in the background. 

 

When you exit a VM, do you save the state, or shut it down?  When you fire up a VM, do you commence from the saved state, or boot it up?  If you use saved states, part of me wonders if shutting down the VM for exit, and fresh boot from start would also resolve the issue?  OTOH, that kinda defeats the purpose of a VM, IMO.  Even if shutdown/boot resolves your VM issues, it is just SO MUCH EASIER and quicker to use the -Syyu flag.  I think Arch in a VM is a textbook case for the use of -Syyu flags, as those mirrors apparently need to be re-synced before the update/upgrade in order to avoid the potential corruption between mirrors.  I'm not familiar enough with Arch to know WHY, but apparently it's the solution?!

[raymac46]

I always shut down and reboot. I just don't like the idea of saved state; maybe it's OK but it reminds me of those sleep/hibernate modes in Windows that always get me in trouble.

Anyhow grateful for your support and ideas. I don't think I really appreciated what the -Syyu flag did before now.

[raymac46]

Well..another update. This time some of the video stack, watched dracut rebuild the kernel modules, rebooted and everything was OK. This was the type of update that usually torched the VM. Maybe we are on to something. Pacman -Syyu FTW.

[Hedon James]

11 hours ago, raymac46 said:

Well..another update. This time some of the video stack, watched dracut rebuild the kernel modules, rebooted and everything was OK. This was the type of update that usually torched the VM. Maybe we are on to something. Pacman -Syyu FTW.

Cool!

 

Your shutdown/reboot of VMs blows my theory though....at least the way I presented it.  Obviously, re-syncing mirrors is having a positive effect.  Equally obvious, it's reasonable to conclude that out-of-sync mirrors were the cause of your issues.  The question is WHY they get out of sync, and I just don't know enough about Arch to say.  My best GUESS is that Arch ranks mirrors and they remain in that order, but the mirrors don't always sync in the same sequence/order, so the hierarchy of latest packages can shift?  And maybe that's because of your "unique" geographic location?

 

What we DO know, is that your Arch on metal doesn't have this problem.  I'm guessing this is because your bare-metal Arch is always on, and you're informed of package updates/upgrades in real-time, as pamac/octopi/pacman is informed of them.  Your VM, OTOH, is "down" for long periods of time (maybe hours, maybe days) when you shut it down.  At shutdown, your packages are up to date with your ranked mirrors.  During shutdown period, the mirrors update packages but don't sync simultaneously, or consistently.  Maybe Ottawa syncs every 24 hours, but nearby #2 mirror is less busy and syncs every 6 hours.  While Ottawa is mirroring package Foobar 2024.10.31 (which is a dependency for recently updated Program 2.1) updates have been pushed and the most recent package is Foobar 2024.12.06-2, which Ottawa won't receive until midnight tonite.  In the meantime, Ray updates today and gets the updated Program 2.1 package, but not the most recent dependency, as mirror syncs haven't been completely pushed; as a result his updated Program 2.1 breaks, because he doesn't have the most recent dependency Foobar 2024.12.06-2, which is a fix for the bug that breaks Program 2.1.  Tomorrow it wouldn't be a problem, but today it is.  In all fairness, tomorrow will be another package, as yet unknown.  At least that's my THEORY.  Maybe SB has other thoughts?

 

I started using the -Syyu flag almost immediately upon learning about Arch.  Perhaps that explains why I have never had an issue with Arch, regardless of whether it was in VB, or VMM?  The threads and articles I read suggest the "use case scenario" for these flags is when you SUSPECT corruption of repos on your system.  I didn't want to allow corruption in the first place, and could't see any reason why the -Syyu flag would cause harm in lieu of the -Syu flag.  I have seen users discourage the use of -Syyu as a default set of flags, as it uses up bandwidth and "clogs" the mirrors, especially if all users adopt that practice.  But I reasoned that I was using a VM and that I was only updating/upgrading 1x a month; maybe 2x if I'm ambitious; maybe every other month if forgetful; but averaging ONCE A MONTH.  In my mind, using the -Syyu flags may use more server bandwidth, but I'm still using less bandwidth per month than daily/weekly Arch upgraders.  I think you can see how I rationalized "hogging" bandwidth on occassion.  And apparently, that is the reason I have been able to farm an Arch VM for so long without issue?!  It had very little (nothing?!) to do with VB or VMM...it had everything to do with the pacman flags used to update/upgrade VMs on an infrequent basis?  Maybe SB has thoughts on this one also?

 

I enjoy learning more about Arch because it's "vanilla", and for that reason, I think Arch packages represent "baseline" or "normal" behavior.  The 'Buntus may behave like "this", and the Fedoras may behave like "that", but that's because of the mods that the distro maintainers make to those packages.  Gotta look to Arch if you know what the DEFAULT behavior is.  JMO...

[raymac46]

Lots to chew on in your post and I don't disagree that Arch is probably the most "fundamental" Linux distro.

That said, I don't think our root cause is distro-centric (since Arch and EOS run fine for *years* on the rails.) Nor is it VM-centric (since other distros and for that matter Windows work OK in VBox per se.)

It has to be some combination of a bleeding edge distro that updates frequently over a worldwide set of mirrors AND an inherently quirky virtualization program with its guest utils and other "features."

I think that combination will give more opportunities for Murphy's Law to apply.

Now our next experiment in relativity will be to wait a week or so, let the updates pile up, and do a real Mother of an upgrade and see how it goes.

 

 

[securitybreach]

I am pretty sure I mentioned it already but -Syy should never be the solution as all you are doing is refreshing the entire package list instead of just the installed ones. From man pacman:

 

Quote

 

   -y, --refresh
       Download a fresh copy of the master package database from the server(s) defined in pacman.conf(5). This should typically be used each time you use --sysupgrade or -u. Passing two --refresh or -y flags will force a refresh of all package databases, even if they appear to be up-to-date.

   -y, --refresh
       Download fresh package databases from the server. Use twice to force a refresh even if databases are up to date.

 

 

Normally, -Syy is only used to update the package list after choosing a new mirror. Now you can also use -Syyu to update your mirrorlist and then update at the same time.

[securitybreach]

Anyway, if it works whatever

[Hedon James]

15 hours ago, securitybreach said:

I am pretty sure I mentioned it already but -Syy should never be the solution as all you are doing is refreshing the entire package list instead of just the installed ones. From man pacman:

 

 

Normally, -Syy is only used to update the package list after choosing a new mirror. Now you can also use -Syyu to update your mirrorlist and then update at the same time.

Not sure how Arch ranks mirrors (distance between ISPs?  Ping speeds?)  but my theory is that Ray is between 2 mirrors, and those mirrors are on 2 different sync schedules; so Ray is only getting partial upgrades when he syncs packages on his VM.  This is only a problem for guys like us, who only run a VM on occasion; and that VM is a rolling release.

 

In that use-case scenario, I think using the -yy flag to sync and upgrade is ESSENTIAL.  Especially when Ray has reproduced his failure multiple times, and the -yy flag prevents the failure (so far, knock on wood!).

 

I wonder if Ray's experience can be tied to an observation of MY Arch VM.  As already stated, I built my Arch VM in VirtualBox years ago, and migrated it to VMM (thanks SB for the recommendation!), where it has continued to perform without issue...probably going on close to 10 years now?!  Typically, I do not shutdown when I exit, nor reboot when I start; I only do that when new kernel presents.  Other than those cases, I usually just exit by "save the state" and start from the "saved state" when opening.  I have noticed that the Arch VM re-opens from the prior date of the prior session.  And I do have NTP enabled.  If I let the Arch VM "idle" long enough, the date will sort itself; sometimes in a minute, sometimes in 5 minutes, sometimes much longer.  (it's not really idle, it's busy in the background...but I use the word idle to mean that I refrain from doing anything)  I'm usually opening ArchVM simply to update/upgrade, and I can't be waiting extended periods for Arch to sort its date, which is kinda critical to the process, so I take to the CLI and manually adjust the time, then switch back to NTP before commencing with the update/upgrade.

 

If I were doing this more frequently, like every day, every other day, or every week, perhaps it wouldn't even be an issue.  But Arch notices and gets fussy when the internal clock says Nov 1 (the prior session) and the current date is Dec 1.  There is SOMETHING about Arch that causes it to be slow to update it's internal clock.  I've got Arch, Debian, Lubuntu, OpenSUSE, Manjaro, Endeavour, Vanilla, and Windows VMs.  Only Arch, Manjaro, and Windows exhibit the delay in updating the date/time to current date/time.  Maybe Endeavour, but I haven't run it long enough to say for certain.  All others update the date/time within seconds of opening the VM.  Any idea why Arch is slower (lower priority?) to update the date/time when VM is re-opened?

 

Whatever the reason, I'm thinking that MAYBE Ray & I are experiencing the same issue with Arch in a VM.  But it's not a problem for me because it's glaringly obvious to look at date/time and see the elapsed time differential, so I deal with THAT before I update/upgrade.  It's a problem for Ray because he updates MUCH more frequently than I do, so the date/time differential isn't as obvious.  This could also possibly tie into my theory about Ray being between 2 mirrors?

 

Thoughts?  Is it possible that Arch updates date/time differently than other distros?  And THAT is the underlying issue?  In the meantime, it seems that the -Syyu flags solve Ray's issue (knock on wood); but then the REAL question becomes "why does the repo database require re-syncing?"  Seems like my observation regarding date/time in Arch would also explain the differences between a bare-metal and VM installation?

 

[securitybreach]

That's what rankmirrors is for:

 

Quote

The pacman-contrib package provides a Bash script, /usr/bin/rankmirrors, which can be used to rank the mirrors according to their connection and opening speeds to take advantage of using the fastest local mirror.

 

https://wiki.archlinux.org/title/Mirrors#List_by_speed

 

it's been a part of pacman for over a decade and mentioned in the installation guide.

[securitybreach]

Reflector can automate this on every pacman upgrade:

 

Quote

• Reflector — Retrieves the latest mirrorlist from the MirrorStatus page, filters and sorts them by speed and overwrites /etc/pacman.d/mirrorlist. Provides automation with a systemd service and timer.

https://xyne.dev/projects/reflector/ || reflector

 

[securitybreach]

Also:

 

Quote

Arch Linux is a rolling release distribution. That means when new library versions are pushed to the repositories, the Developers and Package Maintainers rebuild all the packages in the repositories that need to be rebuilt against the libraries. For example, if two packages depend on the same library, upgrading only one package might also upgrade the library (as a dependency), which might then break the other package which depends on an older version of the library.

That is why partial upgrades are not supported. Do not use:

• pacman -Sy package
• pacman -Sy followed by pacman -S package (Note the absence of -Su in the installation of the package.)
• pacman -Syuw (Note that pacman -Syuw does imply the same risks like pacman -Sy, as it will update the pacman sync database without installing the newer packages.)

When refreshing the package database, always do a full upgrade with pacman -Syu. Note that if pacman -Syu does not perform the upgrade because of an error, the end result is the same as running pacman -Sy. Therefore, the error must be resolved and the upgrade operation completed as soon as possible

 

https://wiki.archlinux.org/title/System_maintenance#Partial_upgrades_are_unsupported

[raymac46]

Well this has been interesting.

I attempted a large update with pacman -Syyu and got a number of failed to commit transaction (conflicting files)

So I did a forced overwrite

sudo pacman --overwrite "*" -Syyu

Everything seemed to work OK but then when  rebooted there was a problem with the kernel and the system would not boot.

So I did arch-chroot from the ISO and used dracut-rebuild to fix the kernel and then reinstalled GRUB and configured GRUB. Now reboot from the system drive works again.

There is still some weirdness going on in VirtualBox.

[raymac46]

I might add that I just upgraded my old Lenovo Flex2-15D that runs EOS on the rails and it went as smooth as could be.

Edited December 9, 2024 by raymac46

[Hedon James]

1 hour ago, raymac46 said:

Well this has been interesting.

I attempted a large update with pacman -Syyu and got a number of failed to commit transaction (conflicting files)

So I did a forced overwrite

sudo pacman --overwrite "*" -Syyu

Everything seemed to work OK but then when  rebooted there was a problem with the kernel and the system would not boot.

So I did arch-chroot from the ISO and used dracut-rebuild to fix the kernel and then reinstalled GRUB and configured GRUB. Now reboot from the system drive works again.

There is still some weirdness going on in VirtualBox.

that IS interesting.  this seems to prove the theory that local package database and mirror repos are somehow conflicting.  Referencing my prior post about the time/date in Arch VM taking a while to sort itself when I open the Arch VM, I'm thinking there's a clue in that post; in that the LOCAL package database time/date is out of sync with mirror time/date.  This doesn't happen with bare-metal installs, because they're constantly synced in "real time".  But my Arch VM takes awhile to sync the correct time/date, so I usually manually intervene to force the updated time/date.  I only experience this in my Arch-related VMs.  My Debian/'Buntu VMs sync the correct date/time immediately upon opening the VM, no matter how long it's been since I last opened the VM.  That's my latest theory...

 

Now that you have Endeavour repaired, and know how to keep it going in VirtualBox, may I ask that you pay attention to the date/time of the VM, in relation to the ACTUAL date/time.  Let's test THAT theory?!

[raymac46]

OK I'll check on the clock when I next boot up the VM.

It seems that about the only common theme is that in the process of updating the installation certain files get corrupted. The symptoms of that corruption vary though. This is the first time I got the errors about conflicting files. 

Also I was able to force through the update but in the process the kernel was borked. But then I was able to recover with the help of the ISO and I haven't been able to do that before.

The use of --overwrite is a blunt force object and I wouldn't want to do that on any system that was important to me.

[raymac46]

Checked time and the VM is exactly the same as the host. No asymmetry at all.

[Hedon James]

13 hours ago, raymac46 said:

Checked time and the VM is exactly the same as the host. No asymmetry at all.

I checked my Endeavour VM and the date/time were exactly as the host system, despite not using EndeavourVM since Dec 1.  Safe to say that Endeavour doesn't exhibit the described time/date symptoms that I notice in my Arch & Manjaro VMs.

 

Sorry we're not making more progress toward solving the case, but we are crossing suspects off the list.