After months of being idle, ran many updates on Klipper/Voron. After reboot "emergency mode" stuff showed up.

[DennisM]

So yesterday I did all the updates necessary to start things up again with my 2.4r2, after months of sitting idle.

Rebooting gives me:

You are in emergency mode. After logging in, "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit" to boot into default mode.
Give root password for maintenance
(or press Control-D to continue):

Hmm. Doing the journalctl -xb and looking through it I do see this in there:

Feb 06 09:15:00 mainsailos systemd-fsck[139]: rootfs contains a file system with errors, check forced.
Feb 06 09:15:19 mainsailos systemd-fsck[139]: rootfs: Duplicate or bad block in use!
Feb 06 09:15:19 mainsailos systemd-fsck[139]: rootfs: Multiply-claimed block(s) in inode 43385: 451746
Feb 06 09:15:20 mainsailos systemd-fsck[139]: rootfs: Multiply-claimed block(s) in inode 134618: 451746
Feb 06 09:15:33 mainsailos systemd-fsck[139]: rootfs: (There are 2 inodes containing multiply-claimed blocks.)
Feb 06 09:15:33 mainsailos systemd-fsck[139]: rootfs: File ... (inode #43385, mod time Mon Apr  4 06:43:28 2022)
Feb 06 09:15:33 mainsailos systemd-fsck[139]:   has 1 multiply-claimed block(s), shared with 1 file(s):
Feb 06 09:15:33 mainsailos systemd-fsck[139]: rootfs:         /home/pi/klipper_logs/crowsnest.log (inode #134618, mod time Tue Feb  6 09:10:57 2024)
Feb 06 09:15:33 mainsailos systemd-fsck[139]: rootfs:
Feb 06 09:15:33 mainsailos systemd-fsck[139]: rootfs: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
Feb 06 09:15:33 mainsailos systemd-fsck[139]:         (i.e., without -a or -p options)
Feb 06 09:15:33 mainsailos systemd-fsck[139]: fsck failed with exit status 4.

I'm hoping this isn't a "bad SD card on a raspberry pi" situation. Anyway, I can't seem to run fsck at all as it says everything is mounted and can't run.

Anyone else come across this? I'd hate to pull the pi or try and boot from a USB linux boot setup, but maybe that's what's needed?

I'm thinking about pulling the SD card out and "cloning" it with BalenaEtcher to a new SD card, which maybe would ignore/fix/skip the bad blocks?

 

Thanks for any tips here,

 

Dennis

 

[Dirk]

I am a dos / windows guy. But this really looks like you have some bad blocks on your 'hard' drive. And has nothing to do with your klipper update.

I would suggest checking your hard drive for bad sectors and repair. I am sure linux has an equivalent... let me see... here is a good one you could follow: https://phoenixnap.com/kb/fsck-command-linux

Sure our dos chksdsk c : /F  is a bit easier...

I would first find out if you have any bad sectors on the SD card and fix it, before undertaking any other steps, like copying the sd card or reinstalling mainsail-os. 

[DennisM]

OK, so I pulled the SD card and mounted it via USB on another linux box. I ran fsck to fix all the errors (there were a lot, so I'm not hopeful the setup is still good).

But viola, the system starts up without any issues and even looks like it's ready to use via the Klipperscreen interface on the printer.

BUT the mainsail web UI isn't working. When I try to access it from my laptop, I get the "favicon" and Mainsail on the browser tab, but there is nothing on the screen. So it's not timing out or anything, just not displaying anything on the web page.

Sound familiar to anyone?

Also, nginx IS running on http port 80, so it's listening.

[Dirk]

You will probably find somewhere in the klipper/mainsail logs that one or other component is missing or not working. Probably due to the corrected errors on your sd card, certain files are corrupt or non-functioning.

If you think your sd card is good and you just had a corrupted system because you shut down wrongly, I will suggest to re-install klipper and the whole bit. KIUAH will be very helpfull to remove and reinstall everything quickly for you. Before doing so, do not forget to backup your configuration files and later place it in the correct place so you can run your system as if nothing has happened.

 

If the SD card is a cheap one, I would suggest re installing a new system.

 

 

[cjkennedy72]

Just for posterity... Sorry guys, been stupidly busy at work for the last few weeks.

@DennisM

On a Rpi or any Linux distro, you need to unmount and remount the root partition. Here are the commands, if you find yourself in this situation again. Of course replace "/dev/rootpartition" with the appropriate device, often "/dev/sda" or /dev/sdb"... On the RPi i just checked, its "/dev/mmcblk0p2". You can find this out using either "lsblk" or "blkid". I have had using "mount -o remount,ro /" work, but not 100% of the time.

sudo su
mount -o remount,ro /dev/rootpartition 
fsck /dev/rootpartition
reboot

I do agree with @Dirk that you probably should reinstall Klipper, Mainsail, and Moonraker. KIAUH is a helpful utility, andI have had this happen a couple of times. Its always my own fault for just switching the printer off with disregard for the RPi. I can usually just reinstall and come back up. But, I have had it not workout and ended up reflashing the SD Card. That same card is still in that printer to this day. You, like me, probably just have filesystem corruption. That doesn't mean the media is bad, it means that a write or metadata (more likely) update was in progress on a file that is either part of the OS or any of the 100s of packages installed to support Klipper.The point I am getting to is that you could end up chasing errors for far longer than anyone would want when, assuming you have a backup of your config files, a quick re-flash and reinstall will get you going faster.
 

Edited February 24 by cjkennedy72

[DennisM]

Thanks for all the updates and useful information everyone!

I ended up reflashing a new SD card and starting over, with the exception of all the configs, which I copied over.

There were some changes needed from deprecated settings and such, and my moonraker notifications broke (and are still broken, not sure what to do there) but everything else it working and the printer is back up and functional.

I'm going to pull the SD card soon and use a utility to literally clone the card to an image file for possible restoration in the future if needed.

Also, I had troubles with the moonraker database history file, but got that sorted as well. That was a little complicated but I needed to fire up the old system again with the old card and run an SQL dump of the data to a file, copy it to the new system and re-import the SQL data to get all of my printing history and such from before.

That involves this stuff:

On old system:
mdb_dump -f backup.txt -a .moonraker_database

FTP/SCP the file to the new Pi, then:
mdb_load -f backup.txt -s -T ~/.moonraker_database

 

Edited February 24 by DennisM