MCU disconnects mid-print

[rokapet]

Hi All!

 

I have an issue that I am troubleshooting right now, and I am posting here because I am starting to feel I am getting nowhere.

 

 - Config (Trident 300):

 - RPi 4 (4GB)

 - BTT Octopus v1.1 (F446)

 - BTT EBB36 v1.2 (G0B1) via Can Bridge of the Octopus board

 - MeanWell RSP-200-24 PSU

 

Issue: For the last few days/prints, my printer stops mid-print with an error stating that the connection to EBBCan mcu (that is EBB36) is lost, and a firmware reset is needed in order for the printer to return to the online state. My Trident has been running solid for more than a year, apart from one thing (well, multiple things, but I thing this one is the potentilly significant in this story): frequent undervoltage warnings on the RPi. It was powered by the appropriate pins of the Octopus board through the GPIO pins, but whatever I did (swap the cables to thicker ones, duplicate them, so there are 2x power and 2x GND pins), nothing helped, I also have 5V LED strips as case ligting, that are connected to the Neopixel pins of the Octopus board. They work fine, and I am aware that both the RPi 5V and the LEDs 5V consume the same power, but turning off the LEDs never helped with the undervoltage issue, at least not mid or long term.

To circumvent this issue, I decided to purchase a separate power supply for the RPi (just as I should have long ago, accordig to the build manual), a MeanWell RS-25-5. I installed it a week or so ago (with an USB-C plug), and indeed, I had no undervoltage issue since. But the lost mcu issue appeared (the first 2 or 3 prints were successfuly, but none afterwards), and I suspect that the new power supply has something to do with it. At least it is a coincidence, and there was no other modification in the printer recently.

 

At first, I thought it is a software bug, but after upgrading EVERYTHING (Klipper, Moonraker, Mainsail, OS, etc., you name it, I reflashed both of the boards as I had Klipper v0.11 previously), there is literally nothing i my printer now that still has an installable update. Disconnect issue persisted.

The next obvious thing would be the CAN cable, but that I consider unlikely. My print fail after 1-2 hours after starting the print, but the most suspicious log I have found (that made me think the new power supply might have something to do with the issue is not from Klipper, but from the OS kernel log:

 

Nov 17 16:16:14 trident kernel: [16475.890059] usb 1-1.4: USB disconnect, device number 4
Nov 17 16:16:14 trident kernel: [16475.890275] gs_usb 1-1.4:1.0 can0: Couldn't shutdown device (err=-19)
Nov 17 16:16:14 trident kernel: [16476.136181] usb 1-1.4: new full-speed USB device number 5 using xhci_hcd
Nov 17 16:16:14 trident kernel: [16476.247727] usb 1-1.4: New USB device found, idVendor=1d50, idProduct=606f, bcdDevice= 0.00
Nov 17 16:16:14 trident kernel: [16476.247763] usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Nov 17 16:16:14 trident kernel: [16476.247780] usb 1-1.4: Product: stm32f446xx
Nov 17 16:16:14 trident kernel: [16476.247795] usb 1-1.4: Manufacturer: Klipper
Nov 17 16:16:14 trident kernel: [16476.247808] usb 1-1.4: SerialNumber: 250024000251313133353932
Nov 17 16:16:14 trident kernel: [16476.256480] gs_usb 1-1.4:1.0: Configuring for 1 interfaces
Nov 17 16:16:14 trident kernel: [16476.510442] IPv6: ADDRCONF(NETDEV_CHANGE): can0: link becomes ready

 

It is apparently not the EBB36 that goes AWOL, but it is the Octopus board that disappears from the USB of the RPI for a second, and as it is the CAN bridge, it takes the whole can0 interface, CAN network and EBB36 with itself, and for some reason, Klipper only recognises the latter. The most obvious reason to a disappearing USB device would be an underpowered USB port, and as I wrote above, the start of this issue coincides with the installation of the RS-25-5.

 

What else I tried:

 - Checked and increased the voltage of the RS-25-5 power supply from the factory 5.05V to 5.15V. Not helped.

 - Tried to run a print with case LEDs off. I know they (and the Octopus board in general) draw the power from the 24V power supplly, but as it doesn1t hurt, I wanted give it a shot. This one is running now.

What else I have in mind:

 - Replace the home made RPi power cord with one that has a factory installed USB-C plug. I have no undervoltage, but who knows... Need to order such a cable, or sacrifice one that I have.

 - Replace the USB cable between RPI and Octopus. Might do this too, though the present cable has been OK for more than a year.

 

Any other thoughts, gentlemen?

 

 

 

[rokapet]

Maybe I was wrong about the RS-25-5 power supply and/or LEDs... Just had another mid-print fail, but this time it was "TMC ‘extruder’ reports error: GSTAT: 00000001 reset=1(RESET)" (no USB disconnect this time), then another USB disconnect, but this time when the printer was sitting idle.

 

Maybe it is the EBB after all. Didn't want to completely disassemble my hotend!

[mvdveer]

It sounds like a loose wire or crimp that triggers on movement of the toolhead

[mvdveer]

5 hours ago, rokapet said:

The next obvious thing would be the CAN cable, but that I consider unlikely.

Could still be. It sounds like something  is failing when there is movement. If your canbus cable is in a cable chain, then the wire may have fractures somewhere along. May be at the connector as well. Worth checking.

[rokapet]

@mvdveer Thanks! It could indeed be the CAN cable (it is umbilical, not in chain), though the last error occured while nothing was moving.

 

However, I realized I made a stupid mistake. Despite of the build manual instructions and common sense, the DC ground of the 24V and 5V power supplies are not connected. That is the No. 1 culprit now for me (better said, I hope this is it, as this would be the easiest to fix ). Unfortunately it is going to take a few days to test this theory, as there are works now in my home that will include power cuts, so I am not in a situation to start multi-hour prints. Will continue towards the end of the week...

[claudermilk]

I would also suspect cables. That or the Octopus board itself.

My Trident uses that Octopus & a RPi 4 and I power both the Pi & all my LEDs from the Octopus 5V rail. I'm using both the NeoPixel port and BLTouch port. I did get a few undervolt warning early on but adding a second 5V wire resolved that. The printer has run for over 3 years without an electrical hitch related to this setup. My main difference is I'm not running Canbus.

[rokapet]

On 11/18/2024 at 3:10 PM, claudermilk said:

I would also suspect cables. That or the Octopus board itself.

My Trident uses that Octopus & a RPi 4 and I power both the Pi & all my LEDs from the Octopus 5V rail. I'm using both the NeoPixel port and BLTouch port. I did get a few undervolt warning early on but adding a second 5V wire resolved that. The printer has run for over 3 years without an electrical hitch related to this setup. My main difference is I'm not running Canbus.

Hi! I tried adding 2nd 5V and GND cables earlier, didn't help. Partly this, and partly that I read that powering the RPi via GPIO bypasses the voltage regulation circuit of the RPi made me make the move to install the separate 5V power supply for the RPi. a move I am starting to regret, as now my printer stops and shuts down within a minute after startup even before doing anything, or during the startup sequence (Z-Tilt, for example), or mid-print...

 

Anyway, since my last post:

 - I added the missing DC ground cable between the power supplies, to make no difference.

 - Checked all the plugs around the RPi and Octopus board, nothing seems to be loose.

 - Plugged the Octopus board to a different USB port on the RPi. Made no difference either.

 - Replaced the USB cable between the Octopus and the RPi. Apparently since this, the USB disconnect issues disappeared from the RPi linux log files, but Klipper still shuts down claiming that the communication to the EBB board has been lost.

 - A new thing today that RPi occasionally does not come up to the wireless network when started (it is used wireless, no eth cable). Then on the next reboot, it does. This has happened twice. When adjusting the 5V power supply, the output voltage once was 5.45V for about half a minute (for the time of measurement), then I quickly turned it down. As I read, RPi should be able to handle voltages up until 5.5V, so this is probably not relevant, especially because I had issues before this as well.

 

Any clue how I could test if the CAN cable is broken? I can re-crimp both ends, but if it is broken somewhere in the middle, then that will not be much of a relief. For a complete replace, I need to order appropriate wires, as I am out of them... Even though the issue is present without any movement too! I may replace the RPi, as I don't like the wireless network issue it developed, and I have a spare one laying around.

[rokapet]

Replaced RPi. USB disconnect is back. 

[mvdveer]

7 hours ago, rokapet said:

Replaced RPi. USB disconnect is back. 

I still think it is in the CAN cable connection. Test with a multimeter across the ends of all 4 cables to see if there is continuity whilst moving the connections. (Unplug both ends and test 24V, GND, CanL CanH.

[rokapet]

16 hours ago, mvdveer said:

I still think it is in the CAN cable connection. Test with a multimeter across the ends of all 4 cables to see if there is continuity whilst moving the connections. (Unplug both ends and test 24V, GND, CanL CanH.

Bullseye!

Took a deep breath, and started to disassemble the umbilical cord last evening. It is in a sleeve, and it's end on the EBB mount is secured by a screw clamp, but once the clamp was removed, the MicroFit plug unplugged from the socket on the EBB, and the wires freed from the sleeve, one of the power cords immediately got completely separated from the plug. The wire strains were broken just where the MicroFit pin was crimped. Can hardly imagine how it could work even for a second... Apparently the clamp was not as tight as it should have been, and the wires were able to move all the way down until the plug, until one of them finally broke at the weakest point, at the crimp. Enter IWS-3220M, broken pin ejected from the plug, new pin crimped to the wire and reseated into the plug, and this morning I have a 3 hour print completed without any hickups! So all in all, it was far less painful to find and repair the fault than I first anticipated.

 

Apparently the installation of the RS-25 power supply precisely coincided with the breakage of the umbilical wire. In my earlier post, thinking that it is the cause of all bad things, I wrote that I started to regret the install of the RS-25. I do not any more, as the undervoltage issues went away. I also consider replacing the RPi yesterday a positive thing, partly because I replaced a 4GB one with a 2GB one (2GB is still more than enough for Klipper, and I can use the 4GB one for other things where more RAM can be required and beneficial), and partly because during the replace, I could tidy up wiring and reorganize parts a bit, so that the RPi gets more airflow from the cooling fan. This means RPi temp went down ~10 degrees, into the 30-is degrees celsius range.

Finally, everything seems to be all right now. Thank you for the suggestions and help from everyone involved!

[claudermilk]

Hooray, you found it! That's one of the biggest PITA failures to track down. Chased that on my printer, and on my previous car (that was a saga that my mechanic still has PTSD from).

[mvdveer]

Excellent - back in business!!