SKR Mini E3 v.2 - dead by stupid mistake!

[eugen360]

Hello good friends!

I'm writing this lines only to announce a really stupid mistake I've done!

WITHOUT reading end documenting like I usually do in such circumstance, yesterday I've introduced in my fresh, new build Voron printer, this little module: https://biqu.equipment/products/btt-ups-24v-v1-0-resume-printing-while-power-off-module?_pos=1&_sid=656c815f9&_ss=r 

BTW I'm using Klipper and this function of resume printing on power loss (as I read it afterwards) is not yet implemented! But I didn't know at that time... Maybe, it wouldn't destroyed my MCU if I'd use the same connector as for the normal 24V BUT as you know there are 2 possibilities to power this module and also because  of the limited space available, I put this UPS on the other port... Connected correctly (+) and (-)...

No smoke, no nothing, the Mainsail has started as usual but Klipper after a while gave this "unable to connect to MCU" error ... so basically that's all... SKR is useless now. Desperately, I've tried to re-flash the firmware but  it was pointless...

I'm still amazed of the "wonderful" work I've done!  

[smirk]

Sorry to hear of your loss, but it is always good to share. Even without resume support in Klipper i would not have expected that to fry your MCU. Unless the UPS was labelled incorrectly? Are there any LEDS on the SKR Mini E3 at all or is is completely dead? Right at this second I cannot remember if there are alternate means to connect to an SKR Mini - or even reflash it. It is sometime possible to bypass the boot loader and reprogram the board via the ISCP header (but that assumes there is any life on the board and that it has some sort of "programming header")

It is perhaps cold comfort, but personally I would not bother with these resume-on-power loss situations (granted I am lucky enough to have a reliable power supplier) but given all the complexities and vageries of the motion system and the way GCODE is processed I very much doubt you would ever get a truly reliable recovery from a power failure. There would always been some flaw left in the model.

[eugen360]

My friend!

Thanks for your answer! I'm attaching a picture of the board... the 24V line was connected on the left side and the UPS next to it. There are 2 LED's working on the module: one marked "Power" and the other one which lights up only when SD card is inserted. Next to the "Power LED" is the "Staus LED" witch lights up I guess only when Klippy show status "Ready" but in my case this one isn't light up of course...

When I've inserted the card with the firmware.bin file there wasn't any LED flashing like it normally should. But on the computer, in the Machine section, interestingly, I was able to make update of all the components... Klipper, Mainsail and another one, I don't remember... (maybe Moonraker).

[claudermilk]

Thanks for sharing. That's unfortunate what happened.

I have thought about the BTT UPS and recovery modules. But then thought it would be better to just get a normal stand alone UPS to plug the printer into. Power here is stable, so it's not been a priority.

[smirk]

Ok, doesn't sound like the board is completely dead. If there's some lights on there. Personally I would ditch the UPS and get that out of the equation. Your picture was interesting as (I presume) you  have the UPS connected to the second  12/24v port on the board. It's interesting because I can find no real reference to what that port is in the BTT documentation. The primary power port is listed as DC IN (which makes sense) but the other port is simply listed as 12/24v so I cannot tell if it is a feed, if it's really connect or anything. Rather dissapointingly the BTT manual for the E3 mini v2.0 shows a UPS in use but only connected to the outage-detection pin but not which power connectors to use.

The UPS 2.0 (24v) product page lists some connections for different models of MCU, and they are all connected "in-line" with the main DC feed into the MCU. This makes sense. However, in the advert there is a picture of an E3 mini (granted the v1.2 version):

 

This arrangement strikes me as rather odd, personally I would not do it, as the UPS would directly discharge into the main power supply. Yes it's also connected to the MCU as it's all part of the same circuit but why not just plug the UPS into the main DC-IN power connector so it's "in-line". Howewer, this UPS thing is not for me. As @claudermilk has observed he'd use a "proper" UPS, which could be a challenge sizing in itself (how much run time do you want). I am probably fortunate but, apart from times when I've kicked a power cable ouit, I have never lost a print to a power outage.

 

Anyway back to the point of this post. I don't think your MCU is necessarily dead/fried. Personally I've never reflashed a Mini E3 (by-passing the regular boot-loader) but I have had to reflash plenty of other types of board when the bootloader has been erased (or the board never came with one). It is eminently possible.

A bit of googling and I found this piece on reddit about reflashing an E3 mini v2 without a boot loader

However, in the spirit of keeping it simple, (if it was me) I would take the UPS out of play and disconnect it. I would then reformat the SD card as FAT32 and then try flashing the firmware.bin file that you would've complied up with klipper and see how that goes. NB: the stock firmware.bin files from BTT and that one from the reddit site will not be Klipper firmware so simply won't speak to Mainsail. They'll probably be full-blown Marlin, so you could connect the board to a PC and try running something like Pronterface which would confirm the USB comms between the MCU and a PC were working.

 

 

[eugen360]

Yes, trowing away the BTT UPS was the first thing I've done! Then re-flashing the firmware was the  next in line... but without success. I have tried now re-plashing a Marlin flavor firmware from that link you kindly provided and the board appears in PC Device management and the Pronterface does connect but I can't "move either of the axis". With this firmware a new LED light up on the board, the green one in the middle... see attached.

So I can confirm, the board isn't totally dead but totally paralyzed! I'm sorry, don't want to sound negative or something but ... I'm a little bit down...

And I needed so much this printer this week... the kids are preparing a charity event at church, a Creativity Fair and I've promised them some 3D objects...

[smirk]

Totaly understand the frustration. We've all been there at some point when a printer fails (generally at the worst time) and seems beyond repair.

 

The fact that you've got lights and have been able to reflash marlin, plus communications using pronterface are all really good signs. The fact that none of the motors are moving is not necessarily terrible and could be because of a couple of non-fatal reasons. It could be as daft as the firmware wasn't compiled with TMC driver support! My main guess would be end-stops. Ordinarily, you're not going to be able to move any of the motors without homing them, and you're not going to be able to home anything if the MCU is misinterpreting the end-stops. The advantage of klipper is the config (like how to read endstops) is stored in an easily readable text file (printer.cfg). In Marlin the config is part of the firmware that got flashed to the board, so unless you have the original source there is no way of guessing what the config actually is.

I don't think there is to much value in fretting about Marlin at this stage - you've proved the MCU is actually living.

You could just as a little extra sanity checking issue a  M122 gcode command into pronterfaces console window. This is the Marlin GCODE for querying the TMC drivers. Just entering M122 should (hopefully) list a bunch of settings  relating to the drivers. That would be fabulous (if there's no errors) but might not work if the stock firmware didn't actually include TMC driver support!

Personally at this stage knowing that the SD card works, the boot loader works and the MCU is "alive". I would now try flashing the Klipper firmware once again. Perhaps as a sensible step it might be best to compile the firmware from fresh for your E3 Mini board the Voron documents have a nice guide if you need a refresher.

 

[eugen360]

I've tried 3 times to re-flash the firmware using directly the saved firmware from the very first time build. But now I've redone the firmware image on Raspberry. The problem is that "serial" is missing on "ls /dev" command, see attached. There are only "serial0" and "serial1". If the board is recognized by the Pi than "serial" must appear in the list...

I'm attaching also a print-screen of the error in Mainsail.

[smirk]

2 hours ago, eugen360 said:

If the board is recognized by the Pi than "serial" must appear in the list...

Yup, that's correct. I wonder if the issue is with the PI rather than the MCU? Unless you were running pronterface from the PI (when it successfully connected to the Marlin firmware). I'll make a dangerous assumption, that perhaps you used another machine to run pronterface and connect to the Marlin firmware.....

The creation of the "virtual" serial device drivers is a function of the operating system (Linux) rather than klipper, so in theory if you reflashed the MCU back to Marlin it should then show up in Linux within /dev/serial/by-id (or by-path), fair enough Mainsail would still grumble. That, however, would prove the PI/USB port/USB cable was fine (as I say, I'm assuming you hadn't effectively already done by using pronterface from the PI)

The klipper firmware is pretty slim there's not much to it as all the grunt work is done by the software on the PI, so there won't be a lot to go wrong. I think I have an E3 somewhere so I'll give that a go and see if it gives me any ideas.

[eugen360]

8 hours ago, smirk said:

I'll make a dangerous assumption, that perhaps you used another machine to run pronterface and connect to the Marlin firmware.....

Guilty as charged! Being focused on the MCU I've connect it with my PC, Pronterface was already there for years, soo .... It's getting better and better... Forgot to mention that I've also re-flashed the Pi yesterday, before of course, flashing the MCU. But if the Pi is the problem here than it's even worse, because these are no longer available...

My God, what have I done...

[eugen360]

I've just had a look at you profile Smirk.. Happy belated birthday my friend!

[smirk]

1 hour ago, eugen360 said:

Happy belated birthday my friend

Thank you, I'm now of an age where I do my best to forget these things. Sadly I am now more than old enough to figure on government lists for vaccinations and various screening efforts.

 

2 hours ago, eugen360 said:

I've connect it with my PC, Pronterface

Phew, that gives an avenue for investigation. First up, quick'n'easy, try the USB cable that you used between the PC and the MCU (that's a known-good cable) between the PI and the MCU. Try all the USB ports on the PI (I can't remember if they are all shared on the same USB bus or if they are shared in pairs) so it could be a bit of overkill but just in case there's a duff USB port on the PI. Having said that, I don't think I've ever seen a USB port go bad over the years but I've seen plenty of shonky USB cables. The other test to make sure the PI's USB ports are fine is plug a USB flash drive into one of them (assuming you have one) or any known-good USB device and see what the PI says.

If that still doesn't show anything up I'd try putting the MCU back to Marlin firmware. You've kinda got a known good setup between the MCU (with Marlin) the PC with Pronterface and that USB cable. Then try the PI with the Marlin-flashed MCU (and the known-good USB cable) - as I say Mainsail will still bork but you should see entries in the /dev/serial folder (or even doing an lsusb command should display the Marlin flashed MCU, just in case it picks an unusual device interface).

If that works then we can focus back on the Klipper firmware (perhaps some of the parameters for the compilation are wrong, boot-loader off-set is an important one, as well as the CPU type of the MCU).

[smirk]

2 hours ago, eugen360 said:

But if the Pi is the problem here than it's even worse, because these are no longer available...

What sort of PC do you have? Klipper (Mainsail, etc) don't just need a PI to run.

[atrushing]

3 hours ago, eugen360 said:

But if the Pi is the problem here than it's even worse, because these are no longer available...

I will be trying out the BTT CB1 on my next printer build. The price and availability are very reasonable and, from the YT reviews, it sounds like their Klipper image works.

But certainly follow the advise from @smirk. I have seen USB ports fail and you could also try UART (probably on the TFT pins) to connect your SKR Mini to the Pi.

[smirk]

13 minutes ago, atrushing said:

The price and availability are very reasonable

And they've got their "black Friday" sale on for the next few days.

[smirk]

I was thinking some more about the BTT UPS option and why it's not a reasonable option for the Klipper world (Beyond the fact the klipper developers have no plans to create resume-on-power-loss code). This is not helpful for the issue at hand, merely me recording my musings .....

The BTT UPS is really built on the assuumption that the only electronics in the printer are the MCU and regular printer gubbins (steppers, heaters and fans), in other words, I suspect a relatively "simple" printer. Given how much power heaters can suck I would be amazed the UPS device could deliver anywhere enough charge from it's capacitors to keep the system alive for the vital seconds required to park the head (unless of course the power-failure code in Marlin automatically turns off the heaters the moment a power outage is detected).

However, with a kliipper set up (even if the code was there) there's a hell of a lot more electronics at play (MCU, Raspberry PI, perhaps a fancy touch screen, as well as the printer gubbins). I think that extra electronics would violate the [simplistic] assumptions behind the BTT UPS (i.e. small MCU only and regular printer gubbins). Things get even more impractical when you consider the often greater additional complexity of a Klipper set up - e.g. multiple MCU (including CANBUS); LEDS; schlond poofa, etc.

[eugen360]

Wow, lot of info! So...

4 hours ago, smirk said:

Sadly I am now more than old enough to figure on government lists for vaccinations and various screening efforts.

Government's use to do a lot of mistakes and obviously there's room for more...  Clearly, no one should put their hopes in them, especially with this "underground" politics...

4 hours ago, smirk said:

I'm now of an age where I do my best to forget these things.

Gen. '75 over here...

4 hours ago, smirk said:

Try all the USB ports on the PI

I've tried 2 for sure but maybe even 3, anyway I will test them all ! This raise me a question: is it possible that some current has "backflow" from that "UPS" over the USB port/cable to Pi?

4 hours ago, smirk said:

What sort of PC do you have? Klipper (Mainsail, etc) don't just need a PI to run.

Windows PC (desktop). But if we take out Pi board from equation it means the computer has to be "On" all the time I want to print something... Well, for the moment will be super Ok, without doubt!

2 hours ago, smirk said:

I would be amazed the UPS device could deliver anywhere enough charge from it's capacitors to keep the system alive for the vital seconds required to park the head

The current stored in the capacitors has to provide power to the electronics only to memorize the exact position (X,Y,Z) and move the tip of the nozzle a little bit away from the printed part. Doesn't matter where, say 10 mm in X or Y axis direction, away from the printed part.  I've tested this personally on I3 MK3S printer and it works... And they are using only the capacitors from the Rambo board  no extra high capacity cap.

 

@atrushing

Didn't know about CB1 but definitely I'll take a look, thanks! Also, have to search for documentations over UART!

 

Edited November 23, 2022 by eugen360

[eugen360]

Maybe I'm wrong... maybe Prusa uses the "left over" current from PSU. Makes more sense, there is I think, at least 2 bigger caps. I did not search for it, don't consider is very important...

[smirk]

3 hours ago, eugen360 said:

This raise me a question: is it possible that some current has "backflow" from that "UPS" over the USB port/cable to Pi?

I suppose it could but I think unlikely. The MCU USB port is the common item and it [probably] would've been fried by the 24v in the process.

3 hours ago, eugen360 said:

Windows PC (desktop). But if we take out Pi board from equation it means the computer has to be "On" all the time I want to print something

I was thinking partly as a stop gap (if the PI is troublesome) and also as an additional (granted not trivial) diagnostic as we know the MCU and PC talk to one another (via pronterface). I was hoping you would say "Linux" , there does seem to be a way of running Klipper under (more modern) Windows using Microsoft's Windows for Linux subsystem but I have not tried those instructions and might be unlikely to (as I generally run Windows as a VM under Linux). If I'm feeling particularly masochistic I might try that to see if it does work (generally trying to coax anything *NIX to run under Windows is tortuous).

 

I think for the moment in diagnostic terms, try the different PI USB ports, try an alternate USB cable (the one from the PC to MCU) and then last resort try Marlin FW on the MCU with the PC USB cable to the PI (you know that Marlin works and the PC<->MCU USB cable works).

3 hours ago, eugen360 said:

Maybe I'm wrong... maybe Prusa uses the "left over" current from PSU.

I don't think you'll be wrong, there will be a certain amount of charge left within the PSU (depends on how large it's various capacitors are) for a large/heavy weight power supply there can be a significant amount of residual charge (but those would not be used on a 3D printer). The residual charge would certainly be enough to write some variables (x,y,z co-ordinates) to non-volatile memory. Plenty of systems (like cache controllers on disks) use capacitors to store just enough charge to safeguard memory. The trouble with electricity it'll go where it's needed/consumed rather than where you want it. I'm just not convinced (without a complete/full UPS) that there'd be enough stored charge to power the PI (running klipper which is doing heafty work); motors and MCU (plus ancilliary stuff) to safely and reliably store results and move the head out of the way. There's also the issue of inherent inaccuracies in the motion system that means you'll only be able to get back to "nearish" or "close enough" to the point of failure.

[eugen360]

Right now I finished testing all the USB ports of the Pi. Using a different cable and also the blue cable that come with the MCU. No communication whatsoever...

But I've tested the USB ports also with Web Cam and 3 different USB drives, all are recognized and appear under lsusb command.

[smirk]

11 minutes ago, eugen360 said:

with the MCU. No communication

Was that with the Marlin firmware as well?

So the PI is working, the USB ports (and cables) are working, Linux is detecting stuff so that's OK. The MCU was fine (with Marlin and speaking to Pronterface)......that kinda leaves the Klipper firmware itself.

 

[eugen360]

6 minutes ago, smirk said:

Was that with the Marlin firmware as well?

I have tried only the communication between the MCU - PC and not MCU - PI when on Marlin ... but I could repeat the test I guess.

It happens to have at hand an Orange Pi, not sure if I can find it's power supply (working on 3V if I remember correctly)... should I try to install software on this one?

[smirk]

Right, I was playing with My SKR Mini E3 v2.0. The attached firmware flash is the one I've just compiled and used with it.

 

Question - if you disconnect the USB from the MCU and then power the printer on is there any life on the board?

 

The reason I ask, what I noticed when I was playing with my E3. The USB connection alone was good enough to power the board. Good enough for my laptop to see the board and detect the USB connection; good enough to flash the board with the new firmware and make all the LEDs blink on the board. However when I connected it to one of my PIs (running klipper) , simply using the USB cable, Klipper grumbled that it could not contact the MCU. It wasn't until I wired the MCU into the 24v supply that klipper would speak to the newly flashed MCU.

firmware.bin

[smirk]

Check the fuse on the E3 board, it should be transparent enough to see if the link is intact. I removed the fuse from my board, and thanks to the USB it still kinda appeared live but clearly wasn't.

[eugen360]

17 minutes ago, smirk said:

Check the fuse

This I've checked the very first time after the "happy" incident. The fuse is Ok, visually and with multi-meter checked. In the picture where I show the green LED, the MCU is powered on 24V only.

[eugen360]

Hello again,

In order to finish this thread I'll post a picture showing the printer working (well, you have to believe me it was working when I took the pic  ) but with the brand NEW board... the old one is gone!

Although the problems seems to be far from gone (neopixel not working anymore, filament blockage due to heat creep), at least it is working... kind of!