Canbus - "Timer too close" mcu SHUTDOWN

[mvdveer]

 

Well, an interesting development that may be related to the broken symlink issue.

Background: The second Voron I build was a V2.4r2 300mm model (Fully self sourced) which started giving errors with the heaterbed not heating at expected levels, and some Raspi under voltage warnings. I suspected the two may be related. Was also time to switch to CanBus on this model. Fixed the Raspi power supply issue, replaced my aluminium build plate (had the original cut locally when I started the build) with a mandala rose magnetic plate and installed canboot with canbus.

Flashing the boards took all but 30min - changed the Klipper settings and off she went printing. Too good to be true - started getting "timer too close" errors, occasionally when homing, persistently when doing input shaping and prints failed after about 2minutes due to this. Done 4 Canbus printers - first time I experienced this. (Though read a lot about it)

Did a lot of reading on why this may happen, and despite very good advice from @Andrew Cremins (see here), I used the MCU_Timing script to alter the timing (Increase TRSYNC_TIMEOUT = 0.025 to 0.05) - obviously it did not help - not even a little!. Good advice, thanks Andrew, but I am stubborn.

Checked the canbus output - no errors, no dropped packages --Mmmmhhh - See below

Quote

5: can0: <NOARP,UP,LOWER_UP,ECHO> mtu 16 qdisc pfifo_fast state UP mode DEFAULT group default qlen 10

    link/can  promiscuity 0 minmtu 0 maxmtu 0

    can state ERROR-ACTIVE restart-ms 0

  bitrate 1000000 sample-point 0.750

  tq 62 prop-seg 5 phase-seg1 6 phase-seg2 4 sjw 1

  gs_usb: tseg1 1..16 tseg2 1..8 sjw 1..4 brp 1..1024 brp-inc 1

  clock 48000000

  re-started bus-errors arbit-lost error-warn error-pass bus-off

  0          0          0          0          0          0         numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

    RX: bytes  packets  errors  dropped missed  mcast   

    38901      5376     0       0       0       0       

    TX: bytes  packets  errors  dropped carrier collsns

    6680       1101     0       0       0       0       

 

Back to the raspi terminal and thought I may reflash Klipper on both the Octopus and the SB2040.

I had their UUID's and Klipper was fully functional.

Surprise surprise - I got an error when trying to flash using the UUID's:

Quote

 

pi@V…..:~/canboot/scripts $ python3 flash_can.py -i can0 -u 1a763f3218be -f ~/firmware/octopus_1.1_klipper.bin

Sending bootloader jump command...

Resetting all bootloader node IDs...

Checking for canboot nodes...

ERROR:root:Can Flash Error

FlashCanError: Unable to find node matching UUID: 1a763f3218be

 

Whaaat? - did a UUID check:

Quote

 

pi@V….:~/canboot/scripts $ python3 flash_can.py -i can0 -q

Resetting all bootloader node IDs...

Checking for canboot nodes...

Query Complete

 

No canboot nodes - but Klipper is not complaining, it is working. The reason being that the raspi sees the device, as can be seen by issuing the following command

sudo dmesg -w 

I get the following output:

Quote

[    4.829680] usb 1-1.4: Product: stm32f446xx

[    4.829695] usb 1-1.4: Manufacturer: CanBoot

[    4.829708] usb 1-1.4: SerialNumber: 0B0010000A50534841313020

 

So the octopus is present, seen by the raspi/klipper for what it is.

Now what?

Thinking outside of the square, I decided to check my version of Debian. As part of flashing canboot to the boards, I normally do a 

apt update
apt upgrade

To my surprise, I had the version installed causing the whole issue with loosing connection to mcu by a broken symlink.

Quote

 

pi@V.....:~ $ apt show udev

Package: udev

Version: 247.3-7+rpi1+deb11u2

 

Ran the patch as described in the beginning of this post.

Successfully completed input shaping, homing and all other checks without a complaint from the canbus interface.

Rebooted, fired up the printer, sliced a print and......

NO MORE mcu shutdown because of "timer too close" . Has been happily printing for the past 3hours.

So maybe, just maybe this bug in the operating system causes more than just a broken symlink. 

 

[devilmastah]

Too bad this is not helping for me at all, i am running against the exact same issue. Here my Octopus 1.1 is crashing consistant when homing Z. 
Getting rid of Zhop helps a bit then it crashes after the first BLtouch touchdown...

Recompiled everything to be sure the core speeds are right. Did the patch as well. No chance. 
I am running 3 Z motors with 2209s which propably does not help. This midday i will receive a Pi 4, as i read that some people had luck changing the host computer even though its the MCU complaining. 

Any ideas would be welcome, kind of lost here, build and designed my own corexy, concept worked, now want to finish the main build 

 

[mvdveer]

55 minutes ago, devilmastah said:

I am running 3 Z motors with 2209s which propably does not help.

Should not matter. The V2.4 runs 4 Z motors - and I also have 2209 steppers. I run the E-Motor, SB Led's, X-Endstop, hotend and part cooling fan from the Canbus, connected to the Octopus board through CanH and CanL, with 24V from the power supply.  The canH and can L cables are twisted and soldered to a RJ12 jack, plugged into the octopus port. Raspi 4B 4G connected to octopus via USB. Raspi powered via gpio header pins 2 AND 4 for 5V and 6 for GND.

What is your electronics setup? Some discussions have suggested running the Canbus board via USB connected to the Raspi and 24V supplied independently. (USB+Power)

[devilmastah]

I am running the Octo 1.1 and Ebb 42 of  the same 24V supply, for the rest more or less same setup as there.
The Octo is running with usb from a pi 3b+ so flashed with USB to Can bus bridge as communication interface. 

Only difference being that my can H and L are not twisted atm, might try that but i would expect to see an error for mcu EBBcan then instead of "mcu" which is the octopus board. When i test everything seperate it works just fine.

Do you know by any chance what the max baud rate for Canbus is in this case? Might want to try increasing it. 

Edit: 

tried with a baud of 2 000 000, it does not like that at all, had to reflash everything back to 1 mil.
Back to square 1. 

Edit2:

Made a twisted pair cable of some ethernet cable runing power externally, no difference, also have no dropped or missed packets so already assumed that

Edit3: 

tried different SD card for pi, no luck, also tried powering pie from 5.2 volt bench supply, there is no current spikes when thishappens, its runnning on an average 450mA with spikes to 550

 

Edit 4 and 5:

"Well, pi 4 fixed all issues out of the box" he said happy. Until i ran shaper_calibrate 

 

Edited August 4, 2023 by devilmastah

[mvdveer]

11 hours ago, devilmastah said:

max baud rate for Canbus

No reason to go higher than 1 000 000. 

 

11 hours ago, devilmastah said:

Well, pi 4 fixed all issues out of the box" he said happy. Until i ran shaper_calibrate 

Will have to read a bit more and give it some thought. 

[devilmastah]

Well i ended up ditching the whole canbus idea on the octopus, i still had a u2c from btt lying here which i wanted to send back. 
So octopus is on serial again, the ebb42 via the u2c, this fixed most of the issues and the printer is sort of working now, first print crashed with this time an error for the Ebbmcu, had to reduce microstepping of the extruder from 16 to 4 to fix it. 
This all feels really beta to me. 

What i noticed, and am wondering, i am running fluid atm, and i get like 4 miljion times ok per second in the console, to the point that after a emergency stop it takes 5 minutes for them to clear up. I dont know what they mean but i cant imagine it helps all this stuff has to be processed, same goes for klippy log that goes to 10 megabytes in a matter of minutes. Any way to disable logging all together to check if that improves things?

Edit:
Ditched Fluidd and Moonraker, back to octoprint. all issues gone ... i now highly suspect moonraker.

As for the above, the Ok messages are acknowlegements to gcode commands as far as i understand it now, octoprint does the same thing, but in octoprint it does not lag after a while, fluidd and moonraker build up a huge delay. I suspect over the queuelength of the canbus. 

 

Edited August 5, 2023 by devilmastah

[Tuct]

Hi, 

I decided to use the EBB36 and my SKR PICO flashed as USB2CAN with a small transiver and run into similar issues:
"Timer too close" mcu SHUTDOWN

It happened sometimes a few minutes into the print other times hours...

I checked my wiring and the configuration, When using an MCU in CAN bus bridge mode 1000000 baus rate is recommended especially if the MCU is also controlling some motors!

What solved it for me (waiting for the U2C head from BTT) was

* using a spare RP2040 with a can transceiver (Waveshare SN65HVD230) and Klipper in can bus bridge mode https://www.klipper3d.org/CANBUS.html#usb-to-can-bus-bridge-mode

My Pi (BTT PI) is connected to the main MCU (SKR Pico) via usb, this drives everything exceept the printhead, the RP2040 is also connected to the PI and via the transceiver to CAN and the EBB36.

With this setup i had not "Timner too close" for the last 15h of printing.

My hunch is that using the CAN bus bridge mode on the main MCU was too much and the CAN bus bridge had not enough CPU?....

 

Maybe this helps someone!

 

 

[PFarm]

On 7/30/2023 at 2:23 AM, mvdveer said:

 

Did a lot of reading on why this may happen, and despite very good advice from @Andrew Cremins (see here), I used the MCU_Timing script to alter the timing (Increase TRSYNC_TIMEOUT = 0.025 to 0.05) - obviously it did not help - not even a little!. Good advice, thanks Andrew, but I am stubborn.

When I switched to the Manta 8P I was getting consistent "time out" errors I changed the above changes to the mcu.py but still got the errors, not sure if this was the correct thing to do but also changed the TRSYNC_SINGLE_MCU_TIMEOUT = 0.500. Has worked great ever since.

[NOG]

I have a small farm of Voron 2.4's all with canbus/tap/sensorless homing/etc..., most are running Octopus/Pi but one is a Manta M8p/CB1 build

I've lost over 4 months fighting with canbus insanity - which I won't waste everyone's time documenting

TTC error is a catch-all error (like a windows BSOD crash) which doesn't really tell you much about the root-cause - it would be really nice if klipper dev team would be more verbose in their logs re: rootcauses

MCU.py timeout fix - addresses timeouts during HOMING process

Initially, I had great success using the Octopus-based canbus interface, then after 3 weeks, it throws TTC errors (only after 12+ hour printjobs).  I tried waveshare hats and they were unreliable as well.  I finally got a dedicated BTT U2c 2.1 adapter (updated the firmware as directed in community), and the TTC errors stopped

I retrofitted the rest of my farm to use U2c adapters, and fast-forward 2+ months...

I started to see random RPI CPU throttling warnings on one of my printers, replaced the 5v PSU, and slightly overvoltaged it (+5% 5v->5.25v) to help ensure RPI was getting necessary power.  CPU throttling warnings continued... then I started getting TTC errors (frequently at this point - after 20 minutes printing vs 12+ hours previously)

I removed the RPI (and 5v PSU) and replaced it with a BTT Pi v1.2, rebuilt the OS, and imported all config settings -- it's running reliably now (1+ month)

tl;dr (quick and dirty)

I don't suggest using RPI boards, instead, use a BTT pi v1.2 instead - run the Klipper image from BTT (same version as Manta M8p) - it just works

If you must use a RPI, suggested tweaks:
- Use an ethernet cable - disable WIFI and Bluetooth (can contribute to cpu lag issues)
- Install a heatsink and set the CPU speed to max - NOT overclocking it (disable adaptive speeds)
- Raspian OS build - 32bit build only (64bit build caused a lot of instability w/ timing/communications with canbus toolhead MCU - maybe fixed now, but I don't know)
- Camera - use a Logitech c270 (supported natively by kernel - keeps cpu/kernel dependencies low/manageable, without increasing cpu load)

Run a dedicated u2c adapter - BTT u2c 2.1 has been solid for me

....

If a new build, consider a Manta M8p + CB1 - the canbus interface works reliably (no u2c required).  note: heatsink does not come with cb1, but can be bought online (suggested)

Best,

Paul