Weirdness, could use some help

[G_T]

Hi Folks,

Sorry this post is a bit long, and I apologize for asking a question on here. I'm the guy working on the hot printer in the build logs, but I hit an issue that isn't related to heat that has me scratching my head.

I had what seemed to be a Pt1000 failure so I swapped in a new one. Octopus v1.1. Klipper. Started life as an LDO kit but has been heavily modified, though the electrical is essentially per the LDO kit. It has been working fine for a while.

The first symptoms from the old Pt1000 was reading about 30C higher than the other two sensors in the chamber when all was cold. But it would still print. Then the old sensor started getting jittery readings and before long the firmware would just shut the printer down. It became nonfunctional in short order.

So I replaced the Pt1000. Same initial temp readings. But, temperature reading was consistent. I did a temperature stairstep of the hot end leaving it at each temperature for a while. Nice flat temperature reading lines. So all seemed well.

That is until I ran a new PID_CALIBRATE. You know what a good calibrate looks like? Nice consistent sawtooth as it swings +/- the target temperature? Well throw some distortion onto the sawtooth and that was what I saw. But it did generate calibration values. So I tried a print, 305C extruder temp printing PC-CF like I've been doing recently.

It died in short order, reading fairly wild instantaneous temperature swings on the hot end. And in reality it wasn't quite hot enough to extrude the plastic.

Then I tried pushing the bed around, jiggling wires, that sort of thing. It was back just reading perhaps 30C high in the cold state and no evidence at all of any wire or connection issues.

So I moved the sensor to a different port on the Octopus board. That didn't have any effect. Same results, all looks good except for the temperature shift.

I tried to mess around with settings in printer.cfg but putting in things that were wrong would result in Klipper crashing out and putting in my normal settings made it happy. Nothing learned there.

Then I turned on the part cooling fan. Instantly, and I do mean instantly, the temperature reading - while cold - dropped to essentially match the temperature readings of the other thermistors in the chamber. Turned the fan off and it stayed happy.

Then I turned on the hot end. It was warming up nicely with a smooth curve, looking good, until it wasn't. Klipper blew up  again. Restarted, temp reads high but now by perhaps 20C instead of 30C. Turned on part cooling fan, and it instantly readjusted to read like the other sensors.

I could say some colorful things, but figured to ask here if anyone has seen anything like this or has any ideas? Each time I cycle the part cooling fan on and off, the temperature reading does come out slightly different, and then stays consistent at the new temperature. Each time I start from powered off, the reading is off by somewhere around 30C but corrects when I turn on that fan and then stays corrected. Each time I really heat the hot end, it results in a Klipper crash fairly soon since the temperature readings go crazy.

If this isn't the right place to put this post, can a moderator relocate it?

Thanks everyone,

Gerald

 

[GarthSnyder]

I haven't experienced this, and you sound like you know what you're doing, so I hesitate to even comment. (Typical tech support: "Are you sure the computer is plugged in?"). But let's cover the basics:

• Did you set the correct sensor type in Klipper?
• Did you check to see if a PT1000 requires a different jumper/resistance to be set on whatever board is attached to the PT1000?
• Did you measure the actual resistance with a multimeter at various temperatures? 
• Did you spin around three times and consecrate your soul to Satan?

Item 4 is optional.  And #3 is, pretty much, also. I have actually never heard of a thermistor or RTD failing, though I'm sure it happens at some rate. But you didn't get two consecutive bad PT1000s.

Edited June 13, 2024 by GarthSnyder

[claudermilk]

I'll assume nothing else  changed in the config to trigger this. How is the thermistor plugged in? Can you plug it directly to the Octopus & eliminate the wire harness in between the controller and thermistor? From the behavior I wonder if the harness has a failure; that can cause all kinds of weird behavior.

[dshess]

Your point about graph distortions on PID tuning make me think of a printer I Klipperized and showed similar problems.  The temp readings were steady when the heater was on, then when it hit the target and turned the heater off, the temps would jump up 5C-7C (or more) in the next second.  Similar behaviour on the way down, it would be cooling 1C or 2C per second, then when the heater was turned back on it would gap down 5C or 7C or so.  The result was various odd behaviours, because PID tuning really expects a continuous result like physics would predict!  My "fix" was to set the heater to 80% power.

I suspect that there was a current leakage of some sort when the heater was live, which was influencing the reference voltage for the thermistor.  The effect was VERY consistent, the only reason I list a range for the jumps is that Klipper logs only give 1s resolution, so I don't have precise data - it might have jumped 8C-10C instantaneously and still be consistent with what I was seeing.  During printing it was steady like expected with no particular jumping around, so I don't think it was a wiring problem in the middle.  My multimeter didn't see any potentials on the heater block.  This was an old Melzi board with some known ground-plane issues, so I didn't debug it further.

[Yes, that sounds nervous-making.  I did add a MOSFET for the bed heater, and checked the video every 15 or 20 minutes when that printer was printing.  It was printing ABS parts so I only needed like 40 hours out of it.]

[G_T]

When you don't have a clue what is wrong, fix the bug you know and see what happens. Any sufficiently complicated problem is usually the intersection of two or more bugs. (can you tell I'm a software engineer?)

So, the neopixels had never worked right. They would shut off pretty shortly after turning the printer on. I've got plenty of light anyway and never cared much for them. So I ripped them out along with their wiring.

Now when I power the printer the PT1000 reads about 1 degree hotter than the other two in the chamber. I can cycle the part cooling fan on and off and the reading doesn't change at all. I'm running thermal tests now.

Heated the bed, and the PT1000 ramps up slowly since it is not far from the bed. Very smooth curve. Cycled the part cooling fan, no measurable effect. So far, so good!

Heating hot end to 50C. Very fast rise, 7C overshoot. I know the PID calibration has to be garbage at this point, so all I'm looking for is smooth response and consistent readings. Anyway 50C is when the heat break fan turns on and that didn't cause any ripples. Nor did cycling the part cooling fan. The temperature eventually stabilized hovering very close to 50.0C. So far, so good!

100C. 3.3C overshoot. Stabilized on temp faster. Cycled the parts cooling fan full on. About 0.3C temp drop, which recovered when fan was turned back off.

150C. 1.3C overshoot. Etc. Looking good!

200C. 1.7C overshoot. Fast recovery. Part cooling fan caused about 12C drop that recovered. With lack of good PID calibrate, this might be ok, or might not be ok.

300C. Heating is slower at higher temps than at lower temps. Heating curve showing more curve and not as steep. No overshoot. Cycled fan, about 3C change initially.

350C. Slow heating. IMHO the heater cartridge is perhaps a little wimpy. The combo is rated for 500C but IMHO it couldn't get there. It's struggling to get to 350C! I'd say printing rather slowly at 330C is about the max this heater is good for. I pulsed the part cooling fan on and off briefly and it didn't make much difference.

0C. Looking for smooth cooling curve.

My theory is the 5V regulator on the Stealthburner board that supplies power to the neopixels was failed and sucking excess current causing issues with the Octopus. I took power away from that circuit when I took out the wiring and the neopixels.

Now the heater cartridge shows as wimpy, but that could just be the heater cartridge. Perhaps 50W isn't enough for the temperatures I'm looking to run, and only barely enough for the temps I've been running. Otherwise it looks fine, at least before I fix the PID calibration.

Running PID_CALIBRATE with a terget of 290C. Really the chamber is only at 55C right now so it's on the cool side. I just want to get a ballpark PID and see how the temp trace looks during calibration. Generally 290C is also a little below my usual printing temp. But it's in the ballpark.

BTW, I've always had trouble getting this printer to be happy in PID_CALIBRATE. It's been hit or miss. This is the cleanest it has ever looked from this printer. Normally it would take me 2 or 3 tries. Not this time! And 290C might be the highest target I've gotten it to calibrate on.

The first pic is of the stairstep temp test. You can see the slight ripple in the middle of the flat zones when I cycled the part cooling fan. The second pic is from the PID calibration.

Moral of the story - If you don't need it in the printer, don't leave it in the printer!

One evening of testing does not lead to a conclusion, but perhaps the problem is solved. I'll report back later. Tomorrow night hopefully I'll get the printer hot and be printing which makes a good test.

Gerald

[G_T]

5 hours ago, dshess said:

Your point about graph distortions on PID tuning make me think of a printer I Klipperized and showed similar problems.  The temp readings were steady when the heater was on, then when it hit the target and turned the heater off, the temps would jump up 5C-7C (or more) in the next second.  Similar behaviour on the way down, it would be cooling 1C or 2C per second, then when the heater was turned back on it would gap down 5C or 7C or so.  The result was various odd behaviours, because PID tuning really expects a continuous result like physics would predict!  My "fix" was to set the heater to 80% power.

I suspect that there was a current leakage of some sort when the heater was live, which was influencing the reference voltage for the thermistor.  The effect was VERY consistent, the only reason I list a range for the jumps is that Klipper logs only give 1s resolution, so I don't have precise data - it might have jumped 8C-10C instantaneously and still be consistent with what I was seeing.  During printing it was steady like expected with no particular jumping around, so I don't think it was a wiring problem in the middle.  My multimeter didn't see any potentials on the heater block.  This was an old Melzi board with some known ground-plane issues, so I didn't debug it further.

[Yes, that sounds nervous-making.  I did add a MOSFET for the bed heater, and checked the video every 15 or 20 minutes when that printer was printing.  It was printing ABS parts so I only needed like 40 hours out of it.]

40 hours...  Hmmm, sounds like you were printing a 2.4 or Trident!

[TheBlackMambata]

Don't know if I understood well... But are you using a pt1000 for measuring the chamber temperature?

[dshess]

3 hours ago, G_T said:

40 hours...  Hmmm, sounds like you were printing a 2.4 or Trident!

V0.2 parts plus skirts and tophat (V0.1 kit).  On a 2016 bedslinger in a cardboard enclosure, so I didn't exactly print full plates.

[G_T]

19 hours ago, TheBlackMambata said:

Don't know if I understood well... But are you using a pt1000 for measuring the chamber temperature?

No, chamber senor doesn't need that kind of temperature range. I'm using an ATC Semitec 104NT-4-R025H42G with the temperature range set from 0 to 100C max. I might have to bump that max at some point but not quite yet.

Gerald

[G_T]

Well, it turns out the problem is not solved. I don't know why it worked perfectly for a while last night. I don't know why it wasn't working today. Example temp curve from today's testing shown in the first pic.

I did look at the system tab and found troubling things. I did a hard reset on the things indicated as corrupt, and then upgraded everything except Klipper. I'm hesitant to update Klipper but it may be the firmware is corrupted. Do you folks think I should update Klipper, or not?

I'm guessing the repositories having been shown as corrupted probably means the Pi's memory card is flaky?

Note, problem is still not solved. I've watched the temperature jump a large amount with no input, just sitting there idle. I pulled the Stealthburner front off while it was running... no effect. So at least I know nothing in there is the reason.

I have ordered a replacement LDO Hartk breakout board and plan to swap it, just in case it is something to do with that board. I have jiggled connectors etc a lot and not gotten a temp sensor response, except once (of many attempts) triggering the Z endstop by raising the print head on the Voron Tap. But it didn't reproduce.

I don't have a root cause. Leading options so far are memory card problems, corrupted firmware, Hartk board, sick Octopus. But I'm still in the dark.

No printing tonight... Well, at least I know that in the worst case if the hot end heater sticks full on (due to malfunctioning temp readings), it can't get things hot enough to cause a problem for this printer.

Gerald

[mvdveer]

15 minutes ago, G_T said:

Do you folks think I should update Klipper, or not?

I think you should be fine updating klipper. I have just updated my micron and VZBot with a beacon probe. Important if you have a beacon probe, to first update beacon software, then flash the beacon probe with the latest firmware

./beacon_klipper/update_firmware.py update all

and then only update klipper.

My switchwire with Voron TAP updated without an issue, as did my V0's with "klicky-type "probes.

So yes - you can safely update klipper. May not solve your problem though. To me it sounds like a hardware failure or corrupt system.

[G_T]

I've updated klipper but not the firmware on the octopus. Currently I'm making a backup of the raspberry pi SD card just in case.

I know of at least one case where I emergency stopped the printer via the power switch. I've shut it down a few times that way too. That's not a good way to treat the Pi. I could have caused some corruption by doing so, depending on what was going on at that time. Checking through the recent portion of the dmesg I did see one mention of a corrupted logfile that was replaced. That was probably me hitting the power switch while a file write was in progress. The Pi is running an fsck with repair each boot, which might catch and fix minor problems.

The other corruption noted in the pic in my previous post is much more troubling because I don't have an explanation for it. So I may end up replacing the SD card or even the Pi. TBD.

I'm on a linux laptop right now, and I'm making the backup via the following command:

   ssh -X my_pi_account_name@my_ip_address "sudo -S dd if=/dev/mmcblk0 bs=1M | gzip -" | dd of=~/pibackup.img.gz

With of course my_pi_account_name and my_ip_address obscured. The backup is currently running. It takes a little while.

I'd hate to have to recreate the Pi install and config from scratch! Hence the backup. I should have made a backup of some form months ago. Heck, even my reference install notes were stored on the Pi, which is a very bad plan.

Gerald

[atrushing]

1 hour ago, G_T said:

Heck, even my reference install notes were stored on the Pi, which is a very bad plan.

I recently did the same with my OpenMediaVault server.. It took me at least a month of following random guides to get it running properly again. Now it is running Trilium Notes in Docker so I can maintain and organize notes from my browser with links and config snippets.

[G_T]

I replaced the little breakout board on the print head. Initially the temps were a bit closer though still not right. Then in testing it faulted. Now I can't get Klipper to stay up at all so I can't see what sensor(s) or whatever might be having the problem. Is there any way to set up Klipper to NOT DISCONNECT ON EVERY ERROR IT ENCOUNTERS??? I've worked on software for critical systems for decades. This would never have been acceptible behavior. Disable power to devices but provide sensors and maintain connectivity - that would be much more reasonable behavior. Is there any way to configure something like this? Or set it so it just doesn't quit? When testing I can hit emergency stop if/when necessary.

All I get is MCU 'mcu' shutdown: ADC out of range. I don't know which one. I'm not sure how to set limits such that any value is acceptible. If I comment out the limits, it won't start anyway since it requires the limits be set!

Grrrrr

Gerald

 

PS - I apologize. This is just a bit frustrating. I'm hoping there is a config setting I can use to make this easier to debug.

Edited June 16, 2024 by G_T

[TheBlackMambata]

On 6/15/2024 at 4:13 AM, G_T said:

I'm using an ATC Semitec 104NT-4-R025H

Well, I was pretty sure that kind of sensor wouldn't work well with that range neither. Did it work well before? I'm currently using a small temperature and humidity sensor for the chamber, a HTU21D connected via i2c to my raspberry. It works well but I had stability issues cause the sensor is really sensitive and it needs cables solder on it, a low cables lengths etc... . Probably now there are better solutions.

Edited June 16, 2024 by TheBlackMambata

[G_T]

Yep, that's the chamber temperature sensor. The hot end sensor is a PT1000. I use a PT1000 because I print over 300C most of the time.

I got through the last problem. I took the stealthburner board back off and flush cut the pins on the back of the board. Then reassembled. Now it is running... still with the temperature issues with the hot end temperature sensor, but running. I've torqued on all the wires and the numbers don't change from that, so I think electrically it is fine at that board. In other words, swapping the board didn't make any real difference, except perhaps the plugs are a bit tighter being a newer board.

Sitting there stationary with neither the bed nor the hot end having been powered, current temperature readings are:

Extruder 96.6C

Heater Bed 23.4C

Chamber Temp 23.3C

Octopus 32.2C

Raspberry Pi 49.7C

The door is open, and everything is cool. On a previous power cycle, the Extruder measured 122.5C. I'll power down and restart, and... well, it didn't actually change this time. In previous testing it did change by up to 20C, except when it read much higher. Then sometimes it would drop down to only about 20-30C high. Another power cycle, and now it reads about 98C.

Essentially what I see is that through firmware restarts or power cycles, sometimes the number comes back the same, sometimes it doesn't. The range over which it can come back is from 20C high to 100C high, but occasionally it can jump another 100C or more. And then it can also drop back down from that often just a few seconds or a minute later. But there isn't consistency. And it can do all these things just sitting there while cold. It seems to have more issues with large upwards temperature jumps when hot though or when running for longer periods of time. Not sure thhough about that. Being hot isn't required.

Quick swapped the old PT1000 back (just at the plug; didn't install), did the required firmware restart, and the temperature reading didn't change. I don't think there is anything wrong with the sensor, or the wiring up at the Stealthburner side of things.

Gerald

 

PS - Funny how when you mention a Raspberry Pi this forum drops in an amazon link.

Edited June 16, 2024 by G_T

[G_T]

I've replaced the breakout board in the electronics bay. When I did I found there was a connection issue with the large cable connecting to the board. While I was at it, I replaced the rear Z motor that had a failing bearing.

After these updates, the temperature readings are all consistent. I'm heat soaking it now. But, the issue may well be solved - a connector problem.

Gerald

[G_T]

It looks like I'll have to recalibrate to get a good print, but temp behavior has been consistent now and survived testing pretty well. So hopefully, that ends this thread!

Gerald