[harrison]

If I understand correctly, the system keeps ticking over and doing measurements, and it thinks the pumps are going fine, but they have actually stopped?

Indeed this is something unlike any issues I have seen. What version of the code/software are you using? Is there anything else non-standard running on your beaglebone?

My best guess at a software reason would be if somehow the pump modulation threads are failing somehow, or the system is reaching some maximum in number of threads, but I don’t know why this would be occuring in your case. It has never been an issue for ours (or as far as I know, for other peoples’). Perhaps you can try putting a print statement into the pumpmodulation (or after that, into the setPWM function) which will print something out when the device attempts to communicate via I2C with the pumps? THat way you would be able to see if there was some software issue (I.e. the communications to the pumps are not even being attempted after the freeze)

In terms of hardware, the pumps are “downstream” of an I2C repeater in each reactor, which potentially could be locking up somehow. I recently made a new software setup V2.0 (see the software page) that can overcome some such errors. But, it seems unlikely that this would be the cause of your problem since all three reactors’ pumps are locking up at the same time (at least I think this is the case from your message?), which would be highly unlikely if it was a reactor-specific hardware fault.

Potentially if it is a hardware issue the code could be adjusted to trigger the watchdog circuit intermittently (which powers down the reactors, which happens when you restart the code), but this wouldn’t solve it if the threading was at fault.

[harrison]

Hi Wilson,

You say you provided 3.3V where it “should be giving it”. Does this mean you are trying to replicate the watchdog signal?

0x60 is indeed one of the PWM chips. If I recall correctly 0x70 is a second I2C address that is reserved by the pwm chip, but we don’t communicate with it directly.

0x1b is the thermometer on the beaglebone control board. You can see what every address corresponds to looking at lines 346-358 (and line 383 for the multiplexer) in the app.py file (current version on github: https://github.com/HarrisonSteel/ChiBio/blob/master/app.py)

I am not sure of a solution to your last point. This is not something I have seen or heard of happening before. I must say at this point it seems you are chasing after some tricky hardware issues – which ultimately should not be solved by Labmaker.

I’ll upload the latest version of the schematics to the site now…

[harrison]

OK. If you have a SD card can you follow version 2 of the software setup guide? This will completely flash the beaglebone with both operating system and software. In theory it doesn’t require you to log into the beaglebone, just power it on with the SD card inserted. Then, you can remove the SD card and connect it as before.

[harrison]

Hello Yzhang,

Did you ever get it to communicate with the Control Computer via Putty? If not – then my best guess would be that you haven’t got appropriate drivers for the Beaglebone Black on your PC. The Beaglebone itself is an off-the-shelf part, so if you cannot communicate to that either it is a driver issue, or (perhaps, but seems very unlikely) the Beaglebone has a manufacturing fault.
Can you get it to connect on a different PC?

[harrison]

I just saw the second post – it is suspicious that you can’t see the multiplexer on the I2Cdetect command.
Could you try running the I2cdetect commands again, and afterward execute the command “dmesg” in the linux terminal, and then copy here the bottom 10 lines? This will ideally tell us whether it is having I2C communication faults, or something stranger is happening…

[harrison]

Dear Wilson,

It seems unlikely it could be down to the software since as you said you tried several versions.
Perhaps the Beaglebone is at fault, but this does seem unlikely.
Do you have an oscilloscope by any chance? To test this the ideal method would be to look at the I2C bus under a scope and check that it is sending data as you expect.

When you run the test above is there any reactor connected? If yes – what happens if you disconnect all reactors from the beaglebone control board?

Harrison

[harrison]

Great!
There is also a new software guide on the website which means you don’t even need to download any packages – it comes with them all included.

[harrison]

Hi Matt,
Sorry to hear you are having these issues – from the description of the causes leading up to it this sounds like there may have been some significant hardware faults in that reactor. Whatever was wrong with it eventually led to a short in the circuit controlling the Blue LED (which incidentally is the most powerful of them) and now it is permanently on at maximum power (higher than intended by the software!)
I’d recommend getting in touch with Labmaker to tell them it came with a hardware fault and ask if they can replace it…

[harrison]

Thanks! That is about what was expected.

Here is a Linux image:
https://www.dropbox.com/s/kvk55gfz88z6u5g/ChiBio-OS-2021-04-06.img.xz?dl=0
From my tests this new version will solve these issues by changing the 2 second timeout to be ~50ms, hence the occasional pump comms failure would not have much of an impact on operation.

To put it on your device you can use the same process as in the software setup guide (which i will update soon): Use Balenaetcher to flash the image to a micro-SD card, then put this into the beaglebone. It will then flash the memory on the beaglebone. Then you are done! In this case the image should include the entire Chi.Bio config including the operating code for the device.

Note that this will wipe any experimental data or other changes you have on your device so copy anything you want to keep off it first…

[harrison]

Note that this will not make the error message go away entirely – rather, it will help the system recover quickly when there are errors hence minimising their impact…

[harrison]

Thanks – so, it is better, but not perfect! My next fix is a nuclear one: Build a new Linux Kernel with changes in the hardware drivers so that the two second failure is reduced to ~<0.1 seconds, hence making little impact on an experiment. I'm working on this now, will let you know once I have it up and running. It will mean flashing the board with a new operating system that implements such fixes...

[harrison]

I’ve been again trying to replicate this today – I have made another small change on the Github to try to remedy one failure mode I observed, let me know how that goes for you.

[harrison]

Then please run a similar test to before – basically just turn the pumps off and on a load of times and copy across the PuTTY log here.

[harrison]

Hmm, I am surprised by this. That fix completely removed the issue when I could replicate it on our end.
So to confirm, you are still seeing messages like:
2021-03-03 03:37:24.318060 Failed Pumps comms 1 times on device M2
2021-03-03 03:37:26.367069 Failed Pumps comms 2 times on device M2
Namely – I am interested in the fact that it is happening twice in succession and almost exactly 2 seconds apart?

To diagnose your issue, could you go to line 1413 (after the line that says tries=tries+1) in app.py and insert the following command:

print(str(datetime.now()) + ‘ Failed with rw = ‘ + str(rw) + ‘ hl = ‘ + str(hl) + ” data1 = ” + str(data1) + ” data2 = ” + str(data2))

I want to see exactly what transmission it is trying to make when it fails…

[harrison]

Hi Nobuhiro,
I came across a similar error to this and figured out a fix to hopefully catch these issues – check out the new version of app.py on the Github.
Harrison

[Author]

Posts