Frequent connection errors and communication failures

[harrison]

Hmm, I am surprised by this. That fix completely removed the issue when I could replicate it on our end.
So to confirm, you are still seeing messages like:
2021-03-03 03:37:24.318060 Failed Pumps comms 1 times on device M2
2021-03-03 03:37:26.367069 Failed Pumps comms 2 times on device M2
Namely – I am interested in the fact that it is happening twice in succession and almost exactly 2 seconds apart?

To diagnose your issue, could you go to line 1413 (after the line that says tries=tries+1) in app.py and insert the following command:

print(str(datetime.now()) + ‘ Failed with rw = ‘ + str(rw) + ‘ hl = ‘ + str(hl) + ” data1 = ” + str(data1) + ” data2 = ” + str(data2))

I want to see exactly what transmission it is trying to make when it fails…

[harrison]

Then please run a similar test to before – basically just turn the pumps off and on a load of times and copy across the PuTTY log here.

[harrison]

I’ve been again trying to replicate this today – I have made another small change on the Github to try to remedy one failure mode I observed, let me know how that goes for you.

[mattkratz]

Hey Harrison, finally got around to testing your new app.py (got my 2nd vaccine dose and had to take a couple days off). Seems to be working properly, with constant turning on and off of pumps not producing any errors. I also did a quick run where it automatically manipulates the pumps and it still seems to be running well. I’m gonna let it run over night to test it a bit more.

[mattkratz]

Hey Harrison, false alarm, I’m still getting the same error (tandem pump comm failures, separated by 2s) albeit at a much lower frequency, once every 1-2 hours instead of once every 5 minutes. I’m running the same app.py with your debug line right now and will update you ASAP.

[harrison]

Thanks – so, it is better, but not perfect! My next fix is a nuclear one: Build a new Linux Kernel with changes in the hardware drivers so that the two second failure is reduced to ~<0.1 seconds, hence making little impact on an experiment. I'm working on this now, will let you know once I have it up and running. It will mean flashing the board with a new operating system that implements such fixes...

[harrison]

Note that this will not make the error message go away entirely – rather, it will help the system recover quickly when there are errors hence minimising their impact…

[mattkratz]

Hey Harrison, here’s the terminal output of the debug run. Experiment starts at 17:17 in the 2nd panel, previous stuff is just process starting up.

[harrison]

Thanks! That is about what was expected.

Here is a Linux image:
https://www.dropbox.com/s/kvk55gfz88z6u5g/ChiBio-OS-2021-04-06.img.xz?dl=0
From my tests this new version will solve these issues by changing the 2 second timeout to be ~50ms, hence the occasional pump comms failure would not have much of an impact on operation.

To put it on your device you can use the same process as in the software setup guide (which i will update soon): Use Balenaetcher to flash the image to a micro-SD card, then put this into the beaglebone. It will then flash the memory on the beaglebone. Then you are done! In this case the image should include the entire Chi.Bio config including the operating code for the device.

Note that this will wipe any experimental data or other changes you have on your device so copy anything you want to keep off it first…

[Author]

Posts