Frequent connection errors and communication failures

[sclamons@gmail.com]

We’ve been running experiments intermittently for a while now, but in the last month we’ve started seeing lots and lots of communication failures and connection problems. Out of three reactors we’ve tried using, one currently can’t connect at all, one encounters frequent “Failed Pumps comms <#> times on device <device>” until it crashes, and the third encounters the “Failed Pumps comms” error but less frequently.

Here’s typical console output from our recent runs:

root@beaglebone:~/chibio# bash cb.sh
[2021-03-15 15:05:54 +0000] [19135] [INFO] Starting gunicorn 19.9.0
[2021-03-15 15:05:54 +0000] [19135] [INFO] Listening at: http://192.168.7.2:5000 (19135)
[2021-03-15 15:05:54 +0000] [19135] [INFO] Using worker: sync
[2021-03-15 15:05:54 +0000] [19139] [INFO] Booting worker with pid: 19139
2021-03-15 15:05:57.891418 Starting watchdog
2021-03-15 15:06:00.246724 Initialising devices
2021-03-15 15:06:00.616026 Initialised M1, Device ID: 61453598001245557550
2021-03-15 15:06:01.258371 Start Up Complete
2021-03-15 15:09:30.786435 Failed transmission test on Pumps 1 times consecutively on device M1
2021-03-15 15:17:29.171288 OD Measurement exception on LASER650
2021-03-15 15:20:32.595893 Failed Pumps comms 1 times on device M1
2021-03-15 15:20:34.675745 Failed Pumps comms 2 times on device M1
2021-03-15 15:26:33.051630 Failed Pumps comms 1 times on device M1
2021-03-15 15:26:35.123637 Failed Pumps comms 2 times on device M1
2021-03-15 15:32:33.483680 Failed Pumps comms 1 times on device M1
2021-03-15 15:32:35.539636 Failed Pumps comms 2 times on device M1
2021-03-15 15:35:33.747586 Failed Pumps comms 1 times on device M1
2021-03-15 15:35:35.795639 Failed Pumps comms 2 times on device M1
2021-03-15 15:35:46.451609 Failed Pumps comms 1 times on device M1
2021-03-15 15:35:48.531649 Failed Pumps comms 2 times on device M1

We’ve seen this with two different pump boards (both of the ones we’ve tried), and many combinations of power supply and USB cable. Any idea what might be causing all of these errors?

One non-standard thing we’ve been doing — we’ve been testing different different ways to oxygenate the reactor, including continuously pumping in air. So some of our runs have one of the pumps running full time. Could this be breaking something?

[sclamons@gmail.com]

Control board is v1.1.
Pump boards are both v1.1.
The reactor that can’t connect at all is a v1.0, the other two are v1.2.

[harrison]

Hello,
It looks like the device is unhappy with several things – the OD measurement/spectrometer, and also the pumps.
One possibility is that you have been very unlucky and gotten several reactors with electrical faults in them – meaning they cause these errors intermittently.

But, perhaps more likely is that you are having issues with the moisture sensing circuit – the narrow tracks on the top of the device. These are designed to deliberately crash the system if it encounters liquid, in case whatever protocol has been programmed is overflowing the device. This sensor can manually power down the device, which can lead to various different error messages (since the next command sent will fail). It might be possible that your pumping in air is in rare circumstances leading to drops being ejected from the lid and landing on the moisture sensors.

To fix this I would suggest trying pumping out air. You will get the same effect (i.e. replacement of head-space gas), but that way all the moisture will be travelling out through the pump, rather than directly out of the lid where it may then encounter the moisture sensors.

Harrison

[mattkratz]

Hey Harrison,

I’m a lab mate of Sam who’s also been working quite heavily with the Chi bios. Couple of comments, that specific console output Sam posted was from a run with an empty chi bio i.e. no tube, with the motors just pumping ethanol in a closed loop in a separate container. Furthermore, in the experiments where we grow stuff, we’re using the 3D printed sealed cap you guys made, so unless we very sloppily screw on the cap/tubes, it seems like liquid droplets exiting due to air pumping is unlikely.

Additionally, that specific console output that Sam posted may have been the result of using a power supply that was 12 V 1A and not 12V 3A. Nonetheless, when we run with a proper power supply, we still get several pump comm errors popping up. In several previous runs, this pump comm error will eventually occur 11 times consecutively and then shutdown the device (console output below *). We always see this pump comm error intermittently (1 to 2 failures in series every 10 minutes or so), but I’ve only seen the hard shutdown on 16 hr + runs.

I also wanted to elaborate on some specific configurations we’ve looked at:
– We’ve used brand new reactors with the same pump board(s) and control board. We still get intermittent pump comm errors.
– We’ve used a brand-new control and pump board with both old and new reactors. The old V1.2 reactor (the one Sam references) initializes and has the pump comm errors. New reactors have the same pump comm errors at similar frequencies to the old V1.2 reactor.
– We’ve also tried two different brands of micro-usb cables, still same pump comm errors.

All the above were performed with an empty chi bio or with a chi bio with a flask of media (no air pumping). To further add, I’ve just ran three chi bios simultaneously (empty cultures) using a new control pump board and we got the following errors overnight (same console output as below, now with three devices :P). We didn’t get the serial error that leads to device shutdown, and I’m going to let them run another evening to see if we can get this serial error to occur.

*
2021-03-03 03:21:26.078056 Failed Pumps comms 1 times on device M2
2021-03-03 03:21:28.126103 Failed Pumps comms 2 times on device M2
2021-03-03 03:21:28.234715 Failed Pumps comms 1 times on device M2
2021-03-03 03:23:05.194113 Failed Pumps comms 1 times on device M2
2021-03-03 03:23:07.262103 Failed Pumps comms 2 times on device M2
2021-03-03 03:23:47.646114 Failed Pumps comms 1 times on device M2
2021-03-03 03:23:49.694203 Failed Pumps comms 2 times on device M2
2021-03-03 03:28:51.269245 Failed Pumps comms 1 times on device M2
2021-03-03 03:33:05.086061 Failed Pumps comms 1 times on device M2
2021-03-03 03:33:07.134103 Failed Pumps comms 2 times on device M2
2021-03-03 03:35:43.226033 Failed Pumps comms 1 times on device M2
2021-03-03 03:35:45.278092 Failed Pumps comms 2 times on device M2
2021-03-03 03:36:17.950060 Failed Pumps comms 1 times on device M2
2021-03-03 03:36:19.998105 Failed Pumps comms 2 times on device M2
2021-03-03 03:36:52.990063 Failed Pumps comms 1 times on device M2
2021-03-03 03:36:55.038106 Failed Pumps comms 2 times on device M2
2021-03-03 03:37:24.318060 Failed Pumps comms 1 times on device M2
2021-03-03 03:37:26.367069 Failed Pumps comms 2 times on device M2
2021-03-03 03:37:43.362097 Failed Pumps comms 1 times on device M2
2021-03-03 03:37:45.438092 Failed Pumps comms 2 times on device M2
2021-03-03 03:40:05.542024 Failed Pumps comms 1 times on device M2
2021-03-03 03:40:07.614096 Failed Pumps comms 2 times on device M2
2021-03-03 03:42:12.094059 Failed Pumps comms 1 times on device M2
2021-03-03 03:42:14.142102 Failed Pumps comms 2 times on device M2
2021-03-03 03:52:12.778016 Failed Pumps comms 1 times on device M2
2021-03-03 03:52:14.846093 Failed Pumps comms 2 times on device M2
2021-03-03 03:55:25.725839 Failed Pumps comms 1 times on device M2
2021-03-03 03:55:27.774103 Failed Pumps comms 2 times on device M2
2021-03-03 03:58:45.046032 Failed Pumps comms 1 times on device M2
2021-03-03 03:58:47.102145 Failed Pumps comms 2 times on device M2
2021-03-03 04:06:01.406067 Failed Pumps comms 1 times on device M2
2021-03-03 04:06:03.454103 Failed Pumps comms 2 times on device M2
2021-03-03 04:13:26.974064 Failed Pumps comms 1 times on device M2
2021-03-03 04:13:29.022103 Failed Pumps comms 2 times on device M2
2021-03-03 04:14:46.238016 Failed Pumps comms 1 times on device M2
2021-03-03 04:14:48.286090 Failed Pumps comms 2 times on device M2
2021-03-03 04:19:18.462058 Failed Pumps comms 1 times on device M2
2021-03-03 04:19:20.510101 Failed Pumps comms 2 times on device M2
2021-03-03 04:29:47.486029 Failed Pumps comms 1 times on device M2
2021-03-03 04:29:49.534096 Failed Pumps comms 2 times on device M2
2021-03-03 04:31:25.214155 Failed Pumps comms 1 times on device M2
2021-03-03 04:31:27.262107 Failed Pumps comms 2 times on device M2
2021-03-03 04:33:34.878150 Failed Pumps comms 1 times on device M2
2021-03-03 04:33:36.926103 Failed Pumps comms 2 times on device M2
2021-03-03 04:44:10.270116 Failed Pumps comms 1 times on device M2
2021-03-03 04:44:12.318107 Failed Pumps comms 2 times on device M2
2021-03-03 04:44:51.494197 Failed Pumps comms 1 times on device M2
2021-03-03 04:44:53.566090 Failed Pumps comms 2 times on device M2
2021-03-03 04:45:17.614121 Failed Pumps comms 1 times on device M2
2021-03-03 04:45:19.678200 Failed Pumps comms 2 times on device M2
2021-03-03 04:54:20.255675 Failed Pumps comms 1 times on device M2
2021-03-03 04:54:20.303669 Failed Pumps comms 2 times on device M2
2021-03-03 04:54:20.326110 Failed Pumps comms 3 times on device M2
2021-03-03 04:54:20.348655 Failed Pumps comms 4 times on device M2
2021-03-03 04:54:20.371241 Failed Pumps comms 5 times on device M2
2021-03-03 04:54:20.393793 Failed Pumps comms 6 times on device M2
2021-03-03 04:54:20.416273 Failed Pumps comms 7 times on device M2
2021-03-03 04:54:20.438787 Failed Pumps comms 8 times on device M2
2021-03-03 04:54:20.461300 Failed Pumps comms 9 times on device M2
2021-03-03 04:54:20.483801 Failed Pumps comms 10 times on device M2
2021-03-03 04:54:20.506252 Failed Pumps comms 11 times on device M2
Failed to communicate to a device 10 times. Disabling hardware and software!
[2021-03-03 04:54:20 +0000] [32547] [INFO] Shutting down: Master
[2021-03-03 04:54:20 +0000] [32547] [INFO] Reason: App failed to load.

occasionally also get failed PWM comms and failed transmission tests on Pumps

**

[mattkratz]

Woops meant to send my previous post as a reply to you

[harrison]

Hmm, OK so you have tried many permutations of reactor-pump-computer, and it seems the issue persists.
I now understand better your experiment, so agree it is probably not an issue with the moisture sensing.

Given that it seems all the errors you see now are Pump related, I suppose it points to issues with your pump boards.

I assume you have told Labmaker about this? In the past they have been good about replacing faulty devices. I know at some point they found there were several pump boards on which the micro-USB connector was making a very weak connection (due to insufficient solder). Perhaps you have boards from this batch.

So, one option would be ask them to replace the pump boards. Alternatively, if you (or one of your colleagues) has good soldering skills you can easily fix the connections yourself. I assembled dozens of these by hand, if you are experienced at soldering small components and have good equipment it is not particularly difficult. If you want to try that let me know and I can give you more detailed guidance.

[mattkratz]

We wanted to check on the forums if it was potentially a fixable non-hardware issue. Now that it seems it’s very likely that it is a hardware issue we’ll probably contact lab maker to see if we can the pump boards replaced. Thanks for the advice and feedback, I’ll keep you updated on how things evolve!

[harrison]

I’m afraid I think it is most likely down to manufacturing – unless you make crazy changes to the code/setup I am unclear on why this would happen absent a hardware issue, given it works on many hundreds of other Chi.Bios

[harrison]

Hello – just to let you know I think this issue may be arising from recent updates made to some of the packages used by the software. Specifically, the way in which it recovers from errors in the I2C bus communiations. Could you possibly let me know what version of the Debian operating system is installed on your device, and what version of the Adafruit packages are on there? (these can be found by executing “pip3 list” in PuTTy).

[mattkratz]

Hey harrison, sorry its taken so long to get back.

lsb_release -a (debian OS version) output is:

root@beaglebone:~/chibio# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster

pip3 list output is:

root@beaglebone:~/chibio# pip3 list
Package Version
————- ——-
Adafruit-BBIO 1.1.1
Click 7.0
colorama 0.3.7
decorator 4.3.0
distro 1.3.0
Flask 1.0.2
itsdangerous 0.24
Jinja2 2.10
MarkupSafe 1.1.0
numpy 1.16.2
pip 18.1
pyctrl 0.4.3
rcpy 0.5.1
scipy 1.1.0
setuptools 40.8.0
Werkzeug 0.14.1

[mattkratz]

FYI, also did this on another control board (has same warning messages and behavior) we have and outputs are the same

[harrison]

OK – could you try rolling back the debian operating system to version 10.0? I think this download link should work:
https://debian.beagleboard.org/images/bone-eMMC-flasher-debian-10.0-iot-armhf-2019-07-07-4gb.img.xz

It seems many of the errors come from the way the updated operating system is handling I2C commands. In my lab we are running them all on Debian 10.0 and no issues – but the latest version (probably which you have installed) is 10.4 and this acn cause strife!

[harrison]

Note to do this you would need to run through the software installation instructions available on this site – you flash it with the new operating system then run the Chi.Bio setup script again.

[harrison]

Hey both – I just made a change on the operating system (see the Github). If you copy the new python file (app.py) across to your device the issue should be resolved (you don’t even need to worry about chaning the Debian OS version!).
Harrison

[mattkratz]

Hey Harrison, I tried replacing the old app.py with the new app.py. I also deleted the associated pyc of the old app.py. We’re still getting the same error messages unfortunately… Is there anything else I ought to do beyond adding the new py file? I also tried completely replacing the root directory with the git hub repo and still the same results. I also had a closer look and confirmed that the specific Debian version is indeed 10.1.

[Author]

Posts