pump and multiplexer errors

[mpatterson]

Hi, I have started getting “Failed to disconnect multiplexer” errors during runs. The “failed pump comms” errors are not new, but for the past month or so they have occurred more frequently. These errors have never caused Chi.Bio to stop. Some of these errors are similar to those reported by uthsc a few months ago, but I have not yet seen an I2C error, so is this a different problem or could it still be the same? None of the usb connections are loose and nothing appears to be dirty. I actually just cleaned the moisture sensing rings on M4 this week.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Oct 16 16:17:39 2022 from 192.168.7.1
root@beaglebone:~# cd chibio/
root@beaglebone:~/chibio# bash cb.sh
[2022-10-16 16:18:24 +0000] [1802] [INFO] Starting gunicorn 20.1.0
[2022-10-16 16:18:24 +0000] [1802] [INFO] Listening at: http://192.168.7.2:5000 (1802)
[2022-10-16 16:18:24 +0000] [1802] [INFO] Using worker: sync
[2022-10-16 16:18:24 +0000] [1805] [INFO] Booting worker with pid: 1805
2022-10-16 16:18:32.844615 Starting watchdog
2022-10-16 16:18:35.333982 Initialising devices
2022-10-16 16:18:35.587023 Initialised M0, Device ID: 38921214501243657614
2022-10-16 16:18:36.347729 Initialised M1, Device ID: 17422214321243657614
2022-10-16 16:18:37.046044 Initialised M2, Device ID: 40980108961243657614
2022-10-16 16:18:37.739387 Initialised M3, Device ID: 22538214021243657614
2022-10-16 16:18:38.434773 Initialised M4, Device ID: 35846214461243657614
2022-10-16 16:18:39.130829 Initialised M5, Device ID: 46094628061728357670
2022-10-16 16:18:39.827951 Initialised M6, Device ID: 50190214001243657614
2022-10-16 16:18:40.523677 Initialised M7, Device ID: 36881626021728357670
2022-10-16 16:18:41.136030 Start Up Complete
2022-10-16 16:20:12.155978 Set item: Thermostat to value 18.0 on M0
2022-10-16 16:20:29.213053Calibrated OD
2022-10-16 16:20:35.581115 Set item: OD to value 0.7 on M0
2022-10-16 16:20:42.677625Calibrated OD
2022-10-16 16:20:46.557318 Set item: Volume to value 20.0 on M0
2022-10-16 16:21:01.054410 Set item: OD to value 0.7 on M1
2022-10-16 16:21:07.623173Calibrated OD
2022-10-16 16:21:08.015417 Set item: Volume to value 20.0 on M1
2022-10-16 16:21:09.297543 Set item: Thermostat to value 18.0 on M1
2022-10-16 16:21:11.815715 Set item: OD to value 0.7 on M2
2022-10-16 16:21:19.015939Calibrated OD
2022-10-16 16:21:19.711860 Set item: Volume to value 20.0 on M2
2022-10-16 16:21:21.048002 Set item: Thermostat to value 18.0 on M2
2022-10-16 16:21:59.290479 Set item: OD to value 0.7 on M3
2022-10-16 16:22:06.675060Calibrated OD
2022-10-16 16:22:07.619049 Set item: Volume to value 20.0 on M3
2022-10-16 16:22:08.859111 Set item: Thermostat to value 18.0 on M3
2022-10-16 16:22:12.171417 Set item: OD to value 0.7 on M4
2022-10-16 16:22:19.075722Calibrated OD
2022-10-16 16:22:19.579866 Set item: Volume to value 20.0 on M4
2022-10-16 16:22:20.659587 Set item: Thermostat to value 18.0 on M4
2022-10-16 16:22:22.875949 Set item: OD to value 0.7 on M5
2022-10-16 16:22:33.223143Calibrated OD
2022-10-16 16:22:33.764817 Set item: Volume to value 20.0 on M5
2022-10-16 16:22:34.749370 Set item: Thermostat to value 18.0 on M5
2022-10-16 16:22:37.013237 Set item: OD to value 0.7 on M6
2022-10-16 16:22:43.893774Calibrated OD
2022-10-16 16:22:44.525995 Set item: Volume to value 20.0 on M6
2022-10-16 16:22:45.509497 Set item: Thermostat to value 18.0 on M6
2022-10-16 16:22:47.533664 Set item: OD to value 0.7 on M7
2022-10-16 16:23:02.454755Calibrated OD
2022-10-16 16:23:03.230525 Set item: Volume to value 20.0 on M7
2022-10-16 16:23:04.430826 Set item: Thermostat to value 18.0 on M7
2022-10-16 16:41:23.716822 Failed Pumps comms 1 times on device M2
2022-10-16 16:41:23.824672 Failed Pumps comms 2 times on device M2
2022-10-16 16:47:21.844693 Failed Pumps comms 1 times on device M1
2022-10-16 16:47:21.952686 Failed Pumps comms 2 times on device M1
2022-10-16 17:03:25.240653Failed to disconnect multiplexer on device M2
2022-10-16 17:12:27.584690 Failed Pumps comms 1 times on device M3
2022-10-16 17:12:27.692784 Failed Pumps comms 2 times on device M3
2022-10-16 17:40:22.432696 Failed Pumps comms 1 times on device M0
2022-10-16 17:40:22.540793 Failed Pumps comms 2 times on device M0
2022-10-16 18:33:30.044675Failed to disconnect multiplexer on device M2
2022-10-16 18:41:30.448713Failed to disconnect multiplexer on device M2
2022-10-16 19:00:31.680726 Failed Pumps comms 1 times on device M2
2022-10-16 19:00:31.788663 Failed Pumps comms 2 times on device M2
2022-10-16 19:08:34.108694 Failed Pumps comms 1 times on device M3
2022-10-16 19:08:34.216675 Failed Pumps comms 2 times on device M3
2022-10-16 19:09:36.620675Failed to disconnect multiplexer on device M4

[mpatterson]

As an update, I did just reclean all moisture sensors with alcohol and all USB and power cables and ports with compressed air, but I am still getting the errors.

[harrison]

Hi Mpatterson,

It is an error arising during the I2C digital communications, often due to (for example) the signal from the reactor to the pump having a lot of noise in it potentially leading to issues. If you have already cleaned the moisture sensors and cable ports there isn’t a whole lot you can do with it. Shorter micro-USB cables would help, as would taking the reactors away from any kind of massive electrical noise source if there was one nearby, though probably this is unlikely.
If it is still running happily enough I suppose let it go as-is. If it gets worse maybe try sourcing some different micro-usb cables and swap them out, perhaps the ones you have are from a dodgy manufacturing batch…

Harrison

[mpatterson]

Okay, we have everything in a cooling incubator so we can run at low temperature. Do you think that would be enough?

[harrison]

I am not sure I follow – i.e. are you proposing that low temperature will help the digital communications?

If it is just doing occasional errors I would leave it as is. If you think it is getting worse then I would consider replacing the microUSB cables as you may have bad ones.

[mpatterson]

No, sorry, I mean we have all of the reactors, control computer, and pumps inside of a Binder cooling incubator. I was asking whether the incubator itself would be enough of an electrical noise source to be causing these errors.

[harrison]

I wouldnt have thought so. If it is getting _worse_ over time it certainly seems unlikely to be the culprit.

[mpatterson]

Okay thanks, I will try replacing the cables.

[Author]

Posts