Failed multiplexer, got replacement from lab maker, error still persists

[wilsonh]

I keep getting Failed Multiplexer Comms and Failed to recover multiplexer on device M0 with my setup even when my bioreactor is not plugged into the T0 spot on the Beaglebone’s attached multiplexer board. I was getting this error before and contacted lab maker about it and they sent out a replacement multiplexer board. After swapping out the board I am having the same issue so I am concerned that it may be something wrong with the Beaglebone? I did the most recent software setup v2.0 and had no issues flashing the image from the SD card to the EMMC on the Beaglebone thinking that maybe the software had gotten messed up somehow? A long shot I know, but I am just lost as to what could be wrong. My only idea that I have left currently is that maybe some of the i2c functionality somehow is damaged on the Beaglebone so it is not communicating correctly with the multiplexer board… Another idea is that maybe the software is trying to communicate with the wrong i2c address, but I would have thought that would not be an issue especially with flashing this newest image to the Beaglebone. I have attached the terminal output for reference. Any help is greatly appreciated.

root@beaglebone:~/chibio# bash cb.sh 
[2021-04-05 15:01:45 +0000] [1883] [INFO] Starting gunicorn 20.1.0
[2021-04-05 15:01:45 +0000] [1883] [INFO] Listening at: http://192.168.7.2:5000 (1883)
[2021-04-05 15:01:45 +0000] [1883] [INFO] Using worker: sync
[2021-04-05 15:01:45 +0000] [1886] [INFO] Booting worker with pid: 1886
2021-04-05 15:01:53.071445 Starting watchdog
2021-04-05 15:01:55.549497 Initialising devices
2021-04-05 15:01:55.612228 Failed Multiplexer Comms 1 times
2021-04-05 15:01:55.668118 Failed Multiplexer Comms 2 times
2021-04-05 15:01:55.724195 Failed Multiplexer Comms 3 times
2021-04-05 15:01:55.759914Failed to recover multiplexer on device M0
2021-04-05 15:01:55.816174 Failed Multiplexer Comms 4 times
2021-04-05 15:01:55.852042Failed to recover multiplexer on device M0
2021-04-05 15:01:55.908153 Failed Multiplexer Comms 5 times
2021-04-05 15:01:55.943934Failed to recover multiplexer on device M0
2021-04-05 15:01:56.200067 Failed Multiplexer Comms 6 times
2021-04-05 15:01:56.235917Failed to recover multiplexer on device M0
2021-04-05 15:01:56.292202 Failed Multiplexer Comms 7 times
2021-04-05 15:01:56.327925Failed to recover multiplexer on device M0
2021-04-05 15:01:56.384143 Failed Multiplexer Comms 8 times
2021-04-05 15:01:56.419924Failed to recover multiplexer on device M0
2021-04-05 15:01:56.476127 Failed Multiplexer Comms 9 times
2021-04-05 15:01:56.511958Failed to recover multiplexer on device M0
2021-04-05 15:01:56.568142 Failed Multiplexer Comms 10 times
2021-04-05 15:01:56.603933Failed to recover multiplexer on device M0
2021-04-05 15:01:56.660142 Failed Multiplexer Comms 11 times
2021-04-05 15:01:56.695928Failed to recover multiplexer on device M0
2021-04-05 15:01:56.752219 Failed Multiplexer Comms 12 times
2021-04-05 15:01:56.787935Failed to recover multiplexer on device M0
2021-04-05 15:01:56.844124 Failed Multiplexer Comms 13 times
2021-04-05 15:01:56.879935Failed to recover multiplexer on device M0
2021-04-05 15:01:56.936134 Failed Multiplexer Comms 14 times
2021-04-05 15:01:56.971951Failed to recover multiplexer on device M0
2021-04-05 15:01:57.028155 Failed Multiplexer Comms 15 times
2021-04-05 15:01:57.063930Failed to recover multiplexer on device M0
2021-04-05 15:01:57.120151 Failed Multiplexer Comms 16 times
2021-04-05 15:01:57.155937Failed to recover multiplexer on device M0
2021-04-05 15:01:57.212122 Failed Multiplexer Comms 17 times
2021-04-05 15:01:57.247920Failed to recover multiplexer on device M0
2021-04-05 15:01:57.304155 Failed Multiplexer Comms 18 times
2021-04-05 15:01:57.339935Failed to recover multiplexer on device M0
2021-04-05 15:01:57.396151 Failed Multiplexer Comms 19 times
2021-04-05 15:01:57.431942Failed to recover multiplexer on device M0
2021-04-05 15:01:57.488151 Failed Multiplexer Comms 20 times
2021-04-05 15:01:57.523941Failed to recover multiplexer on device M0
2021-04-05 15:01:57.580126 Failed Multiplexer Comms 21 times
2021-04-05 15:01:57.615934Failed to recover multiplexer on device M0
2021-04-05 15:01:57.617266Failed to communicate to Multiplexer 10 times. Disabling hardware and software!
[2021-04-05 15:01:57 +0000] [1883] [INFO] Shutting down: Master
[2021-04-05 15:01:57 +0000] [1883] [INFO] Reason: App failed to load.
root@beaglebone:~/chibio# 

[wilsonh]

I also ran i2cdetect -l to see what i2c busses were present. Here is the output from running i2cdetect 0; i2cdetect 1; i2cdetect 2:

root@beaglebone:~/chibio# i2cdetect 0; i2cdetect 1; i2cdetect 2
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n] y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                                                 
10:                                                 
20:                                                 
30: -- -- -- -- 34 -- -- --                         
40:                                                 
50: UU -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60:                                                 
70:                                                 
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-1.
I will probe address range 0x03-0x77.
Continue? [Y/n] y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                                                 
10:                                                 
20:                                                 
30: -- -- -- -- -- -- -- --                         
40:                                                 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60:                                                 
70:                                                 
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-2.
I will probe address range 0x03-0x77.
Continue? [Y/n] y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                                                 
10:                                                 
20:                                                 
30: -- -- -- -- -- -- -- --                         
40:                                                 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60:                                                 
70:                                                 
root@beaglebone:~/chibio# 

[harrison]

Dear Wilson,

It seems unlikely it could be down to the software since as you said you tried several versions.
Perhaps the Beaglebone is at fault, but this does seem unlikely.
Do you have an oscilloscope by any chance? To test this the ideal method would be to look at the I2C bus under a scope and check that it is sending data as you expect.

When you run the test above is there any reactor connected? If yes – what happens if you disconnect all reactors from the beaglebone control board?

Harrison

[harrison]

I just saw the second post – it is suspicious that you can’t see the multiplexer on the I2Cdetect command.
Could you try running the I2cdetect commands again, and afterward execute the command “dmesg” in the linux terminal, and then copy here the bottom 10 lines? This will ideally tell us whether it is having I2C communication faults, or something stranger is happening…

[wilsonh]

I wrote a python script to switch the multiplexer to the correct bioreactor manually and then took apart the reactor and manually supplied 3.3V to where the regulator should have been giving it. After running i2cdetect -y -r 2 and I was able to see two other devices:

root@beaglebone:~# i2cdetect -y -r 2
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- 1b -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: 60 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: 70 -- -- -- 74 -- -- --

I am now just trying to figure out if the 0x60 and 0x70 I am seeing are valid, I’m thinking that the the 0x60 is the (PCA9685) PWM chip and I cannot figure out what the B1 is on the PCB, but maybe that is the other one? The zip on the website that contains the PCB files are all V1.0, while my bioreactor is V1.2. Would I be able to get the updated files in a .zip so that I could know what these parts are?

Also, a weird thing that I am running into is after attempting to run bash cb.sh and it failing the no I2C device works until I unplug the beagle bone and re-power it back up. Must be that when I CTRL-C some master device is holding onto the bus? Any ideas on why this might be?

[harrison]

Hi Wilson,

You say you provided 3.3V where it “should be giving it”. Does this mean you are trying to replicate the watchdog signal?

0x60 is indeed one of the PWM chips. If I recall correctly 0x70 is a second I2C address that is reserved by the pwm chip, but we don’t communicate with it directly.

0x1b is the thermometer on the beaglebone control board. You can see what every address corresponds to looking at lines 346-358 (and line 383 for the multiplexer) in the app.py file (current version on github: https://github.com/HarrisonSteel/ChiBio/blob/master/app.py)

I am not sure of a solution to your last point. This is not something I have seen or heard of happening before. I must say at this point it seems you are chasing after some tricky hardware issues – which ultimately should not be solved by Labmaker.

I’ll upload the latest version of the schematics to the site now…

[diegort]

Hello,

I’m having exactly the same issue. I just upgraded to version 2 recently and did not have the problem before. Basically after a few hours the following error shows up. I tried different reactors and tried connecting at different position in the board already. After this error, a restart of the beaglebone is necessary to connect succesfully again to the reactors.

root@beaglebone:~/chibio# bash cb.sh
[2021-05-25 23:44:29 +0000] [1808] [INFO] Starting gunicorn 20.1.0
[2021-05-25 23:44:29 +0000] [1808] [INFO] Listening at: http://192.168.19.146:50                                                                             00 (1808)
[2021-05-25 23:44:29 +0000] [1808] [INFO] Using worker: sync
[2021-05-25 23:44:29 +0000] [1811] [INFO] Booting worker with pid: 1811
2021-05-25 23:44:37.407105 Starting watchdog
2021-05-25 23:44:39.888825 Initialising devices
2021-05-25 23:44:40.128411 Initialised M4, Device ID: 13328179951295245632
2021-05-25 23:44:40.730010 Initialised M5, Device ID: 20492179931295245632
2021-05-25 23:44:41.330431 Initialised M6, Device ID: 36877182841295245632
2021-05-25 23:44:41.932494 Initialised M7, Device ID: 30731182901295245632
2021-05-25 23:44:42.408232 Start Up Complete
2021-05-25 23:45:13.716769 Set item: OD to value 0.3 on M4
2021-05-25 23:45:28.547329Calibrated OD
2021-05-25 23:45:48.433007 Set item: OD to value 0.3 on M5
2021-05-25 23:45:50.344274Calibrated OD
2021-05-25 23:46:11.277548 Set item: Pump2 to value 1.0 on M6
2021-05-25 23:46:42.458999 Set item: Pump1 to value 0.3 on M6
2021-05-25 23:47:24.757056Calibrated OD
2021-05-25 23:47:26.238169 Set item: OD to value 0.3 on M6
2021-05-25 23:48:04.022843Calibrated OD
2021-05-25 23:48:07.219585 Set item: OD to value 0.3 on M7
2021-05-25 23:52:42.713755 OD Measurement exception on LASER650
2021-05-25 23:52:43.719083 OD Measurement exception on LASER650
2021-05-26 03:50:51.686861 Failed Multiplexer Comms 1 times
2021-05-26 03:50:51.708952 Failed Multiplexer Comms 2 times
2021-05-26 03:50:51.730926 Failed Multiplexer Comms 3 times
2021-05-26 03:50:51.732701Failed to recover multiplexer on device M6
2021-05-26 03:50:51.754886 Failed Multiplexer Comms 4 times
2021-05-26 03:50:51.756565Failed to recover multiplexer on device M6
2021-05-26 03:50:51.778473 Failed Multiplexer Comms 5 times
2021-05-26 03:50:51.780180Failed to recover multiplexer on device M6
2021-05-26 03:50:52.002358 Failed Multiplexer Comms 6 times
2021-05-26 03:50:52.004130Failed to recover multiplexer on device M6
2021-05-26 03:50:52.026185 Failed Multiplexer Comms 7 times
2021-05-26 03:50:52.027877Failed to recover multiplexer on device M6
2021-05-26 03:50:52.049914 Failed Multiplexer Comms 8 times
2021-05-26 03:50:52.051628Failed to recover multiplexer on device M6
2021-05-26 03:50:52.073609 Failed Multiplexer Comms 9 times
2021-05-26 03:50:52.075300Failed to recover multiplexer on device M6
2021-05-26 03:50:52.097247 Failed Multiplexer Comms 10 times
2021-05-26 03:50:52.099008Failed to recover multiplexer on device M6
2021-05-26 03:50:52.120956 Failed Multiplexer Comms 11 times
2021-05-26 03:50:52.122974Failed to recover multiplexer on device M6
2021-05-26 03:50:52.144971 Failed Multiplexer Comms 12 times
2021-05-26 03:50:52.146805Failed to recover multiplexer on device M6
2021-05-26 03:50:52.168780 Failed Multiplexer Comms 13 times
2021-05-26 03:50:52.170637Failed to recover multiplexer on device M6
2021-05-26 03:50:52.192613 Failed Multiplexer Comms 14 times
2021-05-26 03:50:52.194457Failed to recover multiplexer on device M6
2021-05-26 03:50:52.216590 Failed Multiplexer Comms 15 times
2021-05-26 03:50:52.218547Failed to recover multiplexer on device M6
2021-05-26 03:50:52.240536 Failed Multiplexer Comms 16 times
2021-05-26 03:50:52.242353Failed to recover multiplexer on device M6
2021-05-26 03:50:52.264334 Failed Multiplexer Comms 17 times
2021-05-26 03:50:52.266102Failed to recover multiplexer on device M6
2021-05-26 03:50:52.288144 Failed Multiplexer Comms 18 times
2021-05-26 03:50:52.289783Failed to recover multiplexer on device M6
2021-05-26 03:50:52.311702 Failed Multiplexer Comms 19 times
2021-05-26 03:50:52.313394Failed to recover multiplexer on device M6
2021-05-26 03:50:52.335340 Failed Multiplexer Comms 20 times
2021-05-26 03:50:52.337095Failed to recover multiplexer on device M6
2021-05-26 03:50:52.359108 Failed Multiplexer Comms 21 times
2021-05-26 03:50:52.360792Failed to recover multiplexer on device M6
2021-05-26 03:50:52.362062Failed to communicate to Multiplexer 10 times. Disabli                                                                             ng hardware and software!
[2021-05-26 03:50:52 +0000] [1808] [INFO] Shutting down: Master
[2021-05-26 03:50:52 +0000] [1808] [INFO] Reason: App failed to load.
root@beaglebone:~/chibio# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n] Y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:
10:
20:
30: -- -- -- -- 34 -- -- --
40:
50: UU -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60:
70:
root@beaglebone:~/chibio# i2cdetect 1
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-1.
I will probe address range 0x03-0x77.
Continue? [Y/n] Y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:
10:
20:
30: -- -- -- -- -- -- -- --
40:
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60:
70:
root@beaglebone:~/chibio# i2cdetect 2
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-2.
I will probe address range 0x03-0x77.
Continue? [Y/n] Y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:
10:
20:
30: -- -- -- -- -- -- -- --
40:
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60:
70:

[harrison]

Dear Diego,

Thank you for raising this. I’ve taken a look at trying to come up with additional fixes. Consequently, I have pushed a new version of app.py to the GitHub here:
https://github.com/HarrisonSteel/ChiBio/blob/master/app.py
Could you download this and try to run it on your device? It has several small changes that hopefully fix this issue (though it is difficult for me to diagnose it precisely).

That said, you are the first person I have seen have any new issue introduced after updating to the V2.0 software. Are these multiplexer-induced crashes only appearing after this upgrade?
Also, I am interested in what you said that “a restart of the beaglebone” is necessary to fix the problem. Does this mean that simply restarting the Chi.Bio software (i.e. with bash cb.sh) does not fix it? How about if you restart the chi.bio code (but leave the beaglebone on) and also power cycle the reactors? It seems very strange that there is a problem that requires the beaglebone to be power cycled, since this implies it may be somehow an issue with the underlying operating system (which should be pretty robust…). Out of interest what Control computer PCB version do you have (is it V1.0, V1.1, or V1.2?)

[Author]

Posts