Failed to communicate to Multiplexer 20 times

[a]

I get the following error when I run bash cb.sh. Please help.

root@beaglebone:~# cd chibio
root@beaglebone:~/chibio# bash cb.sh
[2021-04-05 19:20:58 +0000] [1340] [INFO] Starting gunicorn 20.1.0
[2021-04-05 19:20:58 +0000] [1340] [INFO] Listening at: http://192.168.7.2:5000 (1340)
[2021-04-05 19:20:58 +0000] [1340] [INFO] Using worker: sync
[2021-04-05 19:20:58 +0000] [1555] [INFO] Booting worker with pid: 1555
2021-04-05 19:21:19.715912 Starting watchdog
2021-04-05 19:21:22.204815 Initialising devices
2021-04-05 19:21:22.306181 Failed Multiplexer Comms 1 times
2021-04-05 19:21:22.362156 Failed Multiplexer Comms 2 times
2021-04-05 19:21:22.418782 Failed Multiplexer Comms 3 times
2021-04-05 19:21:22.453957Failed to recover multiplexer on device M0
2021-04-05 19:21:22.510171 Failed Multiplexer Comms 4 times
2021-04-05 19:21:22.545952Failed to recover multiplexer on device M0
2021-04-05 19:21:22.602287 Failed Multiplexer Comms 5 times
2021-04-05 19:21:22.637946Failed to recover multiplexer on device M0
2021-04-05 19:21:22.946126 Failed Multiplexer Comms 6 times
2021-04-05 19:21:22.982080Failed to recover multiplexer on device M0
2021-04-05 19:21:23.038139 Failed Multiplexer Comms 7 times
2021-04-05 19:21:23.073954Failed to recover multiplexer on device M0
2021-04-05 19:21:23.130173 Failed Multiplexer Comms 8 times
2021-04-05 19:21:23.166015Failed to recover multiplexer on device M0
2021-04-05 19:21:23.222233 Failed Multiplexer Comms 9 times
2021-04-05 19:21:23.258013Failed to recover multiplexer on device M0
2021-04-05 19:21:23.314232 Failed Multiplexer Comms 10 times
2021-04-05 19:21:23.349960Failed to recover multiplexer on device M0
2021-04-05 19:21:23.658210 Failed Multiplexer Comms 11 times
2021-04-05 19:21:23.693994Failed to recover multiplexer on device M0
2021-04-05 19:21:23.750163 Failed Multiplexer Comms 12 times
2021-04-05 19:21:23.785946Failed to recover multiplexer on device M0
2021-04-05 19:21:23.842276 Failed Multiplexer Comms 13 times
2021-04-05 19:21:23.877956Failed to recover multiplexer on device M0
2021-04-05 19:21:23.934176 Failed Multiplexer Comms 14 times
2021-04-05 19:21:23.969962Failed to recover multiplexer on device M0
2021-04-05 19:21:24.026226 Failed Multiplexer Comms 15 times
2021-04-05 19:21:24.062050Failed to recover multiplexer on device M0
2021-04-05 19:21:24.370132 Failed Multiplexer Comms 16 times
2021-04-05 19:21:24.406074Failed to recover multiplexer on device M0
2021-04-05 19:21:24.462206 Failed Multiplexer Comms 17 times
2021-04-05 19:21:24.497943Failed to recover multiplexer on device M0
2021-04-05 19:21:24.554173 Failed Multiplexer Comms 18 times
2021-04-05 19:21:24.589960Failed to recover multiplexer on device M0
2021-04-05 19:21:24.646151 Failed Multiplexer Comms 19 times
2021-04-05 19:21:24.681978Failed to recover multiplexer on device M0
2021-04-05 19:21:24.738182 Failed Multiplexer Comms 20 times
2021-04-05 19:21:24.773954Failed to recover multiplexer on device M0
2021-04-05 19:21:24.830282 Failed Multiplexer Comms 21 times
2021-04-05 19:21:24.865998Failed to recover multiplexer on device M0
2021-04-05 19:21:24.867547Failed to communicate to Multiplexer 20 times. Disabling hardware and software!
[2021-04-05 19:21:24 +0000] [1340] [INFO] Shutting down: Master
[2021-04-05 19:21:24 +0000] [1340] [INFO] Reason: App failed to load.

[a]

I think I2C has something to do with it.

[harrison]

Yes, most likely is the moisture sensor track was tripped by then wet (or now dry) residue. I’d suggest cleaning with ethanol and a q-tip. Look at the troubleshooting guide in the manual for more info.

[a]

Thanks for the reply!
Is the moisture sensor track a device on the main reactor side?
I’m going back to the university tomorrow and will try to clean it up as advised.
I get a multiplexer error both when the Control Computer is connected to the main reactor with a cable and when it is not.
Is it possible that the moisture sensor track trip is also the cause of this error when the Control Computer and Main reactor are not connected by cable?
If there are other factors that could be causing this, please let me know.

(This error occurred last Thursday while I was doing an experiment so I thought it could be caused by a trip.)

[harrison]

If it happens when no reactor is connected then it implies it might be a fault in thhe control computer. If that is the case I suggest contacting Labmaker as it may be defective and they should be able to fix it.

[a]

I rewrote part of the Python code as follows and ran it and got the following error output. I am suspicious about the “device or resource busy” part, but has this ever happened before? Or is it still caused by a device failure?

#We now connect the multiplexer to the appropriate device to allow digital communications.
tries=0
while(tries!=-1):
try:
print(“Attempting Multiplexer communication…”)
sysItems[‘Multiplexer’][‘device’].write8(int(0x00),int(sysItems[‘Multiplexer’][M],2)) #We have established connection to correct device.
print(“Multiplexer communication successful.”)
check=(sysItems[‘Multiplexer’][‘device’].readRaw8()) #We check that the Multiplexer is indeed connected to the correct channel.

if(check==int(sysItems[‘Multiplexer’][M],2)):
tries=-1
else:
tries=tries+1
time.sleep(0.02)
print(str(datetime.now()) + ‘ Multiplexer didnt switch ‘ + str(tries) + ” times on ” + str(M))

except Exception as e:
# 例外が発生した場合のデバッグログ
print(f”Exception occurred: {e}”)
tries = tries + 1
time.sleep(0.02)
print(str(datetime.now()) + f’ Failed Multiplexer Comms {tries} times’)

# except: #If there is an error in the above.
# tries=tries+1
# time.sleep(0.02)
# print(str(datetime.now()) + ‘ Failed Multiplexer Comms ‘ + str(tries) + ” times”)
if (tries>2):
try:
sysItems[‘Multiplexer’][‘device’].write8(int(0x00),int(0x00)) #Disconnect multiplexer.
print(str(datetime.now()) + ‘Disconnected multiplexer on ‘ + str(M) + ‘, trying to connect again.’)
except:
print(str(datetime.now()) + ‘Failed to recover multiplexer on device ‘ + str(M))
if (tries==5 or tries==10 or tries==15):
toggleWatchdog() #Flip the watchdog pin to ensure it is working.
GPIO.output(‘P8_15’, GPIO.LOW) #Flip the Multiplexer RESET pin. Note this reset function works on Control Board V1.2 and later.
time.sleep(0.1)
GPIO.output(‘P8_15’, GPIO.HIGH)
time.sleep(0.1)
print(str(datetime.now()) + ‘Did multiplexer hard-reset on ‘ + str(M))

if tries>20: #If it has failed a number of times then likely something is seriously wrong, so we crash the software.
sysItems[‘Watchdog’][‘ON’]=0 #Basically this will crash all the electronics and the software.
out=0
print(str(datetime.now()) + ‘Failed to communicate to Multiplexer 20 times. Disabling hardware and software!’)
tries=-1
os._exit(4)

root@beaglebone:~/chibio# bash cb.sh
[2021-04-05 23:28:34 +0000] [2540] [INFO] Starting gunicorn 20.1.0
[2021-04-05 23:28:34 +0000] [2540] [INFO] Listening at: http://192.168.7.2:5000 (2540)
[2021-04-05 23:28:34 +0000] [2540] [INFO] Using worker: sync
[2021-04-05 23:28:34 +0000] [2543] [INFO] Booting worker with pid: 2543
2021-04-05 23:28:41.544312 Starting watchdog
2021-04-05 23:28:44.028674 Initialising devices
Attempting Multiplexer communication…
Exception occurred: [Errno 110] Connection timed out
2021-04-05 23:28:44.133983 Failed Multiplexer Comms 1 times
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:44.194260 Failed Multiplexer Comms 2 times
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:44.253813 Failed Multiplexer Comms 3 times
2021-04-05 23:28:44.288253Failed to recover multiplexer on device M0
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:44.345900 Failed Multiplexer Comms 4 times
2021-04-05 23:28:44.380247Failed to recover multiplexer on device M0
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:44.437912 Failed Multiplexer Comms 5 times
2021-04-05 23:28:44.472268Failed to recover multiplexer on device M0
2021-04-05 23:28:44.725255Did multiplexer hard-reset on M0
Attempting Multiplexer communication…
Exception occurred: [Errno 110] Connection timed out
2021-04-05 23:28:44.826896 Failed Multiplexer Comms 6 times
2021-04-05 23:28:44.864357Failed to recover multiplexer on device M0
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:44.921920 Failed Multiplexer Comms 7 times
2021-04-05 23:28:44.956249Failed to recover multiplexer on device M0
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:45.013899 Failed Multiplexer Comms 8 times
2021-04-05 23:28:45.048260Failed to recover multiplexer on device M0
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:45.105883 Failed Multiplexer Comms 9 times
2021-04-05 23:28:45.140260Failed to recover multiplexer on device M0
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:45.197740 Failed Multiplexer Comms 10 times
2021-04-05 23:28:45.232284Failed to recover multiplexer on device M0
2021-04-05 23:28:45.485275Did multiplexer hard-reset on M0
Attempting Multiplexer communication…
Exception occurred: [Errno 110] Connection timed out
2021-04-05 23:28:45.585974 Failed Multiplexer Comms 11 times
2021-04-05 23:28:45.620403Failed to recover multiplexer on device M0
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:45.677917 Failed Multiplexer Comms 12 times
2021-04-05 23:28:45.712259Failed to recover multiplexer on device M0
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:45.769905 Failed Multiplexer Comms 13 times
2021-04-05 23:28:45.804273Failed to recover multiplexer on device M0
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:45.861920 Failed Multiplexer Comms 14 times
2021-04-05 23:28:45.896267Failed to recover multiplexer on device M0
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:45.953904 Failed Multiplexer Comms 15 times
2021-04-05 23:28:45.988262Failed to recover multiplexer on device M0
2021-04-05 23:28:46.241132Did multiplexer hard-reset on M0
Attempting Multiplexer communication…
Exception occurred: [Errno 110] Connection timed out
2021-04-05 23:28:46.337937 Failed Multiplexer Comms 16 times
2021-04-05 23:28:46.372237Failed to recover multiplexer on device M0
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:46.430300 Failed Multiplexer Comms 17 times
2021-04-05 23:28:46.468259Failed to recover multiplexer on device M0
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:46.525957 Failed Multiplexer Comms 18 times
2021-04-05 23:28:46.560253Failed to recover multiplexer on device M0
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:46.617894 Failed Multiplexer Comms 19 times
2021-04-05 23:28:46.652281Failed to recover multiplexer on device M0
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:46.709793 Failed Multiplexer Comms 20 times
2021-04-05 23:28:46.744330Failed to recover multiplexer on device M0
Attempting Multiplexer communication…
Exception occurred: [Errno 16] Device or resource busy
2021-04-05 23:28:46.801909 Failed Multiplexer Comms 21 times
2021-04-05 23:28:46.836761Failed to recover multiplexer on device M0
2021-04-05 23:28:46.838319Failed to communicate to Multiplexer 20 times. Disabling hardware and software!
[2021-04-05 23:28:46 +0000] [2540] [INFO] Shutting down: Master
[2021-04-05 23:28:46 +0000] [2540] [INFO] Reason: App failed to load.

[harrison]

I am not sure – I expect if you have rewritten it it might have caused other issues I haven’t seen before. If the system doesn’t start with zero reactors plugged in then it very much sounds like a hardware fault.

[giacomo]

We had the same error. Did you manage to solve the issue by contacting Labmaker? @a

[Author]

Posts