App failed to load. Not sure if this is a software or a hardware problem.

[yzhang]

Ended the previous session due to the OD value suddenly jumped to twice the set value after the run was initiated. But could not establish communication with the chi bio reactor since. Attempts to establish communication using Putty consistently gave the following error message:

Failed to recover multiplexer on device M2
Failed to communicate to Multiplexer 10 times. Disabling hardware and software!
Shutting down: Master
Reason: App failed to load

Any idea how to solve this issue? Thank you!

Best,
Yi

[harrison]

Hi Yi,

The most likely explanation is that there is some moisture or residue on the sensing tracks (the closely spaced silver lines) at the top of the reactor. These are designed to sense if there is any overflow or other issue with the pumping – but if you accidentally get some liquid on them it can cause you problems. I’d disconnect the reactor from the power/connections and then carefully wipe down the top with alcohol (even if it LOOKS clean), then give it another go.

[yzhang]

Thank you for your suggestions. I tried to use ethanol to clean the sensing track and the rest of the reactor chamber carefully, but the problem persisted. It gave the same error message when I tried to establish communication. Any other suggestions? I reinstalled the latest image for the beagle board but it did not help either. Is there we could know whether this is a software problem or a hardware problem? Thank you!

Best,
Yi

• This reply was modified 3 years, 5 months ago by yzhang.

[harrison]

In that case it seems highly likely the problem is hardware related. If the software worked _once_ it should work again.
From what you said – it was working at one point, right? Did anything else happen that might have caused damage to it? For example liquid spill, or knocking it over, or breaking off one of the micro-USB connectors?
I assume you have only one reactor and therefore couldn’t try plugging a different reactor into your control computer to see if you can identify the issue?

[harrison]

Another check – have you tried booting the software with no reactor plugged in?

[yzhang]

Nothing out of the ordinary happened. The reactor was running OK for 36h but the OD sensor started to become unstable. It eventually threw off the automatic OD maintenance algorithm due to the instability. Once this sudden jump of the OD was observed, we had to stopped the program and the error occurred after trying to reestablish communication.
Yes, I have only one reactor but planning on getting more if we could get it to work. Any suggestions to fix the problem? Do we have to need to send the reactor back to lab maker to get it fixed? or can we borrow one from the lab maker while we are trying to figure out the problem? Thank you!

[yzhang]

Tried that and it seemed that we can still communicate with the beagle board using PC. But still the same error message.

[andrewletten]

Hi Harrison,

We are getting the same error unfortunately (Failed to recover multiplexer on device M0). This is our first time trying to run Chi.Bio so unclear if it’s a software or hardware issue. I’ve written to Lab Maker to see what they suggest but wanted to check what is the expected difference if booting the software with the reactor plugged in or not (as suggested above)? As far as I can see in our case it doesn’t affect the error message. Note that we have tried switching between different reactors, pumps, cables and power supplies (and reinstalling the software).

Many thanks for any suggestions.

Andrew

[2021-04-05 15:03:03 +0000] [1914] [INFO] Starting gunicorn 20.1.0
[2021-04-05 15:03:03 +0000] [1914] [INFO] Listening at: http://192.168.7.2:5000 (1914)
[2021-04-05 15:03:03 +0000] [1914] [INFO] Using worker: sync
[2021-04-05 15:03:03 +0000] [1917] [INFO] Booting worker with pid: 1917
2021-04-05 15:03:11.321165 Starting watchdog
2021-04-05 15:03:13.799128 Initialising devices
2021-04-05 15:03:13.861102 Failed Multiplexer Comms 1 times
2021-04-05 15:03:13.917096 Failed Multiplexer Comms 2 times
2021-04-05 15:03:13.973270 Failed Multiplexer Comms 3 times
2021-04-05 15:03:14.008920Failed to recover multiplexer on device M0
2021-04-05 15:03:14.065179 Failed Multiplexer Comms 4 times
2021-04-05 15:03:14.100981Failed to recover multiplexer on device M0
2021-04-05 15:03:14.157125 Failed Multiplexer Comms 5 times
2021-04-05 15:03:14.192934Failed to recover multiplexer on device M0
2021-04-05 15:03:14.449065 Failed Multiplexer Comms 6 times
2021-04-05 15:03:14.484918Failed to recover multiplexer on device M0
2021-04-05 15:03:14.541123 Failed Multiplexer Comms 7 times
2021-04-05 15:03:14.576912Failed to recover multiplexer on device M0
2021-04-05 15:03:14.633112 Failed Multiplexer Comms 8 times
2021-04-05 15:03:14.668940Failed to recover multiplexer on device M0
2021-04-05 15:03:14.725143 Failed Multiplexer Comms 9 times
2021-04-05 15:03:14.760920Failed to recover multiplexer on device M0
2021-04-05 15:03:14.817120 Failed Multiplexer Comms 10 times
2021-04-05 15:03:14.852978Failed to recover multiplexer on device M0
2021-04-05 15:03:14.909172 Failed Multiplexer Comms 11 times
2021-04-05 15:03:14.944919Failed to recover multiplexer on device M0
2021-04-05 15:03:15.001160 Failed Multiplexer Comms 12 times
2021-04-05 15:03:15.036936Failed to recover multiplexer on device M0
2021-04-05 15:03:15.093116 Failed Multiplexer Comms 13 times
2021-04-05 15:03:15.128928Failed to recover multiplexer on device M0
2021-04-05 15:03:15.185164 Failed Multiplexer Comms 14 times
2021-04-05 15:03:15.220922Failed to recover multiplexer on device M0
2021-04-05 15:03:15.277138 Failed Multiplexer Comms 15 times
2021-04-05 15:03:15.312930Failed to recover multiplexer on device M0
2021-04-05 15:03:15.369159 Failed Multiplexer Comms 16 times
2021-04-05 15:03:15.404920Failed to recover multiplexer on device M0
2021-04-05 15:03:15.461102 Failed Multiplexer Comms 17 times
2021-04-05 15:03:15.496921Failed to recover multiplexer on device M0
2021-04-05 15:03:15.553128 Failed Multiplexer Comms 18 times
2021-04-05 15:03:15.588916Failed to recover multiplexer on device M0
2021-04-05 15:03:15.645136 Failed Multiplexer Comms 19 times
2021-04-05 15:03:15.680922Failed to recover multiplexer on device M0
2021-04-05 15:03:15.737153 Failed Multiplexer Comms 20 times
2021-04-05 15:03:15.772919Failed to recover multiplexer on device M0
2021-04-05 15:03:15.829137 Failed Multiplexer Comms 21 times
2021-04-05 15:03:15.864924Failed to recover multiplexer on device M0
2021-04-05 15:03:15.866208Failed to communicate to Multiplexer 10 times. Disabling hardware and software!
[2021-04-05 15:03:15 +0000] [1914] [INFO] Shutting down: Master
[2021-04-05 15:03:15 +0000] [1914] [INFO] Reason: App failed to load.

[harrison]

Dear YZhang
Can you be a bit more specific with regard to your response:
“Tried that and it seemed that we can still communicate with the beagle board using PC. But still the same error message.”
Does this mean that you:
1. Unplugged reactor from control board
2. At that point you could communicate with the beaglebone itself
3. But, you could NOT start the Chi.Bio software (even though no reactor was in place?)

As to the reactor – Labmaker should be able to send you a new one which you can then use (i.e. don’t have to wait for the existing one to be fixed).

• This reply was modified 3 years, 5 months ago by harrison.

[harrison]

Hi Andrew,
I see, that is annoying…
So you are finding that even if you have no reactor plugged in to the control board it fails to start, and you see the errors above?

The “correct” booting sequence is usually to have the reactors + pumps connected and powered on (i.e. with wall power supplies) when you start up the device. However, I was asking YZhang to try starting the code without anything connected as this would test if the problem is in the reactor, or in the control board.

If you are finding that the software doesn’t start even when there is nothing connected (i.e. no reactors) to the control computer it is indicative that the control PCB might have a fault.

[andrewletten]

Hi Harrison,

Yep, that’s right, we see those errors irrespective of whether the reactors are plugged into the control board or not. Hopefully Lab Marker can send us a new control PCB.

Thanks for your help!
Andrew

[harrison]

Morning,
Yes – if it is failing without anything plugged in to the control board it is indicative of the control board being faulty. (presuming you have the latest software image installed – did you do this?)

Harrison

[andrewletten]

Yep, we have the 2021-06-18 version installed. The good news is that Labmaker have just emailed to say they are going to send a replacement PCB.
Best,
Andrew

[yzhang]

Hi Harrison!

Thank you for your suggestions!
I tried to establish communication while unplugging the reactor from control board. This is what I saw. It seems to be working but once I plugged in the reactor, the same error message showed up again.
So I guess the problem originated from the reactor? Lab Maker rep said they will fix the problem after we send it back. Can I requested a replacement for the reactor while the old one is being fixed?

login as: root
Pre-authentication banner message from server:
| Debian GNU/Linux 10
|
| BeagleBoard.org Debian Buster IoT Image 2020-08-25
|
| Support: https://bbb.io/debian
|
| default username:password is [debian:temppwd]
|
End of banner message from server
root@192.168.7.2’s password:

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Jun 6 14:36:50 2021 from 192.168.7.1
root@beaglebone:~# ls
chibio
root@beaglebone:~# cd chibio
root@beaglebone:~/chibio# bash cb.sh
[2021-07-06 14:42:34 +0000] [3291] [INFO] Starting gunicorn 20.1.0
[2021-07-06 14:42:34 +0000] [3291] [INFO] Listening at: http://192.168.7.2:5000 (3291)
[2021-07-06 14:42:34 +0000] [3291] [INFO] Using worker: sync
[2021-07-06 14:42:34 +0000] [3294] [INFO] Booting worker with pid: 3294
2021-07-06 14:42:41.886960 Starting watchdog
2021-07-06 14:42:44.369160 Initialising devices
2021-07-06 14:42:44.463984 Start Up Complete

[harrison]

Hi Yzhang,

Yes – if the software does start without the reactor connected it seems the reactor is most likely the source of issues. As I said the most common cause of such a problem is moisture (even just dried liquid somewhere hiding) on the moisture sensing tracks – but if not that then it may well be a fault in the hardware that they (Labmaker) will have to fix.

I think Labmaker will hopefully be able to send you a replacement device rather than spending so long fixing it, you will have to ask them – I am not involved in their business or manufacturing process.

[Author]

Posts