Home Forums Software App failed to load. Not sure if this is a software or a hardware problem.

Viewing 15 posts - 1 through 15 (of 26 total)
  • Author
    Posts
  • #1267
    harrison
    Keymaster

    Hi Yi,

    The most likely explanation is that there is some moisture or residue on the sensing tracks (the closely spaced silver lines) at the top of the reactor. These are designed to sense if there is any overflow or other issue with the pumping – but if you accidentally get some liquid on them it can cause you problems. I’d disconnect the reactor from the power/connections and then carefully wipe down the top with alcohol (even if it LOOKS clean), then give it another go.

    #1268
    yzhang
    Participant

    Thank you for your suggestions. I tried to use ethanol to clean the sensing track and the rest of the reactor chamber carefully, but the problem persisted. It gave the same error message when I tried to establish communication. Any other suggestions? I reinstalled the latest image for the beagle board but it did not help either. Is there we could know whether this is a software problem or a hardware problem? Thank you!

    Best,
    Yi

    • This reply was modified 3 years, 9 months ago by yzhang.
    #1270
    harrison
    Keymaster

    In that case it seems highly likely the problem is hardware related. If the software worked _once_ it should work again.
    From what you said – it was working at one point, right? Did anything else happen that might have caused damage to it? For example liquid spill, or knocking it over, or breaking off one of the micro-USB connectors?
    I assume you have only one reactor and therefore couldn’t try plugging a different reactor into your control computer to see if you can identify the issue?

    #1271
    harrison
    Keymaster

    Another check – have you tried booting the software with no reactor plugged in?

    #1272
    yzhang
    Participant

    Nothing out of the ordinary happened. The reactor was running OK for 36h but the OD sensor started to become unstable. It eventually threw off the automatic OD maintenance algorithm due to the instability. Once this sudden jump of the OD was observed, we had to stopped the program and the error occurred after trying to reestablish communication.
    Yes, I have only one reactor but planning on getting more if we could get it to work. Any suggestions to fix the problem? Do we have to need to send the reactor back to lab maker to get it fixed? or can we borrow one from the lab maker while we are trying to figure out the problem? Thank you!

    #1273
    yzhang
    Participant

    Tried that and it seemed that we can still communicate with the beagle board using PC. But still the same error message.

    #1274
    andrewletten
    Participant

    Hi Harrison,

    We are getting the same error unfortunately (Failed to recover multiplexer on device M0). This is our first time trying to run Chi.Bio so unclear if it’s a software or hardware issue. I’ve written to Lab Maker to see what they suggest but wanted to check what is the expected difference if booting the software with the reactor plugged in or not (as suggested above)? As far as I can see in our case it doesn’t affect the error message. Note that we have tried switching between different reactors, pumps, cables and power supplies (and reinstalling the software).

    Many thanks for any suggestions.

    Andrew

    [2021-04-05 15:03:03 +0000] [1914] [INFO] Starting gunicorn 20.1.0
    [2021-04-05 15:03:03 +0000] [1914] [INFO] Listening at: http://192.168.7.2:5000 (1914)
    [2021-04-05 15:03:03 +0000] [1914] [INFO] Using worker: sync
    [2021-04-05 15:03:03 +0000] [1917] [INFO] Booting worker with pid: 1917
    2021-04-05 15:03:11.321165 Starting watchdog
    2021-04-05 15:03:13.799128 Initialising devices
    2021-04-05 15:03:13.861102 Failed Multiplexer Comms 1 times
    2021-04-05 15:03:13.917096 Failed Multiplexer Comms 2 times
    2021-04-05 15:03:13.973270 Failed Multiplexer Comms 3 times
    2021-04-05 15:03:14.008920Failed to recover multiplexer on device M0
    2021-04-05 15:03:14.065179 Failed Multiplexer Comms 4 times
    2021-04-05 15:03:14.100981Failed to recover multiplexer on device M0
    2021-04-05 15:03:14.157125 Failed Multiplexer Comms 5 times
    2021-04-05 15:03:14.192934Failed to recover multiplexer on device M0
    2021-04-05 15:03:14.449065 Failed Multiplexer Comms 6 times
    2021-04-05 15:03:14.484918Failed to recover multiplexer on device M0
    2021-04-05 15:03:14.541123 Failed Multiplexer Comms 7 times
    2021-04-05 15:03:14.576912Failed to recover multiplexer on device M0
    2021-04-05 15:03:14.633112 Failed Multiplexer Comms 8 times
    2021-04-05 15:03:14.668940Failed to recover multiplexer on device M0
    2021-04-05 15:03:14.725143 Failed Multiplexer Comms 9 times
    2021-04-05 15:03:14.760920Failed to recover multiplexer on device M0
    2021-04-05 15:03:14.817120 Failed Multiplexer Comms 10 times
    2021-04-05 15:03:14.852978Failed to recover multiplexer on device M0
    2021-04-05 15:03:14.909172 Failed Multiplexer Comms 11 times
    2021-04-05 15:03:14.944919Failed to recover multiplexer on device M0
    2021-04-05 15:03:15.001160 Failed Multiplexer Comms 12 times
    2021-04-05 15:03:15.036936Failed to recover multiplexer on device M0
    2021-04-05 15:03:15.093116 Failed Multiplexer Comms 13 times
    2021-04-05 15:03:15.128928Failed to recover multiplexer on device M0
    2021-04-05 15:03:15.185164 Failed Multiplexer Comms 14 times
    2021-04-05 15:03:15.220922Failed to recover multiplexer on device M0
    2021-04-05 15:03:15.277138 Failed Multiplexer Comms 15 times
    2021-04-05 15:03:15.312930Failed to recover multiplexer on device M0
    2021-04-05 15:03:15.369159 Failed Multiplexer Comms 16 times
    2021-04-05 15:03:15.404920Failed to recover multiplexer on device M0
    2021-04-05 15:03:15.461102 Failed Multiplexer Comms 17 times
    2021-04-05 15:03:15.496921Failed to recover multiplexer on device M0
    2021-04-05 15:03:15.553128 Failed Multiplexer Comms 18 times
    2021-04-05 15:03:15.588916Failed to recover multiplexer on device M0
    2021-04-05 15:03:15.645136 Failed Multiplexer Comms 19 times
    2021-04-05 15:03:15.680922Failed to recover multiplexer on device M0
    2021-04-05 15:03:15.737153 Failed Multiplexer Comms 20 times
    2021-04-05 15:03:15.772919Failed to recover multiplexer on device M0
    2021-04-05 15:03:15.829137 Failed Multiplexer Comms 21 times
    2021-04-05 15:03:15.864924Failed to recover multiplexer on device M0
    2021-04-05 15:03:15.866208Failed to communicate to Multiplexer 10 times. Disabling hardware and software!
    [2021-04-05 15:03:15 +0000] [1914] [INFO] Shutting down: Master
    [2021-04-05 15:03:15 +0000] [1914] [INFO] Reason: App failed to load.

    #1275
    harrison
    Keymaster

    Dear YZhang
    Can you be a bit more specific with regard to your response:
    “Tried that and it seemed that we can still communicate with the beagle board using PC. But still the same error message.”
    Does this mean that you:
    1. Unplugged reactor from control board
    2. At that point you could communicate with the beaglebone itself
    3. But, you could NOT start the Chi.Bio software (even though no reactor was in place?)

    As to the reactor – Labmaker should be able to send you a new one which you can then use (i.e. don’t have to wait for the existing one to be fixed).

    • This reply was modified 3 years, 9 months ago by harrison.
    #1276
    harrison
    Keymaster

    Hi Andrew,
    I see, that is annoying…
    So you are finding that even if you have no reactor plugged in to the control board it fails to start, and you see the errors above?

    The “correct” booting sequence is usually to have the reactors + pumps connected and powered on (i.e. with wall power supplies) when you start up the device. However, I was asking YZhang to try starting the code without anything connected as this would test if the problem is in the reactor, or in the control board.

    If you are finding that the software doesn’t start even when there is nothing connected (i.e. no reactors) to the control computer it is indicative that the control PCB might have a fault.

    #1278
    andrewletten
    Participant

    Hi Harrison,

    Yep, that’s right, we see those errors irrespective of whether the reactors are plugged into the control board or not. Hopefully Lab Marker can send us a new control PCB.

    Thanks for your help!
    Andrew

    #1279
    harrison
    Keymaster

    Morning,
    Yes – if it is failing without anything plugged in to the control board it is indicative of the control board being faulty. (presuming you have the latest software image installed – did you do this?)

    Harrison

    #1280
    andrewletten
    Participant

    Yep, we have the 2021-06-18 version installed. The good news is that Labmaker have just emailed to say they are going to send a replacement PCB.
    Best,
    Andrew

    #1281
    yzhang
    Participant

    Hi Harrison!

    Thank you for your suggestions!
    I tried to establish communication while unplugging the reactor from control board. This is what I saw. It seems to be working but once I plugged in the reactor, the same error message showed up again.
    So I guess the problem originated from the reactor? Lab Maker rep said they will fix the problem after we send it back. Can I requested a replacement for the reactor while the old one is being fixed?

    login as: root
    Pre-authentication banner message from server:
    | Debian GNU/Linux 10
    |
    | BeagleBoard.org Debian Buster IoT Image 2020-08-25
    |
    | Support: https://bbb.io/debian
    |
    | default username:password is [debian:temppwd]
    |
    End of banner message from server
    root@192.168.7.2’s password:

    The programs included with the Debian GNU/Linux system are free software;
    the exact distribution terms for each program are described in the
    individual files in /usr/share/doc/*/copyright.

    Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
    permitted by applicable law.
    Last login: Sun Jun 6 14:36:50 2021 from 192.168.7.1
    root@beaglebone:~# ls
    chibio
    root@beaglebone:~# cd chibio
    root@beaglebone:~/chibio# bash cb.sh
    [2021-07-06 14:42:34 +0000] [3291] [INFO] Starting gunicorn 20.1.0
    [2021-07-06 14:42:34 +0000] [3291] [INFO] Listening at: http://192.168.7.2:5000 (3291)
    [2021-07-06 14:42:34 +0000] [3291] [INFO] Using worker: sync
    [2021-07-06 14:42:34 +0000] [3294] [INFO] Booting worker with pid: 3294
    2021-07-06 14:42:41.886960 Starting watchdog
    2021-07-06 14:42:44.369160 Initialising devices
    2021-07-06 14:42:44.463984 Start Up Complete

    #1282
    harrison
    Keymaster

    Hi Yzhang,

    Yes – if the software does start without the reactor connected it seems the reactor is most likely the source of issues. As I said the most common cause of such a problem is moisture (even just dried liquid somewhere hiding) on the moisture sensing tracks – but if not that then it may well be a fault in the hardware that they (Labmaker) will have to fix.

    I think Labmaker will hopefully be able to send you a replacement device rather than spending so long fixing it, you will have to ask them – I am not involved in their business or manufacturing process.

    #1289
    jhendricks
    Participant

    Hi and thanks for making and supporting such a cool product. It’s a very generous contribution to the community.
    I’m a colleague of yzhang’s and I’m trying to help him get back up and running while we wait on a replacement “blue PCB” from Labmaker. Sadly, I don’t think that the cape is the source of the issue. In the interest of speed, I’ve purchased an entire replacement setup (and most of the components to assemble 4 more reactors but that’s another story.) Shipping to the US is very slow so I’m hoping that you can point me in the right direction to get this back up and running ASAP.

    I’ve done some troubleshooting with an oscilloscope, logic analyzer and ohmmeter and wanted to share what I’ve found so far.
    1) We currently only have one bioreactor, so nothing to compare it to.

    2)The power supply I received with the bioreactor was DC 7V 1.5A, not 12V 3A
    I haven’t read all of the datasheets yet but the power requirements for the TPS54427 6V regulators came to mind:
    “Power Supply Recommendations
    The TPS54427 is designed to operate from input supply voltage in the range of 4.5 V to 18 V. Buck converters
    require the input voltage to be higher than the output voltage. in this case the maximum recommended operating
    duty cycle is 65%. Using that criteria, the minimum recommended input voltage is Vo/0.65.” So 7 volts puts it at 85% duty cycle, probably OK since there’s a thermal overload cutoff.

    3)Our problem exists on all ports (T0-T7.)

    4)Our problem exists whether the bioreactor is connected to power or not.

    5)After hearing that Labmaker was sending a replacement “blue PCB,” I removed the multiplexer with hot air and carefully soldered on a replacement PCA9548A. That didn’t make any difference.

    6)I’ve used a logic analyzer and an oscilloscope with 10X probes to make sure that probe capacitance isn’t an issue. Rise times seem fine.

    7)I haven’t connected the pump for any of this.

    8)I’ve watched the signals with both ‘bash cb.sh’ and various i2cget/set/detect/dump commands to emulate the script. The results are the same: writing to the multiplexer when the channel has been set to a port with a reactor makes a mess of the SDA and SCL lines until it’s changed to a different channel.

    9)I2C signals (as measured on the BeagleBone pins) are fine with the bioreactor control cable disconnected.
    Zoomed out:
    bash-cb-sh-0x74-0x00-0x80-no-cable
    Zoomed in:
    bash-cb-sh-0x74-0x00-0x80-no-cable-zoomed

    10)I2C signals (as measured on the BeagleBone pins) go OK until they hit a multiplexer channel that is connected to the bioreactor. At this point, both the SDA and SCL lines will get stuck around 1.7V.
    Zoomed way out (note that the signal goes from the usual 0-3.3v, then gets stuck at 1.7V for a while, then goes down to almost, but not quite zero):
    bash-cb-sh-cable-plugged-in-zoomed-way-out
    Zoomed out:
    bash-cb-sh-0x74-0x00-0x80-with-cable
    The moment everything goes wrong:
    bash-cb-sh-0x74-0x00-0x80-with-cable-zoomed

    11)Once the multiplexer is connected to a channel with the bioreactor, any attempt to read/write to the multiplexer will fail e.g. i2cset -y 2 0x74 0x00 0x01 Error: Write failed, as long as it is connected to a channel containing a bioreactor. If the cable is removed, it will recover, e.g. i2cdump -y 2 0x74 will work. If a different channel is selected and the cable is reconnected, then everything is fine again.

    12)The temperature sensor at 0x1b is not affected until the multiplexer is connected to a channel with the bioreactor. After that, it displays the same symptoms e.g. i2cset -y 2 0x1b 0x00
    Error: Write failed.

    13)On the “Laser Measure” board, the resistance between SDA & GND (and maybe SCL & GND? I can’t remember) is ~300 ohms. I measured it multiple times, with multiple ohmmeters, in both directions. That seems very wrong. SDA & SCL to 3.3V are something like 10k ohm IIRC.

    14)Disconnecting the 5 pin cable that connects the “Side Board” to the “Laser Measure” board, prevents the 1.7V lockup issue from occurring. I haven’t been brave (foolish?) enough to attempt powering the bioreactor with this cable disconnected.

    15)Yzhang reported unusual OD measurements leading up to the failure.

    All of this, particularly points 13-15, point towards some failure on the “Laser Measure” board.

    If you’ve got any tips on how to narrow it down to a specific component, which I would then replace, I would appreciate it very much. It can’t be worse than the TSSOP-24 multiplexer that I swapped for no reason ;P

    Thanks for your time, and sorry about the giant wall of text.

Viewing 15 posts - 1 through 15 (of 26 total)
  • You must be logged in to reply to this topic.
Log in/Register
Scroll to top