Cannot start system / Failed Pumps comms

[Timomaria]

Hello,

I just received one reactor with pumps and was able to use it for 2 days successfully. Now I can no longer start it, I always get “Failed Pumps” messages and the system shuts down.

root@beaglebone:~/chibio# bash cb.sh
[2022-02-06 18:44:10 +0000] [1820] [INFO] Starting gunicorn 20.1.0
[2022-02-06 18:44:10 +0000] [1820] [INFO] Listening at: http://192.168.7.2:5000 (1820)
[2022-02-06 18:44:10 +0000] [1820] [INFO] Using worker: sync
[2022-02-06 18:44:10 +0000] [1823] [INFO] Booting worker with pid: 1823
2022-02-06 18:44:18.464797 Starting watchdog
2022-02-06 18:44:20.954171 Initialising devices
2022-02-06 18:44:21.044415 Failed Pumps comms 1 times on device M0
2022-02-06 18:44:21.066876 Failed Pumps comms 2 times on device M0
2022-02-06 18:44:21.089212 Failed Pumps comms 3 times on device M0
2022-02-06 18:44:21.111356 Failed Pumps comms 4 times on device M0
2022-02-06 18:44:21.133512 Failed Pumps comms 5 times on device M0
2022-02-06 18:44:21.155877 Failed Pumps comms 6 times on device M0
2022-02-06 18:44:21.177989 Failed Pumps comms 7 times on device M0
2022-02-06 18:44:21.200499 Failed Pumps comms 8 times on device M0
2022-02-06 18:44:21.222598 Failed Pumps comms 9 times on device M0
2022-02-06 18:44:21.244858 Failed Pumps comms 10 times on device M0
2022-02-06 18:44:21.266991 Failed Pumps comms 11 times on device M0
2022-02-06 18:44:21.288817Failed to communicate to a device 10 times. Disabling hardware and software!
[2022-02-06 18:44:21 +0000] [1820] [INFO] Shutting down: Master
[2022-02-06 18:44:21 +0000] [1820] [INFO] Reason: App failed to load.
root@beaglebone:~/chibio#

I did software updated with fresh flahsed sd card as described in the software manual but it does not work.
Pumps are connected to the reactor as described in the hardware.

What shall I do? Can you please help me?

Thanks for your help, Timo

[harrison]

Hi Timo,
Likely the problem is moisture dried on the moisture sensing tracks (parallel silver lines) on the top of the reactor.
Get a piece of tissue with ethanol/cleaning alcohol and wipe down these tracks thoroughly all around the top of the reactor then try again.

[saurabhmalani]

Hi Harrison,

I’m facing the exact same issue as Timo. I was running the reactor well for 4 days, it occasionally threw a “Failed Pumps Comms” message intermittently but otherwise worked fine. Now, it does not start at all.

If I do bash cb.sh without the reactor connected, it loads and I can look up the GUI at 192.168.7.2:5000. I can connect the reactor then click Scan Device, my reactor pops up. I can change all the parameters, including pump rates, but if I click “Start Experiment”, or click ‘Switch’ on any pump, it instantly throws 11 Failed Pumps Comms messages and crashes.

If I do bash cb.sh with the reactor connected, it immediately crashes with the same 11 Failed Pumps Comms messages.

There was no spill in the reactor, but I cleaned the moisture sensing tracks with ethanol for good measure anyway, but to no avail.

I’ve tried multiple ports for the reactor but same thing.

I have the latest software installed, including the latest ones from the github repo.

What can I do further?

[harrison]

Hello,
Thanks for the helpful diagnosis.
The one thing not quite mentioned is – when you start without the reactor, then connect reactor, then scan devices – if you THEN try to do control something other than the pumps (for example, turn on a LED) does it work? I assume based on what you said the answer is yes but it was not explicit.
Is this only a problem on one reactor? Do yo have others working fine?

It sounds to me as if there is a hardware fault in your pump board, or at the very least, somewhere downstream of the digital connection that goes out to the pump board. Could you take a look at the soldering under the pump board and check if anything looks suspect? Also, look at the microUSB connectors on both ends of that connection, are these all OK, not broken?
In parallel it might be worthwhile contacting Labmaker to tell them it seems the pump board is defective, and request a new one.

[saurabhmalani]

Hi Harrison,

If I start without the reactor, then connect reactor, then scan devices, I can do things like change set points, I can measure OD and get a reasonable value. I can click ‘switch’on the LED and the app does not crash, though I actually do not observe the LED turning on (though if I’m able to measure a OD, clearly the LED must have been turning on?)

I only have one reactor and one pump board, I was hoping to get some good data from it to justify getting the full complement of 8 reactors.

I’ve used different micro USB cables, I don’t ~ seem ~ to see anything amiss in the electronics at the bottom of the pump board.

The pumps have worked well for two runs, aside from some intermittent “Failed Pumps Comms” that usually seems to quickly resolve automatically.

Also – another thing that I’m not sure if it’s relevant to this issue, when I plug in the reactor to the power supply, the heat plate immediately starts heating up even before an experiment is started (i.e. no temperature target is set), not sure if it’s remembering the setting from a previous run, but it gets pretty hot – is this normal behavior?

I’ve dropped LabMaker an email too about this.

[harrison]

Hello,
Thanks for the additional information. If it is measuring OD then that means the Laser is working – but not necessarily the LED.
This might make sense if the fault is in the PWM chip in the main reactor, which incidentally also controls the hot plate.
One possibility is that there is a bad solder joint on some of the pins of that chip, and consequently the software is not able to communicate with it. Hence it somehow gets stuck in a state where the LEDs don’t respond and the hot plate is on.

It is however quite strange that this would just “break”, in the sense that it was previously working fine, and then stopped working all of a sudden. Can you think of anything that might have happened after those runs where it was working fine that may have caused a problem? (e.g. it being dropped, cleaned, etc).

It does sound like the fault is in hardware so indeed Labmaker would be the best to talk to about a replacement.

[Author]

Posts