Errors

[Nuha]

Hello,

I intent to run experiments for 48-72 hours, but I have been running into errors hours into the experiment causing the system to shutdown. I read through the forum in hardware titled Frequent Communication Errors and tried debugging using the methods discussed but still run into other errors. I don’t think it’s a hardware issue. I think it has to do more with software as I noticed the errors were more frequent after I flashed with the updated operating system and github files. Also, the pumps when turned on manually, stop on their own despite the switch being still on. My colleague in the lab ran the pumps manually directly using the python script without the GUI, and she faced no problems keeping them on overnight.

The frequent error I have been getting with and without OD regulation (i.e with and without pumps), with different reactor and pump combinations:
client_loop: send disconnect: Broken pipe

Another:

2021-04-06 18:30:54 +0000] [1797] [CRITICAL] WORKER TIMEOUT (pid:1800)
[2021-04-06 18:30:54 +0000] [1800] [INFO] Worker exiting (pid: 1800)
client_loop: send disconnect: Broken pipe

I have gotten this once, in multiple lines:
[2021-04-06 17:22:22 +0000] [4344] [INFO] Handling signal: winch
followed by:

2021-04-06 17:26:47.395853 Failed PWM comms 11 times on device M1
2021-04-06 17:26:47.417743Failed to communicate to a device 10 times. Disabling hardware and software!

I have also gotten this once and decided to stop working with the reactor in M1, it has also stopped being recognizable when scanning for devices:

2021-04-05 18:52:59.699182 Failed AS7341 comms 1 times on device M1
2021-04-05 18:52:59.721036 Failed  AS7341 in I2CCom while trying to send 157 and 0
2021-04-05 18:52:59.725827 Failed AS7341 comms 1 times on device M1
2021-04-05 18:52:59.747385 Failed  AS7341 in I2CCom while trying to send 158 and 0
2021-04-05 18:52:59.752460 Failed AS7341 comms 1 times on device M1
2021-04-05 18:52:59.774103 Failed  AS7341 in I2CCom while trying to send 159 and 0
2021-04-05 18:52:59.778910 Failed AS7341 comms 1 times on device M1
2021-04-05 18:52:59.800450 Failed  AS7341 in I2CCom while trying to send 160 and 0
2021-04-05 18:52:59.805176 Failed AS7341 comms 1 times on device M1
2021-04-05 18:52:59.826679 Failed  AS7341 in I2CCom while trying to send 128 and 1
2021-04-05 18:52:59.829577AS7341 measurement failed on M1
2021-04-05 18:52:59.833238 Failed AS7341 comms 1 times on device M1
2021-04-05 18:52:59.854721 Failed  AS7341 in I2CCom while trying to send 169 and 4
2021-04-05 18:52:59.859800 Failed AS7341 comms 1 times on device M1
2021-04-05 18:52:59.881674 Failed  AS7341 in I2CCom while trying to send 128 and 0
2021-04-05 18:52:59.896671 Failed AS7341 comms 1 times on device M1
2021-04-05 18:52:59.918163 Failed  AS7341 in I2CCom while trying to send 128 and 1
2021-04-05 18:52:59.922774 Failed AS7341 comms 1 times on device M1
2021-04-05 18:52:59.944242 Failed  AS7341 in I2CCom while trying to send 175 and 16
2021-04-05 18:52:59.949056 Failed AS7341 comms 1 times on device M1
2021-04-05 18:52:59.970593 Failed  AS7341 in I2CCom while trying to send 0 and 0
2021-04-05 18:52:59.975511 Failed AS7341 comms 1 times on device M1
2021-04-05 18:52:59.997379 Failed  AS7341 in I2CCom while trying to send 1 and 0
2021-04-05 18:53:00.002182 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:00.023691 Failed  AS7341 in I2CCom while trying to send 2 and 0
2021-04-05 18:53:00.028566 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:00.050147 Failed  AS7341 in I2CCom while trying to send 3 and 0
2021-04-05 18:53:00.054830 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:00.076298 Failed  AS7341 in I2CCom while trying to send 4 and 2
2021-04-05 18:53:00.081253 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:00.102796 Failed  AS7341 in I2CCom while trying to send 5 and 0
2021-04-05 18:53:00.107358 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:00.129111 Failed  AS7341 in I2CCom while trying to send 6 and 0
2021-04-05 18:53:00.133794 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:00.155302 Failed  AS7341 in I2CCom while trying to send 7 and 0
2021-04-05 18:53:00.160149 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:00.181745 Failed  AS7341 in I2CCom while trying to send 8 and 16
2021-04-05 18:53:00.186490 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:00.208098 Failed  AS7341 in I2CCom while trying to send 10 and 0
2021-04-05 18:53:00.213019 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:00.234515 Failed  AS7341 in I2CCom while trying to send 11 and 0
2021-04-05 18:53:00.239265 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:00.260924 Failed  AS7341 in I2CCom while trying to send 12 and 0
2021-04-05 18:53:00.265654 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:00.287124 Failed  AS7341 in I2CCom while trying to send 13 and 0
2021-04-05 18:53:00.291768 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:00.313373 Failed  AS7341 in I2CCom while trying to send 14 and 32
2021-04-05 18:53:00.318143 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:00.339630 Failed  AS7341 in I2CCom while trying to send 15 and 0
2021-04-05 18:53:00.344664 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:00.366160 Failed  AS7341 in I2CCom while trying to send 16 and 0
2021-04-05 18:53:00.370981 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:00.392467 Failed  AS7341 in I2CCom while trying to send 17 and 16
2021-04-05 18:53:00.397166 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:00.418651 Failed  AS7341 in I2CCom while trying to send 18 and 0
2021-04-05 18:53:01.358601 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:01.380364 Failed  AS7341 in I2CCom while trying to send 148 and 0
2021-04-05 18:53:01.385288 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:01.406804 Failed  AS7341 in I2CCom while trying to send 149 and 0
2021-04-05 18:53:01.411653 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:01.433337 Failed  AS7341 in I2CCom while trying to send 150 and 0
2021-04-05 18:53:01.438297 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:01.459841 Failed  AS7341 in I2CCom while trying to send 151 and 0
2021-04-05 18:53:01.464797 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:01.486321 Failed  AS7341 in I2CCom while trying to send 152 and 0
2021-04-05 18:53:01.491624 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:01.513534 Failed  AS7341 in I2CCom while trying to send 153 and 0
2021-04-05 18:53:01.518415 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:01.539958 Failed  AS7341 in I2CCom while trying to send 154 and 0
2021-04-05 18:53:01.544824 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:01.566461 Failed  AS7341 in I2CCom while trying to send 155 and 0
2021-04-05 18:53:01.571350 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:01.593091 Failed  AS7341 in I2CCom while trying to send 156 and 0
2021-04-05 18:53:01.598172 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:01.619736 Failed  AS7341 in I2CCom while trying to send 157 and 0
2021-04-05 18:53:01.624731 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:01.646257 Failed  AS7341 in I2CCom while trying to send 158 and 0
2021-04-05 18:53:01.650996 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:01.672532 Failed  AS7341 in I2CCom while trying to send 159 and 0
2021-04-05 18:53:01.677695 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:01.699246 Failed  AS7341 in I2CCom while trying to send 160 and 0
2021-04-05 18:53:01.703971 Failed AS7341 comms 1 times on device M1
2021-04-05 18:53:01.725577 Failed  AS7341 in I2CCom while trying to send 128 and 1
2021-04-05 18:53:01.728358AS7341 measurement failed on M1
2021-04-05 18:53:01.729704AS7341 measurement failed twice on M1, setting unity values
2021-04-05 18:53:01.733295 Failed PWM comms 1 times on device M1
2021-04-05 18:53:01.755382 Failed PWM comms 2 times on device M1
2021-04-05 18:53:01.777505 Failed PWM comms 3 times on device M1
2021-04-05 18:53:01.799408 Failed PWM comms 4 times on device M1
2021-04-05 18:53:01.821557 Failed PWM comms 5 times on device M1
2021-04-05 18:53:01.850572 Failed PWM comms 6 times on device M1
2021-04-05 18:53:01.872953 Failed PWM comms 7 times on device M1
2021-04-05 18:53:01.895395 Failed PWM comms 8 times on device M1
2021-04-05 18:53:01.917565 Failed PWM comms 9 times on device M1
2021-04-05 18:53:01.939539 Failed PWM comms 10 times on device M1
2021-04-05 18:53:01.961701 Failed PWM comms 11 times on device M1
2021-04-05 18:53:01.983409Failed to communicate to a device 10 times. Disabling hardware and software!

I’m going to try flashing with the old operating system to see if I run into the same problems. The last time I used the chibios were last year in March and these errors are all new to me. The experiment used to run overnight with no problems. Please let me know if you have any suggestions or possible explanations. Thank you!

[harrison]

Hello,

It seems there might be a few different things at play here.

For the Broken Pipe and winch errors – these are errors (If I recall correctly) arising from the actual connection between the Beaglebone Black and your computer controlling it (i.e. PuTTy is unhappy). For example, your computer might be set to de-power or shut down USB devices/connections after some period of inactivity. Hence it is causing issues (which would explain why your colleague is able to run things properly from the python but not the GUI). For this case I would strongly recommend trying another computer, or trying to figure out what it is that yours is unhappy with.

The above one would expect to be similar regardless of any hardware fault (i.e. same error would appear for every reactor).

The second thing you mention (those errors + M1 not working) is indicative of a hardware fault in that particular reactor, potentially due to liquid ingress, or bad hardware assembly in the first place. I would recommend taking the sides off that reactor and looking at the circuit to see if there is any obvious damage. If it is an assembly fault Labmaker might be able to replace it for you

Harrison

[Nuha]

I thought the same regarding the computer, but I disregarded it since (1) I’m using the same computer I used last year (which is an iMAC not PC) and it’s the first time I see the errors (2) if it has to do with inactivity I expect the disconnection to happen around the same time every time I run an experiment, but it is variable and it has only gotten worse (I ran an experiment before updating everything and it stopped after 26 hours, and after updating I’m getting these errors within an hour or less into the experiment).

Updates for the iMAC have also been installed recently, not sure if that plays a role. I’ll double check and try with a different computer just to be sure. Thank for the prompt response!

[harrison]

To be honest I have never used a mac myself and have no idea what rare/unexpected errors might occur. In our lab we have reactors on both windows and linux PCs without such issues.

[Nuha]

So it turned out there was downtime on the iMAC, when adjusted I stopped seeing the errors I mentioned thankfully. But, I do get one error not too often that stops the whole thing, “Failed to disconnect multiplexer”. This error also becomes more often when I connect more than 2 reactors to the beaglebone. I also get pump connection errors hours into the experiment sometimes, but rarely does it stop the experiment. Any suggestions to why that is happening? I’m thinking to try different beaglebones, and try a new computer afterwards to compare.

[harrison]

That error can happen if there is a bad connection somewhere in the electronics – or if the system overflows any liquid which lands on the moisture sensing tracks that causes it to crash (deliberately in that case). Note that it could even be that water condenses on the test tube exterior and then ends up on the moisture sensing (silver lines on top surface) and triggers it.
What version of reactors are you using? I assume as above you have the latest version of the Beaglebone software?

[Author]

Posts