Several reactors failing to connect to BB/control board

[harrison]

Hello,

For the first challenge – that some of them won’t start when connected – I would say the most likely issue is indeed the moisture sensor circuitry being tripped (internally or externally) including perhaps by a defect in the hardware manufacturing. If this is the case then it can show up at happening on M0 at startup (EVEN IF nothing is connected to M0) since that is the first reactor that is probed by the initialisation script. The second log you posted looks like it has the same root cause (i.e. multiplexer not connecting because something is shorted on the I2C lines), but it is strange that it then leads to a Kernel panic. This is not something we saw before updating to the new Kernel – so I would be very interested to know if it happens mid-experiment (rather than as part of the thing crashing for a separate reason).

Out of interest, what control board (V1.1? V1.2?) do you have?

You mentioned a light is always on in one of the reactors – is it on at very high power? Or just low/medium power? If the former then it is likely an issue in the OmAmp upstream of it, whereas if the latter it is probably that one of the resistors that provides bias to the (LED) current regulation circuit is not soldered perfectly.

It sounds like in general you have had several issues which may stem from manufacturing faults – for which the best solution (unless you want to pull everything apart and find the defect yourself) is probably to ask for a replacement from Labmaker! Sorry that this has been so difficult to fix/debug…

[mattkratz]

Hey Harrison,

With regards to the first problem, the error with the kernel panic occurred when running the initialization script. Additionally, we are using the V1.1 of the control board, out of curiosity, what are the major changes between the two versions? With regards to the reactor LED that is permanently on, it does seem to be on at very high power, although I haven’t probed it in the API as I’m afraid it may end up melting the source piece if I leave it on too long (occurred with a previous reactor). I’ll get into contact with labmaker looking for replacements.

[harrison]

The main differences are that V1.2 has an extra ability to cycle power to the multiplexer in error cases. However, this should have no impact if your system is failing to start due to some hardware fault…

[john.saxon]

Hi Harrison,

I’ve been troubleshooting a set of ChiBio reactors at the Letton Lab. During an experiment we came across an error like what you were describing; a Kernel panic mid-experiment. This is the error message we logged:

Message from syslogd@beaglebone at May 5 02:36:29 …
kernel:[52117.512683] Internal error: : 1028 [#1] PREEMPT SMP ARM

… [another 12 similar messages]

Message from syslogd@beaglebone at May 5 02:36:29 …
kernel:[52117.698950] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000

Message from syslogd@beaglebone at May 5 02:36:29 …
kernel:[52117.774528] Code: e34cc11a e50bc034 e5d00001 e0813310 (e1d340b0)

After which the reactor on M2 failed to disconnect:

…
02:36:34 Failed Multiplexer Comms 20 times
02:36:34Failed to recover multiplexer on device M2
02:36:34 Failed Multiplexer Comms 21 times
02:36:34Failed to recover multiplexer on device M2

We have gotten several similar errors after trying to run the reactors for a few days with the thermostats active. Do you think this could be the moisture sensors? And if so would you know a way to fix the issue?

[harrison]

Hello John,

It might be a possibility that the moisture sensor is causing some such problems. Have you ever seen condensation forming on the vial? It might happen depending on what the climate/humidity is like in your lab…

But, I’d have to say more likely is it is an issue with the underlying operating system which is fundamental to the Beaglebone. I have had one device in the past that gave similar PREEMPT SMP ARM error and it was fixed by flashing the beaglebone with a clean operating system, i.e. following the software setup instructions on this site. I don’t know if some beaglebones (the microcontroller) are themselves less reliable than others, but my experience indicates that might be the place (some cause issues for no apparent reason, potentially pointing to manufacturing defects). So, if none of the above works you could also try changing the beaglebone for which you can find a spare fairly cheap at many places online…

Harrison

[john.saxon]

We’ve seen condensation on the inside of the vial, but none on the outside. I will try flashing the beaglebone with a clean operating system. Thanks heaps for your help.

John

[Author]

Posts