pumps stopped working in the middle of the night

[andreyshur]

Hi,

We are having a very strange and frustrating problem. Normally we are setting up an OD regulation run on 3 reactors overnight. Everything seems fine for hours, then sometimes we get Multiplexer comms or Pump comms error messages (perhaps 1 or 2). Then, the pump stops working, and of course with the pump not working, the OD stops being regulated and the culture grows through the roof. It is very interesting though that the program does not crash, the i2c seems to be working fine as all three reactors are reporting OD and temperature and fluorescence data. But, the pumps are just not doing anything. A simple reset (closing the script and running it again from the .sh file) solves the problem, and everything works as it should again.
Any hints as to what could be the cause of this? We are pulling our hair out trying to replicate this problem but it happens maybe once every 5 hours.

[harrison]

If I understand correctly, the system keeps ticking over and doing measurements, and it thinks the pumps are going fine, but they have actually stopped?

Indeed this is something unlike any issues I have seen. What version of the code/software are you using? Is there anything else non-standard running on your beaglebone?

My best guess at a software reason would be if somehow the pump modulation threads are failing somehow, or the system is reaching some maximum in number of threads, but I don’t know why this would be occuring in your case. It has never been an issue for ours (or as far as I know, for other peoples’). Perhaps you can try putting a print statement into the pumpmodulation (or after that, into the setPWM function) which will print something out when the device attempts to communicate via I2C with the pumps? THat way you would be able to see if there was some software issue (I.e. the communications to the pumps are not even being attempted after the freeze)

In terms of hardware, the pumps are “downstream” of an I2C repeater in each reactor, which potentially could be locking up somehow. I recently made a new software setup V2.0 (see the software page) that can overcome some such errors. But, it seems unlikely that this would be the cause of your problem since all three reactors’ pumps are locking up at the same time (at least I think this is the case from your message?), which would be highly unlikely if it was a reactor-specific hardware fault.

Potentially if it is a hardware issue the code could be adjusted to trigger the watchdog circuit intermittently (which powers down the reactors, which happens when you restart the code), but this wouldn’t solve it if the threading was at fault.

[andreyshur2]

I will try the error messages and then get back to you.

Any chance we could get a schematic of the chi.bio electronics? You posted the gerber files but these do not include a schematic of how everything is connected.

Thanks,

Andrey Shur

[harrison]

Have emailed you 🙂

[Author]

Posts