overnight experiment stopping unexpectedly

[uthsc]

We left an experiment running over night and aren’t sure why it stopped. It looks like errors starting appearing soon after launch even though the lab tech reported it failing sometime later. Can someone please help figure out what’s going on? Let me know what additional info you need. Thank you.

[2021-09-24 06:18:38 +0000] [1799] [INFO] Starting gunicorn 20.1.0
[2021-09-24 06:18:38 +0000] [1799] [INFO] Listening at: http://192.168.7.2:5000 (1799)
[2021-09-24 06:18:38 +0000] [1799] [INFO] Using worker: sync
[2021-09-24 06:18:38 +0000] [1802] [INFO] Booting worker with pid: 1802
2021-09-24 06:18:46.741920 Starting watchdog
2021-09-24 06:18:49.224585 Initialising devices
2021-09-24 06:18:49.436996 Initialised M0, Device ID: 2970188232538057612
2021-09-24 06:18:49.953033 Start Up Complete
2021-09-24 06:20:57.269464Calibrated OD
2021-09-24 06:32:56.353830Failed to disconnect multiplexer on device M0

Message from syslogd@beaglebone at Sep 24 06:32:56 …
kernel:[ 966.734842] Internal error: : 1028 [#1] PREEMPT SMP ARM

Message from syslogd@beaglebone at Sep 24 06:32:56 …
kernel:[ 966.828018] Process irq/36-44e0b000 (pid: 110, stack limit = 0xbb817f 5f)

Message from syslogd@beaglebone at Sep 24 06:32:56 …
kernel:[ 966.834750] Stack: (0xdac13eb8 to 0xdac14000)

Message from syslogd@beaglebone at Sep 24 06:32:56 …
kernel:[ 966.839128] 3ea0: c11a7054 c1607a40

Message from syslogd@beaglebone at Sep 24 06:32:56 …
kernel:[ 966.847346] 3ec0: 00000000 daae0840 dc330800 ffffe000 00000000 dabfeb 40 c01afda0 00000001

Message from syslogd@beaglebone at Sep 24 06:32:56 …
kernel:[ 966.855564] 3ee0: dac13f04 dac13ef0 c0a82c38 c0a828a0 dabfeb40 dc3308 00 dac13f24 dac13f08

Message from syslogd@beaglebone at Sep 24 06:32:56 …
kernel:[ 966.863781] 3f00: c01afdcc c0a82c24 dc330800 dabfeb64 ffffe000 000000 00 dac13f74 dac13f28

Message from syslogd@beaglebone at Sep 24 06:32:56 …
kernel:[ 966.871998] 3f20: c01b0144 c01afdac c10ea2d0 c1506e08 c15de2e1 dac13f 40 c0d44b48 00000000

Message from syslogd@beaglebone at Sep 24 06:32:56 …
kernel:[ 966.880216] 3f40: c01afefc 754b2d89 c015ffe4 dabfe580 dabfe2c0 000000 00 dac12000 dabfeb40

Message from syslogd@beaglebone at Sep 24 06:32:56 …
kernel:[ 966.888433] 3f60: c01affe0 dc305bec dac13fac dac13f78 c01604f4 c01aff ec dabfe59c dabfe59c

Message from syslogd@beaglebone at Sep 24 06:32:56 …
kernel:[ 966.896651] 3f80: 00000000 dabfe2c0 c0160388 00000000 00000000 000000 00 00000000 00000000

Message from syslogd@beaglebone at Sep 24 06:32:56 …
kernel:[ 966.904869] 3fa0: 00000000 dac13fb0 c01010e8 c0160394 00000000 000000 00 00000000 00000000

Message from syslogd@beaglebone at Sep 24 06:32:56 …
kernel:[ 966.913086] 3fc0: 00000000 00000000 00000000 00000000 00000000 000000 00 00000000 00000000

Message from syslogd@beaglebone at Sep 24 06:32:56 …
kernel:[ 966.921304] 3fe0: 00000000 00000000 00000000 00000000 00000013 000000 00 00000000 00000000

Message from syslogd@beaglebone at Sep 24 06:32:56 …
kernel:[ 966.996888] Code: e34cc11a e50bc034 e5d00001 e0813310 (e1d340b0)

Message from syslogd@beaglebone at Sep 24 06:33:07 …
kernel:[ 978.000856] Disabling IRQ #36

[harrison]

Hello, what operating system version are you using? Has it been updated to the latest variant from the Chi.Bio website?
I have observed an error somewhat similar to this in the past but updated the OS in order to try to cover it off. If you are still seeing it that is worrying as it means something might still be amiss…

Harrison

[uthsc]

Thank you very much for the reply. We did load the latest OS available before starting experiments, but I’ll have to check the version number as I’m unsure. I’ll go ahead and load the latest version available on the Chi.Bio website and see if we still get the errors. I’ll be in touch either way. Thanks again.

[harrison]

Great, let me know.
Also, by OS I mean the underlying Beaglebone operating system (i.e. Debian Linux), not just the Chi.Bio software on Github. The underlying OS is the one causing these faults, we have been trying to patch it…

[Author]

Posts