Spontaneous reboot while flying and hanging shutdown issues

[Rob Thomson]

Difficulty is knowing even if it exists in opentx,

I have never had the issue, but could be more luck than anything else.

I tend to fly for a an hour , then power down. Have a break... Fly again etc.

So really... No way of having any certainty on it.

I do however suspect there could be something weird - a brand of sdcard?

Maybe try without the sdcard and see if it happens!

Rob

Sent from my GT-I9300 using Tapatalk 2

[Reacher10]

infrequently is the key word...I had 2 in one day a few hours apart and now on day 3 without any.

[kaos]

yep, my models are all powered and only has a flying time about 5-9 min 12 min max, so most of the time the tx is on while not flying. All the previous reboot was while the tx was sitting there doing nothing while I was playing with new update, no stick movement and all of them happened while I was not even playing with menu or set up it just sits on my desk. Only this time 2 days ago it happened in mid fly. The SD card was there since day one I had the SKY9X board, but that does not entirely rule out it being the source, since so many things related to SD access have been changed.

[kaos]

infrequently is the key word...I had 2 in one day a few hours apart and now on day 3 without any.

In my case it could be wks between reboot. :-S Before 2 days ago, it must have at least 4 - 6 wks since last reboot.

edit: just run the batt down to 6.4 volt (I reset the warning voltage as low as possible) while flying my Q-bot micro. no reboot. I should be happy or sad.
will recharge and let it run again.

[Kilrah]

But, if there are many using 2S or 3S to power the SKY9X board and the group using ERSKY9X are the only group that experiencing reboot while the group using Open9x have none. That would imply the problem is more likely from fW than hardware.

Not necessarily... we have what, 2 or 3 people with problems? Might simply be that those have a HW problem, and just happen by chance to all be using ersky9x...
Chances are tiny that it's a big ersky9x problem, because there would have been more reports since Kaos had his first problems 4 months ago...

Will really need to try and understand when and why, and compare both FWs...

[RCHH]

@kaos - you say the first reboot was when you had BT and haptic connected. Try pulling them and see if you reboot. They are not mission critical for you I assume? Worth a try.

[kaos]

I will try all kinds of ways for sure. Now I just want to get some reboot data so Mike can have something to narrow it down.
yes, anything is possible, just have to narrow it down little by little.
RCHH: yes, when the 1st reboot started I only had BT and haptic, but they are on there since day one I started using SKY9X board without issue till Dec. 20th.
NO I am not blaming on anything, just trying to figure out why this could happen. At one time I even tried while the tx is on, I hold the TX tight with both my hands and shook it as hard as I can (with a servo tester program running) just short of flying out of my hands. Nothing happened.
if it is a hard ware issue, I assume I should expect it would happen more often since I have been using it so long all this time. If it is a software issue, then it may be only certain condition or code is running/or a combination code was running causing it and that is hard to catch. I even suspected whether there may be a radio frequency interfering with the board under certain condition, but the reboot happened (although total is only 7-8 times, no more than 10 times for sure, all this time) in single use of a Flysky or a Frysky DJT module. Because it happens so infrequently I can't tell it happens more with Flysky module or Frysky module.

I DO notice when I use a flysky module when the antenna at the back is pointing toward the tx front side I can hear spker humming noise, once the antenna is straight up, that noise goes away almost nothing. So whether this is because readio frequency is interfering the circuit, i don't know either.

Kilrah: yes, there are only few people reporting this, but you really have to watch it to know it is happening or while you are flying. it reboots really fast. The link will reestablish quick, without voice announcement and splash screen upon reboot, if I am flying a plane while just cruising, I may not even noticed it or thought it may run into a gust of wind and gone. But if you are flying a cp heli, it will show for sure and most likely crash it.

I set my splash screen on and each model will announce the model name and switch position upon booting. that is how I noticed it the last time when I was flying my micro cp heli 2 days ago. initially, i see the heli just slides to the one side quick I thought I did something wrong cause I am just setting up this new TOY. it is darn sensitive and not quite set up right yet. but the voice announcement told me it is rebooting. Before I lay my eyes on the screen the reboot is finished and the heli is laying on my carpet. I don't think I am going to look at the screen while the heli is sliding fast while tx is rebooting.

[Kilrah]

you really have to watch it to know it is happening or while you are flying. it reboots really fast. The link will reestablish quick, without voice announcement and splash screen upon reboot

Now yes, but as mentioned by Mike above this mechanism was only added in r163 5 days ago - all reboots before that would have the splash and warnings. So it's been that way for several months...

[MikeB]

I've committed r165. I managed to find ways to force various reboots due to power fails and watchdog to test the actual reboot operation. I found it did not always skip the splash screen, bug introduced with "UNEXPECTED SHUTDOWN" processing recently. r165 fixes that.
(Previous to r163, a watchdog reboot DID skip the splash screen properly).

I've also reduced the watchdog timeout from 1.5 seconds to 0.5 seconds, so a watchdog reboot will be 1 second quicker. The watchdog should re-trigger every 0.02 seconds, so 0.5 seconds means there is a problem.

I've also checked a number of places where a timer (10mS) is used, and while the compiled code looks OK, I have changed it to make sure.

For the technical, this relates to using unsigned arithmetic for comparison of timers. On the Mega64, 16-bit unsigned values work fine, as the 'natural' integer size is 16-bits, but on the ARM the natural integer size is 32-bits, so 16-bit unsigned values get promoted to 32-bit signed values. This can cause an incorrect comparison. It was one of these that I fixed recently that could cause the shutdown problem.

I've still had no reboots, been running for 6 hours today.

As an observation, if, whatever the reboot is caused by, does NOT interrupt the power to the Tx module, that will remain transmitting, and the receiver will remain bound, so as soon as the ARM processor starts sending data, control will be regained immediately. I doubt anyone will be able to notice if, during a reboot, the receiver loses connection to the Tx.

Mike.

[kaos]

you really have to watch it to know it is happening or while you are flying. it reboots really fast. The link will reestablish quick, without voice announcement and splash screen upon reboot

Now yes, but as mentioned by Mike above this mechanism was only added in r163 5 days ago - all reboots before that would have the splash and warnings. So it's been that way for several months...

yes, I think Mike did that for r163 and on but for people using r162 or prior , they can turn splash screen off (i have it on just my splash screen looks cool and not every model people will have voice announcement for model name, I think. Mines all have one.


Mike: Are you suggesting we use r165 instead of r164 from now? It still gives you the data you want right? if so I will load the r165 in the next few hrs.

Just recharged the batt and flew 3 packs on my Q-bot micro each pack is 7:30 min, no reboot - I expect so too (but a lot of crash by pilot error ). again, it happens so infrequent. I think next time it will happen would be days from now if whatever Mike did is not affecting what it was happening.

[Reacher10]

I concur Mike, in fact I was gonna look down that same street, sniff the wind, bend my neck in that very same direction. I thought perhaps, just perhaps the gonkolater tock banks could have been interrupting each other therefor end up in gibberish coming from ports.

But as you say, 32 bits, you see, you see right there that leads you to the same thing I naturally saw and really 32, 33 bits, whatever it takes but thats just conversely the same. You put your big toe right on the splinter...nice Mike.

We'll get this figured out...I'll pump some thought into few other obscure capillaries as well.

[MikeB]

Yes, use r165, r164 does not skip the splash screen on a watchdog reboot, it used to, but I got the logic wrong when adding the bit about "UNEXPECTED SHUTDOWN".
r165 will also reboot from a watchdog reset faster.

Mike.

[kaos]

Done. running r165.

[ReSt]

At one time I even tried while the tx is on, I hold the TX tight with both my hands and shook it as hard as I can (with a servo tester program running) just short of flying out of my hands.

If you suspect a poor solder joint, take a haptic motor, power it with a battery and while it is running, press the motor against different positions of the board.

This will vibrate the board, so that a bad solder joint should get changing contact levels.


Reinhard

[G550Ted]

Or borrow a self-contained device from an intimate friend to probe any suspect areas.

Ted

[kaos]

wow, just had a reboot after flying 3 pack on q-bot micro and sitting there with TC off and led off. all of a sudden it say "the system is ready , enjoy fly" 'acro mode'.
I looked at the watch dog only show one number '2' at lower right corner.
Tx voltage : 7.8V running r165
Anything else I can provide?
edit: a little more info:
CPU temp. 24c Max 26C
current 15x Max 241
interesting thing is my clock show 16:44 while it is 18:21 , I am pretty certain it was only a few minutes off in the morning or yesterday. And when I try to reset it, it will not reset, 'menu long' does nothing.

Or borrow a self-contained device from an intimate friend to probe any suspect areas.

volunteer to be tested so I know the device does shake something loose?

[Reacher10]

No reboots but the RTC stopped again. I went to see if I could set it to the correct time and I could and it was running again.

[kaos]

yep my clock stopped at 16:44, I think that is the time I upload the r165. and I can't reset it , it just stopped a that time, not running.
I did not restart the tx again after the reboot, just in case Mike needs some other info from the tx.

[Reacher10]

Mine stopped again agout 50 minutes later..not that I use it, just that it mean something. It won't reset. OK, I just re flashed r165 and the clock is running and it's the correct time without me setting it.

[RCHH]

Man you guys are suffering. Reboots and RTC problems. Not good. Are you both using 2S batteries?

[kaos]

I am using 2S lipo as of now.

[cmanley]

I'm using 3S LiFe.

[Kilrah]

Hmm... RTC is on the coprocessor. Could coprocessor communication cause timeouts and thus the reboot? Maybe a bug in the coprocessor firmware where it stops responding at times? What coprocessor fw version are you running?

Sent via mobile

[MikeB]

So you got a watchdog reset. I'll go over the RTC code. The RTC runs on the co-processor and the ARM has to keep getting the time from it. Maybe there is a possibility of a hang up there. Interesting that the RTC keeps time so the co-processor is still 'live'.

For information, the volume setting uses the same interface as the RTC, so if the RTC stops clocking up, try changing the volume please, to see if the I2C interface is frozen.

The co-processor version is displayed at the bottom of the main version screen. I have (1)02 on my rev B board, never updated it since I started using the Rev 3 board, I'll update to the latest version.

Mike.

Edit: Co-proc now at (1)05, test running.

[MikeB]

I've just checked between ersky9x and openTx regarding the code to access the co-processor. The driver code is the same, apart from the odd name change.
The only difference is ersky9x reads the coprocessor 4 times per second, openTx only once per second.
If there is a problem, it is perhaps 4 times more likely to happen on ersky9x.
I'll see if I can do a test and read it more frequently!

Mike.

Edit, Just had a watchdog reboot myself!

[RCHH]

Mike, my system is still on the 02 copro. No reboots whatsoever. Did yours reboot once you went to 05? Been running now since Friday. Yesterday in the window CPU temp maxed at 49degC in the sun. Not a heat issue.

[MikeB]

Yes, no problems with 02, just loaded 05 today, got a reboot within half an hour.

Mike.

Edit: OK, I've got a test running where I read the co-processor 50 times per second instead of 4.
The most likely cause of a watchdog reset is an interrupt routine that does not clear the cause of the interrupt, The interrupt routine then just keeps executing. The problem now is to find it!

[RCHH]

If I load 05 and get reboots, is it possible to revert to 02 and see if it reboots then?

[Reacher10]

My co processor is 04. When the clock stops again I'll check to see is I can change the volume.

OK, so you had one happen also Mike, I am really surprised because I just had a feeling you wouldn't get one.
Something is amiss thats for sure.

Quick question, I was unable to flash the co processor so Brent did it for me. How can I erase it? It's a neat feature but I really don't need it.

[MikeB]

Haven't had another reboot yet in 2.5 hours at the faster reading rate!

The coprocessor TinyApp.bin version 02 is somewhere on the repository, just need to find which revision it belongs to (might be 116).

Reacher10: You can't easily erase the co-processor. You would need to program it directly (not using SAM-BA) with the bootloader. The ARM will still try to read the co-processor though.

Mike.