ERSKY9X Coding

[MikeB]

OK, I have the fix for the cluster size, its related to the multiple "mount" operations I changed earlier.
I think it is actually just 7 filenames looking at the code again.
There is 8Mbyte of external SDRAM on the Horus, some of which is used to store the screen image, but there is plenty left unused. I'm going to see if I can read the whole directory into this, then I won't need to keep accessing the SD card. I can't make mine restart whatever I do!

Mike

[jhsa]

Mike, if you found out what the differences are between your board and the production one, wouldn't that help? I mean, don't you have access to that information??

João

Sent from my BLN-L21 using Tapatalk

[ozphoenix]

Sorry, did not see this post until just now - I didn't see that we had 'rolled over' to a new page

Glad you found the cluster size issue which had popped up - it was a little perplexing to see that one suddenly appear.

I'd be glad if the Horus was able to make use of the extra resources, but I'd be nervous to just make use of this space and then never understand why the SD was failing, anyway - one day, it might come back to bite us in another way, if we don't find out the 'root cause'. Accessing the SD at any time should not be biting us. But, let's proceed and we can see where it takes us Since OpenTX is open-source, is there any way to look at what code gyrations they had to do with the SD card, just to see if it caused them any headaches?

BTW - I changed the Model Names folder contents to 7 files and could not get a restart.

Regards,
ozphoenix

OK, I have the fix for the cluster size, its related to the multiple "mount" operations I changed earlier.
I think it is actually just 7 filenames looking at the code again.
There is 8Mbyte of external SDRAM on the Horus, some of which is used to store the screen image, but there is plenty left unused. I'm going to see if I can read the whole directory into this, then I won't need to keep accessing the SD card. I can't make mine restart whatever I do!

Mike

[MikeB]

João: Yes I do have the differences, I can't see why they should cause this problem.
ozphoenix: Yes, I do want to find why this is failing.
When you get a restart, do you have everything "freeze" for about 3 seconds, then restart?
I'm busy this morning, then I've got another (unrelated) task I need to look into as well. I swapped the interface board to the display, now my backlight is working OK again.

I may need to add something to locate where the code is "hanging", particularly if you have the "3 second" freeze.

Mike

[ozphoenix]

Good Morning/Evening Mike,
All good and understood - and, glad your radio is now displaying more reliably

When you get a restart, do you have everything "freeze" for about 3 seconds, then restart?

No, everything is working normally, I'm scrolling through the list and the screen goes black (or, sometimes, snow lines) for about 1-2 secs (and the LED goes out) then the screen is refreshed to the Main Model Page (EDIT: Main Model Flying Page) and all buttons and keys work as normal.

In the next few minutes, I'll try to post a video, in my next post.

Just as you have something else to do this morning, I am also about to shut down for the night, but I'll try to make the video and post it before I go.

regards,
ozphoenix

João: Yes I do have the differences, I can't see why they should cause this problem.
ozphoenix: Yes, I do want to find why this is failing.
When you get a restart, do you have everything "freeze" for about 3 seconds, then restart?
I'm busy this morning, then I've got another (unrelated) task I need to look into as well. I swapped the interface board to the display, now my backlight is working OK again.

I may need to add something to locate where the code is "hanging", particularly if you have the "3 second" freeze.

Mike

[ozphoenix]

Wow! I just made a video and got a restart within about 15 lines from the top of the list - but I guess it shows all of the sequence. See attached file.

[MikeB]

I just looked closely at the low level driver (from STM) for the Sd card. The interrupt priority was set too high, blocking the 10mS interrupt, among others, that extended the watchdog timeout.
New test version posted.

Mike

[ozphoenix]

Great find, Mike - if that's it, this would be a very good result.

I'll be able to test the new version in about 15 minutes or so and get back to you with a result.

I wonder how/if this relates to your unit acting so differently to mine?

Regards,
ozphoenix

I just looked closely at the low level driver (from STM) for the Sd card. The interrupt priority was set too high, blocking the 10mS interrupt, among others, that extended the watchdog timeout.
New test version posted.

Mike

[MikeB]

The watchdog is set to half a second I think, but I use software to extend it for some operations. As long as my SD card operations complete within the half a second (could be smaller directories to search etc.), it works OK. If yours takes longer, and the extended watchdog timer is blocked, then you get the reboot.

Mike

[ozphoenix]

Ok, understood - I was very hopeful that this was the fix, but after a few seconds of scrolling, I got another restart - repeated the test and got the same results (and data)
BOOT REASON=WATCHDG (96)
The Cluster Size is 32
The DEBUG information (is very scant):
0001 0000 (that's all she wrote).

Sorry to have to write that

regards,
ozphoenix

The watchdog is set to half a second I think, but I use software to extend it for some operations. As long as my SD card operations complete within the half a second (could be smaller directories to search etc.), it works OK. If yours takes longer, and the extended watchdog timer is blocked, then you get the reboot.

Mike

[MikeB]

Did you get a freeze and a 3 second delay now or was it the same as before?
My current thought is there is a memory access to an invalid address taking place, in which case you don't get the 3 second delay. Let me see if I can detect that.

Mike

[ozphoenix]

Mike,
I re-checked my steps - re-downloaded the file, verified the file dates and sizes, re-loaded to the radio, re-flashed the version. Same thing - no freeze when it crashes, scrolling along, instant flash to black, reload screen, REASON=WATCHDOG, DEBUG same as before (very little detail).

regards,
ozphoenix

Did you get a freeze and a 3 second delay now or was it the same as before?
My current thought is there is a memory access to an invalid address taking place, in which case you don't get the 3 second delay. Let me see if I can detect that.

Mike

[MikeB]

Test version posted that should detect invalid memory accesses, it will just put a different value on the BOOT REASON display.
If this is happening I'll need to either read through the (STM) driver code, or see if I can get information relating to the invalid access.

Most of the other debug was related to the missing model image, so I've removed that as it is now working.

Mike

[ozphoenix]

Understood on the debug information reduction.
Hard fault version downloaded, got restart.
BOOT REASON=WATCHDOG (96)
DEBUG 0001 0000
regards,
ozphoenix

Test version posted that should detect invalid memory accesses, it will just put a different value on the BOOT REASON display.
If this is happening I'll need to either read through the (STM) driver code, or see if I can get information relating to the invalid access.

Most of the other debug was related to the missing model image, so I've removed that as it is now working.

Mike

[MikeB]

Do you have the modules turned off still? If not, please turn them off and test. The interrupt being reported on a watchdog reboot is the telemetry serial data from the internal module.

Mike

[ozphoenix]

The modules are both turned off at the beginning of the test - I just made sure of that and re-tested and still got the restart - after the restart, the Internal XJT has been automatically returned (by the radio) to the 'enabled' state.

BOOT REASON=WATCHDOG but now, after I made SURE the XJT was off before the test, I have 00B8 on the WATCHDOG screen and 0001 0000 on DEBUG

I have to go out for about 2.5 hours so can now until answer questions on my phone, not do any further testing, until I return.

Will check back in when I return.

regards,
ozphoenix

Do you have the modules turned off still? If not, please turn them off and test. The interrupt being reported on a watchdog reboot is the telemetry serial data from the internal module.

Mike

[MikeB]

I've put up a new test version. I've added some more detection the SD interrupts and unexpected interrupts.
Changes done to model and radio settings may not be saved to the SD card if you get a watchdog reboot, I have an idea how to handle this, but need some time to implement it.
I put over 250 files in my model name voice directory and cannot get a failure on my radio!

This will likely be the last test version today, and I won't have time to progress things until tomorrow afternoon.

Mike

[ozphoenix]

Ok, Mike - thank you.

Will test it when I get home and post the results, for when you do get a chance to get back to it.

Enjoy your time away from it - don't even think about scrolling, in the meantime

Regards,
Peter M.

[ozphoenix]

Downloaded and installed, set Internal XJT to 'off' before testing, got restart.
BOOT REASON
WATCHDOG
01E2 020B 01ED
00BA
0

Cluster Size 32
DEBUG
0001 0000

Regards,
ozphoenix

I've put up a new test version. I've added some more detection the SD interrupts and unexpected interrupts.
Changes done to model and radio settings may not be saved to the SD card if you get a watchdog reboot, I have an idea how to handle this, but need some time to implement it.
I put over 250 files in my model name voice directory and cannot get a failure on my radio!

This will likely be the last test version today, and I won't have time to progress things until tomorrow afternoon.

Mike

[MikeB]

New test version posted. I've extended the watchdog timeout to 4 seconds (from 0.5 seconds). You should get a "freeze" for 4 seconds if things "hang up" while waiting for the watchdog to kick in.
The SD card driver has a number of places where it waits for an SD access to complete. I've added markers that should appear in the BOOT REASON screen (where it shows 00BA above) if it is stuck in one of those.

Mike

[ozphoenix]

Thanks, Mike - welcome back!

Now, as you predicted, different behaviour during the restart:
1. Scrolling as normal through list until...
2. Information flashes off the screen, which then goes to a grey/white blank (no text) background
3. The grey/white background stays on the screen for about 4 seconds
4. Then screen goes to black and restart occurs.

BOOT REASON WATCHDOG
01E2 020B 01ED
8103 0002 0204
2400 0000 0

DEBUG
0001 0000

Hope that helps.
Regards
ozphoenix

New test version posted. I've extended the watchdog timeout to 4 seconds (from 0.5 seconds). You should get a "freeze" for 4 seconds if things "hang up" while waiting for the watchdog to kick in.
The SD card driver has a number of places where it waits for an SD access to complete. I've added markers that should appear in the BOOT REASON screen (where it shows 00BA above) if it is stuck in one of those.

Mike

[ozphoenix]

While you've been asleep, openrcforums.com has been down almost all the time hope it stays up, now

[MikeB]

I know, I was going to ask this last night:
When you have the grey/white background does this:
1. Cover the whole display?
2. Is it the same as your set background colour?
3. What colour is the LED on the top of the radio? From the video you posted it seems stuck on a single colour (red or blue rather than purple which is both together).

If I could just get my radio to fail in the same way it would be easier to investigate!
I'm going to need to add some complex monitoring to work out what is happening.

Are you always using the "Model Voice" file list for these tests. If so, please try selecting a file for a voice alert.

Mike

[ozphoenix]

Hi again, Mike - good for us to be back online

1. Yes, the grey display covers the whole display area every time
2. My back ground is a light blue colour with some slight grey 'notes' (according to my wife and daughter - remember, I have a slight red/green/brown colour-blindness) -- the background colour numbers selected are: Red:0, Green:15, Blue:30. When the display flashes at the beginning of the failure, it is a much more noticeably grey (more pale) colour than my 'normal' background colour -- it is a similar grey to that which I see if I hold the radio at an angle downwards (and away from me at the bottom) and look at the normal-colour display from an acute angle (from the LED top edge of the radio) - the display changes colour as I rotate the radio away from me at the bottom, but that is just the observance angle of the LEDs, I believe.
3. When I start the scroll, the LED is purple. Coincident with the display going to the blank grey display, the LED changes to a red colour and holds that red colour for the period of the 'grey' display. Both the display and the LED go black after the 4-5 sec delay, then the normal display returns and the LED returns to purple.
4. Yes, until now, I have been focussing on the Model Voice file because that's what gets the restart debug data you need. Tonight, I tried the Voice Alert audio file name list, as you now suggested - first failure I got was 'snow lines' on the screen, which I could only clear by a power cycle - it did not progress to a watchdog restart within 20 seconds of scrolling and I worried about possible screen damage, so I did the power cycle. After the power cycle, another 'snow line' display occurred within about 5-7 seconds of scrolling and this immediately (less than 1 second) was followed by a full (i.e. what I have been seeing before) restart with the following info:
BOOT REASON WATCHDOG
01E2 020B 01ED
8103 0002 0204
2400 0000 0

DEBUG
0001 0000

I do believe that the restart occurs regardless of which file name list you are using for the scrolling.

I am having trouble tonight attaching a new sample of a restart video (which includes your new delays and with a bit longer 'scrolling time' before the restart, to let you look around a bit) - the openrcforums site keeps failing to move the file (it is only 16.9MB in size) -- here is a link to a safe Dropbox folder - please retrieve the file from there but let me know if you are having any trouble recovering it:

Link to restart video

regards,
ozphoenix

I know, I was going to ask this last night:
When you have the grey/white background does this:
1. Cover the whole display?
2. Is it the same as your set background colour?
3. What colour is the LED on the top of the radio? From the video you posted it seems stuck on a single colour (red or blue rather than purple which is both together).

If I could just get my radio to fail in the same way it would be easier to investigate!
I'm going to need to add some complex monitoring to work out what is happening.

Are you always using the "Model Voice" file list for these tests. If so, please try selecting a file for a voice alert.

Mike

[MikeB]

I can't upload a new test file either, it is being blocked.

Perhaps PM me your e-mail address.

Mike

[ozphoenix]

Done.

I can't upload a new test file either, it is being blocked.

Perhaps PM me your e-mail address.

Mike

[MikeB]

I've e-mailed a new test version. This has 2 changes.
While reading the filenames I've blocked any task scheduling.
I've added a timer generating an interrupt as an "early watchdog". This saves the stack in preserved memory at the point the watchdog would have rebooted the radio. The data is displayed on the debug menu, 4 rows of 8 numbers being the stack pointer value itself and 15 32-bit values from the stack. If you still get a restart, this may tell me where the processor was executing.

Mike

[jhsa]

I sent Rob an email about not being able to upload files. I am sure he will take care of it soon

João

[ozphoenix]

Got it - thanks!
Verified XJT off, got restart.
WATCHDOG
01E2 020B 01EF
8103 0002 0204
2400 0000 0

DEBUG
2000 2078 0000 0000 0000 0000 0400 0000
0000 0000 0000 000E 0804 5087 0802 C9DC
6100 0200 0804 5087 0000 3BEF 0000 0000
2000 0488 0000 0000 2000 6D4B 2000 2678

rechecked those numbers - they are as presented.

regards,
ozphoenix

I've e-mailed a new test version. This has 2 changes.
While reading the filenames I've blocked any task scheduling.
I've added a timer generating an interrupt as an "early watchdog". This saves the stack in preserved memory at the point the watchdog would have rebooted the radio. The data is displayed on the debug menu, 4 rows of 8 numbers being the stack pointer value itself and 15 32-bit values from the stack. If you still get a restart, this may tell me where the processor was executing.

Mike

[ozphoenix]

Thanks, João - much appreciated.
regards,
ozphoenix

I sent Rob an email about not being able to upload files. I am sure he will take care of it soon

João