Taranis dropping telemetry packets

[mrexcitement]

Hi , i have made my own vario/accelerometer/voltage monitor which sends the data by the frsky protocol and it works fine on frsky's FLD-02 lcd display and i have also tested it with the android dashboard app and also no problems.
The FLD-02 was what i was using before i received my Taranis and it never missed a data packet but on my taranis it misses data packets.
I set up a test where i was sending a 1 second count out for all hub data, so on the Taranis all telemetry values should increment by 1 once a second.
This shows clearly when a packet doesn't show on the taranis telemetry screen, I even dropped it back so i was sending only the altitude packet (5e 10 01 00 5e 21 00 00 5e) incremented by 1 once a second.
In this test it seems to randomly miss packets on all telemetry data at different times, anywhere from 1 to 5 in a row.
Eg some telemetry values update while others haven't from the same data packet.
Same test on FLD-02 lcd and it never misses a packet.
I have also looked at the data stream coming out of the transmitter module (original setup dht module in hitec transmitter) using the fddlight window app and it shows all user data with none missing so i know i'm not
overloading the receivers buffer and it transmits all data with no missed packets.
I am using D4r-ii receiver and sending data in the original frsky protocol (not s.port).
I have experimented with different timings thinking it was a timing issue but the fact the taranis can't seem to reliably show the data packets even when it's just 1 value once a second.
I am wanting to use the data logging functions to record to sd card but with missing data this is not usable
I have been searching on the net for the last 2 weeks with no one talking about simular issue's, so i am thinking it's still in the way i send my data ( but works on FLD-02 ? )
I guess i'm asking if there are any known issue's like this or possible bugs in opentx for Taranis.

Any help would be greatly appreciated

Cheers

[gohsthb]

Have you tried logging the data to the SD card anyway? I think there might be some filtering in the display code.
-Gohst

[mrexcitement]

Actually i haven't checked if the data logging is recording without dropped packets, i will do this tonight.
I just presumed what i was seeing would be whats logged.
Thanks
Even with display filtering i wouldn't have thought it would stop updating for up to 5 seconds, i guess i will find out.

Cheers

[Kilrah]

Even if you use a D receiver the Taranis' internal module talks sport protocol. This updates less often, but still usually at least twice a second.
But it is not "transparent" as there is code in the module that converts the old data format to the new one. So if you're sending non "standard" values they might be dropped.

Sent from my SC-03E using Tapatalk

[mrexcitement]

Thanks for the suggestions, i actually wanted to use the S.PORT protocol but it's not available to the public yet.
I am using the frsky protocol as per the frsky protocol v2 data sheet and as far as i can tell all my values are within the ranges specified in the data sheet.
Yet it works fine on the FLD-02 LCD , which is what i did all my original development work on. I would have expected to see missing packets on the FLD-02 if i was sending non standard data.
I'll play some more tonight after work and reply if no joy.

Cheers

[mrexcitement]

Just did some logging tests and the actual log files just verified what I have seen on the Taranis display, missing data.
As per the Frsky protocol i'm sending frame 1 every 200ms and frame 2 every second.
I am logging at 0.2s.
As an example i'm sending altitude gps as a 1 second count and in the log It will show anywhere from 1 to 4 seconds of the second count not changing.
I presume the Frsky hub system non S.PORT works with the Taranis with no dropped packets ?
If so then I must be doing something different to Frsky.
Maybe has something to do with the conversion to S.Port inside the Taranis, wish they would release the S.Port protocol.
Anyways thanks for the input
I'll go over my code again and double check all values are within stated limits.

Cheers

[alex]

Hi, i saw a similar problem and i think i have a solution. I wanted to send the telemetry data from the Taranis to my PC. Therefore i modified the firmware on my Taranis so that it outputs the raw SPORT telemetry bytes on the serial port in the battery bay. When i decoded the SPORT frames on my PC i discoverd that there was a significant amount of corrupted packets.

I did some debugging and it turned out that the UART in the Taranis CPU that recieves the SPORT data sets the ORE-Flag (overrun error) from time to time. This is caused by the timer 1 interrupt blocking the uart receive interrupt for too long. The timer 1 interrupt (TIM1_CC_IRQn) controlls the generation of the PPX frames.

I wrote a patch that makes the timer 1 interrupt handler preemtable, so that it can no longer block the uart receive interrupt. For me, this fixes the problem with the corrupted SPORT frames. The patch is attached.
But i don't no if the patch has negative side effects for the generation of the PPX frames. It would be great if a opentx expert could have a look on this.

Alex.

[MikeB]

I've only had a very quick look at your patch, it seems you are raising the interrupt priority of the USART above the timer. Since the USART interrupt simply reads the data and puts it into a FIFO, it doesn't take long to execute.
The timer interrupts are not critical, the entire PXX frame is sent using DMA, the interrupt only triggers setting up the pulses and starting the frame transmission. For PPM, the interrupt does load the time of the next transisition, but the actual transisition is actioned by the hardware so again the interrupt isn't critical, just needs to be handled before the next edge, which is a minimum of 100uS later. I don't think the USART interrupt will take that long!

I did the original writing of the PXX/PPM generation.

Mike.

[Kilrah]

This is caused by the timer 1 interrupt blocking the uart receive interrupt for too long.

How did you determine that, and what openTx revision is your current tree based on?

We have had tons of problems with the ORE during development, but never really found the source of it, we could only patch it to "hide" it. Until we handled the ORE error we would actually even get occasional crashes/reboots. Now the ORE is handled and the CRC check on the SPORT data is implemented, which should in theory not let a wrong packet go through.

Of course the interrupt scenario was considered, but not only did our attempts at playing with priorities never change a thing but they should not be a problem in the first place - after all ORE only happens if the last received byte has not been read when the next one has finished arriving, which at 57600bps means a whopping 173us. There is nothing that should be holding an interrupt for that long, actually that CPU can calculate the entire mixer in about that time...

But I know there must still be an occasional problem as even myself get some random spurious false alarms such as an "A2 Low" warning when nothing would justify it (model on the ground with a full battery).

We'll try to test the patch anyway, there is still hope...

[MikeB]

I wasn't aware you had problems with the SPORT serial data. On ersky9x, I used DMA to guarantee data was read OK. It may well be possible (just had a quick look) to use the DMA on the STM to do the same thing. Run the DMA with double buffering and it runs continuously filling first one buffer, then the other, then back to the first. Poll the buffers/DMA and pick up the data that has arrived. The DMA buffers function as the FIFO instead of the one used in the interrupt.

Mike.

[MikeB]

Just put some debug in the "ersky9x for Taranis" I'm working on. I'm getting USART overrun errors (none of the other possible errors like noise). The USART interrupt is quite low down the default priority order, so if you get a SYSTICK interrupt, along with timer interrupts (PXX, trainer PPM etc), it could well get held off for 173 uS.

Mike.

[MikeB]

I've just added:
NVIC_SetPriority( TIM8_CC_IRQn, 2 ) ; // Lower priority interrupt
and similar for ALL the timer interrupts to do with pulse generation. The default priority is 0 (highest) for all interrupts.
Run the code for a couple of minutes and got NO overrun errors. This is on "ersky9x for Taranis", but the pulse generation code and the SPort interrupt code is pretty much the same as openTx.

In theory, we probably should set a specific interrupt priority for every interrupt that is enabled.

On ersky9x originally I did set the odd interrupt to a lower level.

Mike.

[bertrand35]

Good catch! I wonder if it would be worth to work a little bit on these pulses interrupts to have them as small as possible.

Bertrand

[MikeB]

I had a look at setupPulsesPXX to se if I could make it quicker and the answer was not really. I'm going to add a timer to it to see how long it is taking to execute, then I can look into speeding it up.

We have 16 priority levels for interrupts 0 (highest) to 15 (lowest). I suggest we add:
NVIC_SetPriority( XXXXX_IRQn, 7 ) ;
to every interrupt that is enabled, to set them all to a mid level, then we can adjust individual interrupts as required to a higher or lower level.

Mike.

[MikeB]

My test version of "ersky9x for Taranis" shows:
extern "C" void TIM1_CC_IRQHandler()
taking 206 uS to execute, not including the interrupt entry and exit overhead. The code is virtually the same as on openTx.
setupPulsesPXX()
takes a long time to run, and is called from this interrupt.

Mike.

Actually, my code is probably faster, I haven't implemented the second (external) module yet.
I've tweaked:
putPcmPart()
and saved 11 to 12 uS in execution time.

[Kilrah]

Cool, thanks.
Yep then it will really have to be taken out of the interrupt.

[MikeB]

I don't see the need to remove it from the interrupt, just lower the priority and it is OK.
I just forced:
putPcmPart()
to be always inline. The interrupt time dropped to 160uS at a cost of only 80 bytes of code.

Mike.

[alex]

How did you determine that, and what openTx revision is your current tree based on?

To find the reason for the OREs i made a short debug output in at the start and end of each interrupt and when the ORE occured. It turnd out that the timer 1 interrupt was always the last interrupt bevor the USART receive interrupt where the ORE flag was set. I did these tests with r2856, i think.

I don't see the need to remove it from the interrupt, just lower the priority and it is OK.

This is my first time working on the code for a ARM processor, but i think lowering the priority with NVIC_SetPriority is not enough. The USART receive interrupt has to be able to interrupt the running timer 1 interrupt (i.e. preempt it).

The NVIC has two sorts of priority. The subpriority is used to choose which interrupt is serviced first if two or more interrupt occur at the same time or are queued for execution. But only if a interrupt has a higher preemtion priority it can interrupt a running interrupt handler.
The stm32f4 has 4 Bits for the interrupt priority an with NVIC_PriorityGroupConfig you can define how many of those Bits are for the preemtion priority an how many are for the subpriority.
May be you already knew that, but as it took some time for me to understand it, i just wanted to mention it here.

And that is what i did in my patch: I configured the NVIC to use one Bit for preemtion priority and 3 Bit for subpriority and gave the USART receive interrupt a higher preemtion priority than the timer 1 interrupt.

Alex.

[alex]

Cool, thanks.
Yep then it will really have to be taken out of the interrupt.

That would be the best solution if it is possible.

Edit:

The interrupt time dropped to 160uS at a cost of only 80 bytes of code.

Great that should be fast enough (<173us). So you can forget about preemtion and just use NVIC_SetPriority().

Alex.

[MikeB]

This is my first time working on the code for a ARM processor, but i think lowering the priority with NVIC_SetPriority is not enough. The USART receive interrupt has to be able to interrupt the running timer 1 interrupt (i.e. preempt it).

Taken from "STM_M3progMan"
"When the processor is executing an exception handler, the exception handler is preempted if a higher priority exception occurs. If an exception occurs with the same priority as the exception being handled, the handler is not preempted, irrespective of the exception number. However, the status of the new interrupt changes to pending."

and
"To increase priority control in systems with interrupts, the NVIC supports priority grouping.
This divides each interrupt priority register entry into two fields:
● An upper field that defines the group priority
● A lower field that defines a subpriority within the group.
Only the group priority determines preemption of interrupt exceptions. When the processor is executing an interrupt exception handler, another interrupt with the same group priority as the interrupt being handled does not preempt the handler,"

So, as I understand it, setting the priority of the timer interrupt to 2, means all interrupts of priority 0 WILL preempt it.

This certainly seems to be the case for my code as I was getting many OREs before I changed the timer 1 priority (and timer 8 priority). After changing the priority I didn't get any OREs, and this was BEFORE I made the interrupt run faster.

We still need the preemption even if the interrupt is 160uS because other interrupts may also occur to hold the USART interrupt off. For interrupts that have the same priority, those with a lower exception number are processed first.

I still like the idea that ALL interrupts are set to a priority of 7 as a default, then specific interrupts are given a higher or lower priority as necessary. The processor default is thel ALL interrupts are priority 0.

Mike.

[alex]

Yes, you are right. I just found out that the reset value of the PRIGROUP field is 0b0000. For the stm32 that means you have only preemption priorities and no sub priorities. So priority 0 does indeed preemt priority 2.
I was a bit confused as when i first tried to decreas the priority of the timer 1 interrupt that didn't work for me. But i must have made another mistake back then. I just tried it again and it works.
Your idea about setting the priority to 7 as default sound good to me.

[mrexcitement]

Hi, this sounds very interesting , it could be the reason for why i am loosing telemetry packets. Is it possible to send me a firmware version with this incorporated so i can test ?

Cheers

[alex]

I built a firmware with the changed interrupt priorities MikeB suggested. It is based on r2834 with these changes:

0001-Define-Interrupt-priorities-to-prevent-SPORT-UART-re.patch

(5.93 KiB) Downloaded 237 times

I sent you the firmware via PM.

Alex.

[mrexcitement]

Thank you Alex, i will test and report back.

[mrexcitement]

It works, i now finally have reliable telemetry with my project, no dropped packets. I am very happy, thanks again Alex.
Just to be sure i tried the original firmware and yours and every time the original firmware dropped packets and yours doesn't.
I won't fly with it, i guess i will have to wait until it makes it into the next firmware release.

Cheers