Possible new flashing method

[w4kac]

Great work Mike. Thanks!

[skylark]

THANK YOU MIKE. Genius can never be created, it is inbred. Great, I am up and running again. Stan.

[jtaylor]

Looks good Mike, Now a couple of usage questions. I compiled the code from Kilrah's git tree using BOOTLOADER=YES and I get a bootloader_ramBoot.bin file in the bootloader directory and opentx.bin in src. I assume I flash the bootloader.bin file and can then use it to flash the firmware. How do I download the bootloader? Companion? dfu-util?

Also, to remove the ramBoot bootloader and revert back to the old system is it just load old code with Companion (or new code compiled with BOOTLOADER=NO).

Jim

[MikeB]

Assuming Kilrah has followed my method (I believe he has), the bootloader is embedded into opentx.bin. Just flash the opentx.bin using dfu and the bootloader is in the first 32K of the flash.
There will need to be an update to Companion to use the bootloader fully.

Mike.

[Kilrah]

Yes I have. BOOTLOADER=YES builds the bootloader in its own folder, runs the python script to make the lbm that is put in opentx folder, includes it, and chooses the appropriate linker script.

[jtaylor]

Thanks for the explanations. I expected the BOOT=YES and BOOT=NO opentx.bin file sizes to be different (boot=yes to be 32k larger) and was surprised when they were the same size. Sorry for the stupid questions, I don't really understand this embedded stuff......but I'm trying.

Jim

[Kilrah]

Thanks for the explanations. I expected the BOOT=YES and BOOT=NO opentx.bin file sizes to be different (boot=yes to be 32k larger) and was surprised when they were the same size.

I do have the size difference here. But that's maybe because the option is BOOTLOADER=NO. As it's defined as YES by default in the makefile in that branch, if you added BOOT=NO in the command line you simply declared a new unused symbol instead of overwriting the default, and thus still had the bootloader compiled in.

[MikeB]

Currently, I understand, openTx, while implementing the bootloader I have written, are not using the specific code I produced that is dedicated to building the bootloader. Instead, they are using the same sources that are used to build openTx itself.

I specifically chose to keep all the sources for the bootloader separate from any application to make sure the bootloader remains VERY stable, and does NOT get modified accidently due to changes in the application.

I feel it is VERY important that the bootloader should remain UNCHANGED, unless there is a specific change required to its functionality.
The code I have has been quite well tested and, in my opinion, could be used as it is without needing any re-compiling.

I STRONGLY feel the bootloader is NOT part of the application, and should actually be treated as a separate program, and built as such. The resulting binary file is then just added at the front of the application for ease of distribution.

If the bootloader is re-compiled every time the application is compiled, then this will also mean it needs to be fully tested every time. If it is kept separate, then it will only need testing if it is changed.

Normally, with any processor with a bootloader, you don't change the bootloader every time you load the application!

Mike.

[bertrand35]

Today there is no bootloader in [next], so no risk!

I am ok with what you are stating above, but I don't like too much the idea of having 100+ duplicated files inside the same project. It seems to me unmaintainable. Instead I will be ok with the bootloader binary commited in the repository. Then we won't need to recompile it every time (or force the user to do it and break things).

[MikeB]

But my point is it isn't really the same project! That is why it SHOULD have its own source files, so they DON'T change when the application changes files, so they are NOT really duplicated, it IS a different project.

Mike.

[Rob Thomson]

Goto agree.

It is essentially a BIOS for a computer.

Installed. And updated ONLY if there is some sort of issue that is broken in it.

[MikeB]

So let's just use the bootloader from the sub-directory I have in ersky9x (creating it its own project if necessary), then you have a copy of the binary to include in the main firmware. You won't have ANY duplicated files then, they will be somewhere else.

Mike.

[kn4ud]

Great work Mike. I put my V3 together last February and have been using r159 since. I get back on this site after a year and find things I would have never dreamed of. I now have 106 on the coprocessor and r201 and it works great. Now this loading did come with some issues of errors on my part but me and the radio both survived it. All I can say is WOW, is there no limits to what an be done with this radio with the programming genius of MikeB.

[MikeB]

Sadly, the openTx team have ignored the suggestion to keep the bootloader compatible across platforms/firmware.
They have chosen to completely integrate the source files into openTx and then change various things to make it less compatible and less safe for users.
In the interest, it appears, of keeping fewer source files, they are including code in the bootloader that could erase the bootloader, and including code in the main program (by not using a separate maintenance mode) that could erase the main program.

By not implementing "maintenance mode" as a separate task that runs in isolation, flashing the bootloader becomes less safe, and more could go wrong, thus losing the bootloader and causing users to have to revert to Zadig.

Even simple things like the firmware sub-directory name. I created a perfectly good bootloader, and chose the name "Firmware". Instead of keeping this, they have chosen to change it to "FIRMWARES". On asking why, the only reason was "well all the other sub-directories are plural". This doesn't seem to be helpful to users to me. The bootloader, as far as I'm concerned, is supposed to be compatible across platforms, the original SKY boards, the 9XR-PRO and Taranis. I'm particularly concerned with the 9XR-PRO that will ship with a bootloader already installed. It means that if a user updates the bootloader from an openTx binary, they will find the firmware sub-directory they need to use now has a different name!

I personally, fail to see why the bootloader binary I produced originally couldn't have been used directly. I have provided the original sources on Github as their own repository, completely separate from any specific application code, here: https://github.com/MikeBland/OpenRcBootloader.

I mention all this because I spent quite a lot of time writing and testing the bootloader, and am happy it works reliably. Only changes that are specifically done to the bootloader sources on Github will affect the bootloader operation.

With the way the openTx team have integrated the bootloader with the openTx sources, there is now potential for changes to the application code to chnage the operation of the bootloader. It is therefore possible for the bootloader to change in a way that makes it unreliable, but that may not be noticed as users won't need to update the bootloader every time they update the main code. It could then cause bootloader problems when then bootloader is updated. Some of the code is from STM provided source file libraries. If these are updated, then again, the bootloader could change unexpectedly. I strongly feel, for the benefit of users, the bootloader should remain unchanged unless a specific bug is found that needs fixing.

I also find an element of "double standards". When FrSky were taking openTx and creating their own copy, the openTx team threw their hands up in horror at what was being done, yet they seem to be quite happy to do something similar in reverse in the case of the bootloader.

I hope the bootloader on Taranis, and the updating of the bootloader from the main application do work reliably, but I haven't tested either.

I have tried to convince them to use a bootloader that is the same across all hardware platforms, and all firmwares, but it seems to have fallen on deaf ears.

I do feel users deserve consideration, rather than developers saying "I don't want lots of duplicated files". With the sources elsewhere, they wouldn't have duplicated files anyway.

The bootloader has been used extensively by several users testing the Beta version of the 9XR-PRO. No problems reported, and no one saying it is difficult to use or needs improving.

Mike.

Edit: Another thing. I have code that prevents writing to the flash memory unless you start at the beginning with a valid file. This seems to no longer exist in the openTx version, so the bootloader could write to the flash memory when it shouldn't.

[Daedalus66]

I can't comment specifically on the planned OpenTX approach, but I can say that Mike's Ersky9x boot loader is brilliant and has worked flawlessly through numerous upgrades on the 9XR Pro. It is easy to understand and use. A blessed relief from SAM-BA and its drivers!

[jhsa]

+1.
Many, many updates, eeprom readings and writings, accessimg card nearly everyday, etc.
The thing is rock solid..

Joao

Sent from my GT-I9195 using Tapatalk

[bertrand35]

I don't think that anybody in OpenTX team removed any copyrights and broke any point of the GPL license (which FrSky did when we "threw our hands up in horror", and which is resolved now). We did a lot of small bugfixes (I sent you all the corrections by email), cosmetics (some screens were not nice on Taranis), and added some more features (today eeprom backup / restore) to the bootloader. I am still hoping that a 'maintenance mode' won't be needed at all, perhaps I am wrong, we will see later.

We forked from er9x for this exact same reason some time ago: we like to do our own thing. Yes we took a lot of code from er9x/ersky9x (which took a lot from th9x etc.), but I also see a lof of things that have been copied from OpenTX. That's what GPL allows and that's good!

Bertrand.

EDIT: yes the flash binary validity is checked

[MikeB]

It looks like you have removed the EEPROM backup. I certainly don't think the bootloader is a place for such a function, maybe EEPROM restore is OK in the bootloader.

You may check the binary validity, but I believe you have removed the flash writing block I included. When using the USB from the PC, the flash memory is mapped to a file. It is quite possible for the PC to do a random access to this file, and so write to the flash memory. With my blocking code, this would be blocked, thus protecting the firmware. The flash memory is only unblocked if writing starts at the beginning with a valid file.

I see you have also removed the delay I included to only update the display every 20mS, instead of every 10mS. I put this in because the bit bashing of the display update took longer than 10mS, so the 10mS timing would become upset over time. Has this been removed because of the processor clock change to 120Mhz and the display updates faster? If so, I would ask if you have checked a datasheet for the display to be certain it it will work reliably at the resulting faster clock speed. I've never seen a datasheet for the LCD.

Off topic, but related to the processor clock change, has the ADC timing been checked. I think the ADC clock has been doubled, so all the sample times etc. will become halved. This may compromise the accuracy of the resulting conversions.

I take it users are not going to be helped by rationalising the firmware directory name to "FIRMWARE" and openTx will persist in using "FIRMWARES" simply on the grounds of "all directories are plural".

Regarding "maintenance mode", I strongly feel the operation of updating the bootloader should be considered a "special" operation, and therefore not something users can easily access, possibly by mistake. "maintenance mode" provides such an operation, in addition to avoiding potential problems of trying to write to flash memory with the rest of the firmware operating. Subsequently adding the ability to update SPort devices will also be much simpler if it doesn't have to "fight" for the required hardware resources. I also think this should only be accessed in a "special" mode.

"We like to do our own thing" is perfectly reasonable, in my opinion, but provided it is qualified with "for the benefit of the users".
"We like to do our own thing" because we want to and users don't matter if it confuses them is rather less reasonable.

Another change done to the bootloader to make it openTx specific only is in "isEepromStart()" to validate the start of an EEPROM write. I have chosen a different EEPROM file block size for ersky9x for Taranis. The bootloader code I have supports both EEPROM formats, openTx bootloader only supports openTx, yet again, WHY?

Mike.

[bertrand35]

It looks like you have removed the EEPROM backup. I certainly don't think the bootloader is a place for such a function, maybe EEPROM restore is OK in the bootloader.

The EEPROM backup is inside the firmware (General Settings - VERSION page), not inside the bootloader. Instead this is the EEPROM restore which is in the bootloader.

Another change done to the bootloader to make it openTx specific only is in "isEepromStart()" to validate the start of an EEPROM write. I have chosen a different EEPROM file block size for ersky9x for Taranis. The bootloader code I have supports both EEPROM formats, openTx bootloader only supports openTx, yet again, WHY?
Mike.

That was not done on purpose! I will have a look to the ersky9x/Taranis EEPROM format later today.

[MikeB]

My tests:

openTx:
size = buffer[1] + ( buffer[2] << 8 ) ;
bs = buffer[5] ;
if ( ( size == 0x0100 ) && ( bs == 0x40 ) )

Ersky9x:
size = buffer[1] ;
bs = buffer[3] ;
if ( ( size == 0x0080 ) && ( bs == 0x80 ) )

Again, I recommend replacing the "FlashBlocked" variable and associated code to provide protection for the firmware.

Mike.

[MikeB]

I had missed off blocking the EEPROM in the X9D/Taranis version of the bootloader. The Github source files now have this omission fixed.
This means that writing to the EEPROM memory is blocked, particularly from a PC on the USB connection, until writing starts with a valid first block.

This provides added protection to the EEPROM against attempts to write randomly to it using the file image.
Users could accidently write to the file images incorrectly. With the disk containing the EEPROM and Firmware files displayed, a user can simply use a delete operation to apparently remove the files. This won't actually delete any data, but would then allow the PC to write to both EEPROM and flash memory. Copying a file to the disk could then write completely incorrect data to both the EEPROM and flash memories.
The blocking code helps prevent this, as both EEPROM and Flash would not be written.

It is strongly recommended that this feature should be replaced/added to the openTx Taranis bootloader, or, the Github bootloader sources used to make things compatible.

I've also reduced the display update rate to 25 times/second on the Taranis. This provides more processor time for real processing, and the user doesn't actually see any difference.

Mike.

[bertrand35]

It is strongly recommended that this feature should be replaced/added to the openTx Taranis bootloader, or, the Github bootloader sources used to make things compatible.

If my memory is right, I already did it

[bertrand35]

I've also reduced the display update rate to 25 times/second on the Taranis. This provides more processor time for real processing, and the user doesn't actually see any difference.

Last measurements on OpenTX showed this same refresh rate. Anyhow the mixer is not in the same task than the LCD refresh in OpenTX, and has a nicer priority.

[bertrand35]

Off topic, but related to the processor clock change, has the ADC timing been checked. I think the ADC clock has been doubled, so all the sample times etc. will become halved. This may compromise the accuracy of the resulting conversions.

Yes this has been checked.

[bertrand35]

"We like to do our own thing" is perfectly reasonable, in my opinion, but provided it is qualified with "for the benefit of the users".

Indeed!
https://github.com/opentx/opentx/issues/milestones

[MikeB]

Off topic, but related to the processor clock change, has the ADC timing been checked. I think the ADC clock has been doubled, so all the sample times etc. will become halved. This may compromise the accuracy of the resulting conversions.

Yes this has been checked.

This hasn't changed:
#define SAMPTIME 2 // sample time = 15 cycles
This hasn't changed:
ADC->CCR = 0 ; //ADC_CCR_ADCPRE_0 ; // Clock div 2

But isn't the peripheral bus clock doubled? So the 15 cycles sample time has been halved.

The LCD refresh rate I'm refering to is in the bootloader. There is no mixer and no tasks running there so I don't understand your reference.

Regarding users, I'm refering to the firmware directory name. It is too late for me to change as I have released code to HK for the 9XR-PRO with the bootloader in it, also for the SKY boards. This uses the name "Firmware". Surely it makes sense for openTx to use the same name. If users of the 9XR-PRO want to flash openTx, they will be using the bootloader already in it. This is such a simple thing to make it easier for users, why insist on a different name?

Don't forget to add the same blocking protection to the EEPROM.

Mike.

[Kilrah]

Peripheral clock has not been doubled. The buses were already running at max rated speed, so the doubling of the main clock was compensated by a halving of the peripheral bus clocks. So only the CPU core is running faster.

EEPROM check is present.

[MikeB]

OK, good on the ADC.

I can see an EEPROM check, but no code to block, unlike the flash. I only added the EEPROM block to the X9D/Taranis code yersterday, it was in the SKY versions, but I missed it in the X9D/Taranis version, so was never in the code being used in openTx.

I would also like to see the flash_driver.cpp file at least conditionally compiled so the bootloader only includes code to erase the main application blocks and the main application only has code to erase the bootloader blocks. It is added protection against anything going wrong.

Mike.

[Kilrah]

The EEPROM write lock is part of the underlying file system code that was already there when mass storage was in the main fw.

Regarding the flash erase it is indeed easy to take out the unrelated ifs for each side in the erase function, and can agree about the "good practice".
But on the other hand, my point of view is that considering that as a serious point of failure is heavily overrated... If we can't trust a rather simple piece of code not to go erase a block it shouldn't just because it has the possibility of doing so, then how could we trust the firmware and its much more complex ramifications and potential failure points to fly a model at all...
When put in perspective the highly improbable scenario of the bootloader erasing itself, which would lead to the minor nuisance of having to reflash it with dfu-util is a million of times less likely than a little bug somewhere that could cause a crash of a real model... I.e. It's hard to understand why the reason to get all crazy about it


Envoyé de mon iPad à l'aide de Tapatalk

[ReSt]

Believe in Murphy:
if an error might happen, it will happen.
and it will happen at a time where you never, never expect it to happen.

Isn't it good practice to solve possible errors as soon as possible and then forget about them?

Reinhard