19074: cpu/esp8266: build the SDK bootloader from source r=benpicco a=gschorcht
### Contribution description
This PR is a takeover of PR #17043, which is rebased to the current master and includes some corrections that became necessary after rebasing.
**Copied from description of PR #17043:**
We had four versions of pre-built bootloaders for the esp8266 with different settings of logging and color logging. These bootloaders were manually built from the SDK and shipped with RIOT-OS source code. However there are more settings that affect the bootloader build that are relevant to the app or final board that uses this bootloader. In particular, flash size and flash speed is important for the bootloader to be able to load an app from a large partition table at the fastest speed supported by the board layout and flash chip.
Another example is the UART baudrate of the logging output from the bootloader. The boot ROM will normally start at a baud rate of 74880 (depending on the crystal installed), so it might make sense to keep the UART output at the same speed so we can debug boot modes and bootloader with the same terminal.
This patch builds the `bootloader.bin` file from the ESP8266 SDK source code. The code is built as a module (`esp8266_bootloader`) which at the moment doesn't generate any object code for the application and only produces a `bootloader.bin` file set to the `BOOTLOADER_BIN` make variable for the `esptool.inc.mk` to flash.
The code needs to be compiled and linked with custom rules defined in the module's Makefile since the `bootloader.bin` is its own separate application.
The `BOOTLOADER_BIN` variable is changed from a path relative to the `$(RIOTCPU)/$(CPU)/bin/` directory to be full path. This makes it easier for applications or board to provide their own bootloader binary if needed.
As a result of building the bootloader from source we fixed the issue of having a large partition table.
### Testing procedure
Use following command to flash the application with STDIO UART baudrate of 115200 baud.
```
BAUD=74880 USEMODULE=esp_log_startup make -C tests/shell BOARD=esp8266-esp-12x flash
```
Connect with a terminal programm of your choice (unfortunatly `picocom` and `socat` don't support a baudrate close to 74880), for example:
```
python -m serial.tools.miniterm /dev/ttyUSB0 74880
```
On reset, the `esp8266-esp-12x` node shows the ROM bootloader log output
```
ets Jan 8 2013,rst cause:2, boot mode:(3,7)
load 0x40100000, len 6152, room 16
tail 8
chksum 0x6f
load 0x3ffe8008, len 24, room 0
tail 8
chksum 0x86
load 0x3ffe8020, len 3408, room 0
tail 0
chksum 0x79
```
as well as the second-stage bootloader built by this PR (`ESP-IDF v3.1-51-g913a06a9ac3`) at 74880 baudrate.
```
I (42) boot: ESP-IDF v3.1-51-g913a06a9ac3 2nd stage bootloader
I (42) boot: compile time 11:25:03
I (42) boot: SPI Speed : 26.7MHz
...
I (151) boot: Loaded app from partition at offset 0x10000
```
The application output is seen as garbage since the `esp8266-esp-12x` uses 115200 as baurate by default.
To see all output at a baudrate of 74880 baud, you can use the following command:
```
CFLAGS='-DSTDIO_UART_BAUDRATE=74880' BAUD=74880 USEMODULE=esp_log_startup make -C tests/shell BOARD=esp8266-esp-12x flash
```
If the application is built without options, the ROOM bootloader output will be 74880 baud and the second stage bootloader and application output will be 115200 baud.
### Issues/PRs references
Fixes issue #16402
Co-authored-by: iosabi <iosabi@protonmail.com>
Co-authored-by: Gunar Schorcht <gunar@schorcht.net>
Instead of using a fixed position of the image file in the flash, the variable `FLASHFILE_POS` is used which allows to override the default position of the image in the flash at 0x10000.
We had four versions of pre-built bootloaders for the esp8266 with
different settings of logging and color logging. These bootloaders were
manually built from the SDK and shipped with RIOT-OS source code.
However there are more settings that affect the bootloader build that
are relevant to the app or final board that uses this bootloader. In
particular, flash size and flash speed is important for the bootloader
to be able to load an app from a large partition table at the fastest
speed supported by the board layout and flash chip.
Another example is the UART baudrate of the logging output from the
bootloader. The boot ROM will normally start at a baud rate of 74880
(depending on the crystal installed), so it might make sense to keep
the UART output at the same speed so we can debug boot modes and
bootloader with the same terminal.
This patch builds the bootloader.bin file from the ESP8266 SDK source
code. The code is built as a module (esp8266_bootloader) which at the
moment doesn't generate any object code for the application and only
produces a bootloader.bin file set to the BOOTLOADER_BIN make variable
for the esptool.inc.mk to flash.
The code needs to be compiled and linked with custom rules defined in
the module's Makefile since the bootloader.bin is its own separate
application.
The `BOOTLOADER_BIN` variable is changed from a path relative to the
`$(RIOTCPU)/$(CPU)/bin/` directory to be full path. This makes it easier
for applications or board to provide their own bootloader binary if
needed.
As a result of building the bootloader from source we fixed the issue of
having a large partition table. Fixes#16402.
This commit includes the following changes:
- the default flash mode in bootloader should be dio and not dout
- the default flash configuration for ESP32 has to be exported to be also visible in cpu/esp32/bootloader/Makefile
- the comments in cpu/esp8266/Makefile.include have to be removed so that the flash configuration do not contain trailing white spaces
The former FLASH_MODE_{DOUT,DIO,QOUT,QIO} defines are replaced by the corresponding CONFIG_FLASHMODE_{DOUT,DIO,QOUT,QIO} and CONFIG_ESPTOOLPY_FLASHMODE_{DOUT,DIO,QOUT,QIO} as used by the ESP-IDF. This is also needed for the migration of defining flash mode in Kconfig.
Module `esp_idf_nvs_flash` uses C++ code. Since `esp_idf_nvs_flash` module is always enabled on ESP8266, the permanent dependency on `cpp` is correct. But on ESP32, the `esp_idf_nvs_flash` module is only enabled if `esp_wifi_any` is used. Only in that case the compilation should depend on module `cpp`.
If the WiFi interface is enabled by module `esp_wifi_any`, binary SDK libraries use the RTT. Therefore, `ztimer` must not use `periph_rtt`as backend, if the WiFi interface is enabled by module `esp_wifi_any`.
`ztimer_core` functions have to reside in IRAM for timing reasons and to be available also when the IROM cache is disabled. Although the module is called `ztimer`, its object files are generated in directory `ztimer_core`.
The semantics of defining an SSID prefix that overrides the already defined SSID exactly when and only when it is set, and then enabling dynamic SSID generation with that prefix, made handling the parameter definition unnecessarily difficult and hard to understand.
Defining a boolean option that enables dynamic SSID generation, which then simply reuses the defined SSID as a prefix, makes it much more understandable and easier to handle, especially with respect to Kconfig.
cpu/esp_common/syscalls.c💯 error (memleak): Memory leak: mtx
cpu/esp_common/syscalls.c:131: error (memleak): Memory leak: rmtx
cpu/esp_common/syscalls.c:365: error (comparePointers): Subtracting pointers that point to different objects
cpu/esp_common/thread_arch.c:355: error (comparePointers): Comparing pointers that point to different objects
cpu/esp8266/startup.c:59: error (comparePointers): Subtracting pointers that point to different objects
The functions of the `esp*/freetos` libraries must be placed in IRAM because they can be called when the IROM cache is disabled. While the functions implemented in `cpu/esp8266/freetos/*.c` are already placed in IRAM, the functions implemented in `cpu/esp_common/freetos/*.c` are not placed in IRAM. The reason for this is that the object files of these files are created in the `esp_freertos_common` directory, which is not included in the `esp.riot-os.ld` file because the library is named `esp_freertos_common`.
RIOT-OS uses part of Espressif ESP8266 RTOS SDK to build support for
this CPU. The SDK includes some vendor-provided closed source
pre-compiled libraries that we need to modify to adapt to RIOT-OS
usage. This library modifications was done once and uploaded to a fork
of the vendor repository and was provided as an environment variable.
This patch changes two things:
1. It installs the SDK as a RIOT PKG from the new pkg/esp8266_sdk
directory instead of requiring the user to download it separately.
2. It performs the library modifications (symbol renames) on the pkg
Makefile removing the need to use a fork with the modifications applied
and simplifying the SDK update and future modifications.
This change sets the SDK package version (git SHA) to the same one that
our fork was using as a parent in the vendor repository, meaning that
the output libraries are exactly the same as before.
Tested with
```
ESP8266_RTOS_SDK_DIR=/dev/null USEMODULE=esp_log_startup make -C tests/shell BOARD=esp8266-esp-12x flash
```
and verified that the program works. The boot message now includes:
```
ESP8266-RTOS-SDK Version v3.1-51-g913a06a9
```
confirming the SDK version used.
`/dev/null` in the test is just to make sure that no evaluation of
`ESP8266_RTOS_SDK_DIR` in make is affected by the environment variable
value which would be set to the SDK for people who followed the set up
instructions before this change.
Tested the checkout size:
```bash
$ du -hs build/pkg/esp8266_sdk/
124M build/pkg/esp8266_sdk/
```
