Skip to content
Embedded Template Library
Source
Testing the ETL

Testing the ETL

This document describes how to build and run the ETL test suite locally, inside Dev Containers, and in CI.

Table of Contents

  1. Prerequisites
  2. Running Tests Locally (test/run-tests.sh)
  3. Syntax Checks (test/run-syntax-checks.sh)
  4. Cross-Architecture Testing (.devcontainer/run-tests.sh)
  5. Dev Containers for Native Compilers
  6. CMake Options Reference
  7. CI Checks (GitHub Actions)
  8. Appveyor (Windows / MSVC)
  9. Code Coverage
  10. Generator Tests (scripts/generator_test.py)

Prerequisites

  • CMake ≥ 3.10
  • GCC and/or Clang (any version supported by the project)
  • Make or Ninja (build backend)
  • Docker (only needed for cross-architecture testing via .devcontainer/run-tests.sh)
  • QEMU user-mode (installed automatically inside the cross-arch Docker images)

The project is header-only, so there is no library to compile – the build step compiles the test binary etl_tests which links against a bundled copy of UnitTest++.

Running Tests Locally

The main entry point for local testing is test/run-tests.sh. It iterates over a matrix of compiler / configuration combinations, creates a temporary build-make directory for each one, runs CMake + Make + CTest, and reports coloured pass/fail output (also appended to log.txt).

Synopsis 

cd test
./run-tests.sh <C++ Standard> [Optimisation] [Threads] [Sanitizer] [Compiler] [Verbose]
ArgumentValuesDefault
C++ Standard11, 14, 17, 20, 23, or all(required)
Optimisation0, 1, 2, 30
Threadsany positive integer4
Sanitizers (enable) / n (disable)n (disabled)
Compilergcc, clangall compilers
Verbosev (enable) / n (disable)n (disabled)

Examples 

# Run all C++17 tests with GCC only, optimisation -O0, 8 threads
./run-tests.sh 17 0 8 n gcc

# Run every standard with both compilers, sanitizers enabled, verbose
./run-tests.sh all 0 4 s "" v

What the script does

For every selected C++ standard the script loops over a built-in list of configurations (STL / No STL / Force C++03 / Non-virtual messages / …) for each selected compiler. For every combination it:

  1. Creates a fresh build-make directory inside the configuration’s source subdirectory.
  2. Invokes cmake with the appropriate -D flags (see CMake Options Reference).
  3. Builds via cmake --build . (parallel level controlled by CMAKE_BUILD_PARALLEL_LEVEL).
  4. Runs ctest -V.
  5. Reports success or failure and removes the build directory.

The script exits immediately on the first compilation or test failure.

Test configurations exercised

CompilerConfiguration
GCCSTL
GCCSTL – Non-virtual messages
GCCSTL – Force C++03
GCCNo STL
GCCNo STL – Force C++03
GCCNo STL – Builtin mem functions
ClangSTL
ClangSTL – Force C++03
ClangNo STL
ClangNo STL – Force C++03
ClangNo STL – Builtin mem functions
GCC / ClangInitializer list test
GCC / ClangError macros – log_errors, exceptions, log_errors_and_exceptions, assert_function

Syntax Checks

The script test/run-syntax-checks.sh performs compilation-only syntax checks across multiple C++ standards and configurations. Unlike run-tests.sh, it does not run the test binary – it only verifies that the code compiles successfully. This is useful for quickly validating that header changes do not introduce compilation errors across the supported standard/configuration matrix.

Synopsis 

cd test
./run-syntax-checks.sh <C++ Standard> [Threads] [Compiler]
ArgumentValuesDefault
C++ Standard03, 11, 14, 17, 20, 23, or a (all)(required)
Threadsany positive integer4
Compilergcc, clangall compilers

Examples 

# Check C++17 syntax with GCC only, using 8 threads
./run-syntax-checks.sh 17 8 gcc

# Check all standards with both compilers
./run-syntax-checks.sh a

What the script does

The script operates from the test/syntax_check directory and iterates over the selected C++ standard(s). For each standard and compiler combination it:

  1. Creates fresh build directories (bgcc / bclang).
  2. Invokes cmake with the appropriate -D flags for the configuration.
  3. Builds via cmake --build.
  4. Reports compilation success or failure (logged to log.txt).

The script exits immediately on the first compilation failure.

Configurations checked per standard

For each C++ standard the following configurations are compiled:

CompilerConfiguration
GCCSTL
GCCNo STL
GCCSTL – Built-in traits
GCCNo STL – Built-in traits
ClangSTL
ClangNo STL
ClangSTL – Built-in traits
ClangNo STL – Built-in traits

Cross-Architecture Testing

.devcontainer/run-tests.sh builds and runs the test suite for non-x86_64 architectures using Docker and QEMU user-mode emulation. It is designed to be run from the project root.

Supported architectures

ArgumentTargetEndiannessQEMU binary
armhfARM hard-float (32-bit)Littleqemu-arm-static
i386x86 32-bitLittleqemu-i386-static
powerpcPowerPC 32-bitBigqemu-ppc
riscv64RISC-V 64-bitLittleqemu-riscv64-static
s390xIBM Z (64-bit)Bigqemu-s390x-static

Synopsis 

# From the project root
.devcontainer/run-tests.sh <architecture>

How it works

The script has two phases controlled by a second (internal) argument:

  1. Outside the container (no second argument):

    • Builds a Docker image from .devcontainer/<arch>/Dockerfile.
    • Starts a container, bind-mounting the project at /workspaces/etl.
    • Re-invokes itself inside the container with the inside_container flag.

  2. Inside the container (inside_container):

    • Creates build-<arch> and runs CMake with the appropriate cross- compilation toolchain file (.devcontainer/<arch>/toolchain-<arch>.cmake).
    • Builds with cmake --build . using all available cores.
    • Runs the test suite via ctest --output-on-failure.

The toolchain files set CMAKE_CROSSCOMPILING_EMULATOR so that CTest can run the binary transparently through QEMU.

Example 

# Build & run the armhf test suite
.devcontainer/run-tests.sh armhf

The cross-arch containers build with the following fixed settings:

  • C++23, No STL, sanitizer off, optimisation -O0.

Dev Containers for Native Compilers

The .devcontainer/ directory also provides Dev Container definitions for a wide range of native (x86_64) compiler versions. These are intended for use with VS Code Dev Containers or GitHub Codespaces.

DirectoryCompiler
gcc09 – gcc15GCC 9 through 15
clang7 – clang21Clang 7 through 21

Each subdirectory contains a devcontainer.json that references the shared Dockerfile (.devcontainer/Dockerfile) and passes the appropriate base Docker image via the BASE_IMAGE_NAME build argument (e.g. gcc:15).

The default Dev Container (.devcontainer/devcontainer.json) uses the Microsoft C++ dev-container base image.

To use one of these containers:

  1. Open the repository in VS Code.
  2. Ctrl+Shift+P → Dev Containers: Reopen in Container and select the desired configuration (e.g. Gcc 15).
  3. Use test/run-tests.sh inside the container as described above.

CMake Options Reference

When invoking CMake for the test suite (source directory is test/), the following -D options control the build:

OptionTypeDescription
BUILD_TESTSBOOLMust be ON to compile the test binary.
NO_STLBOOLBuild without the C++ Standard Library.
ETL_CXX_STANDARDSTRINGC++ standard: 11, 14, 17, 20, 23.
ETL_OPTIMISATIONSTRINGCompiler optimisation flag, e.g. -O0.
ETL_ENABLE_SANITIZERBOOLEnable address / undefined-behaviour sanitizers.
ETL_USE_TYPE_TRAITS_BUILTINSBOOLUse compiler built-in type traits.
ETL_USER_DEFINED_TYPE_TRAITSBOOLUse user-defined type traits.
ETL_FORCE_TEST_CPP03_IMPLEMENTATIONBOOLForce the C++03 code paths even on newer standards.
ETL_MESSAGES_ARE_NOT_VIRTUALBOOLUse non-virtual message types.
ETL_USE_BUILTIN_MEM_FUNCTIONSBOOLUse built-in memory functions in No-STL mode.
CMAKE_TOOLCHAIN_FILEPATHToolchain file for cross-compilation.

Minimal manual build example 

cd test
mkdir build && cd build
cmake -DBUILD_TESTS=ON -DNO_STL=OFF -DETL_CXX_STANDARD=20 ..
cmake --build . -j$(nproc)
ctest -V

CI Checks (GitHub Actions)

Every push or pull request to master, development, or pull-request/* branches triggers a comprehensive set of GitHub Actions workflows defined in .github/workflows/.

Workflow matrix

Workflow fileCompilerStandardNotes
gcc-c++11.ymlGCCC++11STL, No STL, Force C++03
gcc-c++14.ymlGCCC++14STL, No STL, Force C++03
gcc-c++17.ymlGCCC++17STL, No STL, Force C++03
gcc-c++20.ymlGCCC++20STL, No STL, Force C++03
gcc-c++23.ymlGCCC++23STL, No STL, Force C++03
clang-c++11.ymlClangC++11STL, No STL, Force C++03
clang-c++14.ymlClangC++14STL, No STL, Force C++03
clang-c++17.ymlClangC++17STL, No STL, Force C++03
clang-c++20.ymlClangC++20STL, No STL, Force C++03
clang-c++23.ymlClangC++23STL, No STL, Force C++03
gcc-syntax-checks.ymlGCCC++03 – C++23Compilation-only syntax checks (no tests run)
clang-syntax-checks.ymlClangC++03 – C++23Compilation-only syntax checks (no tests run)
msvc.ymlMSVC 2022C++17Windows, STL & No STL
gcc-c++23-armhf.ymlGCC crossC++23armhf via QEMU
gcc-c++23-i386.ymlGCC crossC++23i386 via QEMU
gcc-c++23-powerpc.ymlGCC crossC++23powerpc via QEMU
gcc-c++23-riscv64.ymlGCC crossC++23RISC-V 64 via QEMU
gcc-c++23-s390x.ymlGCC crossC++23s390x via QEMU
coverage.ymlGCC—Generates lcov coverage report, deploys to GitHub Pages
generator.yml——Runs the code generator
platformio-update.yml——PlatformIO registry update

Typical CI job structure

Each compiler/standard workflow follows the same pattern:

  1. Checkout – actions/checkout@v4.
  2. Build – set CC/CXX, call cmake with the appropriate -D flags, then make -j.
  3. Run tests – execute ./test/etl_tests -v (or ctest -V for cross- arch jobs).

The cross-architecture CI jobs additionally install a cross-compiler toolchain and QEMU inside a debian:trixie container, use the matching toolchain file from .devcontainer/<arch>/, and run tests via CTest (which delegates to QEMU through CMAKE_CROSSCOMPILING_EMULATOR).

Branches tested

  • master
  • development
  • pull-request/*

All workflows run on both push and pull_request events (types: opened, synchronize, reopened).

Appveyor (Windows / MSVC)

The appveyor.yml at the repository root provides additional Windows CI using Visual Studio 2022. It builds the master branch only.

Configurations tested:

  • Debug MSVC C++14
  • Debug MSVC C++14 – No STL
  • Debug MSVC C++17
  • Debug MSVC C++17 – No STL
  • Debug MSVC C++20
  • Debug MSVC C++20 – No STL

The build uses the VS 2022 solution file at test/vs2022/etl.vcxproj.

Code Coverage

The coverage.yml GitHub Actions workflow generates an lcov coverage report:

  1. Runs test/run-coverage.sh which builds and tests with GCC coverage flags.
  2. Uploads the HTML report as a build artifact (retained for 30 days).
  3. On pushes to master, deploys the report to GitHub Pages.

To generate coverage locally:

cd test
./run-coverage.sh
# Open test/build-coverage/coverage/index.html

Generator Tests

The script scripts/generator_test.py verifies that the code generators in include/etl/generators/ produce output matching the checked-in header files in include/etl/private/.

ETL uses Cog to generate certain repetitive header files (e.g. delegate.h, message_packet.h). The generator templates live in include/etl/generators/*_generator.h and the generated output is committed to include/etl/private/*.h. This test ensures the two stay in sync.

Prerequisites

  • Python 3
  • cogapp – install via pip install cogapp

Synopsis 

python3 scripts/generator_test.py

What the script does

  1. Iterates over every *_generator.h file in include/etl/generators/.
  2. Runs Cog on each generator, outputting to build/generator_tmp/.
  3. Compares each generated file against the corresponding file in include/etl/private/.
  4. Reports success if all files match, or failure if any differ.

The script exits with code 0 on success and 1 on failure.

CI integration

The generator.yml GitHub Actions workflow runs this script automatically on pushes and pull requests to verify generator consistency.

Code Generation for Pre-C++11 Support