Skip to content
Embedded Template Library
Source
Testing the ETL

Testing the ETL

This document describes how to build and run the ETL test suite locally, inside Dev Containers, and in CI.

Table of Contents

  1. Prerequisites
  2. Running Tests Locally (test/run-tests.sh)
  3. Syntax Checks (test/run-syntax-checks.sh)
  4. Cross-Architecture Testing (.devcontainer/run-tests.sh)
  5. Dev Containers for Native Compilers
  6. CMake Options Reference
  7. CI Checks (GitHub Actions)
  8. Appveyor (Windows / MSVC)
  9. Code Coverage
  10. Generator Tests (scripts/generator_test.py)

Prerequisites

  • CMake ≥ 3.10
  • GCC and/or Clang (any version supported by the project)
  • Make or Ninja (build backend)
  • Docker (only needed for cross-architecture testing via .devcontainer/run-tests.sh)
  • QEMU user-mode (installed automatically inside the cross-arch Docker images)

The project is header-only, so there is no library to compile – the build step compiles the test binary etl_tests which links against a bundled copy of UnitTest++.

Running Tests Locally

The main entry point for local testing is test/run-tests.sh. It iterates over a matrix of compiler / configuration combinations, creates a temporary build-make directory for each one, runs CMake + Make + CTest, and reports coloured pass/fail output (also appended to log.txt).

Synopsis 

cd test
./run-tests.sh <C++ Standard> [Optimisation] [Threads] [Sanitizer] [Compiler] [Verbose]
Argument Values Default
C++ Standard 11, 14, 17, 20, 23, or all (required)
Optimisation 0, 1, 2, 3 0
Threads any positive integer 4
Sanitizer s (enable) / n (disable) n (disabled)
Compiler gcc, clang all compilers
Verbose v (enable) / n (disable) n (disabled)

Examples 

# Run all C++17 tests with GCC only, optimisation -O0, 8 threads
./run-tests.sh 17 0 8 n gcc

# Run every standard with both compilers, sanitizers enabled, verbose
./run-tests.sh all 0 4 s "" v

What the script does

For every selected C++ standard the script loops over a built-in list of configurations (STL / No STL / Force C++03 / Non-virtual messages / …) for each selected compiler. For every combination it:

  1. Creates a fresh build-make directory inside the configuration’s source subdirectory.
  2. Invokes cmake with the appropriate -D flags (see CMake Options Reference).
  3. Builds via cmake --build . (parallel level controlled by CMAKE_BUILD_PARALLEL_LEVEL).
  4. Runs ctest -V.
  5. Reports success or failure and removes the build directory.

The script exits immediately on the first compilation or test failure.

Test configurations exercised

Compiler Configuration
GCC STL
GCC STL – Non-virtual messages
GCC STL – Force C++03
GCC No STL
GCC No STL – Force C++03
GCC No STL – Builtin mem functions
Clang STL
Clang STL – Force C++03
Clang No STL
Clang No STL – Force C++03
Clang No STL – Builtin mem functions
GCC / Clang Initializer list test
GCC / Clang Error macros – log_errors, exceptions, log_errors_and_exceptions, assert_function

Syntax Checks

The script test/run-syntax-checks.sh performs compilation-only syntax checks across multiple C++ standards and configurations. Unlike run-tests.sh, it does not run the test binary – it only verifies that the code compiles successfully. This is useful for quickly validating that header changes do not introduce compilation errors across the supported standard/configuration matrix.

Synopsis 

cd test
./run-syntax-checks.sh <C++ Standard> [Threads] [Compiler]
Argument Values Default
C++ Standard 03, 11, 14, 17, 20, 23, or a (all) (required)
Threads any positive integer 4
Compiler gcc, clang all compilers

Examples 

# Check C++17 syntax with GCC only, using 8 threads
./run-syntax-checks.sh 17 8 gcc

# Check all standards with both compilers
./run-syntax-checks.sh a

What the script does

The script operates from the test/syntax_check directory and iterates over the selected C++ standard(s). For each standard and compiler combination it:

  1. Creates fresh build directories (bgcc / bclang).
  2. Invokes cmake with the appropriate -D flags for the configuration.
  3. Builds via cmake --build.
  4. Reports compilation success or failure (logged to log.txt).

The script exits immediately on the first compilation failure.

Configurations checked per standard

For each C++ standard the following configurations are compiled:

Compiler Configuration
GCC STL
GCC No STL
GCC STL – Built-in traits
GCC No STL – Built-in traits
Clang STL
Clang No STL
Clang STL – Built-in traits
Clang No STL – Built-in traits

Cross-Architecture Testing

.devcontainer/run-tests.sh builds and runs the test suite for non-x86_64 architectures using Docker and QEMU user-mode emulation. It is designed to be run from the project root.

Supported architectures

Argument Target Endianness QEMU binary
armhf ARM hard-float (32-bit) Little qemu-arm-static
i386 x86 32-bit Little qemu-i386-static
powerpc PowerPC 32-bit Big qemu-ppc
riscv64 RISC-V 64-bit Little qemu-riscv64-static
s390x IBM Z (64-bit) Big qemu-s390x-static

Synopsis 

# From the project root
.devcontainer/run-tests.sh <architecture>

How it works

The script has two phases controlled by a second (internal) argument:

  1. Outside the container (no second argument):

    • Builds a Docker image from .devcontainer/<arch>/Dockerfile.
    • Starts a container, bind-mounting the project at /workspaces/etl.
    • Re-invokes itself inside the container with the inside_container flag.

  2. Inside the container (inside_container):

    • Creates build-<arch> and runs CMake with the appropriate cross- compilation toolchain file (.devcontainer/<arch>/toolchain-<arch>.cmake).
    • Builds with cmake --build . using all available cores.
    • Runs the test suite via ctest --output-on-failure.

The toolchain files set CMAKE_CROSSCOMPILING_EMULATOR so that CTest can run the binary transparently through QEMU.

Example 

# Build & run the armhf test suite
.devcontainer/run-tests.sh armhf

The cross-arch containers build with the following fixed settings:

  • C++23, No STL, sanitizer off, optimisation -O0.

Dev Containers for Native Compilers

The .devcontainer/ directory also provides Dev Container definitions for a wide range of native (x86_64) compiler versions. These are intended for use with VS Code Dev Containers or GitHub Codespaces.

Directory Compiler
gcc09 – gcc15 GCC 9 through 15
clang7 – clang21 Clang 7 through 21

Each subdirectory contains a devcontainer.json that references the shared Dockerfile (.devcontainer/Dockerfile) and passes the appropriate base Docker image via the BASE_IMAGE_NAME build argument (e.g. gcc:15).

The default Dev Container (.devcontainer/devcontainer.json) uses the Microsoft C++ dev-container base image.

To use one of these containers:

  1. Open the repository in VS Code.
  2. Ctrl+Shift+P → Dev Containers: Reopen in Container and select the desired configuration (e.g. Gcc 15).
  3. Use test/run-tests.sh inside the container as described above.

CMake Options Reference

When invoking CMake for the test suite (source directory is test/), the following -D options control the build:

Option Type Description
BUILD_TESTS BOOL Must be ON to compile the test binary.
NO_STL BOOL Build without the C++ Standard Library.
ETL_CXX_STANDARD STRING C++ standard: 11, 14, 17, 20, 23.
ETL_OPTIMISATION STRING Compiler optimisation flag, e.g. -O0.
ETL_ENABLE_SANITIZER BOOL Enable address / undefined-behaviour sanitizers.
ETL_USE_TYPE_TRAITS_BUILTINS BOOL Use compiler built-in type traits.
ETL_USER_DEFINED_TYPE_TRAITS BOOL Use user-defined type traits.
ETL_FORCE_TEST_CPP03_IMPLEMENTATION BOOL Force the C++03 code paths even on newer standards.
ETL_MESSAGES_ARE_NOT_VIRTUAL BOOL Use non-virtual message types.
ETL_USE_BUILTIN_MEM_FUNCTIONS BOOL Use built-in memory functions in No-STL mode.
CMAKE_TOOLCHAIN_FILE PATH Toolchain file for cross-compilation.

Minimal manual build example 

cd test
mkdir build && cd build
cmake -DBUILD_TESTS=ON -DNO_STL=OFF -DETL_CXX_STANDARD=20 ..
cmake --build . -j$(nproc)
ctest -V

CI Checks (GitHub Actions)

Every push or pull request to master, development, or pull-request/* branches triggers a comprehensive set of GitHub Actions workflows defined in .github/workflows/.

Workflow matrix

Workflow file Compiler Standard Notes
gcc-c++11.yml GCC C++11 STL, No STL, Force C++03
gcc-c++14.yml GCC C++14 STL, No STL, Force C++03
gcc-c++17.yml GCC C++17 STL, No STL, Force C++03
gcc-c++20.yml GCC C++20 STL, No STL, Force C++03
gcc-c++23.yml GCC C++23 STL, No STL, Force C++03
clang-c++11.yml Clang C++11 STL, No STL, Force C++03
clang-c++14.yml Clang C++14 STL, No STL, Force C++03
clang-c++17.yml Clang C++17 STL, No STL, Force C++03
clang-c++20.yml Clang C++20 STL, No STL, Force C++03
clang-c++23.yml Clang C++23 STL, No STL, Force C++03
gcc-syntax-checks.yml GCC C++03 – C++23 Compilation-only syntax checks (no tests run)
clang-syntax-checks.yml Clang C++03 – C++23 Compilation-only syntax checks (no tests run)
msvc.yml MSVC 2022 C++17 Windows, STL & No STL
gcc-c++23-armhf.yml GCC cross C++23 armhf via QEMU
gcc-c++23-i386.yml GCC cross C++23 i386 via QEMU
gcc-c++23-powerpc.yml GCC cross C++23 powerpc via QEMU
gcc-c++23-riscv64.yml GCC cross C++23 RISC-V 64 via QEMU
gcc-c++23-s390x.yml GCC cross C++23 s390x via QEMU
coverage.yml GCC — Generates lcov coverage report, deploys to GitHub Pages
generator.yml — — Runs the code generator
platformio-update.yml — — PlatformIO registry update

Typical CI job structure

Each compiler/standard workflow follows the same pattern:

  1. Checkout – actions/checkout@v4.
  2. Build – set CC/CXX, call cmake with the appropriate -D flags, then make -j.
  3. Run tests – execute ./test/etl_tests -v (or ctest -V for cross- arch jobs).

The cross-architecture CI jobs additionally install a cross-compiler toolchain and QEMU inside a debian:trixie container, use the matching toolchain file from .devcontainer/<arch>/, and run tests via CTest (which delegates to QEMU through CMAKE_CROSSCOMPILING_EMULATOR).

Branches tested

  • master
  • development
  • pull-request/*

All workflows run on both push and pull_request events (types: opened, synchronize, reopened).

Appveyor (Windows / MSVC)

The appveyor.yml at the repository root provides additional Windows CI using Visual Studio 2022. It builds the master branch only.

Configurations tested:

  • Debug MSVC C++14
  • Debug MSVC C++14 – No STL
  • Debug MSVC C++17
  • Debug MSVC C++17 – No STL
  • Debug MSVC C++20
  • Debug MSVC C++20 – No STL

The build uses the VS 2022 solution file at test/vs2022/etl.vcxproj.

Code Coverage

The coverage.yml GitHub Actions workflow generates an lcov coverage report:

  1. Runs test/run-coverage.sh which builds and tests with GCC coverage flags.
  2. Uploads the HTML report as a build artifact (retained for 30 days).
  3. On pushes to master, deploys the report to GitHub Pages.

To generate coverage locally:

cd test
./run-coverage.sh
# Open test/build-coverage/coverage/index.html

Generator Tests

The script scripts/generator_test.py verifies that the code generators in include/etl/generators/ produce output matching the checked-in header files in include/etl/private/.

ETL uses Cog to generate certain repetitive header files (e.g. delegate.h, message_packet.h). The generator templates live in include/etl/generators/*_generator.h and the generated output is committed to include/etl/private/*.h. This test ensures the two stay in sync.

Prerequisites

  • Python 3
  • cogapp – install via pip install cogapp

Synopsis 

python3 scripts/generator_test.py

What the script does

  1. Iterates over every *_generator.h file in include/etl/generators/.
  2. Runs Cog on each generator, outputting to build/generator_tmp/.
  3. Compares each generated file against the corresponding file in include/etl/private/.
  4. Reports success if all files match, or failure if any differ.

The script exits with code 0 on success and 1 on failure.

CI integration

The generator.yml GitHub Actions workflow runs this script automatically on pushes and pull requests to verify generator consistency.

Code Generation for Pre-C++11 Support