Skip to content

[C++] ENABLE_ASAN builds do not instrument TsFile sources #1062

Closed

Description

@ColinLeeo

Problem

ENABLE_ASAN=ON in C++-only builds (the unit-test-cpp Asan matrix) links the sanitizer runtime but does not compile TsFile sources with -fsanitize, so most memory-safety checks are inactive.

The compile flags are appended to CMAKE_CXX_FLAGS in the ENABLE_ASAN block, but that variable is overwritten right before add_subdirectory(third_party):

# cpp/CMakeLists.txt (ENABLE_ASAN block, before third_party)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fsanitize=${TSFILE_SANITIZER_CHECKS} -fno-omit-frame-pointer")
...
# later, unconditionally:
set(CMAKE_CXX_FLAGS "$ENV{CXXFLAGS} -Wall -std=c++${TSFILE_CXX_STANDARD}")
add_subdirectory(third_party)

# only the bindings mode re-applies instrumentation (added by #1047)
if (ENABLE_ASAN AND TSFILE_SANITIZER_SHARED_RUNTIME AND NOT MSVC AND NOT WIN32)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fsanitize=${TSFILE_SANITIZER_CHECKS} -fno-omit-frame-pointer -g")
endif ()

The C++-only jobs use -Denable.asan=ON with TSFILE_SANITIZER_SHARED_RUNTIME=OFF, so TsFile targets keep the reset flags. The executable still gets -fsanitize=... -static-libasan through CMAKE_EXE_LINKER_FLAGS, which is why the runtime is present at all.

MSVC is unaffected: it uses add_compile_options(/fsanitize=address /Zi), a directory property that survives the flag reset.

Measured impact

Verified on Ubuntu 20.04 / GCC 9.4 (same -static-libasan link mechanism as the GCC 13 CI image):

Compile Link Leak detection Heap-use-after-free
without -fsanitize -fsanitize=address -static-libasan (current C++ Asan mode) reported (exit 23) not detected (exit 0)
fully instrumented same reported reported

LeakSanitizer works through malloc/free interception and does not need compile-time instrumentation, so the current job still catches leaks and invalid frees. What is lost is every compiler-inserted check: heap-buffer-overflow, use-after-free/use-after-return, global-buffer-overflow, and all UBSan checks (-fsanitize=undefined instrumentation is never emitted, even though libubsan is linked).

Proposed fix

Re-apply the sanitizer compile flags for TsFile targets (not third_party) after add_subdirectory(third_party) in the static-runtime mode as well, and keep the runtime linkage consistent. The shared-library layout needs a decision, because a second private sanitizer runtime in libtsfile.so is not supported:

CI should also gain a negative control (a test that deliberately triggers a heap-buffer-overflow / use-after-free) so the Asan jobs prove that instrumentation is actually active, similar to the bindings positive controls.

Acceptance criteria

  • With -Denable.asan=ON (shared runtime off), TsFile sources are compiled with -fsanitize=${TSFILE_SANITIZER_CHECKS}.
  • A deliberately introduced use-after-free or heap-buffer-overflow in TsFile code fails the C++ Asan CI job.
  • Existing C++ Asan jobs stay green on Linux and macOS.

Context

Found while adding the Python binding sanitizer job in #1047, which fixed the same flag-reset issue for the bindings mode only. The follow-up is intentionally scoped separately because it changes how the sanitizer runtime is linked in the C++-only test builds.

Activity

  1. added this to the 2.5.1 milestone on Nov 3, 2026
  2. ColinLeeo commented on Nov 3, 2026

    @ColinLeeo
    ContributorAuthor

    Update: the Python binding sanitizer coverage landed via #1047, squash-merged to develop as 7b96800 (the branch used for that PR has been removed). This issue remains open as the C++-only follow-up.

  3. added a commit that references this issue on Nov 3, 2026
    6042764
  4. added a commit that references this issue on Nov 9, 2026
    934fdee
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions