Cephtrace v1.5 Release Notes

Release Date: January 15, 2026

Overview

Cephtrace v1.5 introduces ARM64 architecture support, critical bug fixes for Ceph 19.2.0+ compatibility, and expanded platform coverage with new DWARF JSON files. This release also brings improved testing infrastructure with MicroCeph-based functional tests.

What's New

ARM64 Architecture Support

• Full ARM64/AArch64 Support - Cephtrace now builds and runs on ARM64 systems:
  • Added user_pt_regs support for ARM64 in BPF utilities
  • Implemented ARM64-specific fetch_register function
  • Added ARM64 architecture macros in Makefile
  • GitHub workflow now includes Ubuntu ARM64 build verification

Bug Fixes

• Ceph 19.2.0+ Compatibility - Fixed OSDOp extraction bug in radostrace for Ceph versions >= 19.2.0 by using BPF global variables for SIZE/OFFSETS instead of MICROS
• Timeout Behavior - Fixed osdtrace/radostrace to start timeout countdown after initialization completes, preventing premature exits during slow startups
• Zero Function Address Handling - Added warning and skip behavior when func_addr is zero, preventing crashes
• Output Formatting - Fixed missing space between pg and acting columns in radostrace output

Platform Support

• Debian Trixie (13) - Added DWARF JSON files for Debian's upcoming stable release
• Ceph Caracal 19.2.3 - Added DWARF JSON files for the latest Caracal point release (both Ubuntu and UCA)

Testing Infrastructure

• MicroCeph Functional Tests - New end-to-end functionality tests for radostrace and osdtrace using MicroCeph:
  • Automated test execution in GitHub Actions Ubuntu runners
  • Improved test isolation with pre-installed Ceph package cleanup
  • OSD memory optimization for CI environments

Breaking Changes

None. This release maintains backward compatibility with v1.4.

Contributors

Special thanks to all contributors who made this release possible:

• @taodd - ARM64 support, bug fixes, and core development
• @pponnuvel - MicroCeph testing infrastructure and DWARF JSON files
• @alanbach - Debian Trixie DWARF files

Installation

From Source

git clone https://github.com/taodd/cephtrace.git
cd cephtrace
git checkout v1.5
git submodule update --init --recursive
make
sudo make install

Documentation

• Getting Started Guide
• radostrace Documentation
• osdtrace Documentation
• kfstrace Documentation
• DWARF JSON Files Guide

Requirements

• Linux kernel 5.8+
• Ceph Octopus (15.x) - Squid (19.x)
• For building: clang, gcc, libelf-dev, libdw-dev
• Supported architectures: x86_64, ARM64

Known Issues

None at this time.

What's Next

Looking ahead to v1.6:

• Additional Ceph version support
• More analysis tool features
• Extended platform support

Support

• Report issues: https://github.com/taodd/cephtrace/issues
• Documentation: https://github.com/taodd/cephtrace

Full Changelog: v1.4...v1.5

Cephtrace v1.4 Release Notes

Release Date: November 23, 2025

Overview

Cephtrace v1.4 brings significant improvements to testing infrastructure, build system robustness, platform support, and analysis tools. This release focuses on better developer experience, broader platform compatibility, and enhanced testing capabilities.

What's New

Testing Infrastructure

• JSON-based DWARF Comparison - Implemented a comprehensive Python-based JSON comparison tool (tests/compare_dwarf_json.py) replacing plain diff commands, providing:

  • Structured comparison of function-to-program-counter mappings (func2pc)
  • Detailed validation of function-to-variable-fields mappings (func2vf)
  • Color-coded, human-readable diff output
  • Better error messages for debugging test failures

• Enhanced Test Coverage

  • Added CentOS Stream/RHEL testing support (dwarf-compare-centos.sh)
  • Expanded radostrace DWARF JSON file comparison tests
  • Improved pytest integration with conftest for sample data management
  • Added end-to-end tests for osdtrace analysis tools

• CI/CD Improvements

  • Added GitHub Actions workflow for Rocky Linux builds
  • Integrated tox-based linting and testing in CI pipeline
  • Added multi-release support (Jammy and Noble) in test matrix

Build System

• Compiler Warnings - Relaxed -Werror flag to prevent build failures from warnings while maintaining code quality through linting
• Dynamic BTF Header Generation - Improved ceph_btf_local.h generation from available ceph kernel modules
• Parallel Build Support - Fixed issues preventing parallel builds

Platform Support

• CentOS Stream/RHEL

  • Added version detection utilities for RPM-based systems
  • Included DWARF JSON files for CentOS Stream (18.2.7, 19.2.3)
  • Automated package installation via cephadm in tests

• Ubuntu

  • Added DWARF JSON support for additional Ubuntu versions
  • Improved debug symbol fetching and handling

• MicroCeph - Enhanced support for tracing MicroCeph snap-based deployments with improved library path detection

Analysis Tools

• osdtrace Analysis Enhancements
  • New "infer" method for better latency analysis
  • Added threshold filtering capabilities
  • Fixed percentile display bugs
  • Improved CSV file compatibility while maintaining backward compatibility
  • Better sample data with reduced file sizes

Documentation

• Restructured Documentation
  • Flattened doc directory for easier navigation
  • Added comprehensive guides for:
    • Tracing containerized Ceph (cephadm/rook, Docker, LXD)
    • Tracing MicroCeph snap deployments
  • Enhanced README with Cephalocon 2025 presentation links
  • Added/updated man pages for osdtrace, radostrace, and kfstrace
  • Fixed broken links in documentation

Core Improvements

• DwarfParser - Now uses base name instead of full path as module key for better portability
• kfstrace - Added latency threshold option with unsafe reply logic
• Error Handling - Improved error messages and handling throughout the codebase

DWARF JSON Files

Added support for new Ceph versions:

• Ubuntu: 17.2.9, 19.2.0
• CentOS Stream: 18.2.7, 19.2.3

Installation

From Source

git clone https://github.com/taodd/cephtrace.git
cd cephtrace
git checkout v1.4
git submodule update --init --recursive
make
sudo make install

Debian/Ubuntu Package

# Will be available on package repositories soon

Documentation

• Getting Started Guide
• radostrace Documentation
• osdtrace Documentation
• kfstrace Documentation
• DWARF JSON Files Guide

Full Changelog: v1.3...v1.4

This is a special release for holding sample logs

Release v1.3

🎉 Major Features

New Tool: kfstrace - Kernel-level Ceph Client Tracing

A powerful new tracing tool that monitors Ceph kernel client operations using kprobes, providing visibility into both data and metadata operations.

Key Features:

• No DWARF files required - Works out of the box without debug symbols
• Dual-mode tracing:
  • OSD mode: Traces data operations (reads/writes) to Ceph OSDs
  • MDS mode: Traces metadata operations (lookup, create, setattr, etc.) to MDS servers
  • All mode: Traces both OSD and MDS operations simultaneously
• Detailed metrics:
  • Per-operation latency tracking
  • Two-phase reply timing for MDS operations (unsafe and safe replies)
  • Process information (PID, command name)
  • Object names and detailed operation types
  • Retry attempt counters
• Use cases:
  • CephFS kernel mount performance analysis
  • Kernel RBD client debugging
  • Metadata operation latency investigation

Example usage:

sudo ./kfstrace              # Trace MDS requests (default)
sudo ./kfstrace -m osd       # Trace OSD data operations
sudo ./kfstrace -m all       # Trace both OSD and MDS
sudo ./kfstrace -t 30 -m all # Trace for 30 seconds

Container Support - CentOS Stream

Full support for tracing Ceph processes running inside containers, particularly for cephadm-deployed clusters.

Features:

• Pre-generated DWARF JSON files for CentOS Stream 9 with Ceph 19.2.3
• Process ID targeting with -p <PID> flag
• Version check bypass with --skip-version-check flag
• Comprehensive documentation for containerized tracing workflows

Supported configurations:

• CentOS Stream 9 with Ceph 19.2.3 (pre-generated DWARF files available)
• Any other version (users can generate their own DWARF files)

🔧 Major Improvements

Enhanced Container Process Tracing

• radostrace: Improved library path resolution for containerized processes
• osdtrace: Enhanced process detection and validation for container environments
• Library search path improvements: Added /lib64/ceph to search directories for CentOS Stream
• Root directory handling: Proper chroot support when attaching uprobes to containerized processes

Process Validation & Safety

• Process ID validation to ensure correct process type is being traced
• Check for deleted executables and warn users appropriately
• Improved error messages when tracing wrong process types
• Better handling of edge cases in container environments

DWARF File Management

• Reorganized structure: Files now organized by distribution
  • files/ubuntu/radostrace/ - Ubuntu radostrace DWARF files
  • files/ubuntu/osdtrace/ - Ubuntu osdtrace DWARF files
  • files/centos-stream/radostrace/ - CentOS Stream radostrace DWARF files
  • files/centos-stream/osdtrace/ - CentOS Stream osdtrace DWARF files
• Updated DWARF files: Refreshed for v1.3 binary compatibility
• Version coverage: Added support for newer Ceph versions on Ubuntu

osdtrace Enhancements

• Combined output modes: Full mode (-x) now includes bluestore details automatically
• Enhanced bluestore tracing:
  • Added trace to log_latency_fn function
  • Display all types of bluestore layer latency
  • Added OSD ID in bluestore mode output
• Improved latency tracking:
  • Added mark_flag_point support for Quincy and later
  • Better mark point positioning for accurate measurements
• Better process handling: Only traces specified process when -p is provided

Analysis Tools

Three new/enhanced analysis scripts for post-processing trace data:

1. osdtrace Analysis Script (tools/analyze_osdtrace.py)

• Field filtering and selection
• OSD ID filtering
• Sorting capabilities
• BlueStore operation analysis
• Contributing factor identification
• Statistical summaries

2. radostrace Analysis Script (tools/analyze_radostrace_output.sh)

• Improved output formatting
• Better data extraction
• Enhanced readability

3. File Extension Fixes

• Proper .py extension for Python scripts (better syntax highlighting)
• Executable permissions set correctly

📚 Documentation Improvements

Comprehensive Quick Start Guide

• Ubuntu users: Pre-built binaries with DWARF files
• CentOS Stream users: Container tracing guide with examples
• All three tools: Step-by-step usage instructions

Detailed Output Explanations

• radostrace output: Column-by-column explanation
• osdtrace output: Latency stage breakdown
• kfstrace output: Both OSD and MDS mode explanations
  • OSD mode: Data operation details
  • MDS mode: Two-phase reply protocol explanation

Container Tracing Guide

• Finding process IDs in containerized environments
• Using -p and --skip-version-check flags
• Generating custom DWARF files for unsupported versions
• Important notes about using actual process IDs (not container PIDs)

🐛 Bug Fixes

• Fixed chroot bug in radostrace when tracing containerized processes
• Fixed OSD ID extraction from /proc/<pid>/cmdline
• Fixed DWARF parser marking flag point tracing issues
• Improved library path resolution with proper root directory handling
• Fixed deleted ceph-osd executable detection
• Suppressed verbose cout output from DWARF parser

🔨 Technical Changes

Build System

• Special build rules for kfstrace (doesn't need dwarf_parser)
• BTF generation from ceph kernel module instead of libceph
• Updated Makefile with proper dependencies

Code Quality

• Better error handling and validation
• Improved process detection logic
• Enhanced library search mechanisms
• Code cleanup and restructuring

📦 Available DWARF Files

Ubuntu

Multiple Ceph versions supported across Ubuntu releases:

• Ubuntu 20.04: Ceph 15.2.x
• Ubuntu 22.04: Ceph 17.2.x series (17.2.5, 17.2.6, 17.2.7, 17.2.9)
• Ubuntu 24.04: Ceph 19.2.x series

CentOS Stream 9

• Ceph 19.2.3 (radostrace and osdtrace)

Check the files/ directory for the complete list of available DWARF files.

📋 Requirements

• Kernel: v5.8 or later
• Privileges: Root access required
• For containerized tracing: Host-level access to trace container processes

🙏 Contributors

Special thanks to:

• @pponnuvel - Fix file extension for radostrace analyser
• @nishant-dash - Enhanced osdtrace analysis script
• @alanbach - Debian watch update

📈 Statistics

• 63 commits since v1.2
• 1 new tool added (kfstrace)
• 3 analysis tools improved/added
• 2 platforms now supported (Ubuntu + CentOS Stream)
• Extensive documentation updates

🔗 Installation

Pre-built Binaries (Recommended)

Download the latest binaries directly:

# radostrace
wget https://github.com/taodd/cephtrace/releases/download/v1.3/radostrace
chmod +x radostrace

# osdtrace
wget https://github.com/taodd/cephtrace/releases/download/v1.3/osdtrace
chmod +x osdtrace

# kfstrace (NEW!)
wget https://github.com/taodd/cephtrace/releases/download/v1.3/kfstrace
chmod +x kfstrace

DWARF Files

Download pre-generated DWARF files for your platform:

# Ubuntu example (Ceph 17.2.7)
wget https://raw.githubusercontent.com/taodd/cephtrace/main/files/ubuntu/radostrace/17.2.7-0ubuntu0.22.04.2_dwarf.json
wget https://raw.githubusercontent.com/taodd/cephtrace/main/files/ubuntu/osdtrace/osd-17.2.7-0ubuntu0.22.04.2_dwarf.json

# CentOS Stream example (Ceph 19.2.3)
wget https://raw.githubusercontent.com/taodd/cephtrace/main/files/centos-stream/radostrace/rados-2:19.2.3-0.el9_dwarf.json
wget https://raw.githubusercontent.com/taodd/cephtrace/main/files/centos-stream/osdtrace/osd-2:19.2.3-0.el9_dwarf.json

Build from Source

git clone https://github.com/taodd/cephtrace
cd cephtrace/cephtrace
git checkout v1.3
git submodule update --init --recursive
make

🚀 What's Next

Future plans include:

• More pre-generated DWARF files for CentOS Stream versions
• Additional analysis capabilities
• Performance optimizations
• Enhanced filtering options

Full Changelog: v1.2...v1.3

CephTrace v1.2 Release Notes

🎉 What's New

🔍 Version Compatibility Checking on osdtrace

• Automatic version mismatch detection: Both osdtrace and radostrace now automatically detect version mismatches between DWARF JSON files and target systems
• Smart error handling: Clear warnings when running processes with outdated/deleted executables
• Enhanced user guidance: Better error messages help users understand when to restart services

📦 Debian Package Support

• Full debian packaging: Complete debian packaging infrastructure with proper dependencies
• Installation targets: make install support for system-wide installation
• Man pages: Professional man pages for both osdtrace and radostrace
• Clean build system: Improved Makefile with package generation support

📊 Enhanced Data Export

• CSV export for radostrace: Export tracing data to CSV format for analysis

🚀 Ubuntu Quick Start

• Pre-built binaries: Ready-to-use binaries for Ubuntu users via GitHub releases
• No-installation tracing: Start tracing immediately with pre-generated DWARF files
• Zero dependencies: No need for debug symbols or build tools

🛠️ Improvements

Command Line Interface

• Fixed parameter mapping:
  • -t now properly used for timeout (was conflicting with latency)
  • -l now used for latency threshold
• Inline help support: Added -h flag with comprehensive help text
• Better error handling: More informative error messages and user guidance

Documentation

• Comprehensive README: Complete rewrite with quick start guide and detailed examples
• osdtrace output explanations: Detailed breakdown of all latency stages and operation types
• Usage examples: Real-world examples with actual output samples
• Better organization: Logical flow from installation to advanced usage

📦 Installation

Ubuntu Quick Start (Recommended)

# Download pre-built binary
wget https://github.com/taodd/cephtrace/releases/download/v1.2/radostrace
chmod +x radostrace

# Download DWARF file for your Ceph version
wget https://raw.githubusercontent.com/taodd/cephtrace/main/files/radostrace/17.2.6-0ubuntu0.22.04.2_dwarf.json

# Start tracing immediately
sudo ./radostrace -i 17.2.6-0ubuntu0.22.04.2_dwarf.json

Build from Source

git clone https://github.com/taodd/cephtrace
cd cephtrace
git submodule update --init --recursive
make
sudo make install

Full Changelog: v1.1...v1.2

Download: Release Assets

Report Issues: GitHub Issues

There are situations when people upgrades package in disk but forget to restart the process
The release enforces the version consistency between the imported dwarf json file and in-memory library version, if a mismatch is detected, it will report a message advising a restart and exit.

Initial release of radostrace.
Compatible with Ubuntu Focal (HWE kernel), Jammy, and Noble.