Chapter 24: Testing in Rust

Software testing is essential for verifying code correctness, particularly when refactoring or adding features. Rust’s strong compile-time safety checks eliminate entire classes of bugs prevalent in C and C++, such as use-after-free, null pointer dereferencing, and many buffer overflows. However, these checks primarily ensure memory and type safety, not the correctness of the application’s logic or its adherence to requirements. Therefore, testing remains crucial in Rust for validating behavior, logic, and performance.

This chapter introduces Rust’s integrated testing framework and common practices. We will cover unit, integration, and documentation tests, techniques for running tests selectively, handling expected failures, using test-specific dependencies, and briefly introduce benchmarking. Comparisons to C/C++ testing practices will be made where relevant.

24.1 The Role of Testing in Rust

While Rust’s safety features significantly reduce certain types of bugs, testing is indispensable for building robust software.

24.1.1 Beyond Memory Safety: Validating Logic and Requirements

Rust’s compiler enforces memory safety (preventing dangling pointers, data races) and type safety at compile time. Runtime checks, like array bounds checking, provide further guarantees. This contrasts sharply with C/C++, where such issues often manifest as runtime errors or security vulnerabilities, requiring extensive dynamic analysis tools (like Valgrind) or careful manual checking.

However, the compiler cannot verify that the program’s logic matches the intended behavior or specifications. For instance:

• A financial calculation might use a mathematically incorrect formula, even if it’s memory-safe.
• A network protocol implementation might safely handle bytes but deviate from the protocol standard.
• A function might accept inputs according to its type signature but fail to enforce domain-specific constraints (e.g., requiring positive inputs).

Tests are necessary to confirm that the code behaves correctly according to functional requirements and logical specifications.

24.1.2 Benefits of Integrated Testing

A comprehensive test suite offers several advantages:

• Regression Prevention: Ensures existing functionality isn’t broken by new changes.
• Executable Documentation: Tests demonstrate how code should be used and its expected outcomes.
• Design Guidance: The process of writing tests often encourages more modular and testable code designs.
• Collaboration Safety: Provides a safety net when multiple developers contribute to a codebase.

Unlike C/C++, where testing typically involves integrating external libraries (e.g., CUnit, Google Test, Check) and build system configuration, Rust incorporates testing as a first-class feature of the language and its build tool, Cargo. This significantly lowers the barrier to writing and running tests.

24.2 Writing Basic Tests

In Rust, tests are functions marked with the #[test] attribute. The test runner executes these functions. A test passes if its function completes execution without panicking; it fails if the function panics.

24.2.1 The #[test] Attribute

fn add(a: i32, b: i32) -> i32 {
    a + b
}

#[test]
fn test_addition_success() {
    let result = add(2, 2);
    assert_eq!(result, 4); // Passes if 2 + 2 == 4
}

#[test]
fn test_addition_failure() {
    let result = add(2, 2);
    // This assertion fails because 4 != 5, causing the function to panic.
    assert_eq!(result, 5);
}

• The #[test] attribute identifies test_addition_success and test_addition_failure as test functions.
• Test functions typically take no arguments and return () (the unit type), although returning Result is also possible (see Section 24.5.2).

24.2.2 Assertion Macros

Rust’s standard library provides macros for asserting conditions within tests:

• assert!(expression): Panics if expression evaluates to false. Suitable for simple boolean conditions.
• assert_eq!(left, right): Panics if left != right. This is the most frequently used assertion. Requires that the types implement the PartialEq and Debug traits (the latter for printing values upon failure).
• assert_ne!(left, right): Panics if left == right. Also requires PartialEq and Debug.

These macros can accept optional arguments (after the mandatory ones) for a custom failure message, formatted using the same syntax as println!:

#[test]
fn test_custom_message() {
    let width = 15;
    assert!(width >= 0 && width <= 10, "Width ({}) is out of range [0, 10]", width);
}

24.2.3 Running Tests with cargo test

The command cargo test compiles the project in a test configuration (which includes code marked with #[cfg(test)]) and runs all discovered tests (unit, integration, and documentation tests).

$ cargo test
   Compiling my_crate v0.1.0 (...)
    Finished test [unoptimized + debuginfo] target(s) in ...s
     Running unittests src/lib.rs (...)

running 2 tests
test tests::test_addition_success ... ok
test tests::test_addition_failure ... FAILED

failures:

---- tests::test_addition_failure stdout ----
thread 'tests::test_addition_failure' panicked at src/lib.rs:16:5:
assertion failed: `(left == right)`
  left: `4`,
 right: `5`
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    tests::test_addition_failure

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out;

error: test failed, to rerun pass '--lib'

The output clearly shows test progress, failures with assertion details (values, file, line number), and a final summary.

24.3 Test Organization

Rust’s testing framework encourages separating tests based on their scope: unit tests and integration tests.

24.3.1 Unit Tests

Unit tests verify small, isolated components, typically individual functions or methods, including private ones. They are conventionally placed within the same source file as the code under test, inside a dedicated submodule named tests and annotated with #[cfg(test)].

// In src/lib.rs or src/my_module.rs
pub fn process_data(data: &[u8]) -> Result<String, &'static str> {
    if data.is_empty() {
        return Err("Input data cannot be empty");
    }
    internal_helper(data)
}

// Private helper function
fn internal_helper(data: &[u8]) -> Result<String, &'static str> {
    // ... complex logic ...
    Ok(format!("Processed {} bytes", data.len()))
}

// Unit tests are placed in a conditionally compiled submodule
#[cfg(test)] // Ensures this module is only compiled during `cargo test`
mod tests {
    use super::*; // Import items from parent module (process_data, internal_helper)

    #[test]
    fn test_process_data_success() {
        let result = process_data(&[1, 2, 3]).unwrap();
        assert_eq!(result, "Processed 3 bytes");
    }

    #[test]
    fn test_process_data_empty() {
        let result = process_data(&[]);
        assert!(result.is_err());
        assert_eq!(result.unwrap_err(), "Input data cannot be empty");
    }

    #[test]
    fn test_internal_logic() {
        // Directly test the private helper function
        let result = internal_helper(&[10]).unwrap();
        assert!(result.contains("1 bytes")); // Example check
    }
}

• #[cfg(test)]: This attribute ensures that the tests module and its contents are only included when compiling for tests (cargo test). This avoids including test code in release builds.
• use super::*;: This imports all items (functions, types, etc.) from the parent module (super), making them available within the tests module.
• Testing Private Items: Unit tests can directly access and test private functions and types within the same module (like internal_helper). This is useful for verifying internal implementation details or invariants that are not exposed publicly.

Cargo’s cargo new my_lib --lib command automatically generates a src/lib.rs file with this standard test module structure.

24.3.2 Integration Tests

Integration tests verify the public API of your library crate from an external perspective, mimicking how other crates would use it. They reside in a dedicated tests directory at the root of your project, alongside the src directory.

my_crate/
├── Cargo.toml
├── src/
│   └── lib.rs             // Contains process_data, internal_helper (private)
└── tests/                 // Integration tests directory
    ├── common/            // Optional subdirectory for shared test utilities
    │   └── mod.rs         // Module file for common utilities
    └── api_usage_tests.rs // An integration test file

Each .rs file within the tests directory (e.g., api_usage_tests.rs) is compiled by Cargo as a separate crate. This means each such test file links against your library crate (my_crate in this example) as if it were an external dependency.

Example (tests/api_usage_tests.rs):

// Import the library crate being tested
use my_crate; // Use the actual name defined in Cargo.toml

#[test]
fn test_public_api_call() {
    // Can only call public items (like process_data) from my_crate
    let result = my_crate::process_data(&[1, 2, 3, 4]).unwrap();
    assert_eq!(result, "Processed 4 bytes");

    // Attempting to call private items results in a compile-time error
    // let _ = my_crate::internal_helper(&[1]);
    // Error: function `internal_helper` is private
}

#[test]
fn test_empty_data_error_via_public_api() {
    let result = my_crate::process_data(&[]);
    assert!(result.is_err());
}

Key characteristics of integration tests:

• External Perspective: Integration tests can only access pub items (functions, structs, enums, modules) defined in your library crate. They cannot access private implementation details. This ensures they test the library as an external user would.
• Separate Crates: Because each file directly in tests/ is a distinct crate, they are compiled independently. This rigorously tests the public contract but means that sharing setup code or utility functions across multiple integration test files requires specific handling.

Sharing Code Between Integration Tests

When multiple integration test files (e.g., tests/feature_a_tests.rs, tests/feature_b_tests.rs) need to share common setup code or utility functions, the recommended approach is to place this shared code into a module within a subdirectory of tests. For instance, you could create a directory tests/common/ and place your shared module code in a file named tests/common/mod.rs.

The file name mod.rs is significant here. Using a mod.rs file inside a directory (e.g., module_name/mod.rs) to define the root of a module named module_name is an established pattern in Rust. While modern Rust (since the 2018 edition) often allows for a module_name.rs file to implicitly own a same-named directory for its submodules (e.g., src/my_module.rs and src/my_module/child.rs), the module_name/mod.rs convention is crucial when a directory itself is intended to be the module’s primary source, as is common for organizing shared utilities within the tests directory. When you declare mod common; in an integration test file (which acts as its own crate, like tests/api_usage_tests.rs), and common corresponds to a directory (in this case, tests/common/), Rust specifically expects to find tests/common/mod.rs. This mod.rs file serves as the root or entry point of the common module. Files residing within such subdirectories (like tests/common/mod.rs) are not automatically compiled as separate test crates by Cargo.

If you were to name the file differently, for example, tests/common/my_utils.rs, a simple mod common; declaration would not load it as the root of the common module. tests/common/my_utils.rs could, however, be a submodule if tests/common/mod.rs declared it (e.g., pub mod my_utils;).

Example Structure with Shared Code: (Refer to the directory structure shown at the beginning of section 24.3.2, which includes tests/common/mod.rs)

Shared Utilities (tests/common/mod.rs):

// tests/common/mod.rs
// This file defines the 'common' module's contents because it is named 'mod.rs'
// within the 'common/' directory. It is not compiled as a separate test crate
// due to its location in a subdirectory.

pub fn setup_environment() {
    // ... perform common setup actions ...
    println!("Common setup for an integration test performed.");
}

pub fn create_sample_data() -> Vec<u8> {
    vec![10, 20, 30]
}

// If you had other utility files within tests/common/, for example,
// tests/common/internal_helpers.rs, you would declare them here as submodules:
// pub mod internal_helpers; // This would make common::internal_helpers available.

Using Shared Utilities (e.g., in tests/another_feature_tests.rs):

// tests/another_feature_tests.rs
// This file IS compiled as a separate test crate.

use my_crate; // Assuming 'my_crate' is the library being tested

// Declare the 'common' module. Because 'common' is a directory,
// Rust resolves this to tests/common/mod.rs.
mod common;

#[test]
fn test_another_feature_with_shared_utils() {
    common::setup_environment();
    let data = common::create_sample_data();
    let result = my_crate::process_data(&data).unwrap();
    // Assuming process_data from your lib
    assert!(result.contains("3 bytes")); // Adjust assertion as per actual logic
}

By adhering to the tests/common/mod.rs naming convention, you create a clearly defined common module accessible to all your integration test files, without Cargo treating the shared code as an independent test suite.

Regarding a tests/common.rs file (Not a Subdirectory):

If you were to create tests/common.rs (i.e., a file named common.rs directly within the tests/ directory, not in a common/ subdirectory), Cargo would treat this tests/common.rs file as a separate test crate. If it contained no #[test] functions, it would simply appear as an empty test suite in your test output (e.g., “running 0 tests” for common). While other test files like tests/api_usage_tests.rs could still load its contents using mod common;, this approach is generally less clean as it introduces an unnecessary test target in Cargo’s view. The subdirectory method (tests/common/mod.rs) is preferred for clarity and to avoid this.

Integration Tests for Binary Crates

Integration tests are primarily designed for library crates (--lib). If your project is a binary crate (src/main.rs only), the tests/ directory cannot directly call functions within src/main.rs because a binary doesn’t produce a linkable artifact in the same way a library does that other crates can depend on for testing.

The recommended approach for testing binary applications is to structure the project with a library component:

1. Extract the core logic from src/main.rs into src/lib.rs, exposing public functions. This turns your project into a crate that has both a library and a binary.
2. Keep src/main.rs minimal. Its main responsibilities would be parsing command-line arguments and then calling the public functions exposed by src/lib.rs.
3. Write integration tests in the tests/ directory that target the public API defined in src/lib.rs.

This structure allows the core application logic in src/lib.rs to be thoroughly tested independently of the command-line interface specifics in src/main.rs.

In some cases, you may want to verify not just the library logic, but also that the final binary runs successfully—essentially performing a smoke test of the actual built executable. This ensures the program can be launched and behaves correctly when invoked as an external process, just as an end user would run it.

To do this, you can write an integration test that spawns the binary using std::process::Command. Cargo makes the path to the compiled binary available during testing via the CARGO_BIN_EXE_<name> environment variable, where <name> is the name of your binary target.

Here is a basic example of this pattern:

// Binary entry point (src/main.rs)
use my_crate; // assuming logic lives in src/lib.rs

fn main() {
    my_crate::run_it();
}

// Integration test that runs the binary (tests/integration_test.rs)
use std::process::Command;

#[test]
fn test_main_binary_runs() {
    let bin_path = env!("CARGO_BIN_EXE_my_crate"); //Replace with binary's actual name

    let output = Command::new(bin_path)
        .output()
        .expect("Failed to execute binary");

    assert!(output.status.success(), "Binary exited with an error");
    // Optionally inspect output.stdout or output.stderr here
}

This approach offers a practical way to validate that your binary builds and launches correctly in its final form—complementing unit and integration tests focused on library functionality.

Note: If your binary must be invoked via a shell (e.g., to use shell syntax like pipes, redirection, or quoted arguments), you’ll need to distinguish between operating systems using cfg!, such as cfg!(target_os = "windows"):

// On Windows
Command::new("cmd").args(["/C", bin_path])

// On Unix-like systems
Command::new("sh").arg(bin_path)

24.4 Controlling Test Execution

Cargo provides fine-grained control over the test execution process. By default, cargo test compiles and runs all enabled tests in your project, which includes unit tests, integration tests, and (as we’ll cover in Section 24.6) documentation tests. However, you often need more specific control, such as running a subset of tests by name, focusing only on unit or specific integration tests, or adjusting execution parameters like parallelism and output verbosity. These options are invaluable for efficient development workflows and for managing test suites in continuous integration (CI) environments.

24.4.1 Running Specific Tests

You can selectively run tests based on their names or their location within your project:

• Filter by Name: Run only tests whose names contain a specific substring. The filter applies to the test function’s full path (e.g., module_name::sub_module::test_function_name or simply test_function_name if it’s unique enough).

  # Runs tests with "api" in their name, like test_public_api_call
  # or api_tests::some_test
  cargo test api
  
  # Runs only the test named test_internal_logic within the tests module
  # of your library
  cargo test tests::test_internal_logic
  

  The substring match is case-sensitive.

• Run Specific Integration Test File: Execute all #[test] functions within a particular integration test file located in the tests/ directory. Provide the filename without the .rs extension.

  # Runs all #[test] functions in tests/api_usage.rs
  cargo test --test api_usage
  

• Run Only Library Unit Tests: Execute only the unit tests defined within your library crate (typically in src/lib.rs or modules defined therein). This command will not run integration tests or documentation tests.

  cargo test --lib
  

  Similarly, if you have a binary crate with tests in src/main.rs, you can run them using cargo test --bin your_binary_name. If your package only contains a single binary, cargo test --bins can be used.

24.4.2 Ignoring Tests

Tests that are slow, require specific environments (e.g., network access), or are currently flaky can be marked with the #[ignore] attribute.

#[test]
fn very_fast_test() { /* ... */ }

#[test]
#[ignore = "Requires network access and is slow"] // Optional reason string
fn test_external_service() {
    // ... code that might take a long time or fail intermittently ...
}

• Ignored tests are skipped by default when running cargo test.
• To run only the ignored tests:

  cargo test -- --ignored
  

• To run all tests, including those marked as ignored:

  cargo test -- --include-ignored
  

Note on --: Arguments placed after a standalone -- are passed directly to the test runner executable built by Cargo, not to Cargo itself. Use cargo test -- --help to see options accepted by the test runner, such as --ignored, --include-ignored, --test-threads, and --nocapture. Contrast this with cargo test --help, which shows Cargo’s own command-line options.

24.4.3 Controlling Parallelism and Output

• Parallel Execution: By default, cargo test runs tests in parallel using multiple threads for faster execution. If tests might interfere with each other (e.g., accessing the same file or resource without synchronization) or if sequential execution simplifies debugging, parallelism can be disabled:

  # Run tests sequentially using only one thread
  cargo test -- --test-threads=1
  

• Capturing Output: Standard output (println!) and standard error (eprintln!) generated by passing tests are captured by default and not displayed. Output from failing tests is shown. To display the output from all tests, regardless of their status:

  # Show all stdout/stderr from all tests
  cargo test -- --nocapture
  

24.5 Testing Panics and Errors

Sometimes, the expected behavior of code under specific conditions is to panic or return an error. Rust’s test framework provides ways to verify this.

24.5.1 Expecting Panics with #[should_panic]

If a function is designed to panic for certain inputs (e.g., division by zero, out-of-bounds access on a custom type), you can use the #[should_panic] attribute on a test function. The test passes if the code inside panics and fails if it completes without panicking.

pub fn get_element(slice: &[i32], index: usize) -> i32 {
    // This will panic if index is out of bounds
    slice[index]
}

#[test]
#[should_panic]
fn test_index_out_of_bounds() {
    let data = [1, 2, 3];
    get_element(&data, 5); // Accessing index 5 should panic
}

To make the test more specific, you can assert that the panic message contains a certain substring using the expected parameter. This helps ensure the code panics for the intended reason.

#[test]
#[should_panic(expected = "out of bounds")]
fn test_specific_panic_message() {
    let data = [1, 2, 3];
    get_element(&data, 5);
    // Panics with a message like "index out of bounds: len is 3 but the index is 5"
}

This test passes only if the function panics and the panic message includes the substring “out of bounds”.

24.5.2 Using Result<T, E> in Tests

Test functions can return Result<(), E> instead of (). This allows the use of the question mark operator (?) within the test for cleaner handling of operations that return Result.

• The test passes if it returns Ok(()).
• The test fails if it returns an Err(E).
• The error type E must implement the std::fmt::Debug trait so the test runner can print it upon failure.

use std::num::ParseIntError;

// Function that might return an error
fn parse_even_number(s: &str) -> Result<i32, ParseIntError> {
    let number = s.parse::<i32>()?; // Propagate ParseIntError if parsing fails
    if number % 2 == 0 {
        Ok(number)
    } else {
        // For simplicity, we reuse ParseIntError, though a custom error type is
        // often better. This specific error construction is illustrative;
        // typically you'd define a custom error enum.
        Err("".parse::<i32>().unwrap_err())
        // Create a dummy ParseIntError for odd numbers
    }
}

#[test]
fn test_parse_valid_even() -> Result<(), ParseIntError> {
    let number = parse_even_number("42")?; // Use `?` - test proceeds if Ok
    assert_eq!(number, 42);
    Ok(()) // Return Ok(()) to indicate success
}

#[test]
fn test_parse_odd_returns_err() {
    // We expect an Err, so we don't use `?` or return Result
    let result = parse_even_number("3");
    assert!(result.is_err());
    // Optionally, check the specific error kind if needed
}

#[test]
fn test_parse_invalid_string_fails() -> Result<(), ParseIntError> {
    // This test will fail because parse_even_number returns Err("abc".parse()?)
    // The Err will propagate out, causing the test runner to mark it as failed.
    let _number = parse_even_number("abc")?;
    Ok(()) // This line is never reached
}

Note: You cannot use the #[should_panic] attribute on a test function that returns Result. If you need to test that a function returning Result specifically produces an Err variant, assert this directly using methods like is_err(), unwrap_err(), or pattern matching, as shown in test_parse_odd_returns_err.

24.6 Documentation Tests (doctests)

Rust includes a powerful feature where code examples written inside documentation comments (/// for items, //! for modules/crates) can be compiled and run as tests. This ensures that your documentation examples remain accurate and functional as the underlying code evolves.

/// Calculates the factorial of a non-negative integer.
///
/// Panics if the input `n` is negative.
///
/// # Examples
///
/// ```
/// # use my_crate::factorial; // Hidden setup line
/// assert_eq!(factorial(0), 1);
/// assert_eq!(factorial(5), 120);
/// ```
///
/// This example demonstrates the panic condition:
/// ```should_panic
/// # use my_crate::factorial;
/// // Factorial is not defined for negative numbers
/// factorial(-1);
/// ```
///
/// Example showing compilation only (e.g., for demonstrating type signatures):
/// ```no_run
/// # use my_crate::factorial;
/// let f6: u64 = factorial(6);
/// // No assertion, just compile check.
/// ```
///
/// This block is ignored by the test runner and documentation generator:
/// ```ignore
/// This is not Rust code. It won't be tested or rendered.
/// ```
pub fn factorial(n: i64) -> u64 {
    if n < 0 {
        panic!("Factorial input cannot be negative");
    }
    let mut result: u64 = 1;
    for i in 1..=(n as u64) {
        result = result.saturating_mul(i); // Use saturating_mul for safety
    }
    result
}

When cargo test runs, it extracts these code blocks:

1. It automatically adds extern crate my_crate; (using your crate’s name) if needed.
2. It often wraps the code block in fn main() { ... }.
3. It compiles and runs the code according to the block’s attributes.

• Assertions: Standard assert! macros work within doctests.
• Hidden Lines: Lines starting with # (hash space) are executed during testing but are hidden in the rendered HTML documentation (cargo doc --open). This is ideal for including necessary use statements or setup code that would otherwise clutter the example.
• Attributes: Placed after the opening ```:
  • ``` (no attribute): The code must compile and run successfully (without panicking).
  • ```should_panic: The code must compile and must panic when run.
  • ```no_run: The code must compile, but it is not executed. Useful for examples involving actions with side effects (like filesystem or network operations) or just demonstrating API usage patterns.
  • ```ignore: The code block is completely ignored by cargo test and cargo doc.

Running Documentation Tests Exclusively

While cargo test by default includes documentation tests along with unit and integration tests, you can specifically target only the documentation tests for your library using the --doc flag:

cargo test --doc

This command is useful for quickly verifying that all documentation examples are up-to-date and compile and run correctly, without needing to execute the entire unit and integration test suite.

Doctests are excellent for verifying basic usage examples of your public API but are generally not suitable for complex test scenarios or testing internal implementation details, for which unit or integration tests are preferred.

24.7 Test Dependencies

Tests, examples, or benchmarks might require helper crates not needed by the main application or library code. These dependencies should be specified under the [dev-dependencies] section in your Cargo.toml file.

# Cargo.toml
[package]
name = "my_crate"
version = "0.1.0"
edition = "2021"

[dependencies]
# Regular dependencies used by src/lib.rs or src/main.rs
# Example: serde = { version = "1.0", features = ["derive"] }

[dev-dependencies]
# Dependencies only compiled for tests, examples, benchmarks
# Example: provides improved assertion diffs
pretty_assertions = "1.4"
# Example: helps create temporary files/directories for tests
tempfile = "3.10"

Cargo only compiles dev-dependencies when building targets that might need them (tests, examples, benchmarks). They are not included when a user depends on your library crate, nor are they included in release builds of binary crates unless explicitly used by src/main.rs (which is usually not the case).

Example using pretty_assertions: This popular dev-dependency provides replacements for assert_eq! and assert_ne! that produce colorful, detailed diff output when comparing complex structures, making failures much easier to diagnose.

// In src/lib.rs within #[cfg(test)] mod tests { ... }
// Or in a file within the tests/ directory

#[cfg(test)]
mod diff_tests {
    // Use the enhanced assertion macro from the dev-dependency
    use pretty_assertions::assert_eq;

    #[derive(Debug, PartialEq)]
    struct ComplexData {
        id: u32,
        name: String,
        values: Vec<i32>,
    }

    #[test]
    fn test_complex_data_equality() {
        let expected = ComplexData {
            id: 101,
            name: "Example".to_string(),
            values: vec![1, 2, 3, 4, 5],
        };
        let actual = ComplexData {
            id: 101,
            name: "Example".to_string(),
            values: vec![1, 2, 99, 4, 5], // Mismatch in the middle
        };

        // The standard assert_eq! would show the full structs.
        // pretty_assertions::assert_eq! shows a focused diff.
        assert_eq!(expected, actual);
    }
}

24.8 Benchmarking

Benchmarking measures the execution speed (latency) or throughput of code snippets. It complements testing by tracking performance characteristics, helping to identify regressions, and validating optimizations. Systems programming often requires careful performance management, making benchmarking a valuable tool.

Rust offers several approaches to benchmarking:

• Built-in Benchmark Harness: A basic harness available on the nightly Rust toolchain.
• Dedicated Crates: Third-party libraries like criterion and divan that work on stable Rust and offer more advanced features, statistical analysis, and reporting.

For most comprehensive benchmarking needs, dedicated crates are preferred due to their stability and richer feature sets.

24.8.1 Built-in Benchmarks (Nightly Rust Only)

If you are using the nightly Rust compiler, you can use the language’s built-in, unstable benchmarking support. This can be useful for very simple benchmarks without adding external dependencies.

1. Enable Feature and Import: Add #![feature(test)] to your crate root (usually src/lib.rs or src/main.rs) and import the test crate.
2. Write Benchmark Functions: Benchmark functions are typically placed within a #[cfg(test)] module, similar to unit tests. They are marked with the #[bench] attribute and take a mutable reference to test::Bencher.
3. Use Bencher::iter: Inside the benchmark function, the code to be measured is passed as a closure to b.iter(|| ...).

Example:

// In src/lib.rs or src/main.rs
#![feature(test)] // Required for built-in benchmarks

// This line is only needed if you're putting benchmarks in a file that isn't the
// crate root and needs to explicitly import the `test` crate provided by the compiler.
// For benchmarks within the same file as `#![feature(test)]`, it's often implicitly
// available.
extern crate test;

pub fn expensive_calculation(input: u32) -> u32 {
    // A simple placeholder for a function to benchmark
    (0..input).fold(0, |acc, x| acc.wrapping_add(x))
}

#[cfg(test)]
mod benchmarks {
    use super::*;
    use test::Bencher; // Import the Bencher type

    #[bench]
    fn bench_expensive_calculation(b: &mut Bencher) {
        // The iter method runs the closure multiple times and measures its execution.
        b.iter(|| {
            // Code to benchmark goes here
            // Use test::black_box to prevent the compiler from optimizing away
            // the code being benchmarked if its result isn't used.
            expensive_calculation(test::black_box(1000))
        });
    }
}

Running Built-in Benchmarks: Use the cargo bench command. This command will compile your code in a test configuration (enabling #[cfg(test)]) and run functions annotated with #[bench].

cargo bench

Output is typically printed to the console, showing the time taken per iteration.

Note: The built-in benchmark harness is very basic. It lacks statistical rigor and advanced features found in dedicated crates. Compiler optimizations can also heavily affect benchmark results; using test::black_box around inputs to and outputs from benchmarked code is crucial to prevent the compiler from optimizing away the work. While available on nightly, for comprehensive analysis, consider using criterion or divan.

24.8.2 Benchmarking with criterion (Stable Rust)

criterion is a powerful, statistics-driven benchmarking library for stable Rust. It performs multiple runs, analyzes results statistically to mitigate environmental noise, detects performance changes over time, and can generate detailed HTML reports.

1. Add Dependency and Configure Harness: Add criterion to your [dev-dependencies] in Cargo.toml. You also need to configure Cargo to use criterion’s harness for benchmark targets.

   # Cargo.toml
   [dev-dependencies]
   criterion = { version = "0.5", features = ["html_reports"] }
   # Check for the latest version
   
   # Tell Cargo to use criterion's test harness for benchmarks.
   # 'main_bench' corresponds to the benchmark file benches/main_bench.rs
   [[bench]]
   name = "main_bench" # This is the name of your benchmark target
   harness = false     # Disables the default libtest harness
   

2. Create Benchmark File: Create a file in the benches directory at the root of your project (e.g., benches/main_bench.rs).

   // benches/main_bench.rs
   use criterion::{black_box, criterion_group, criterion_main,
       Criterion, BenchmarkId};
   
   // Example function to benchmark (could be imported from your library)
   fn fibonacci(n: u64) -> u64 {
       match n {
           0 => 0,
           1 => 1,
           n => fibonacci(n - 1) + fibonacci(n - 2),
       }
   }
   
   fn fibonacci_benchmarks(c: &mut Criterion) {
       // Benchmark fibonacci(10)
       // "fib 10" is a unique string ID for this specific benchmark case.
       // This ID is used in reports and when comparing performance over time.
       c.bench_function("fib 10", |b| b.iter(|| fibonacci(black_box(10))));
   
       // Benchmark fibonacci(20) with a different ID
       c.bench_function("fib 20", |b| b.iter(|| fibonacci(black_box(20))));
   
       // You can also benchmark with varying inputs using a group
       let mut group = c.benchmark_group("Fibonacci Numbers");
       for i in [5u64, 10u64, 15u64].iter() {
           // BenchmarkId is used to create a unique ID for each parameter value
           // It takes the group ID, parameter description, and parameter value.
           group.bench_with_input(BenchmarkId::new("Recursive", i), i, |b, i_val| {
               b.iter(|| fibonacci(black_box(*i_val)))
           });
       }
       group.finish();
   }
   
   // The criterion_group! macro defines a benchmark group.
   // The first argument `benches` is the name of the group suite.
   // Subsequent arguments are the benchmark functions to include in this suite.
   criterion_group!(benches, fibonacci_benchmarks);
   
   // The criterion_main! macro generates the main function necessary
   // to run all benchmark group suites defined by criterion_group!.
   criterion_main!(benches);

   • criterion::black_box: A function that acts as an opaque barrier to compiler optimizations, ensuring the benchmarked code is actually executed.
   • Criterion::bench_function("ID", ...): Defines a single benchmark case. The first argument is a string identifier for this benchmark.
   • Criterion::benchmark_group("Group Name"): Allows grouping related benchmarks and comparing different functions or parameters side-by-side.
   • Bencher::iter: Runs the provided closure multiple times to gather timing statistics.

3. Run Benchmarks: Execute cargo bench.

   cargo bench
   

   criterion saves results and generates detailed HTML reports, typically found in target/criterion/report/index.html. These reports include plots and statistical analysis, making it easier to understand performance characteristics and regressions.

24.8.3 Benchmarking with divan (Stable Rust)

divan is a newer benchmarking library (requires Rust 1.75 or later as of Divan 0.1.x) focused on simplicity, low overhead, and ergonomic features like attribute-based benchmark registration and parameterization.

1. Add Dependency and Configure Harness: Add divan to your [dev-dependencies] in Cargo.toml and configure the benchmark harness.

   # Cargo.toml
   [dev-dependencies]
   divan = "0.1" # Check for the latest version
   
   [[bench]]
   name = "app_benchmarks" # Corresponds to benches/app_benchmarks.rs
   harness = false
   

2. Create Benchmark File: Create a file in the benches directory (e.g., benches/app_benchmarks.rs).

   // benches/app_benchmarks.rs
   
   // Example function to benchmark (could be imported from your library)
   fn fibonacci_divan(n: u32) -> u64 {
       if n <= 1 {
           n as u64
       } else {
           fibonacci_divan(n - 1) + fibonacci_divan(n - 2)
       }
   }
   
   fn main() {
       // Run all benchmarks registered in this crate (binary).
       divan::main();
   }
   
   // Simple benchmark for a fixed input.
   // The function itself becomes the benchmark.
   #[divan::bench]
   fn fib_10() -> u64 {
       fibonacci_divan(divan::black_box(10))
   }
   
   // Parameterized benchmark: runs for each value in `args`.
   // Divan automatically handles `black_box` for arguments in many cases.
   #[divan::bench(args = [5, 10, 15])]
   fn fib_params(n: u32) -> u64 {
       fibonacci_divan(n)
   }

   • divan::main(): Initializes and runs all benchmarks registered with #[divan::bench] in the current crate.
   • #[divan::bench]: Attribute macro that marks a function as a benchmark.
   • args = [...]: An option for #[divan::bench] to provide a list of input values for parameterized benchmarks.
   • divan::black_box is available if explicit control over optimization prevention is needed, though divan often applies such measures implicitly for arguments.

3. Run Benchmarks: Execute cargo bench.

   cargo bench
   

   divan outputs benchmark results directly to the console. For more advanced features and configuration options, consult the Divan documentation.

Choosing between criterion and divan often depends on specific needs: criterion is known for its in-depth statistical analysis and historical trend reporting, making it excellent for tracking performance over a project’s lifetime. divan offers a more lightweight and arguably more ergonomic experience for defining and running benchmarks quickly, with good support for parameterization. Both are excellent choices for benchmarking on stable Rust.

24.9 Profiling

While benchmarking measures the performance of specific, isolated code paths, profiling analyzes the runtime behavior of an entire application to identify bottlenecks – sections where the program spends the most time or consumes the most resources (CPU, memory). Profiling is essential for guiding optimization efforts effectively.

Profiling typically involves using external, often platform-specific tools:

• Linux: perf, Valgrind (specifically Callgrind), Heaptrack
• macOS: Instruments (Xcode developer tools)
• Windows: Visual Studio Profiler, Intel VTune Profiler
• Cross-platform: Tracy Profiler

Integrating these tools with Rust builds often involves compiling with debug information (debug = true in Cargo.toml profile, even for release builds intended for profiling) and then running the compiled executable under the profiler’s control.

The Rust Performance Book provides an excellent, detailed guide on various profiling tools and techniques applicable to Rust programs. Covering profiling in depth is beyond the scope of this chapter.

24.10 Inspecting Generated Assembly

For developers transitioning from C or C++ to Rust, understanding the machine code generated by the Rust compiler can be particularly insightful. It allows you to verify how Rust’s high-level abstractions translate to low-level instructions, often demonstrating their “zero-cost” nature. This can also be useful when diagnosing highly specific performance issues or simply satisfying curiosity about the compiler’s behavior.

There are primarily two powerful tools at your disposal for inspecting the generated assembly: the online Compiler Explorer and the local cargo-show-asm utility.

Using Compiler Explorer (Online Tool)

A widely-used online tool for inspecting generated assembly is Compiler Explorer, created by Matt Godbolt, and accessible at https://godbolt.org/. This interactive tool supports Rust (among many other languages) and allows you to write Rust code and instantly see the corresponding assembly output generated by various versions of the Rust compiler for different target architectures. You can also see the output with different optimization levels.

How It’s Useful for Rust Programmers:

• Demystifying Abstractions: See how features like iterators, closures, Option<T>, Result<T, E>, and trait objects compile down. For example, you can observe how an iterator chain often compiles to the same efficient loop as a manually written C-style loop.
• Comparing with C/C++: Directly compare the assembly output of similar Rust and C/C++ snippets side-by-side.
• Learning about Optimizations: Observe the effects of different optimization flags (e.g., -O, -C target-cpu=native) on the generated code.
• Quick Exploration: Ideal for quick tests or sharing code snippets and their assembly output.

Using cargo-show-asm (Local Tool)

For local inspection of your project’s compiled output, the cargo-show-asm subcommand is an invaluable tool. It allows you to view the assembly, LLVM IR (Intermediate Representation), MIR (Mid-level Intermediate Representation), and even WebAssembly (Wasm) generated by rustc directly from your terminal.

Installation:

To use cargo-show-asm, you first need to install it:

cargo install cargo-show-asm

How It’s Useful for Rust Programmers:

• Local Analysis: Inspect the assembly for your specific project’s code, including dependencies.
• Deep Dives: Provides access to various intermediate representations (LLVM IR, MIR), offering deeper insights into the compilation process.
• Integration with Build Process: Can be integrated into local development workflows for performance profiling and optimization.
• Detailed Output: Offers options to filter output, demangle symbols, and control the assembly syntax.

Basic Usage:

To analyze the assembly code for your Rust project, you can use

cargo show-asm

to get a list of all the functions, and then display the assembly with:

cargo show-asm <function_name>

For example, if you have a function named my_function in your crate, you can run:

cargo show-asm my_function

You can also specify modules or types. Use cargo show-asm --help for more options, including viewing LLVM IR (--llvm-ir) or MIR (--mir).

Note that cargo show-asm shows by default the assembly code that results from a build with --release.

Example Exploration: Summing a Slice

Consider comparing the assembly for these two functions, which sum the elements of a slice:

// Example to explore with Compiler Explorer or cargo-show-asm
#[no_mangle]
pub fn sum_with_iterator(slice: &[i32]) -> i32 {
    slice.iter().sum()
}

#[no_mangle]
pub fn sum_with_loop(slice: &[i32]) -> i32 {
    let mut sum = 0;
    for i in 0..slice.len() {
        sum += slice[i];
    }
    sum
}

By pasting this code into Compiler Explorer and selecting a Rust compiler (e.g., a recent stable version with optimizations enabled like -O), you can examine the generated assembly for both functions. Alternatively, you can save this code in a local Rust project (e.g., in src/lib.rs) and run cargo show-asm sum_with_iterator and cargo show-asm sum_with_loop (ensuring your Cargo.toml is configured for a library or a binary with the functions accessible).

You’ll often find that sum_with_iterator produces code that is just as efficient as, or even identical to, sum_with_loop, showcasing Rust’s ability to optimize high-level patterns effectively without manual intervention. Adding the #[no_mangle] attribute to Rust functions helps preserve their original names in the assembly output, making them easier to recognize.

While a deep dive into assembly language is beyond this chapter’s scope, both Compiler Explorer and cargo-show-asm offer accessible ways to “look under the hood.” For systems programmers, these tools can be invaluable resources for building a deeper understanding and confidence in Rust’s performance characteristics and code generation strategies.

24.11 Summary

Testing and benchmarking are integral to developing reliable and efficient Rust software, complementing the language’s compile-time safety guarantees.

• Purpose of Testing: Verifies logical correctness, behavior against requirements, and prevents regressions, going beyond the memory safety enforced by the compiler. Rust’s integrated tooling simplifies test creation and execution compared to typical C/C++ workflows.
• Basic Tests: Functions marked #[test] are run by cargo test. Use assert!, assert_eq!, assert_ne! macros to check conditions. Tests fail on panic.
• Test Organization:
  • Unit Tests: Reside in #[cfg(test)] mod tests within src/ files. Can test private items.
  • Integration Tests: Located in the tests/ directory. Each file is a separate crate testing only the public API.
• Execution Control: Filter tests by name (cargo test <filter>), run specific test files (--test <name>), control parallelism (--test-threads=1), manage output (--nocapture), and skip tests (#[ignore], -- --ignored).
• Testing Failures: Use #[should_panic] (optionally with expected = "...") to verify intended panics. Test functions can return Result<(), E> to use ? and test error paths cleanly.
• Documentation Tests: Code examples (```) in doc comments are tested by cargo test, ensuring documentation stays valid. Use # to hide setup lines.
• Test-Only Dependencies: Specified under [dev-dependencies] in Cargo.toml for helper crates not needed in the final library or binary.
• Benchmarking: Measures code performance. Use stable crates like criterion or divan (cargo bench) for reliable results and analysis.
• Profiling: Identifies performance bottlenecks in the application using external tools. Essential for targeted optimization.

By adopting disciplined testing and benchmarking practices, developers can leverage Rust’s strengths to build software that is not only safe but also correct and performant.