Principles of Chaos Engineering
See also: Fuzzing || Compilers Correctness
Blogs
- Google Testing Blog
- Evolving Understanding About Exploratory Testing
- Exploratory Testing
- How SQLite Is Tested
- How the GNU coreutils are tested
- Principles of Automated Testing
- Refactoring for testability in C++
- Testing in Production, the safe way
Blogs: Series
- Abseil C++ Tips of the Week (TotW)
- James Grenning
- TDD How-to: Get your Legacy C Into a Test Harness
- TDD Guided by ZOMBIES
- Test-Driven Development For Embedded C++ Programmers
- John Regehr
- Kent Beck
- Programmer Test Principles
- “Unit” Tests?
- Nelson Elhage
- William Caputo
Books
- Software Testing: From Theory to Practice
- The Fuzzing Book: Tools and Techniques for Generating Software Tests
Courses
- CMPT 473: Software Testing, Reliability and Security
- Software Analysis and Testing
- Software Testing: How to Make Software Fail
Research
Research: 2010s
- Assertions Are Strongly Correlated with Test Suite Effectiveness
- Assurances in Software Testing: A Roadmap
- AUSTIN (AUgmented Search-based TestINg): An open source tool for search based software testing of C programs
- Compiler Instrumentation for Testing
- Does Refactoring of Test Smells Induce Fixing Flaky Tests?
- Regression Testing Minimisation, Selection and Prioritisation: A Survey
- STADS: Software Testing as Species Discovery
- Testing C++ generic libraries
- The Art of Testing Less Without Sacrificing Quality
- The Oracle Problem in Software Testing: A Survey
- Tools and Language Elements for Testing, Encapsulation and Controlling Abstraction in Large Scale C++ Projects
- When Testing Meets Code Review: Why and How Developers Review Tests
Research: 2000s
- Software Testing and Industry Needs
- Software Testing Research: Achievements, Challenges, Dreams
Research: 1990s
- Bringing Testing Into the Fold
- PIE: A Dynamic Failure-Based Technique
Research: 1980s
- On Testing Non-Testable Programs
Concurrency
- A Survey of Recent Trends in Testing Concurrent Software Systems
- Applications of Synchronization Coverage
- Are Concurrency Coverage Metrics Effective for Testing: A Comprehensive Empirical Investigation
- Cuzz – Concurrency Fuzzing
Continuous Integration
- ci_helloworld: A simple example of how to setup a complete CI environment for C and C++
- https://github.com/ainfosec/ci_helloworld
- A Test a Day Keeps Your Manager Away!
- CppCon 2017; Rian Quinn
- https://www.youtube.com/watch?v=KdJhQuycD78
- "During this session we will step through an open source project designed to demonstrate how to integrate different C++ analysis tools into your CI services. These tools include static analysis (Clang Tidy, Coverity Scan, Codeacy and CppCheck), dynamic analysis (Valgrind and Google's Sanitizers), source formatting (Astyle and Clang Format), documentation (Doxygen), code coverage (Codecov, Coveralls, and LLVM's Software-based Code Coverage), cross platform tests (Windows, Cygwin, Linux, and macOS), compiler tests (GCC, Clang, and Visual Studio) and finally C++ libraries designed to assist in reliability and automated testing (Catch, Hippomocks and the Guideline Support Library). In addition we will openly discuss the advantages and disadvantages of using various analysis tools, how to integrate these tools into existing projects (both large and small) as well as common problems encountered while using these tools autonomously in a CI environment."
- Code Coverage & Continuous Integration
Continuous Integration: Readings
- Automated Reporting of Anti-Patterns and Decay in Continuous Integration
- Continuous delivery: Patterns and antipatterns in the software life cycle
- Continuous Integration: Improving Software Quality and Reducing Risk
- Continuous Integration Theater
- Game Changers - Trunk Based Development
- How to get code coverage from CI
- Modern C++ CI
- Taming Google-Scale Continuous Testing
Continuous Integration: Software
- Azure DevOps C++ build tasks for CMake and vcpkg
- CMake Modern: C++, CMake and Travis-CI Hello World
- Continuous Integration Demo
- GitLab CI for C++ projects
Continuous Integration: Software: AppVeyor
- Intro to AppVeyor - C++ Weekly - Ep 80
- Boost.Hana .appveyor.yml
Continuous Integration: Software: GitHub Actions
- Using GitHub Actions with C++ and CMake
- Speeding up C++ GitHub Actions using ccache
Continuous Integration: Software: Travis
- .travis.yml Examples
- Boost.Hana: clang++, g++, Boost, Valgrind - https://github.com/boostorg/hana/blob/master/.travis.yml
- Catch2: https://github.com/catchorg/Catch2/blob/master/.travis.yml
- meta: https://github.com/Leandros/meta/blob/master/.travis.yml
- MPark.Variant: https://github.com/mpark/variant/blob/master/.travis.yml
- xstd: clang++, g++, binutils, Boost, libc++, lcov, lld, lldb - https://github.com/rhalbersma/xstd/blob/master/.travis.yml
- C++ CI: Clang, Travis, CMake, GTest, Coveralls & Appveyor
- Intro To Travis CI - C++ Weekly - Ep 79
- Continuous integration with Travis CI
- icc-travis: Script to help install Intel C/C++ Compiler on Travis CI.
- Minimal Travis CI example for modern C++
- Richel Bilderbeek
- C++ Travis CI tutorial
- Favorite C++ setup
- TravisTorrent: Synthesizing Travis CI and GitHub for Full-Stack Research on Continuous Integration
Coverage
Coverage: Readings
- A Large-Scale, Longitudinal Study of Test Coverage Evolution
- Applications of Synchronization Coverage
- Assessing Oracle Quality with Checked Coverage
- Automatic Self-Validation for Code Coverage Profilers
- Can Testedness be Effectively Measured
- Code Coverage at Google
- Code Coverage for Suite Evaluation by Developers
- Code Coverage is a Strong Predictor of Test Suite Effectiveness
- Comparing non-adequate test suites using coverage criteria
- International Symposium on Software Testing and Analysis (2013)
- Gligoric, M., Groce, A., Zhang, C., Sharma, R., Alipour, A., Marinov, D.
- https://agroce.github.io/issta13.pdf
- Coverage and Its Discontents
- Coverage is Not Strongly Correlated with Test Suite Effectiveness
- Covrig: A Framework for the Analysis of Code, Test, and Coverage Evolution in Real Software
- Guidelines for coverage-based comparisons of non-adequate test suites
- ACM Transactions on Software Engineering and Methodology (2015)
- Gligoric, M., Groce, A., Zhang, C., Sharma, R., Alipour, A., Marinov, D.
- https://dl.acm.org/citation.cfm?id=2660767
- http://mir.cs.illinois.edu/coco/
- http://users.ece.utexas.edu/~gligoric/papers/GligoricETAL15CoCoJournal.pdf
- How to Misuse Code Coverage
- Hunting for Bugs in Code Coverage Tools via Randomized Differential Testing
- Invasive Software Testing: Mutating Target Programs to Diversify Test Exploration for High Test Coverage
- Mythical Unit Test Coverage
- Not All Coverage Measurements Are Equal: Fuzzing by Coverage Accounting for Input Prioritization
- Selecting Software Test Data Using Data Flow Information
- TestCov: Robust Test-Suite Execution and Coverage Measurement
- Turning Programs against Each Other: High Coverage Fuzz-Testing using Binary-Code Mutation and Dynamic Slicing
- Verification, coverage and maximization: The big picture
Coverage: Software
- afl-cov - AFL Fuzzing Code Coverage
- cmake_catch_coverage
- Code Coverage in Chromium
- CompareCoverage
- CompareCoverage (CmpCov in short) is a simple instrumentation module for C/C++ programs and libraries, which extracts information about data comparisons taking place in the code at run time, and saves it to disk in the form of standard .sancov files. It is based on the SanitizerCoverage instrumentation available in the clang compiler, which itself is tightly related to AddressSanitizer. Specifically, the library implements the instrumentation callbacks defined by the Tracing data flow feature of SanitizerCoverage.
- https://github.com/googleprojectzero/CompareCoverage
- CovNavi: Code coverage navigation and analysis
- cpp-llvm-coverage: How to use LLVM coverage for for C++
- drcov: a DynamoRIO client tool that collects code coverage information
- GNU gcov
- https://gcc.gnu.org/onlinedocs/gcc/Gcov.html
- Writing Better Function Tests with GCOV
- Gcovr
- GCovHTML - Generate reports on code coverage
- Dev Santa Claus Part 2: C++ code coverage metrics with gcov
- Using Gcov and Lcov to generate beautiful C++ code coverage statistics
- Kcov
- OpenCppCoverage: an open source code coverage tool for C++ under Windows
- LLVM
- Clang Source-based Code Coverage
- SanitizerCoverage
- TestCov: Test execution, coverage measurement, and test-suite reduction
Coverage: Mutation
Mutation Analysis, Mutation Coverage, Mutation Testing
Coverage: Mutation: Readings
- Mutation Testing Repository
- Evaluating Test Effectiveness with Mutation Analysis
- A Systematic Literature Review of How Mutation Testing Supports Quality Assurance Processes
- A Systematic Literature Review of Techniques and Metrics to Reduce the Cost of Mutation Testing
- An Analysis and Survey of the Development of Mutation Testing
- An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation that Avoids the Unreliable Clean Program Assumption
- An Industrial Application of Mutation Testing: Lessons, Challenges, and Research Directions
- Are Mutants a Valid Substitute for Real Faults in Software Testing?
- C++11/14 Mutation Operators Based on Common Fault Patterns
- Detecting Trivial Mutant Equivalences via Compiler Optimisations
- How Verified (or Tested) is My Code? Falsification-Driven Verification and Testing
- If You Can’t Kill a Supermutant, You Have a Problem
- Is Mutation an Appropriate Tool for Testing Experiments?
- Mutant Quality Indicators
- Mutation Testing Advances: An Analysis and Survey
- Mutation Testing: From Theory to Practice
- On the Limits of Mutation Analysis
- State of Mutation Testing at Google
- The Care and Feeding of Wild-Caught Mutants
Coverage: Mutation: Software
- MART Framework for Multi-Programming Language Mutation Testing based on LLVM
- Mull: Mutation testing system built on top of LLVM
- Mutate++ - A C++ Mutation Test Environment
- SRCIROR: SRC and IR mutatOR
Coverage: Mutation: Talks
- Testing The Tests: Mutation Testing for C++
Generation
- An Orchestrated Survey on Automated Software Test Case Generation
- Automated Software Test Generation: Some Challenges, Solutions, and Recent Advances
- FSX: fine-grained incremental unit test generation for C/C++ programs
- KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs
- KLOVER (KLee based cOVERage tool, for C/C++)
- KLOVER: Automatic Test Generation for C and C++ Programs, Using Symbolic Execution
- KLOVER: A Symbolic Execution and Automatic Test Generation Tool for C++ Programs
- Symbolic Execution Algorithms for Test Generation
- Symbolic Execution for Software Testing: Three Decades Later
- Test-Comp: International Competition on Software Testing
- Test Generation Using Symbolic Execution
Generation: Random
Random Testing
- A Survey on Adaptive Random Testing
- Adaptive Random Testing
- Adaptive Random Testing: the ART of Test Case Diversity
- John Regehr
- The Central Limit Theorem Makes Random Testing Hard
- Randomly Testing a Static Analyzer
- Burning in a Module with Random Unit Testing
- Oracles for Random Testing
- Random Test Run Length and Effectiveness
- The Confidence Sequence Method: a computer-age test for statistical SLOs
Property-Based Testing
- What is Property Based Testing?
- Finding Property Tests
- Generating Good Generators for Inductive Relations
- How hard is it to guide test case generators with branch coverage feedback?
- How to Specify it! A Guide to Writing Properties of Pure Functions
Property-Based Testing: Metamorphic
Metamorphic Relations, Metamorphic Testing
- Metamorphic Testing
- IEEE/ACM International Workshop on Metamorphic Testing (MET)
- A Survey on Metamorphic Testing
- Automated Metamorphic Testing
- Metamorphic Testing
- Metamorphic Testing 20 Years Later: A Hands-on Introduction
- Metamorphic Testing and Its Applications
- Metamorphic Testing: A New Approach for Generating Next Test Cases
- Metamorphic Testing: A Review of Challenges and Opportunities
- Metamorphic Testing: A Simple Yet Effective Approach for Testing Scientific Software
- Techniques for Testing Scientific Programs Without an Oracle
Reduction
Test-Case Reduction, Minimization
See also: Debugging: Readings: Delta Debugging
Reduction: Readings
- Everything You Ever Wanted To Know About Test-Case Reduction, But Didn’t Know to Ask
- Mitigating (and Exploiting) Test Reduction Slippage
- Notes on Test-Case Reduction
- Reduce Before You Localize: Delta-Debugging and Spectrum-Based Fault Localization
- Rust Bug Minimization Patterns
Reduction: Software
- C-Reduce
- "C-Reduce is a tool that takes a large C, C++, or OpenCL file that has a property of interest (such as triggering a compiler bug) and automatically produces a much smaller C/C++ file that has the same property. It is intended for use by people who discover and report bugs in compilers and other tools that process source code."
- https://embed.cs.utah.edu/creduce/
- Delta Debugging
- halfempty: Fast, Parallel Testcase Minimization
- Shrink Ray: a test-case reducer designed to be effective on a wide range of formats
- Structured Shrinking: Structured shrinking of unknown file formats
Software
- Approval Tests for C++
- Boost.Test
- [Boost].UT: C++20 μ(micro)/Unit Testing Framework
- Catch2
- Cgreen - The Modern Unit Test and Mocking Framework for C and C++
- CppUTest unit testing and mocking framework for C/C++
- CUTE: C++ Unit Testing Easier
- cwrap - A toolset for client server testing
- https://cwrap.org/
- cwrap's mission is to enable developers to test complex network-based and privileged software stacks on UNIX machines with limited network access and without root privileges by providing preloadable libraries to wrap standard libc functions.
- Testing your full software stack with cwrap
- Testing your software stack without root privileges using cwrap
- DeepState: A unit test-like interface for fuzzing and symbolic execution
- doctest
- Google Test
- GUnit - Google.Test/Google.Mock/Cucumber on steroids
- OutputCheck: A tool for checking tool output inspired by LLVM's FileCheck
- RapidCheck: QuickCheck clone for C++ with the goal of being simple to use with as little boilerplate as possible.
Software: Test Doubles
Test Doubles: Faking, Mocking
- Fake Function Framework (fff)
- gen-xfakes
- PowerFake
Software: Test Doubles: Mocking
- FakeIt: a simple mocking framework for C++
- Google Mock: The Google C++ mocking framework
- GUnit.GMock
- Hippomocks: Single-header mocking framework
- Mockitopp: Simple mocking for C++
- Trompeloeil: Header-only C++14 mocking framework
Talks
- Google Test Automation Conference (GTAC)
Talks: 2019
- Back to Basics: Test-driven Development
- Making Testing C++ Binaries Practical @ Facebook Scale: A CI Story
Talks: 2018
- But is it fun? Software Testing in the Video Game Industry
- Ensuring Exception Safety Through Testing
Talks: 2017
- C++ Unit testing - the good, the bad & the ugly
- Microcontrollers in Micro-increments: A Test-driven C++ Workflow for Embedded Systems
- Mocking C++
- Mocking Frameworks considered harmful
Talks: 2016
- Assessing software quality from first principles
Talks: 2015
- Advanced Unit Testing in C & C++
- All Your Tests are Terrible: Tales from the Trenches
- Fast Iteration Tools in the Production of the Talos Principle
Talks: 2014
- Pragmatic Unit Testing in C++
Tags:
native
reading
Last modified 24 November 2023