Having trouble? Reach out on slack: #apm-shared-testing
System-tests is a black-box testing workbench for Datadog tracer libraries. It runs the same tests against every tracer implementation -- Java, Node.js, Python, PHP, Ruby, C, C++, .NET, Go, and Rust -- so shared features stay consistent across languages.
Key principles:
- Black-box testing -- only component interfaces are checked, no assumptions about internals. "Check that the car moves, regardless of the engine."
- Cross-language -- one test validates all tracer libraries.
You need bash, Docker (20.10+), and Python 3.12.
# 1. Build images for the language you want to test
./build.sh python # or: c, java, nodejs, ruby, php, dotnet, golang
# 2. Run the tests
./run.sh # run all default tests
./run.sh SCENARIO_NAME # run a specific scenario
./run.sh tests/test_smoke.py::Test_Class::test_method # run a single testHaving trouble? Check the troubleshooting page.
To understand the test output, see test outcomes and the glossary.
All detailed documentation lives in the docs/ folder. Here is a guided reading order:
| Topic | Description |
|---|---|
| Architecture overview | Components, containers, data flow |
| Scenarios | End-to-end, parametric, SSI, K8s -- what each one tests |
| Weblogs | The test applications instrumented by tracers |
| Glossary | Definitions of SSI, scenario, weblog, pass, fail, and other terms |
| Topic | Description |
|---|---|
| Build | Build options, weblog variants, image names |
| Run | Run options, selecting tests, scenarios, timeouts |
| SSI quick start | Choose and run Docker, AWS, or Kubernetes SSI tests |
| Logs | Understanding the logs folder structure |
| Test outcomes | Reading test results |
| Replay mode | Re-run tests without starting the containers |
| Custom tracer versions | Testing with local tracer builds |
| Troubleshooting | Common issues and how to fix them |
| Topic | Description |
|---|---|
| Add a new test | Step-by-step guide to adding tests |
| Add a new scenario | Creating new test scenarios |
| Enable / disable tests | Activating tests for a library version |
| Manifests | How test activation is declared per library |
| Skip tests | Decorators for conditional skipping |
| Features | Linking tests to the feature parity dashboard |
| Formatting | Linter and code style |
| Troubleshooting | Debugging tips for test development |
| Topic | Description |
|---|---|
| CI overview | Adding system-tests to your CI pipeline |
| GitHub Actions | GitHub Actions workflow details |
| System-tests CI | How the system-tests own CI works |
| Topic | Description |
|---|---|
| Internals overview | Deep-dive index for maintainers |
| End-to-end lifecycle | How e2e scenarios execute step by step |
| Parametric lifecycle | How parametric scenarios execute |
| Interface validation | API reference for validating intercepted traces |
| Topic | Description |
|---|---|
| AI integration guide | Built-in rules for AI-assisted development |
Specific scenarios need extra tools or access. For Docker SSI, AWS SSI, and Kubernetes library injection, start with the SSI quick start. The SSI access guide lists every requirement.
Before submitting a PR, always run the linter (./format.sh). Here are the most common types of contributions, ordered by frequency:
| What you want to do | Guide |
|---|---|
| Activate or deactivate a test for a library | Manifests, enable a test, skip tests |
| Add or edit a test | Add a new test, editing overview |
| Add or edit a scenario | Scenarios guide, scenarios overview |
| Add or edit a weblog | Weblog spec, build options |
| Other changes | Full editing docs, internals |
For testing against unmerged tracer changes, see enable-test.md and binaries.
For file ownership see .github/CODEOWNERS.
Test ownership is defined using the @features decorator (see the feature doc and utils/_features.py).
Drop a message in #apm-shared-testing -- we're happy to help!