Testing: Quick Guide for LLMs
- Purpose: Help an LLM choose and run the right test/quality commands rapidly, and structure new tests correctly.
- Scope: Unit/integration/E2E tests, doctests, coverage, linting, and pre-commit across this repository.
**Test Layout**
- `tests/unit`: Fast, isolated unit tests (prefer default target here).
- `tests/integration`: Cross-module and service integration tests.
- `tests/e2e`: End-to-end flows (slower; may require services/containers).
- `tests/playwright`: UI tests (requires extra setup; keep out of default runs).
**Fast Commands (root)**
- Unit tests: `make test`
- Doctests then unit tests: `make doctest test`
- Coverage (md + HTML + XML + badge + annotated files): `make coverage`
- HTML coverage only (rebuild if needed): `make htmlcov` → `docs/docs/coverage/index.html`
- Smoke test (container build + simple flow): `make smoketest`
- Environment sanity (dotenv parity): `make check-env`
**Selective Runs**
- By name substring: `pytest -k "fragment"`
- By markers: `pytest -m "not slow"` or `pytest -m "api"`
- Single file/test: `pytest tests/unit/test_xyz.py::TestClass::test_method`
**Markers (convention)**
- `slow`, `ui`, `api`, `smoke`, `e2e` — keep default runs fast. Prefer filtering with `-m`.
**Doctests**
- Many framework modules include doctests (e.g., plugin models/manager).
- Commands: `make doctest` then `make doctest-coverage` and `make doctest-check` (enforce target %).
**Coverage**
- Full suite with reports and badges: `make coverage`.
- HTML report only: `make htmlcov` (auto-runs coverage if `.coverage` missing).
**Code Quality & Security**
- Formatting & hooks: `make autoflake isort black pre-commit`
- Static analysis: `make pylint flake8`
- Full lint suite (includes web linters after one-time setup): `make lint` and `make lint-web` (install web linters once if needed)
- Additional checks (varies by repo config): `make bandit interrogate verify check-manifest`
**Pull Request Readiness (recommended)**
- One-liner sequence before opening a PR:
- `make doctest test htmlcov smoketest lint-web flake8 bandit interrogate pylint verify`
**Writing Tests**
- Use pytest; async tests should use `@pytest.mark.asyncio`.
- Keep tests deterministic and isolated; avoid network and real credentials by default.
- Prefer unit tests near logic you modified; only add integration/E2E where behavior spans components.
- Name files `test_*.py`; classes `Test*`; functions `test_*`.
- Use marks to categorize slow/UI/API/E2E tests.
- Follow strict typing; run formatters and linters locally before PRs.
**Plugins (extra note)**
- For native/external plugin testing patterns (unit/integration/MCP), see `llms/plugins-llms.md` (Testing Plugins section) for examples and curl/MCP snippets.
**Coverage-First Workflow (aligns with unit-test-writer)**
- Inspect current tests and structure: `find tests/unit -name '*.py' -type f | sort`.
- Use coverage artifacts to target gaps:
- Run `make coverage` and open `docs/docs/coverage/index.html`.
- Review annotated files generated by `coverage annotate -d .` for untested lines (`.cover` markers).
- Prefer editing existing test files over creating new ones; keep style and fixtures consistent.
- Target uncovered branches first: error paths, exceptions, boundary conditions, conditional logic.
- Use parametrization (`@pytest.mark.parametrize`) to cover multiple inputs succinctly.
- Mock at boundaries (DB, network, external services) with `unittest.mock`/`pytest-mock`.
**Database Safety**
- Tests must not affect the real DB. If a temporary DB is unavoidable for a unit-level check, point to a throwaway SQLite file:
- `DATABASE_URL=sqlite:///./mcp-temp.db pytest -k 'your_test'`
- Prefer pure unit tests with mocked persistence layers for speed and determinism.
**Useful Snippets**
- Run a single test method: `pytest -q tests/unit/path/test_mod.py::TestClass::test_case`.
- Show test node IDs quickly: `pytest --collect-only -q`.
