CHAPTER 56 · ADVANCED4 min read

Testing Strategy and CI

Build a small test portfolio that checks meaningful contracts, integration boundaries, and the installed artifact.

Python 3.12+Free to readModule 13 of 13

YOUR NEXT SMALL WINS

By the end, you’ll be able to…

Choose unit, integration, and acceptance tests by risk.
Use fixtures, temporary directories, and precise assertions.
Define a reproducible CI sequence with package smoke checks.

01Test contracts at different boundaries

A useful test fails when observable behavior breaks. Unit tests target pure transformations and edge cases cheaply. Integration tests exercise real collaborations such as a parser writing to an in-memory database. Acceptance tests run the public command and inspect status, output, and effects. Each layer catches a different class of mistake; a large number of tiny assertions cannot substitute for a missing boundary check.

Choose cases from requirements and failure modes. For a catalog importer, duplicate identifiers, boolean numeric values, atomic rollback, repeated imports, and output ordering matter more than checking an internal helper was called exactly once. Keep tests readable enough to explain the contract. Prefer a small explicit fixture over a giant realistic-looking dataset whose important properties are difficult to see.

02Use the standard library to isolate state

unittest provides assertions, discovery, setup, and cleanup without an external dependency. TemporaryDirectory creates isolated filesystem state that is removed after use. In-memory SQLite is useful for transaction behavior, while a temporary database file checks persistence across connections. These are real local integrations, so a mock of sqlite3 would remove the behavior the test needs to verify.

The example checks normalization, invalid input, and actual file round-tripping. subTest identifies which rejected value failed while allowing the rest to run. The final runner supplies an explicit argv so a notebook or sandbox’s own arguments do not affect discovery. For a project, save tests under tests and run python -m unittest discover -s tests -v from the repository.

test_normalize.py🐍Python

Ctrl + ↵

import json
from pathlib import Path
import tempfile
import unittest

def normalize_title(value):
    if not isinstance(value, str) or not value.strip():
        raise ValueError('title is required')
    return value.strip()

class TitleTests(unittest.TestCase):
    def test_trims_whitespace(self):
        self.assertEqual(normalize_title(' Python '), 'Python')

def test_rejects_invalid_values(self):
        for value in ['', '  ', None, 3]:
            with self.subTest(value=value):
                with self.assertRaises(ValueError):
                    normalize_title(value)

def test_json_file_round_trip(self):
        with tempfile.TemporaryDirectory() as directory:
            path = Path(directory) / 'catalog.json'
            path.write_text(json.dumps({'title': 'Python'}), encoding='utf-8')
            data = json.loads(path.read_text(encoding='utf-8'))
            self.assertEqual(data, {'title': 'Python'})

suite = unittest.defaultTestLoader.loadTestsFromTestCase(TitleTests)
result = unittest.TextTestRunner(verbosity=0).run(suite)
assert result.wasSuccessful()

Output

Click Run or press Ctrl+Enter

Output will appear here after you run the code.

03Mock only the boundary you intend to replace

Inject a fetch callable for core importer tests so live HTTP availability does not control correctness. Separately test the adapter with a controlled local server or response fixture for status, size, and timeout behavior. A mock should preserve the relevant contract, not simply return whatever makes the test pass. Overmocked tests can stay green while real components no longer agree.

Avoid sleeps in concurrency tests when events or explicit completion signals can establish ordering. Do not assert exact timings, random tokens, process IDs, or platform-specific temporary paths unless those values are the requirement. Capture stable outcomes and meaningful error categories. When a bug is fixed, add one targeted regression test that would have failed for the original behavior.

04Make CI reproduce a clean developer workflow

A CI job should select each claimed Python version, create an isolated environment, install the project, run discovery, build artifacts, and smoke-test the wheel in another environment outside the checkout. For a cross-platform CLI, include relevant operating systems. Keep build dependencies distinct from runtime dependencies and record the tool versions or constraints needed for reproducible investigations.

CI is evidence from the tested matrix, not proof of correctness or a production certificate. Keep secrets out of ordinary test jobs and run public fixture tests without external credentials. If a check fails, preserve useful logs without full payloads or environment dumps. Start with this small sequence and add type checking, linting, or security analysis when they address an actual project requirement.

ci-checks.txtShell

# Run in a fresh environment for each supported interpreter:python -m pip install -e .python -m unittest discover -s tests -vpython -m pip install buildpython -m build# Then install the wheel in a second clean environment and run# the installed command's help and fixture acceptance checks.

LET IT CLICK

The ideas worth keeping.

Select test layers according to the behavior at risk.
Use real local dependencies for transaction and filesystem contracts.
CI should exercise clean installation and the built artifact.

TURN UNDERSTANDING INTO SOMETHING YOU CAN DO

Make it yours.

Type and test your code right here in the interactive workspace. Run your solution, check the output, and compare with the worked guide when ready.

Test the error contract

Write validate_minutes and a unittest case proving it accepts zero and rejects True, a string, and a negative integer.

your_turn_1.py🐍Python

Ctrl + ↵

Output

Click Run or press Ctrl+Enter

Output will appear here after you run the code.

Explore a worked solution

one_possible_solution.pyPython

import unittestdef validate_minutes(value):    if type(value) is not int or value < 0:        raise ValueError('minutes must be a nonnegative integer')    return valueclass MinutesTests(unittest.TestCase):    def test_zero(self):        self.assertEqual(validate_minutes(0), 0)    def test_invalid(self):        for value in [True, '5', -1]:            with self.subTest(value=value):                with self.assertRaises(ValueError):                    validate_minutes(value)result = unittest.TextTestRunner(verbosity=0).run(    unittest.defaultTestLoader.loadTestsFromTestCase(MinutesTests))assert result.wasSuccessful()

The cases target the external validation rule, including the bool-is-int trap, rather than inspecting the function’s implementation.

Check stream separation

Test a function that emits JSON to stdout and a warning to stderr. Parse stdout as JSON and verify the diagnostic independently.

your_turn_2.py🐍Python

Ctrl + ↵

Output

Click Run or press Ctrl+Enter

Output will appear here after you run the code.

Explore a worked solution

one_possible_solution.pyPython

import contextlibimport ioimport jsonimport sysdef emit_report():    print(json.dumps({'count': 2}))    print('using fixture data', file=sys.stderr)output, errors = io.StringIO(), io.StringIO()with contextlib.redirect_stdout(output), contextlib.redirect_stderr(errors):    emit_report()assert json.loads(output.getvalue()) == {'count': 2}assert errors.getvalue().strip() == 'using fixture data'

Parsing captured stdout verifies the consumer-facing format. A stray progress print would cause this acceptance check to fail.

A SMALL QUESTION. A USEFUL NEXT STEP.

What stayed with you?

Check the ideas while they’re fresh. Your answers stay in this page and feedback appears as you go.

A LITTLE ACTIVE RECALL0 / 3 answered

01Which test best proves a failed database batch is atomic?

AMock commit and assert it was called.BUse a real temporary or in-memory database and inspect state after failure.CCount helper functions.

02Why avoid live external APIs in core correctness tests?

AHTTP cannot be tested.BAvailability, credentials, and changing data make those tests nondeterministic.CFixtures guarantee the real adapter works.

03A CI matrix passes on Python 3.12 and one OS. What is justified?

AAll future Python versions and platforms work.BThe checked contracts passed in that matrix.CThe project needs no operational review.

⚡CODING CHALLENGE

🔒Required to Mark Lesson Complete

Test the error contract

Write validate_minutes and a unittest case proving it accepts zero and rejects True, a string, and a negative integer.

EXPECTED OUTPUT

[Process exited with code 0 - No output]

challenge.pyPython 3

Ctrl + ↵

Execution Console

Click "Run & Verify" (or press Ctrl + Enter) to test your code against the challenge.

Keep exploring.

The official references behind this chapter, when you want a second angle or the full details.

One chapter closer.

Reading is a start. Trying it yourself makes the connection.

Solve the hands-on coding challenge in the knowledge section to verify your code and unlock chapter completion.