Testing Fundamentals: Unit, Integration, and E2E Tests Done Right

pytest Fundamentals

We have now spent two chapters mocking requestGrade — pretending the grader exists so we could test everything around it. Time to go inside the thing we have been faking.

The grader is a Python service. It pulls a submission out of storage, runs the exercise's cases against it in a sandbox, and returns a report:

# grader/service.py
from grader.storage import fetch_submission
from grader.sandbox import sandbox_for
from grader.exercises import load_cases
from grader.scoring import classify_case, percentage_score

def grade(submission_id):
    submission = fetch_submission(submission_id)
    cases = load_cases(submission.exercise_id)

    with sandbox_for(submission) as box:
        outcomes = [classify_case(*box.run(case)) for case in cases]

    passed = sum(1 for o in outcomes if o == "pass")
    return {"passed": passed, "total": len(cases), "score": percentage_score(passed, len(cases))}

pytest is the framework most Python projects default to, and the philosophy behind it can be summed up as staying out of your way — ordinary functions, a plain assert statement, none of the ceremony unittest and its class hierarchy used to demand. Here it is on the smallest piece of the grader — the function that turns a sandbox exit into a verdict for one case:

# grader/scoring.py
def classify_case(exit_code, stderr):
    if exit_code == 0:
        return "pass"
    if exit_code == 124:      # the sandbox's timeout wrapper killed it
        return "timeout"
    if exit_code == 137:      # SIGKILL — the memory cap was hit
        return "out_of_memory"
    return "fail"

def test_classify_case_passes_on_clean_exit():
    assert classify_case(0, "") == "pass"

def test_classify_case_reports_a_timeout():
    assert classify_case(124, "") == "timeout"

No class to subclass, no self, no assertion methods. Two plain functions whose names start with test_, and pytest finds and runs them.

Plain assert, and why that's enough

Most other frameworks steer you toward a whole catalogue of assertion methods — assertEqual, assertTrue, assertIn, assertGreater — for one reason: in most languages, a bare assert that fails just says "assertion failed" and leaves you to figure out the rest yourself. pytest sidesteps that entire catalogue a different way: at collection time it goes in and rewrites the actual bytecode behind each assert line in a file, which lets it capture what was really sitting on each side of the comparison when something went wrong.

def test_out_of_memory_is_not_a_plain_failure():
    assert classify_case(137, "") == "out_of_memory"   # plain assert — no assertEqual needed

Say classify_case regresses and starts returning "fail" where it shouldn't. The failure output shows both sides of the broken comparison and the exact expression that produced them — E AssertionError: assert 'fail' == 'out_of_memory', differing characters called out — and none of that required reaching for a named matcher anywhere. That's assertion rewriting at work, and it's why pytest code reads like plain Python rather than a dialect built entirely out of assertion-method calls. assert a == b, assert x in y, assert not flag — every one of these just works and produces a genuinely useful error, because pytest examines the expression you actually wrote instead of expecting you to have guessed the one correctly named method for the comparison you had in mind.

Fixtures

A fixture is a function whose job is handing a test whatever setup it needs — and, if it wants to, cleaning that setup back up afterward — delivered to any test that names the fixture as one of its parameters. The grader needs a real one: running a case means writing a submission's files into an actual temporary workspace, and that workspace has to get torn down afterward no matter whether the test passed, failed, or blew up entirely.

import pytest

@pytest.fixture
def workspace(tmp_path):
    box = Sandbox(root=tmp_path, memory_mb=256)
    box.start()
    yield box        # everything before yield is setup; this value is injected into the test
    box.destroy()    # everything after yield is teardown, and it runs even if the test fails

def test_a_solution_that_returns_the_right_answer_passes(workspace):
    workspace.write("solution.py", "def solve(n):\n    return n * 2\n")
    exit_code, stderr = workspace.run(Case(input=21, expected=42))
    assert classify_case(exit_code, stderr) == "pass"

pytest sees the parameter named workspace, finds the fixture with that exact name, and calls it on the test's behalf — nothing in the test body ever calls setup directly. That yield is what splits one function into two acts: everything before it is the setup half, everything after it is the teardown half, and the teardown half runs during cleanup even when the test itself blows up, which is exactly what makes a fixture something you can trust for anything that needs tidying afterward. A sandbox that never gets torn down isn't just untidy — it's a container still holding onto 256 MB, one per failed test, until CI eventually runs out of memory and starts failing tests that have nothing to do with any of this.

(tmp_path in that signature is one of pytest's own built-in fixtures — a fresh temporary directory per test, which pytest creates and cleans up for you. Fixtures can request other fixtures, all the way down.)

Fixture scope

Left alone, a fixture starts over from scratch for every test that asks for it. That default can be dialed up or down, and where you land on that dial affects both whether your tests are correct and how long they take to run:

@pytest.fixture(scope="function")   # default — runs once per test
def workspace(tmp_path):
    ...

@pytest.fixture(scope="module")     # runs once per test FILE, shared by all its tests
def exercise_catalog():
    return load_all_exercises_from_disk()

@pytest.fixture(scope="session")    # runs once for the ENTIRE test run, across every file
def sandbox_image():
    image = build_sandbox_image()   # ~40 seconds; nothing about it is test-specific
    yield image
    remove_image(image)

Going wider buys speed at the cost of isolation — it is never free — and the grader makes that trade-off concrete. sandbox_image is exactly the kind of thing that belongs at session scope: it takes forty seconds to build, nothing about it changes once it exists, and no single test can do anything to it that would affect another. workspace is the opposite case entirely. Widen that one to session scope and every test after the first runs inside a sandbox still holding the previous test's solution.py — so a test that forgets to write its own solution file quietly passes by inheriting someone else's, and only when the suite happens to run in that exact order.

Stick with function scope until there's a specific reason not to, and widen it only on purpose — reserved for setup that's genuinely costly to build and is either unchanging once built, or safe to reliably reset before the next test touches it.

parametrize, the same table in Python

classify_case has the same shape of problem inviteStatus had on the Node side: several branches, and one test body that would otherwise be copy-pasted once per branch. pytest's version of test.each is parametrize, and it is the same idea with the same payoff:

import pytest

@pytest.mark.parametrize("exit_code,stderr,expected", [
    (0,   "",                                    "pass"),
    (1,   "AssertionError: expected 42, got 21", "fail"),
    (124, "",                                    "timeout"),
    (137, "",                                    "out_of_memory"),
    (2,   "SyntaxError: invalid syntax",         "fail"),
])
def test_classify_case(exit_code, stderr, expected):
    assert classify_case(exit_code, stderr) == expected

pytest runs that body once for every tuple in the list and scores each run as its own separate result — test_classify_case[0--pass], test_classify_case[124--timeout], and so on — instead of bundling them into one test that silently stops mattering after whichever case it happens to fail on first. Whenever you catch yourself about to write four or five test functions that look nearly the same and vary only in what goes in versus what should come back out, this is what you reach for instead. It also happens to be the cheapest insurance against a missed branch: four real branches, five rows here — the fifth is a second route into fail — so a new branch added later without a matching row stands out immediately in the diff.