Limitations¶

diffcone is designed to select too many tests rather than too few. These are the cases where it selects more than you might expect, or can't see everything.

Changes that run on import reach every importer.

A change to code that runs when a module is imported (a module-level statement, a constant, a class body, a decorator) selects every target whose module imports that module, directly or indirectly. In a codebase where most modules import a few central ones, such a change selects most tests. Execution evidence narrows this to the tests that actually used the changed code.

Class-level changes affect every method.

A change to a class's body (its bases, decorators or class attributes) affects all of its methods. Removing or redirecting an import affects the whole importing module; adding one affects only the code that uses the new name, unless the imported module runs code of its own that the importing module did not run before.

Calls through objects of unknown type are matched by name.

diffcone doesn't infer types. self.method() and cls.method() are resolved through the class hierarchy, but a call like obj.run() on an object of unknown type is matched against every function and method named run, so it's selected whenever any of them changes. Instance attributes are resolved only when every assignment is a simple one in __init__.

Plugins that collect files of their own.

diffcone reports the collecting plugins it recognises from your configuration. A plugin that collects files just by being installed is invisible to a plan from the code. Whenever run starts pytest, it runs every collected test the plan doesn't know, and a recording captures what pytest really collected.

Dynamic code is bounded by imports, if at all.

A function that uses eval, exec, globals(), vars() or getattr with a computed name is treated as affected by any change in the modules its module imports. A module imported by a computed name (importlib.import_module(name)) can be affected by any change at all.

A file that doesn't parse selects everything.

Any file under your source roots that isn't valid Python 3 or isn't UTF-8 is an analysis error, even a test data file nothing imports. Choose source roots that leave such files out.

Discovery reads the source, so it can't see what plugins do.

A pytest plugin that collects tests by its own rules, or a test base class outside your source roots, can mean diffcone's target list is incomplete. diffcone reports it and the plan exits with code 3 (static discovery). Parametrized tests are selected or skipped as a whole.

Uncommitted analysis is only as stable as your working tree.

A plan of WORKTREE or INDEX describes the files at the moment you planned. The report always marks such plans as uncommitted.