Add terminate_iteration to skip batch handlers - #3826
nightcityblade wants to merge 7 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new Engine control signal to allow a process function to skip ITERATION_COMPLETED handlers for the current batch, preventing downstream handlers from consuming state.output when a batch should be ignored (addressing #996).
Changes:
- Add
Engine.terminate_iteration()and a newshould_terminate_single_iterationflag. - Apply the skip logic to both generator (
_run_once_on_dataset_as_gen) and legacy (_run_once_on_dataset_legacy) execution paths. - Add a regression test that runs in both modes via the existing
interrupt_resume_enabledparameterization.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| ignite/engine/engine.py | Introduces terminate_iteration() and conditionally skips Events.ITERATION_COMPLETED in both run paths. |
| tests/ignite/engine/test_engine.py | Adds a regression test verifying skipped batches do not trigger ITERATION_COMPLETED handlers in both execution modes. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Follow-up validation for 122cee3:
|
|
Also terminate_epoch has extra parameter I notiched skip_epoch_completed, I think terminate_iteration should be consistent with that |
|
Addressed in 54f5a5e. terminate_iteration now accepts skip_iteration_completed, defaulting to False consistently with terminate_epoch, and both execution paths honor it. The focused parametrized tests pass in generator and legacy modes (4 passed), and Ruff check/format pass. |
f5a4edd to
122cee3
Compare
|
@vfdev-5 LGTM |
|
CI note: the only failing check is linkcheck, caused by the pre-existing Polyaxon URL redirect in generated/ignite.handlers.polyaxon_logger.rst (https://polyaxon.com/docs/experimentation/tracking/client/ redirects to /docs/references/python-library/tracking/client/). The PR does not touch that documentation; all code-style, CPU, doctest, MPS, TPU, Horovod, and pyrefly jobs pass. |
|
@nightcityblade thanks for your effort working on the issue. I remember it was not a simple issue as we can not simply skip iterations especially with metrics attached to engines. |
|
Addressed in 2466d55. The terminate_iteration regression in tests/ignite/engine/test_engine.py now verifies both requested cases: skipped batches are excluded from a trainer's RunningAverage and from an evaluator's epoch Average, with the expected values asserted in both engine execution modes. Validation: 185 passed, 10 skipped for the full engine test file; Ruff check and format check passed on the changed test. |
|
sorry, I missed the updates :) |
|
Updated the test in bba1810 to use the default RunningAverage decay while retaining the required identity output transform. The expected value is now 1.04 (only completed iterations 1 and 3 contribute with alpha=0.98). Focused result: 2 passed. |
Co-authored-by: Aaishwarya Mishra <aaishwarymishra@gmail.com>
This reverts commit 5579653.
| trainer = Engine(process) | ||
| completed_iterations = [] | ||
| trainer.add_event_handler(Events.ITERATION_COMPLETED, lambda e: completed_iterations.append(e.state.iteration)) | ||
| RunningAverage(output_transform=lambda output: output).attach(trainer, "running_average") |
There was a problem hiding this comment.
@vfdev-5 dont you think output_transform=lambda output: output should be default here like in Average so we dont have to explicitly pass it when declaring running average?
Fixes #996
Description:
Add
Engine.terminate_iteration()so a process function can ignore the current batch without firingITERATION_COMPLETEDhandlers that consumestate.output. Apply the signal consistently to both the generator and legacy execution paths, and cover both modes with a regression test.Tests:
python -m pytest tests/ignite/engine/test_engine.py -q(185 passed, 10 skipped)ruff check ignite/engine/engine.py tests/ignite/engine/test_engine.pyCheck list: