Repository navigation
Support Aviso V2 ECFLOW-2121 - #476
Open
marcosbento wants to merge 27 commits into
Open
marcosbento wants to merge 27 commits into
marcosbento wants to merge 27 commits into
Conversation
marcosbento
force-pushed
the
task/adopt_aviso_v2
branch
from
October 6, 2026 11:25
e8dfc6a to
6ebc84d
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #476 +/- ##
===========================================
+ Coverage 56.22% 56.54% +0.31%
===========================================
Files 1268 1265 -3
Lines 105366 105752 +386
Branches 15374 15390 +16
===========================================
+ Hits 59247 59795 +548
+ Misses 46119 45957 -162 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Introduce the ENABLE_AVISO option, ON by default, which requires libaviso_ffi 2.4.2 or later. Only ecflow_server links the library, and logs its version at start-up. Re ECFLOW-2121
marcosbento
force-pushed
the
task/adopt_aviso_v2
branch
from
October 6, 2026 12:06
6ebc84d to
b3d943b
Compare
Remove the etcd polling backend and the v1 listener schema. An Aviso attribute now receives notifications from a backend registered by the executable; without one, the node stays queued with a remote error. The v1 options --schema and --polling are rejected, naming 5.19.x as the last release supporting Aviso v1. Checkpoints written by 5.19.x still load, and the serialised form remains compatible. Re ECFLOW-2121
marcosbento
force-pushed
the
task/adopt_aviso_v2
branch
3 times, most recently
from
October 7, 2026 05:34
cae8e60 to
8e6b958
Compare
Each Aviso attribute watches its aviso-server through one client and one watch, authenticated by its --auth file: the key of an ECMWF API credentials file, as a bearer token, or Basic credentials. Every error leaves the node queued with a reason that names 5.19.x as the last release supporting Aviso v1, and the watch is re-created after 60 s, resuming after the last consumed notification. Re ECFLOW-2121
Add a local server implementing the aviso-server watch protocol, used to test the backend and an ecFlow server end to end, without network or container. A re-created watch now resumes after the last delivered notification, or from when the first watch opened, so that nothing published while the watch was down is lost. Re ECFLOW-2121
An Aviso attribute whose URL or credentials variable cannot be resolved no longer fails the begin or requeue of its suite. The node stays queued with a remote error and a reason naming the variable, which a requeue no longer clears; once fixed, a reload of the attribute starts it. Re ECFLOW-2121
Each release consumes exactly one notification, in order; the option --collapse consumes all the notifications received in one release. The notification is held until the node leaves the queued state, and only then is the revision advanced, so that a query of why the node is held, or another dependency holding it, no longer loses it. Re ECFLOW-1984
Changing an Aviso attribute (alter change aviso) now parses the new configuration with the node as its parent, stops the previous one, and starts the new one while the node is queued, so that the attribute keeps working, also after a requeue. Re ECFLOW-2121
The traversal of the whole definition now reaches every node and its attributes, so that a server restart (e.g. from a checkpoint) starts the Aviso attributes of queued nodes, and a halt or shutdown stops them. A queued Aviso task thus keeps being released, once per notification, across a server restart. Re ECFLOW-2121
A node with an Aviso attribute now defines ECF_AVISO_EVENT_TYPE, ECF_AVISO_EVENT_SEQUENCE, ECF_AVISO_EVENT_DATA_IDENTIFIER and ECF_AVISO_EVENT_DATA_PAYLOAD, describing the notification that released it (the latest one, when collapsing). Until a notification releases the node, the variables are empty and the sequence is 0. The notification is kept in checkpoints (option --event), so the variables survive a server restart. Re ECFLOW-2026 Re ECFLOW-1983
The user manual describes the Aviso attribute for Aviso v2 only, with the minimum aviso-client and aviso-server versions, the listener, the credentials file, one release per notification and --collapse, the generated variables, the errors and the fixed 60 s retry delay. It states that a release may be repeated after a server crash. The cookbook adds an end-to-end use case: a script that runs an aviso-server in a container, an ecflow_server and a suite. The client help and the Python reference are regenerated, and the build documentation covers ENABLE_AVISO and AVISO_FFI_ROOT. Re ECFLOW-2121 Re ECFLOW-1984 Re ECFLOW-2026
marcosbento
force-pushed
the
task/adopt_aviso_v2
branch
from
October 7, 2026 07:56
8e6b958 to
25d0c63
Compare
The Aviso v1 reminder no longer names the current release, and a credentials error names its file. A watch that fails to be re-created is reported at once, and clearing an Aviso error no longer clears the remote error flag of a node without one. Unused code is removed, shared test helpers are factored out, and the comments and documentation are aligned with the behaviour. Re ECFLOW-2121
An Aviso attribute is refused on a suite or a family, where no job would ever consume a notification: definitions and check points that hold one fail to load, and alter and the Python API refuse to add it. Tasks and their aliases are unaffected. Re ECFLOW-2121
Starting an attribute that is already watching keeps its configuration, so the active listener and the error match the running watch; reload applies a new one. An attribute added to a queued node with alter is started at once, and a deleted attribute stops watching and clears the remote error flag it raised. Re ECFLOW-2121
The local test server ends a stream with the routine reason max_duration_reached, after which the client library reconnects by itself, or with an error event, after which the backend re-creates the watch following the retry delay. The documentation states that the delay applies only after an error the library does not recover from. Re ECFLOW-2121
The Aviso attribute page states that the revision is kept with the rest of the node state in the check point, so a crash, a forced reboot or a takeover from an earlier check point copy can release a task again for the same notification, and how a task can detect the repeat. Re ECFLOW-2121
With Aviso v1 removed, the listener, credentials and error helpers join the Aviso types, the backend interface becomes BaseAvisoBackend, and the backend based on the aviso-client library becomes AvisoBackend. The test target u_service_aviso_v2 is renamed u_service_aviso_backend. Re ECFLOW-2121
The Debian package is built with Aviso, and the Aviso client library of the build environment is delivered beside it and installed by the image, whose test checks that the server uses it. Re ECFLOW-2121
Re ECFLOW-2121
An invalid value leaves the running watch untouched, and the replaced attribute no longer holds the node in error. A change restarts from the given --revision, as now documented. Re ECFLOW-2121
A mirrored node takes its state from the remote node, which a release by a notification would contradict: definitions and check points that hold both fail to load, and alter and the Python API refuse to add one next to the other. Re ECFLOW-2121
A package build replaces only the Aviso archive of its own architecture, and the ecflow-server image build fails when a library needed by the ecFlow executables is missing. Re ECFLOW-2121
Name every Aviso service test, create the credentials files through the test scaffold, and label the backend tests that run against a local Aviso server as integration tests. Re ECFLOW-2121
Iterate test inputs by reference, which silences a range-loop warning, and drop the report tags from test comments. Re ECFLOW-2121
Document the parameters and results of the Aviso attribute, service and test server declarations, in the project Doxygen style. Re ECFLOW-2121
State that one aviso attribute is allowed per task, and name the fields of the --event option, as the attribute page does. Re ECFLOW-2121
The local Aviso server reports the notifications streamed and the streams ended, so that the tests wait for them with a bounded timeout. Re ECFLOW-2121
The script stops when the aviso-server does not become healthy, and publishes once the watch of the attribute is established. Re ECFLOW-2121
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
As per PR title - see commit messages for details.
Contributor Declaration
By opening this pull request, I affirm the following:
🌦️ >> Documentation << 🌦️
https://sites.ecmwf.int/docs/dev-section/ecflow/pull-requests/PR-476