Skip to content

fix(movingMedian): match graphite-web alignment and NaN handling - #948

Merged
npazosmendez merged 5 commits into
mainfrom
njpm/moving-fix
Oct 19, 2026
Merged

npazosmendez merged 5 commits into
mainfrom
njpm/moving-fix

Conversation

@npazosmendez

@npazosmendez npazosmendez commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #523 and the one-step time shift of the movingMedian output.

movingMedian had its own implementation, with these bugs:

  • The output started one step too early (at from, not at from + step). When you combine it with an aligned series, for example asPercent(x, movingMedian(x, '1h')), the first point and the last point are null.
  • Integer windows did not fetch earlier data, so the first windowSize - 1 points were null.
  • Windows with NaN values gave wrong medians, because github.com/JaderDias/movingmedian does not handle NaN.
  • xFilesFactor was ignored.

The moving package already had a median branch without these bugs (most fixes came from #783). But movingMedian was not registered there, so the branch was never used. This PR registers movingMedian in moving, and removes the movingMedian package and the github.com/JaderDias/movingmedian dependency.

Changes to moving on top of that:

  • Speed: Windowed.Median() copied and sorted the window for each point. It now uses two indexed heaps that skip NaN values, so each push is O(log window).
  • Speed: the xFilesFactor check uses the NaN count of the window, so it does not scan the full window for each point.
  • Fix: when returnNaNsIfStepMismatch is false, data that is returned unchanged now keeps its timestamps. Before, they moved forward by the window size.

The expected values in the new tests come from graphite-web (movingWindow with median).

movingMedian now reads returnNaNsIfStepMismatch from the moving function config. A functionsConfig.movingMedian entry has no effect now.

A couple of unrelated fixes follow in #950

Benchmark

Benchmarks show how the new is a bit slower, but not by orders of magnitude (on the flip side, it's more memory efficient). The trade is correctness. I'm sure this can be optimized further, but it's not my intent here.

goos: linux
goarch: amd64
pkg: github.com/go-graphite/carbonapi/expr/types
cpu: Intel(R) Core(TM) Ultra 7 165U
                        │     old      │                  new                  │
                        │    sec/op    │    sec/op      vs base                │
WindowedMedian/4-14       420.5µ ± 14%    810.2µ ± 30%  +92.69% (p=0.000 n=10)
WindowedMedian/60-14      848.6µ ± 12%   1682.0µ ± 20%  +98.21% (p=0.000 n=10)
WindowedMedian/360-14     1.277m ± 26%    2.489m ±  9%  +94.87% (p=0.000 n=10)
WindowedMedian/600-14     1.539m ± 10%    2.799m ± 17%  +81.89% (p=0.000 n=10)
WindowedMedian/3600-14    2.430m ± 10%    3.742m ± 10%  +53.99% (p=0.000 n=10)
WindowedMedian/60480-14   18.04m ±  7%    19.92m ± 11%        ~ (p=0.063 n=10)
geomean                   1.770m          2.985m        +68.65%

                        │     old      │                 new                  │
                        │     B/op     │     B/op      vs base                │
WindowedMedian/4-14         336.0 ± 0%     288.0 ± 0%  -14.29% (p=0.000 n=10)
WindowedMedian/60-14      2.172Ki ± 0%   3.000Ki ± 0%  +38.13% (p=0.000 n=10)
WindowedMedian/360-14     14.92Ki ± 0%   18.34Ki ± 0%  +22.93% (p=0.000 n=10)
WindowedMedian/600-14     27.92Ki ± 0%   30.34Ki ± 0%   +8.67% (p=0.000 n=10)
WindowedMedian/3600-14    158.4Ki ± 0%   184.1Ki ± 0%  +16.20% (p=0.000 n=10)
WindowedMedian/60480-14   3.413Mi ± 0%   2.797Mi ± 0%  -18.04% (p=0.000 n=10)
geomean                   23.40Ki        25.06Ki        +7.07%

                        │     old     │                new                 │
                        │  allocs/op  │ allocs/op   vs base                │
WindowedMedian/4-14        8.000 ± 0%   5.000 ± 0%  -37.50% (p=0.000 n=10)
WindowedMedian/60-14      15.000 ± 0%   5.000 ± 0%  -66.67% (p=0.000 n=10)
WindowedMedian/360-14     21.000 ± 0%   5.000 ± 0%  -76.19% (p=0.000 n=10)
WindowedMedian/600-14     23.000 ± 0%   5.000 ± 0%  -78.26% (p=0.000 n=10)
WindowedMedian/3600-14    29.000 ± 0%   5.000 ± 0%  -82.76% (p=0.000 n=10)
WindowedMedian/60480-14   49.000 ± 0%   5.000 ± 0%  -89.80% (p=0.000 n=10)
geomean                    20.86        5.000       -76.03%

movingMedian had its own implementation. Its output started one step
too early, integer windows did not fetch earlier data, and windows with
NaN values gave wrong medians. Register movingMedian in the moving
package, which already had a median branch, and remove the old package
and the github.com/JaderDias/movingmedian dependency.

Make Windowed.Median() incremental so that the shared implementation is
close to the speed of the old library on large windows.

Expected test values are generated with graphite-web.
…lesFactor

With returnNaNsIfStepMismatch set to false, the data returned unchanged
had its timestamps moved forward by the window size.

The xFilesFactor check scanned the full window for each point. Use the
NaN count that Windowed already keeps.
The sorted slice moved O(window) values on each push, which was slow for
large windows. Keep the non-NaN values in two indexed heaps, so that each
push costs O(log window).
Keep the movingMedian package as a thin wrapper around moving, so that
movingMedian still reads returnNaNsIfStepMismatch from
functionsConfig.movingMedian.
@npazosmendez
npazosmendez marked this pull request as ready for review October 8, 2026 19:56
@npazosmendez
npazosmendez merged commit 7405d30 into main Oct 19, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] movingMedian doesn't handle gaps

2 participants