Skip to content

perf: cache UTM inverse transforms, MAVLink message info and parameter metadata lookups - #42

Open
userepo wants to merge 3 commits into
Rouniy:masterfrom
Poholos:perf/managed-lookup-fixes
Open

userepo wants to merge 3 commits into
Rouniy:masterfrom
Poholos:perf/managed-lookup-fixes

Conversation

@userepo

@userepo userepo commented Sep 21, 2026

Copy link
Copy Markdown

Three small lookup fixes, one commit each, smallest first. Each replaces linear or per-call work with a cached or indexed lookup and returns the same answers as before, verified by tests that compare the new code against a copy of the old lookup.

1. utmpos.ToLLA inverse transform cache (ExtLibs/Utilities/utmpos.cs)

ToLLA built a projected coordinate system and a transformation for every point; Grid.CreateGrid calls it once per waypoint on every survey slider change, on the UI thread. The inverse transform is now cached per signed zone, mirroring the forward cache in PointLatLngAlt. Zones outside the real UTM range are converted without caching, so values typed into the coordinate editor cannot grow the cache.

2. GetMessageInfo dictionary lookup (ExtLibs/Mavlink/MavlinkUtil.cs)

GetMessageInfo scanned the 350-entry message table linearly, two to three times per parsed packet, so the cost grew with the message id. The global MAVLINK_MESSAGE_INFOS table is now indexed in a dictionary that is rebuilt when the table is replaced (NvModemMavlinkDialect.Register and plugins do this at runtime); any other array keeps the linear scan. First-match, unknown-id and null-array behavior are unchanged.

Residual: an entry edited in place (MAVLINK_MESSAGE_INFOS[i] = ...) is not seen until the array is replaced. Nothing in this repo or in upstream's example plugin does that.

3. Parameter metadata document indexes (ExtLibs/Utilities/ParameterMetaDataRepositoryAPMpdef.cs, ParameterMetaDataRepositoryAPM.cs)

Every metadata lookup walked the XML linearly: all param elements of up to three pdef sources (local overlay plus downloaded document each), then XContainer.Element over thousands of siblings in the legacy file. Empty answers, the common case for Range, Increment, Bitmask and RebootRequired, are never cached, so filling a parameter table cost about 1.4 s every time, partly on the UI thread.

Each pdef document is now indexed by name attribute, keeping document order so scoped and unscoped entries resolve exactly as before; the legacy document is indexed by vehicle and parameter element name. An index is tied to its document and rebuilt when the document is replaced or edited. Empty answers are still not cached, because the metadata downloads asynchronously after first use.

Pre-existing and left alone: a vehicle type with no .apm.pdef.xml on disk still costs a File.Exists per lookup, and _parameterMetaDataXML is an unsynchronized dictionary written from a task continuation.

Measurements

Release build, .NET 10, Windows x64.

Workload Before After
100,000 utmpos.ToLLA calls 197-221 ms 16-26 ms
Grid.CreateGrid, 20,519 points 49 ms 5 ms
10 million GetMessageInfo calls, mixed ids 1,863 ms 146-155 ms
Tlog parse, 2M packets of common low-id messages 627-655 ns/packet 568-585 ns/packet
Tlog parse, 2M packets of ArduPilot-specific messages 1,073-1,096 ns/packet 571-575 ns/packet
Parameter metadata, 1,200 names x 8 keys, first pass 1,714-1,744 ms 27 ms
Same, warm passes 1,371-1,468 ms 9-14 ms

Verification

  • New tests: UtmposTransformCacheTests (11), MavlinkMessageInfoLookupTests (8), ParameterMetadataIndexTests (10), over the packaged metadata files and synthetic documents that cover the ordering rules.
  • 451,176 parameter metadata answers through the public ParameterMetaDataRepository (6,836 names, 11 keys, 6 vehicle types including the SITL, AP_Periph and legacy fallbacks) are byte-identical between old and new code.
  • Full suite green apart from the tests that already fail on Windows on master.

ToLLA built a projected coordinate system and a coordinate
transformation for every point. Cache the inverse transform per signed
zone instead. Zones outside 1-60 are converted without caching so that
zone numbers typed into the coordinate editor cannot grow the cache.
Results are bit-identical to the uncached path.

Grid.CreateGrid on a 20,519-point survey drops from 49 ms to 5 ms.
GetMessageInfo scanned the 350-entry message table linearly, two to
three times per parsed packet, so the cost grew with the message id
(11 ns for HEARTBEAT, 489 ns for ESC_TELEMETRY_1_TO_4). Build a
dictionary per table instance and rebuild it when the table is
replaced, as NvModemMavlinkDialect.Register does at runtime. First
match, unknown-id and null-array behavior are unchanged.

Lookups are now about 14 ns for every id. Parsing a tlog of
ArduPilot-specific messages is about 1.9x faster; a tlog of common
low-id messages about 7% faster.
Every metadata lookup walked the XML documents linearly: a LINQ scan of
all param elements in up to six pdef documents, then XContainer.Element
over thousands of siblings in the legacy file. Answers that are empty,
the common case for Range, Increment, Bitmask and RebootRequired, were
never cached, so filling a parameter table cost about 1.4 s every time,
partly on the UI thread.

Index each pdef document by name attribute, keeping document order so
scoped and unscoped entries resolve exactly as before, and index the
legacy document by vehicle and parameter element name. Indexes are tied
to their document: a replaced document gets a new index, and an edited
pdef document drops its index.

1,200 parameters x 8 keys: 1,714 ms -> 27 ms cold, 1,371 ms -> 13 ms
warm, with identical answers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant