Test analyzer in lazy mode - #276
Conversation
|
@chrishalcrow can you take a look at this one? |
| else: | ||
| self._cache_data = {} | ||
|
|
||
| def cache_data(self): |
There was a problem hiding this comment.
This is cool. Just some real world stuff: on a 30 hr 96 channel analyzer, turning the cache on added 500MB of RAM use per spike amp + scaling. Great for my use case but I'd probably keep the default False.
| def on_decimate_or_cache_change(self, *args): | ||
| if self.settings["cache_data"]: | ||
| self.cache_data() | ||
| else: | ||
| self._cache_data = {} |
There was a problem hiding this comment.
Little thing: when you change e.g. auto_decimate, the GUI will run _on_settings_changed first (which includes a refresh), then on_decimate_or_cache_change. This means the scatter view doesn't refresh when you change decimate values.
I can't think of a neat way to deal with this. A brute force would be to add another refresh here.
There was a problem hiding this comment.
good point. I'll think about it...
There was a problem hiding this comment.
but it will refresh at the next selected unit, right?
There was a problem hiding this comment.
I think you updated the behavior, but I'm not sure how you did it...
| print('Force compute "noise_levels" is needed') | ||
| ext = analyzer.compute_one_extension('noise_levels') | ||
| self.noise_levels = ext.get_data() if ext is not None else None | ||
| self.noise_levels = np.asarray(ext.get_data()) if ext is not None else None |
There was a problem hiding this comment.
Do these also force a materialization?
There was a problem hiding this comment.
yes and on purpose: these arrays are small to keep in RAM, but would require a remote read (for every new unit). Same is done for:
random_spikestemplates_average/stdccgs/bins
|
Some more benchmarks in a real 1352 units / ~25M spikes dataset on s3 (remote access):
Note that lazy + cache (with SpikeInterface/spikeinterface#4806) is as snappy as memory! |
2 participants
Using this: SpikeInterface/spikeinterface#4623
Just adds the option to run tests with
--lazyflagThe only code modification is to avoid copying and extending the spike vector, but keep external arrays of channel indices and random selected spikes.