Repository navigation
Conversation
The cache also works with the manual downloads within the recipes, giving some more sense to the helper package. Cache is invalidated every 10 days so that packages don't lag behind, and can be cleared if necessary for development.
Lets the full cache cycle (collect, merge, save, cleanup, reuse) run on this PR. To be reverted before merging. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Here is the build version of the cookbook in this pull request: documentation.zip, you can view it locally by unzipping |
|
I think having a cache that is not aware of when it is outdated will create confusion. E.g. a bug only appearing when the cache is refreshed, or the opposite. I think hugging face provides a cache functionality (we use it in metatrain I think) that is aware of outdated files. If the reason we are doing this is for hugging face downloads it should be possible to integrate. But if you really really really want to have it as implemented here, I would at least make it refresh before our weekly Monday run, not every 10 days. |
| directory, downloads go through a cache in that directory, keyed by URL, | ||
| so a file fetched once (e.g. by another recipe) is copied from there | ||
| instead of being downloaded again. The cookbook CI uses this to share | ||
| data files across recipes. With the variable unset nothing changes, any |
There was a problem hiding this comment.
What does "nothing changes" mean here?
Avoid "historical overview" style
I thought a lot about this. We keep getting rate limits on huggingface, zenodo, now also on conda because we download conda >40 times. The fact that there might be "delayed fails" is IMO no different from the fact that we get failures every monday morning, but it's annoying that every morning morning we need to relaunch 3 times before no runs fails because of rate limiting. I also feel bad for the insane amount of traffic that we generate downloading torch 60 times |
|
We can do it ok, but I would set the max age to 6 days or something like that, so that at least we ensure that there are no 2 consecutive builds on outdated stuff. Also, I don't have the bandwith to review the details of the implementation, which has lots of pieces, so:
|
The cache also works with the manual downloads within the recipes, giving some more sense to the helper package. Cache is invalidated every 10 days so that packages don't lag behind, and can be cleared if necessary for development.
Made with some help from an LLM