Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
新增
core.skip_store设置(仅 online 模式合法):开启后 SDK 不在本地产生任何文件(无swanlog/、run-*.swanlab、media/、files/、debug/),全部数据直传云端。Related Issue: #1713
使用
设计
协议:proto 全部追加字段,向后兼容——
MediaItem.payload=5(optional,区分空文件与缺失)、SaveRecord.payload=6、CoreSettings.skip_store=10、ProbeSettings.skip_store=13。本地 DataStore 文件格式版本未变,旧swanlog仍可被新版读取与sync。落盘短路:
DataStoreWriter(skip=True)的 open/write/close 全部无副作用(write 仅计数供 close 统计);init跳过一切目录创建;诊断日志只输出终端。媒体:
transform(path=None)时将内容写入MediaItem.payload而非落盘;sender 经MemoryViewReader零拷贝内存直传对象存储,不触碰本地 media 路径。内部 save(config/metadata/requirements/conda):内容按落盘同款编码内联进
SaveRecord.payload(config 复用dump_config,避免云端结构漂移);sender 解析后走 profile 上传。解析失败属确定性脏数据,告警跳过,不进入 Transport 无限重试;上传失败保持既有 ApiError 分类(5xx 重试 / 4xx 跳过)。CUSTOM save:
payload恒空(非空视为协议违约丢弃),从source_path原路径读取上传,不创建本地软链接镜像;policy="live"的 watcher 直接监听源文件所在目录(direct-source 模式,事件路径精确匹配,同源多 name 一对多注册)。probe:通过新增的显式
ProbeSettings.skip_store字段感知模式,metadata/requirements/conda 直接注入 payload,不写files/目录。模式约束:Settings validator 保证 skip_store 仅 online 合法(含
cloud别名归一化、env 注入、merge 降级路径);交互式引导从 online 降级到 offline 时显式关闭 skip_store 并告警,保证离线数据正常落盘。取舍
swanlab sync/swanlab watch不适用;init 时打印警告明示。测试
单元测试与静态检查
uv run pytest tests/unituv run ruff check .uv run basedpyrightcd core && go build ./...make protoprotos/源同步,无额外 diff覆盖点:
基准测试
新增两个基准(
tests/benchmark/),分别衡量本地持久化层与端到端运行时链路的开销差异。1. 本地持久化层
tests/benchmark/sdk/internal/core_python/store/bench_store_skip.py,直接走生产路径CorePython._store_records,对比落盘(skip_store=False)与完全跳过持久化(skip_store=True)的本地工作:bench_metrics_steps.py对齐),落盘写 LevelDB log,skip 仅计数Image.transform),落盘写media/image/(含 fsync),skip 内联MediaItem.payloadCore._handle_custom_save),落盘建软链接镜像,skip 不建本机(macOS / Apple Silicon,Python 3.11,每类重复 3 次取最优):
落盘侧附加指标:标量吞吐 1.97 M rec/s、媒体写入带宽 140.8 MiB/s、
run-*.swanlab34.4 MB、媒体文件 1,000 个 / 32,768,000 B、save 镜像 100 个;skip 侧无任何本地文件。数据完整性:标量可完整回读,媒体文件数/字节数、save 镜像数均与写入一致。2. 端到端(online + mock HTTP)
tests/benchmark/sdk/cmd/bench_skip_store_e2e.py,完整跑swanlab.init → log/log_image/save → finish,HTTP 全部 mock,对比 skip_store 对用户线程延时与端到端吞吐量的影响。为让 producer / finish 两阶段边界确定,record_interval设为大值,上传统一发生在 finish。负载:1000 step × 100 key = 100,000 标量、250 张媒体、50 个 save。
本机(同上,每场景重复 2 次取最优):
run.log平均延时run.logp50 / p95 / p99解读:
run.log延时基本不变(~113 µs),因为生产端本就只做入队。