Repository navigation
Bug: sam local invoke unconditionally rebuilds docker image on every invocation when a layer is declared #9255
Description
Description:
sam local invoke unconditionally rebuilds the Docker image on every single invocation when the function has a declared layer in the template (AWS::Serverless::LayerVersion with a local ContentUri).
Root cause (traced in source, v1.165.0):
samcli/local/docker/lambda_image.py:276-281forces_build_image()wheneverany(layer.is_defined_within_template for layer in downloaded_layers)is true — no hash/content comparison, this is a blanket condition.samcli/lib/providers/provider.py:301setsis_defined_within_template = bool(codeuri)— true for any layer built locally, unconditionally, regardless of whether its content actually changed since the last build/invoke.- The rebuild itself (
_build_image→create_tarball,samcli/lib/utils/tar.py:51-53) re-tars the entire layer directory via Python'starfile, one file at a time, before Docker's build API/cache is ever reached. On our layer (~30MB, ~5300 files undernode_modules) this alone costs 60-90s, every call.
SAM CLI already has the infrastructure to avoid this: sam build itself decides whether to reinstall a layer's dependencies by hashing the manifest (package.json/lockfile) via DependencyHashGenerator in samcli/lib/build/build_strategy.py, not by hashing/re-reading the installed node_modules tree. lambda_image.py's invoke-time rebuild check does not reuse this at all.
Steps to reproduce:
- Template with a function that has
Layers: [!Ref DepsLayer], whereDepsLayerisAWS::Serverless::LayerVersionwith a localContentUriandMetadata: BuildMethod: makefile(i.e. built locally, not referencing a published/remote layer ARN). sam buildsam local invoke MyFunction -e event.json --debug- Immediately run the exact same command again, with no changes to any file.
- Both invocations pay the same "Building image" cost.
Observed result:
Two consecutive invokes, same template, same build, zero changes in between:
=== invoke 1 ===
2026-09-04 19:31:57,863 | Local image is up-to-date
2026-09-04 19:33:24,098 | Mounting /path/.aws-sam/build/MyLambda as /var/task:ro,delegated, inside runtime container
sam local invoke MyLambda -e event.json --debug 1:31.82 total
=== invoke 2 ===
2026-09-04 19:33:29,058 | Local image is up-to-date
2026-09-04 19:34:29,438 | Mounting /path/.aws-sam/build/MyLambda as /var/task:ro,delegated, inside runtime container
sam local invoke MyLambda -e event.json --debug 1:05.05 total
86s and 60s respectively spent between "Local image is up-to-date" and "Mounting" — the silent image-rebuild window — on both calls, despite nothing having changed between them.
Expected result:
sam local invoke should reuse the existing rapid image as much as possible — the same way SAM already skips reinstalling a layer's dependencies during sam build when its manifest hash is unchanged.
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
- OS: macOS 26.6.2 (arm64)
sam --version: SAM CLI, version 1.165.0
{
"version": "1.165.0",
"system": {
"python": "3.14.7",
"os": "macOS-26.6.2-arm64-arm-64bit-Mach-O"
},
"additional_dependencies": {
"container_engine": "Docker(v29.7.2)",
"aws_cdk": "Not available",
"terraform": "1.7.4"
}
}
Activity
roger-zhangg commentedon Sep 4, 2026
Thanks for reporting this issue, we are taking a look and will ship a fix with next SAM CLI release. In the meantime, you can unblock by installing our previous version: https://github.com/aws/aws-sam-cli/releases#release-v1.164.0
roger-zhangg commentedon Sep 4, 2026
Hi @marcoreni I tried to reproduce this issue. Could you help me to confirm if this is actually a regression in version 1.165.0? Based on my understanding the rebuild behavior on layer has been there at least starting 2020. This was due to SAM CLI cannot effectively tell if the layer has changed or not in between of invokes. So it triggers re-build just to be safe.
Similar request: #3725
If what you are experiencing is an actual regression introduced in 1.165.0, please help to try out 1.164.0 and provide more info here.
If 1.164.0 turns out to have similar behavior, we can take this as a feature request but not as a bug.
fix(local): reuse the invoke image when layer content is unchanged
fix(local): reuse the invoke image when layer content is unchanged
marcoreni commentedon Sep 6, 2026
It's not a regression, I was able to track it in the code (with the help of AI), but I didn't check "when" the code was added :)
I also agree that #3725 is similar, however it covers
start-apiand notinvoke.start-apihas a few differences vsinvoke(eg. no "--mount-symlinks" - I will followup on this with a different issue) so I assumed that there are two different "pipelines" and I opened a new issue.I also saw the PR, however I'm not sure that it would help much in our scenario: if the lambda layer includes the dependencies dir (as I assume most people are using layers for), computing the checksum on each run may be almost-as-slow as rebuilding the layer.
May there be a way to specify on "which files/folders" to compute the hash on? Then, one would simply pick lockfile (or equivalent) or whatever they see fit.