Skip to content

Bug: sam local invoke unconditionally rebuilds docker image on every invocation when a layer is declared #9255

Open

Description

@marcoreni

Description:

sam local invoke unconditionally rebuilds the Docker image on every single invocation when the function has a declared layer in the template (AWS::Serverless::LayerVersion with a local ContentUri).

Root cause (traced in source, v1.165.0):

  • samcli/local/docker/lambda_image.py:276-281 forces _build_image() whenever any(layer.is_defined_within_template for layer in downloaded_layers) is true — no hash/content comparison, this is a blanket condition.
  • samcli/lib/providers/provider.py:301 sets is_defined_within_template = bool(codeuri) — true for any layer built locally, unconditionally, regardless of whether its content actually changed since the last build/invoke.
  • The rebuild itself (_build_image → create_tarball, samcli/lib/utils/tar.py:51-53) re-tars the entire layer directory via Python's tarfile, one file at a time, before Docker's build API/cache is ever reached. On our layer (~30MB, ~5300 files under node_modules) this alone costs 60-90s, every call.

SAM CLI already has the infrastructure to avoid this: sam build itself decides whether to reinstall a layer's dependencies by hashing the manifest (package.json/lockfile) via DependencyHashGenerator in samcli/lib/build/build_strategy.py, not by hashing/re-reading the installed node_modules tree. lambda_image.py's invoke-time rebuild check does not reuse this at all.

Steps to reproduce:

  1. Template with a function that has Layers: [!Ref DepsLayer], where DepsLayer is AWS::Serverless::LayerVersion with a local ContentUri and Metadata: BuildMethod: makefile (i.e. built locally, not referencing a published/remote layer ARN).
  2. sam build
  3. sam local invoke MyFunction -e event.json --debug
  4. Immediately run the exact same command again, with no changes to any file.
  5. Both invocations pay the same "Building image" cost.

Observed result:

Two consecutive invokes, same template, same build, zero changes in between:

=== invoke 1 ===
2026-09-04 19:31:57,863 | Local image is up-to-date
2026-09-04 19:33:24,098 | Mounting /path/.aws-sam/build/MyLambda as /var/task:ro,delegated, inside runtime container
sam local invoke MyLambda -e event.json --debug 1:31.82 total

=== invoke 2 ===
2026-09-04 19:33:29,058 | Local image is up-to-date
2026-09-04 19:34:29,438 | Mounting /path/.aws-sam/build/MyLambda as /var/task:ro,delegated, inside runtime container
sam local invoke MyLambda -e event.json --debug 1:05.05 total

86s and 60s respectively spent between "Local image is up-to-date" and "Mounting" — the silent image-rebuild window — on both calls, despite nothing having changed between them.

Expected result:

sam local invoke should reuse the existing rapid image as much as possible — the same way SAM already skips reinstalling a layer's dependencies during sam build when its manifest hash is unchanged.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: macOS 26.6.2 (arm64)
  2. sam --version: SAM CLI, version 1.165.0

{
"version": "1.165.0",
"system": {
"python": "3.14.7",
"os": "macOS-26.6.2-arm64-arm-64bit-Mach-O"
},
"additional_dependencies": {
"container_engine": "Docker(v29.7.2)",
"aws_cdk": "Not available",
"terraform": "1.7.4"
}
}

Activity

  1. added
    stage/needs-triageAutomatically applied to new issues and PRs, indicating they haven't been looked at.
    on Sep 4, 2026
  2. roger-zhangg commented on Sep 4, 2026

    @roger-zhangg
    Member

    Thanks for reporting this issue, we are taking a look and will ship a fix with next SAM CLI release. In the meantime, you can unblock by installing our previous version: https://github.com/aws/aws-sam-cli/releases#release-v1.164.0

  3. roger-zhangg commented on Sep 4, 2026

    @roger-zhangg
    Member

    Hi @marcoreni I tried to reproduce this issue. Could you help me to confirm if this is actually a regression in version 1.165.0? Based on my understanding the rebuild behavior on layer has been there at least starting 2020. This was due to SAM CLI cannot effectively tell if the layer has changed or not in between of invokes. So it triggers re-build just to be safe.

    Similar request: #3725

    If what you are experiencing is an actual regression introduced in 1.165.0, please help to try out 1.164.0 and provide more info here.

    If 1.164.0 turns out to have similar behavior, we can take this as a feature request but not as a bug.

  4. added and removed
    stage/needs-triageAutomatically applied to new issues and PRs, indicating they haven't been looked at.
    on Sep 4, 2026
  5. added 2 commits that reference this issue on Sep 4, 2026
    e810e30
    5fbfb06
  6. marcoreni commented on Sep 6, 2026

    @marcoreni
    Author

    It's not a regression, I was able to track it in the code (with the help of AI), but I didn't check "when" the code was added :)

    I also agree that #3725 is similar, however it covers start-api and not invoke. start-api has a few differences vs invoke (eg. no "--mount-symlinks" - I will followup on this with a different issue) so I assumed that there are two different "pipelines" and I opened a new issue.

    I also saw the PR, however I'm not sure that it would help much in our scenario: if the lambda layer includes the dependencies dir (as I assume most people are using layers for), computing the checksum on each run may be almost-as-slow as rebuilding the layer.

    May there be a way to specify on "which files/folders" to compute the hash on? Then, one would simply pick lockfile (or equivalent) or whatever they see fit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions