[yinggeh]

What does the PR do?

Serves a Model Connect bundle from the TensorRT backend, one request at a time. text_generation and text_continuation take one string or one token-id sequence. structured_ranking takes named numeric tensors. Plan models stay on the existing path.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

triton-inference-server/core#541

Where should the reviewer start?

src/tensorrt.cc for the plan versus model_connect dispatch, then src/model_connect/instance_state.cc for load and execute.

Test plan:

Loaded dazr, ajk50, ajk64, feed, and gpt2 one at a time. Compared each Triton response with the native SDK on the same inputs. Every output matched.

• CI Pipeline ID:

Caveats:

One request at a time. No batching and no streaming. The ranking path is built against the private SDK.

Background

The bundle task is read from the bundle header before the model is loaded.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes GitHub issue: #xxx

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; both previous findings are addressed and no new actionable issue was established.

Summary

The PR adds Model Connect bundle serving alongside the existing plan path.

• Triton loads bundles by the task named in each header.
• Model Connect requests return text or ranking results.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Model initialization] --> B{Model type}
  B -->|Plan| C[Existing plan path]
  B -->|Model Connect| D[Read bundle header]
  D --> E[Load matching task]
  E --> F[Copy one request to host]
  F --> G[Run text or ranking task]
  G --> H[Write response and release request]

Loading

_{Reviews (2) · Last reviewed commit: "fix: Read Model Connect SDK errors after..." · Reviewed by Greptile}

[whoisj]

LGTM with the exception that I don't think any of the new files are being compiled.

Have they been added to the CMakeLists.txt file?