feat(onboarding): port the onboardingV2 skill and add its eval suite - #168
Conversation
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Remove first-flag/, references/1.8-summary.md, and references/1.9-editor-rules.md (the v2 flow has no summary or editor-rules step) and repoint every reference to the parent skill's Step 4.
Skill eval results
Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline. |
Drop the edge case claiming mcp/fm needs no migration, which contradicted the adjacent deprecation prompt, and name the Cursor toggle after the server the config actually writes.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
There was a problem hiding this comment.
Devin Review found 5 new potential issues.
1 security issue and 5 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
2 participants
Summary
Ports the
onboardingV2skill from launchdarkly/experimental-agent-skills intoskills/onboarding, and adds the matching eval suite.mcp/launchdarklyserver and treatsmcp/fmandmcp/aiconfigsas deprecated (migration is always asked, never automatic).first-flag/,references/1.8-summary.md,references/1.9-editor-rules.md.evals/onboardingsuite covering kickoff voice, monorepo ambiguity, MCP deferral, and drift handling; registered in the eval manifest, package scripts, and README.Testing
python3 scripts/validate_skills.py,python3 scripts/generate_catalog.py --check,python3 -m unittest discover -s tests, and(cd evals && npm test)all pass.npm run eval:onboardingscores 4/4 (100%), locally and in CI.Notes
improve/onboarding-ux) also changes onboarding UX and predates the V2 rewrite. It likely conflicts with, or is superseded by, this port.defaultproject after MCP setup, dashboard links interpolating unresolved keys, the wrap-up never routing back to the MCP offer, and the no-account placeholder path conflicting with "do not proceed until initialization is verified"). Left as-is here and answered in the review threads; worth a follow-up upstream.Implementation details
Skill
skills/onboarding/SKILL.mdis the V2 orchestrator, adapted to this repo: name/version stayonboarding, companion-skill installs point atlaunchdarkly/ai-tooling, and nested skills are referenced by their local paths. Additions on top of the source: the SDK key-type rule, the "non-LaunchDarkly dependency changes need approval" edge case, and a references list.mcp-configure(the V2 flow writes no log).mcp/fmneeds no migration (it conflicted with the adjacent deprecation prompt), and Cursor auth telling users to toggle on "LaunchDarkly feature management" when the config written is named "LaunchDarkly".skills/onboarding/README.md(the skill had none, and the eval manifest renders score badges into it); refreshedmarketplace.jsonand the root README description; regeneratedskills.json.Evals
evals/onboarding/promptfooconfig.yamladapts the source suite's three cases (kickoff, monorepo, drift) and adds a case asserting MCP is neither configured nor raised with the user before the flag works. Lean mode (allow_builtins: false) with per-testmax_turnskeeps runs to the graded beats.codebase_contextstates its listing as the result of the repository scan. Without that framing the agent sees no app and every fixture takes the skill's "no runnable app" branch, which is not the state under test.ask-questioncall rather than only the first, and the MCP check is scoped to the calls beforecreate-flag, since the MCP offer at the end of the flow is the correct behavior. The drift rubric reads the assistant turns plus theask-questionoptions, because the choice the skill requires is offered through that tool rather than in prose.force_skill_invocation,expose_ask_question,mock_ask_question_answers, andkickoff_text, so no harness changes were needed.Note
Overview
Replaces the onboarding orchestrator with a quieter, fixed sequence: short kickoff (no roadmap table or upfront account question), repo-detected resume instead of
LAUNCHDARKLY_ONBOARDING.md, work onlaunchdarkly-onboardingwith uncommitted changes, then explore → SDK install → first flag with a live in-app banner reveal → optional MCP.MCP moves to the end. Setup reaches the first flag via dashboard links when needed;
mcp-configureand catalog copy now say MCP is offered only after the flag works. MCP docs emphasize the unifiedmcp/launchdarklyserver, treatmcp/fmandmcp/aiconfigsas deprecated (migrate only after user consent), and probe tools before suggesting editor restart.Removes nested
first-flag/and follow-through references (1.8-summary,1.9-editor-rules); first-flag behavior lives in the parent skill.skills.jsondrops thefirst-flagskill entry and bumps onboarding to 0.3.0.Adds
evals/onboarding(kickoff voice, monorepo package choice, no MCP beforecreate-flag, drift redirect) pluseval:onboarding/:singlescripts and manifest registration; root and skill READMEs describe the new flow.Reviewed by Cursor Bugbot for commit f79ffd4. Bugbot is set up for automated code reviews on this repo. Configure here.
Link to Devin session: https://app.devin.ai/sessions/7227d142944143a4816dc815a2f365f4
Open in Devin Desktop: https://app.devin.ai/desktop/session/7227d142944143a4816dc815a2f365f4?variant=devin
Requested by: @ari-launchdarkly