Skip to content

feat: track template shortcut and subdirectory used with create - #737

Draft
lukegalbraithrussell wants to merge 1 commit into
mainfrom
feat/track-template-subdir-shortcut
Draft

lukegalbraithrussell wants to merge 1 commit into
mainfrom
feat/track-template-subdir-shortcut

Conversation

@lukegalbraithrussell

Copy link
Copy Markdown
Contributor

Changelog

N/A — telemetry only, no change to the user experience.

Summary

This pull request adds the template shortcut and template subdirectory to the telemetry event sent by slack create.

We want to measure how app templates are chosen, and today's telemetry can't answer these questions:

  • Is the agent shortcut used? slack create agent and slack create with "AI apps" picked from the menu send the same event (command: "create", same data.app.template), because positional arguments aren't recorded.
  • Which agent framework is chosen? data.app.template only holds owner/repo, so the Claude Agent SDK and OpenAI Agents SDK variants of a starter (subdirectories of the same repo) can't be told apart.

Changes:

  • Adds data.app.template_shortcut (e.g. agent, empty otherwise) and data.app.template_subdir (e.g. claude-agent-sdk, from --subdir or the selected template) to AppEventData, both omitempty.
  • Redacts PII from template_subdir the same way as template, since custom --subdir values are user input.
  • slack samples runs create internally, so it records these fields too. slack init doesn't use templates and is unchanged.
  • Existing fields are unchanged, so current queries on data.app.template keep working.

Example event data for slack create agent my-app → Starter Agent → Bolt for JavaScript → Claude Agent SDK:

"app": {
  "template": "slack-samples/bolt-js-starter-agent",
  "template_shortcut": "agent",
  "template_subdir": "claude-agent-sdk"
}

Preview

N/A

Testing

  1. Build with make build.
  2. Run ./bin/slack create agent my-agent --verbose, select a starter, a framework, and an agent SDK.
  3. In the FlushToLogstash will POST debug line, confirm data.app.template_shortcut is agent and data.app.template_subdir matches the chosen SDK.
  4. Run ./bin/slack create my-app -t slack-samples/bolt-js-starter-template --verbose and confirm neither new field is present.

Notes

  • The new keys may need to be registered in the logstash/ATS pipeline before they're queryable.

Requirements

Co-Authored-By: Claude svc-devxp-claude@slack-corp.com

Adds template_shortcut and template_subdir to the app event data sent
with telemetry so usage of "slack create agent" and the selected agent
framework can be measured.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
@codecov

codecov Bot commented Oct 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.19%. Comparing base (7a79c88) to head (1cac15b).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #737   +/-   ##
=======================================
  Coverage   78.18%   78.19%           
=======================================
  Files         240      240           
  Lines       18166    18176   +10     
=======================================
+ Hits        14203    14212    +9     
- Misses       3963     3964    +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant