Workshop output should be refuseable slices—not consensus that “we should plug in GPT too”
When brands decide to “add AI,” kickoff often means a demo day: watch competitor chat widgets, then pick a slot for “a bot first.” Three months later cost, hallucinations, and ticket volume rise together. Running an AI product discovery workshop restores the model choice to a product decision: which user jobs deserve automation or assist, who owns failure cost, whether data is reachable, how to eval “good enough,” and what phase one explicitly will not do. KSX nests AI discovery in Discover: workshops force jobs, red lines, eval sets, and slices; Design shapes assistant/workflow UX rather than mascots; Build wires models, tools, and observability; Launch gates on evals, not demo fluency. Below: a reusable agenda, required artifacts, and traps—for support assist, internal knowledge, generative editing, and vertical workflows—not another unbounded “smart CS” slogan.
Mandatory pre-reads: tickets, transcripts, docs, and how humans do it today
AI workshops without evidence decay into sci-fi brainstorming. Collect 72 hours prior: 30–50 high-frequency tickets or sales Q&As, knowledge-base inventory, permission and sensitive-field notes; competitors as risk reference, not a feature shopping list. Business brings “how a human finishes the job today”; eng brings residency and vendor constraints. Missing evidence means reschedule—or demote the meeting to problem collection with no scope freeze.
Job map: assist, draft, retrieve, act—four risk levels
Plot candidate capabilities on four levels: (1) assist comprehension (summarize, explain); (2) draft (human publishes); (3) retrieval-augmented (with citations); (4) act (mutate data, place orders, send messages). Higher levels demand heavier evals and permissions. Allow only one “act” level into deep discussion; demote or drop the rest. Name whether users are internal or external—customer-facing failure cost is higher by default; guardrails and escalation belong in phase one.
Value hypotheses and kill criteria: which numbers stop the build
For each selected job write: baseline time/conversion, expected lift, measurement method, kill criteria (e.g. “if human edit rate >40%, disable auto-send”). Ban “better experience” with no proxy metric. Cost hypothesis separate: rough token/call volume, peaks, whether a non-AI fallback is allowed. Jobs without a clear ROI story go to the parking lot—not the MVP.
Data and permission whiteboard: what is usable, what never enters prompts
Draw system bounds: read-only knowledge, ticket fields, PII, payment/health red lines. Decide redaction rules, tenant isolation, log retention, and whether third-party models are allowed. Internal knowledge (RAG-style) needs document owners and refresh cadence—stale docs are more dangerous than “no AI.” End with an allow/deny data table for legal and security co-sign.
Eval-set owners: no gold labels, no launch date
Name an eval-set owner in the room (usually business + QA). Minimum set: 50–100 real questions, expected points or citations, failure samples (overreach, fabrication, competitor smears). Define severity and launch-blocking failure types. State regression cadence (every prompt/model change). KSX rejects “users will teach it after launch” as the only quality strategy—that makes customers free QA.
UX principles: cite, refuse, handoff, undo
Set UX principles in the workshop: must answers carry sources; when to say “I don’t know”; one-click human handoff with context; how users undo bad actions. Empty/loading states and cancel-during-stream belong in principles, not last-minute eng taste. Brand voice has bounds: humor never overrides a safe refusal.
MVP slice and non-goals: demos are not products
Phase one serves one role, one primary path, one success metric. Explicit non-goals: sitewide entry points, full autonomy, multi-model routing for show, “creative mode” without evals. Timebox (e.g. 6–8 weeks) and gates: eval pass rate, latency, cost ceiling. Demo day may show more; the scope doc only recognizes the slice. Manufacturing-style internal assistants (knowledge retrieval) and public marketing-site features should be separate charters—one prompt does not rule them all.
Agenda timeboxes and KSX artifact templates
Recommended full day: morning evidence readout, job leveling, value/kill criteria; afternoon data permissions, evals and UX principles, MVP and signed non-goals. Half-day only does job leveling + non-goals; evals get a dedicated session. Artifacts: job map, allow/deny data table, eval owner and minimum-set plan, MVP slice, numbered open questions. Need facilitation? Send a one-pager and sample tickets to hi@keshangxian.com—we take AI from slogan to gated product increment through discover→design→build→launch.
Checklist
- 130+ real tickets/Q&As and knowledge inventory prepared before the room
- 2Capabilities leveled assist/draft/retrieve/act with act-level count capped
- 3Each MVP job has baseline, measurement, and kill criteria
- 4Allow/deny data table and eval-set owner assigned
- 5MVP slice, non-goals, and launch gates (eval/cost/latency) written and confirmed
Key takeaways
- Running an AI product discovery workshop: jobs and failure cost first; models and vendors second.
- No eval-set owner means no credible launch date.
- MVP serves one role and one path; demo-day scope must not enter the contract.
FAQ
- Must engineers attend the workshop?
- Someone who can veto infeasible integrations must be present, or you only ship a wish list. If eng has half a day, cluster data-permission and feasibility blocks for them.
- We already bought a model subscription—still need discovery?
- More so. Subscriptions buy invocation rights, not jobs, data, evals, or UX red lines. The workshop keeps budget off the wrong entry point.
- How does KSX AI discovery differ from generic project discovery?
- On top of generic scope tools we add job risk levels, data allow lists, eval gates, and cost hypotheses, with guardrails and human handoff as defaults. Contact hi@keshangxian.com.