Product Sense

Source: Concepts adapted from Ace the Data Science Interview — Chapter 10 (“Product Sense”) by Kevin Huo & Nick Singh. Rewritten in original wording for this course.

Product-focused DS interviews test whether you can partner with PMs: pick metrics, dig into weird changes, brainstorm features, and design honest experiments. Technical chops still matter — but here the bar is judgment, structure, and company context. Pretend you already work there and you’re in a working meeting with a teammate.

Four common question types

Type	Example prompt
Define a metric	How would you measure success of a new launch / feature change?
Diagnose a metric change	Engagement dropped — how do you find the root cause? What about trade-offs with counter-metrics?
Brainstorm product / features	Should we launch X? What features would move metric Y?
Design an A/B test	How would you test feature Z? What pitfalls, and how do you handle them?

Keep these templates in mind as you read the frameworks below.

How to approach any product question

• Clarify — user flow, end users, stakeholders, product & business goals. Even if you researched the company, phrase knowledge as a check-in (“Robinhood’s mission is… this crypto wallet seems aimed at… am I on track?”).
• Scope — these prompts are huge; say what you’re deliberately ignoring for the interview clock.
• Talk out loud — they hired you to hear your thinking.
• Be conversational — check in (“I’ll use avg watch time as the engagement north star — sound good?”).
• Keep goals forefront — answers hang off mission / vision you stated early.
• Bring experience tactfully — real-world parallels help; “this is the only way because Google did it” does not.

Ultimate cheat code: behave like you’re already on the team.

Quiz

Question types & approach

1 / 2

The four staple product-DS question types are…

Pick an answer to see if you are right.

Building product & business sense

Product sense is a skill, not a birthright. Daily habit — when you use a product, ask:

• Who was it created for?
• What main problem does it solve?
• What end-user benefits go beyond the narrow problem?
• How do design / copy reinforce purpose?
• How does it tie to company mission?

Empathy matters. Snapchat vs Instagram look similar at a glance but optimize different jobs (ephemeral authentic sharing vs polished feed consumption). Calibrate with reviews (App Store, Play, Reddit, G2 / Gartner, Amazon) — positives articulate benefits; negatives spark brainstorm ideas.

Business sense (even if “model free cash flow” is rare): follow the money so you work on leveraged problems.

• Monetization model & levers
• KPIs and what moves them
• Landscape — competitors & partners

Homework that levels the playing field against bubble-dwelling interviewers / internal candidates:

1. Business model + revenue mix (earnings / analyst coverage / comps if private)
2. Marketplace or SaaS vocabulary for that company type
3. Engineering blog posts (Uber Eats delivery ETA, food discovery, etc.)
4. Use the product from every side (eater + driver + restaurant portal for Uber Eats)

Yes, it’s a lot — and “Did you try our product? Ideas?” is a real question. Research also upgrades your closing questions.

Metrics: AARRR + engagement

Dave McClure’s pirate metrics (AARRR):

Stage	Question	Example metrics
Acquisition	How do users find you?	New users, signup conversion, CAC
Activation	Great first experience / “aha”?	First order, first post, 10 feed views
Retention	Do they come back?	Monthly retention / churn
Referral	Do they tell others?	Invites × invite conversion = k-factor (k>1 → viral)
Revenue	How do you make money?	LTV, paid upgrades, ads; aim for healthy LTV/CAC

Engagement (often called out even if not a letter in AARRR): intensity / frequency of core actions — time spent, rides booked, DAU / WAU / MAU, stickiness ≈ DAU/MAU.

Do homework for the company type (SaaS → ACV, MRR; marketplaces → GMV, take rate). Lean Analytics is a solid deeper dive.

Compute

Compute: funnel metrics

1 / 5

Story problem — figure out which numbers are the formula inputs, then compute.

Viral k-factor = (invites/user)×(conversion). i=5, c=0.3. What is k? Round to 2 decimals.

Enter a number (at most 2 decimal places).

Practice: metric arithmetic

What makes a metric good or bad?

Bad metrics (Facebook Dating examples):

Flaw	Why it fails
Vanity	Profiles viewed — sounds busy, weak signal of meaningful relationships
Irrelevant	Time spent — great for YouTube, weak for activity-driven dating value
Impractical	# of 3rd dates — hard/impossible to measure reliably
Complicated	Stakeholders can’t explain a move without dissecting the formula
Delayed	# of marriages — too laggy for product decisions

Good metrics reverse those: meaningful (tied to goals, hard to game, actionable), measurable, understandable, timely (leading where possible).

Always pair a north star with guardrail / counter metrics you refuse to trash while optimizing (e.g. cut harmful content without nuking posts / views / comments).

Quiz

Good vs bad metrics

1 / 3

A vanity metric…

Pick an answer to see if you are right.

Framework: define success metrics (3 steps)

Example: “How would you measure success of Facebook Dating?”

1. Clarify product & purpose — business why, casual vs serious dating, audience, standalone vs in-app, UI mechanics, launch market.
2. State product & business goals — synthesize (don’t only ask); tie to mission (meaningful relationships) and ecosystem (engagement → ads, keep people in Meta apps).
3. Define metrics along the funnel — acquisition (signups / profiles), activation (rich profile, first match), engagement (matches, swipes, meet signals), retention, revenue (subs / ads) — and say what matters at launch (often PMF / retention before monetization).

Framework: diagnose a metric change (4 steps)

Example: “Avg comments per Instagram post is declining — troubleshoot.”

1. Scope — definition (ratio? which part moved?), importance, timeframe, magnitude (WoW vs YoY), absolute vs relative.
2. Hypothesize in buckets: accidental (logging bugs), natural (seasonality), internal (UI / ranking / features), external (competitors, macro). Walk up the funnel from local feature → broader surfaces; don’t dump 20 theories — prioritize.
3. Validate — segment (demo, device, cohort), compare likes/shares vs comments, feed vs Stories/Reels, comment composer views.
4. Classify findings: root cause / contributing / correlated symptom / unrelated.

Classic ending: onboarding interstitial teaching “turn off comments” created more comment-disabled posts → average comments/post fell even while commenting health on comment-enabled posts was fine.

Metric trade-offs

“Ads +1% revenue, time spent −3% — ship?” Clarify both metrics, map to strategy, then recommend revert, mitigate (hide ad / better targeting), or accept — often conditional on how hard each % is to earn back. Interviewers want judgment process, not a single “correct” ship bit.

A/B testing in product interviews

After metrics definition you’ll often hear “How would you test it?” Cover four setup steps:

1. Pick primary metric(s) (+ guardrails) — no post-hoc cherry-picking
2. Thresholds — α≈0.05, power≈0.8, MDE with stakeholders
3. Sample size & length — from variance / MDE / traffic; often ≥2 weeks for day-of-week
4. Random assignment — avoid confounding (platform may handle this)

When not to A/B test

No infrastructure, tiny impact vs cost, no traffic, no conviction (random monkey variants), no isolation (logo change hits headlines). Alternatives: UX research, log analysis, ship + retrospective.

Practical pitfalls

Pitfall	Idea
Non-normal metrics	Bootstrap, rank tests, more data
Multiple testing	Bonferroni / FDR / FWER; dig into overlapping experiments
Network effects	Spillover understates effects; cluster / graph partition randomization
Novelty / primacy	Spike or backlash then settle; new-user slices / longer runs / holdouts
Significant ≠ ship	Effect size, guardrails, eng/support cost, bug risk
Holdouts	Tiny long-run control after launch — skip for sensitive safety fixes

Quiz

Product A/B testing

1 / 4

A good experiment setup always includes…

Pick an answer to see if you are right.

Quick synthesis

Topic	Sound bite
Four prompts	Metrics, diagnosis, brainstorm, A/B
Approach	Clarify, scope, converse, mission-first
Prep	Use the product, read blogs/earnings, learn model-type KPIs
Funnel	AARRR + engagement; k-factor; LTV/CAC
Metric quality	Meaningful, measurable, clear, timely + guardrails
Diagnosis	Scope → hypothesize → validate → classify
Experiments	Design upfront; know when not to test; watch networks & novelty

Pair with Statistics for the math of tests and End-to-end ML workflow for shipping judgment.

Next: Product sense problems.