Data Analyst Mock Interview: Metric Diagnosis + A/B Test Drill (2026)

The 4-Dimensional Rubric

• ●Hypothesis rigor: Can you state 3-5 mutually exclusive hypotheses before touching data?
• ●Cohort framing: Do you slice by the dimension most likely to surface the signal (device, geography, cohort, funnel stage)?
• ●A/B design: Do you pick a statistic, a minimum detectable effect, and a sample size?
• ●Synthesis: Can you recommend a next action with a confidence level and a risk?

The Metric Diagnosis Drill

Your dashboard shows day-7 retention down 12% over the last 2 weeks. The interviewer is silent. Where do you start?

• ●Step 1: Validate the metric. Is day-7 retention defined the same way? Any recent pipeline changes?
• ●Step 2: Isolate the segment. Is the drop across all users or a cohort (new signups, specific country, specific device)?
• ●Step 3: Form 3 hypotheses. Product change, marketing change, instrumentation bug. Rank by prior.
• ●Step 4: Test each. Ask for data to eliminate or confirm. If retention dropped only on Android and you shipped an Android release 2 weeks ago, you have your answer.
• ●Step 5: Synthesize. Likely cause, confidence, next action, rollback option.

Common Metric Diagnosis Scenarios

• ●"Checkout conversion dropped 5% on Monday and did not recover. Diagnose."
• ●"DAU looks flat but sessions per user is up 8%. What is going on?"
• ●"Email CTR is up 20% but downstream conversion is flat. Is it real?"
• ●"Revenue per user is stable but ARPPU dropped 15%. What is happening in the user mix?"

A/B Test Design Drill

You propose a product change that you believe will lift day-1 activation by 4 percentage points. Design the test.

• ●Primary metric: Day-1 activation rate.
• ●Secondary metrics: Day-7 retention, activation definition stability, crash rate.
• ●Guardrail: App crash rate should not increase more than 5%.
• ●Minimum detectable effect: 2 percentage points.
• ●Sample size: Calculate for 80% power at 95% confidence.
• ●Randomization: User-level, not session.
• ●Duration: Cover at least one full weekly cycle plus buffer.

Real vs Instrumentation: The Call That Separates Good Analysts

Half of all "metric drops" in production dashboards are actually instrumentation. Before you go deep on product hypotheses, do three checks: did a new event version ship, did a schema change, did a pipeline job fail. An analyst who can rule out instrumentation in 5 minutes is worth 5x one who spends a week chasing a phantom.

SQL and Python Expectations

Most data analyst interviews include a SQL round. Expect window functions, joins, aggregations with case statements, and cohort retention queries. Some teams also test Python (Pandas) for cleaning and quick plots. ZeroPitch's data analyst practice room focuses on the qualitative diagnosis and A/B design rounds, which are the hardest to train for alone.