Scrolls
4 min read
On this page

Surveys & Quantitative Research

Surveys are the complement to interviews, not the replacement. Interviews tell you why five people feel a certain way. Surveys tell you how many of your ten thousand users feel the same way. The mistake most teams make is starting with a survey. You should almost always start with qualitative research — interviews, usability tests, support ticket analysis — and use surveys to validate those findings at scale.

When to Survey

Surveys are the right tool when you need to quantify something you already understand qualitatively.

Good reasons to survey:
  - You ran 8 interviews and found 3 recurring themes.
    Now you need to know which theme affects the most users.
  - You want to measure satisfaction across your entire user base,
    not just the 5 people you interviewed.
  - You need to segment responses by role, company size, or plan tier
    to understand which users feel what.
  - You want to track a metric over time (quarterly satisfaction survey).

Bad reasons to survey:
  - You have no idea what users think and want to "learn about them."
    (Start with interviews instead.)
  - You want to validate a feature idea. ("Would you use X?" will
    get a yes whether or not they would actually use it.)
  - You want to replace talking to users with something faster.
    (Surveys are only useful when you know the right questions to ask.)

Question Design

Survey design is harder than it looks. A poorly worded question produces useless data at scale, which is worse than no data because people trust it.

Avoid Leading Questions

Leading (bad):
  "How much do you enjoy our easy-to-use interface?"
  "Don't you agree that faster load times would improve your experience?"
  "How helpful was our excellent customer support team?"

Neutral (good):
  "How would you describe your experience with the interface?"
  "How satisfied are you with page load times?"
  "How would you rate your most recent support interaction?"

Adjectives like "easy," "excellent," and "helpful" in questions push respondents toward positive answers. Remove them.

Use Scales Consistently

Pick a scale and stick with it throughout the survey. Switching between 5-point and 7-point scales, or between agreement scales and satisfaction scales, confuses respondents and makes data harder to analyze.

Common scales:

5-point Likert (most common):
  1 = Strongly disagree
  2 = Disagree
  3 = Neither agree nor disagree
  4 = Agree
  5 = Strongly agree

5-point satisfaction:
  1 = Very dissatisfied
  2 = Dissatisfied
  3 = Neutral
  4 = Satisfied
  5 = Very satisfied

7-point scales provide more granularity but increase cognitive load.
For most product surveys, 5-point is sufficient.

Binary (for simplicity):
  "Was this helpful? Yes / No"
  Best for in-app micro-surveys where speed matters.

Keep It Short

Survey fatigue is real. Every additional question reduces completion rate. Target under 5 minutes — which means 10-15 questions maximum.

Completion rate by survey length:
  1-3 questions:   85-95% completion
  5-10 questions:  70-80% completion
  15-20 questions: 50-60% completion
  30+ questions:   20-30% completion

The questions at the end of a long survey get low-quality responses
because people are rushing to finish. Better to ask 8 great questions
than 25 mediocre ones.

Question Types & When to Use Them

Multiple choice (single select):
  "What is your primary role?"
  Use when categories are mutually exclusive.

Multiple choice (multi-select):
  "Which features do you use regularly? (Select all that apply)"
  Use when respondents may have multiple valid answers.

Likert scale:
  "I find the product easy to use."
  Strongly disagree ... Strongly agree
  Use for measuring attitudes and perceptions.

Open-ended:
  "What is the biggest challenge you face with [product]?"
  Use sparingly (1-2 per survey). Rich data but hard to analyze at scale.

Ranking:
  "Rank the following features by importance (1 = most important)"
  Use when you need relative priority. Keep the list under 7 items.

Matrix/grid:
  Multiple statements rated on the same scale.
  Use to consolidate related Likert questions. But keep the grid
  small — large grids cause straight-lining (same answer for every row).

Avoid These Mistakes

Double-barreled questions:
  Bad:  "How satisfied are you with the speed and reliability of the product?"
  Good: Ask two separate questions — one for speed, one for reliability.

Loaded assumptions:
  Bad:  "Since you use the reporting feature daily, how would you improve it?"
  Good: "How often do you use the reporting feature?" (ask first)
        "What would you change about it?" (ask second)

Jargon:
  Bad:  "How do you rate our API's idempotency guarantees?"
  Good: "How reliable do you find the integration when the same
         request is sent multiple times?"

Absolutes:
  Bad:  "Do you always use the search feature?"
  Good: "How often do you use the search feature?"
        (Never / Rarely / Sometimes / Often / Always)

NPS, CSAT & CES

Three standard satisfaction metrics are widely used in product management. Each measures something different.

Net Promoter Score (NPS)

Question: "How likely are you to recommend [product] to a friend
           or colleague?" (0-10 scale)

Scoring:
  0-6:  Detractors (unhappy, may churn, may damage brand)
  7-8:  Passives (satisfied but not enthusiastic)
  9-10: Promoters (loyal, will refer others)

  NPS = % Promoters - % Detractors
  Range: -100 to +100

Benchmarks (B2B SaaS):
  Below 0:   Serious problems
  0-30:      Average
  30-50:     Good
  50-70:     Excellent
  70+:       World-class (rare)

NPS measures overall loyalty and brand sentiment. It is a lagging indicator — by the time NPS drops, the damage is already done. Its strength is simplicity and comparability across industries. Its weakness is that it does not tell you why someone is a detractor.

Always follow NPS with an open-ended question: "What is the primary reason for your score?" The qualitative responses are more valuable than the number.

Customer Satisfaction Score (CSAT)

Question: "How satisfied were you with [specific interaction/feature]?"
          (1-5 scale or 1-7 scale)

Scoring:
  CSAT = (Number of satisfied responses / Total responses) x 100
  Typically, "satisfied" = top 2 box scores (4-5 on a 5-point scale)

Benchmarks:
  Below 60%: Poor
  60-75%:    Needs improvement
  75-85%:    Good
  85%+:      Excellent

CSAT is specific and transactional. Use it after a specific interaction: post-purchase, post-support-ticket, post-onboarding. It tells you how well a specific experience performed, not overall sentiment.

Customer Effort Score (CES)

Question: "[Company] made it easy for me to [complete task]."
          (1-7 scale, strongly disagree to strongly agree)

Scoring:
  CES = Average score across respondents

What it measures:
  How much effort the customer had to exert. Lower effort
  correlates with higher loyalty. A customer who solved their
  problem easily is more likely to stay than one who had to
  call support three times.

CES is the best predictor of repeat purchase behavior and loyalty, outperforming both NPS and CSAT in many studies. If you can only measure one thing about a specific interaction, measure effort.

Choosing Between Them

Use NPS when:
  - You want a high-level loyalty/brand metric
  - You need to benchmark against competitors or industry
  - You want to track overall sentiment over time

Use CSAT when:
  - You want to measure a specific experience or feature
  - You need feedback on a particular touchpoint
  - You're comparing satisfaction across different features

Use CES when:
  - You want to measure the friction in a specific workflow
  - You're optimizing self-service or support experiences
  - You want to predict which users will churn

Many companies use all three at different touchpoints:
  - NPS: Quarterly, company-wide
  - CSAT: After specific interactions (support, onboarding)
  - CES: After task completion (checkout, setup, migration)

Sample Size & Statistical Validity

Survey results are only meaningful if you have enough responses and the respondents represent your user base.

How Many Responses Do You Need?

The required sample size depends on your population size, desired confidence level, and margin of error.

For a 95% confidence level with 5% margin of error:

Population size    Required sample
-----------------------------------
100                80
500                217
1,000              278
5,000              357
10,000             370
50,000             381
100,000+           384

Key insight: Once your population exceeds ~10,000, the required
sample barely changes. 384 responses from a 100K user base gives
you 95% confidence with 5% margin of error.

Response Bias

The people who respond to surveys are not a random sample of your users. They are disproportionately:

Over-represented:
  - Very satisfied users (fans who want to help)
  - Very dissatisfied users (people with complaints)
  - Highly engaged users (they see the survey prompt)
  - Users with strong opinions (motivated to respond)

Under-represented:
  - Casual users (do not care enough to respond)
  - Churned users (no longer see your prompts)
  - Users who are "fine" (no strong opinion either way)
  - Non-English speakers (if survey is English-only)

This bimodal distribution means survey results often overstate both satisfaction and dissatisfaction while missing the silent majority. Acknowledge this limitation when presenting results.

Common Pitfalls

  • Surveying before doing qualitative research — if you do not know what questions to ask, the survey will ask the wrong ones. Interviews first, surveys second.
  • Leading questions — any question that contains adjectives, assumptions, or emotional language will skew results. Review every question for neutrality.
  • Survey fatigue — over-surveying destroys response rates and annoys users. Limit to one survey per user per quarter.
  • Treating all responses equally — a survey of power users tells you different things than a survey of new users. Always segment.
  • Ignoring non-response bias — the 85% who did not respond may feel very differently from the 15% who did. Acknowledge this limitation.
  • Analysis paralysis on sample size — for directional insights, 50-100 responses is often enough. You do not need 384 responses to identify that "performance" is the top complaint.
  • Asking "would you use this?" — people are terrible at predicting their future behavior. They will say yes to be helpful. Measure behavior, not stated intent.
  • Skipping the open-ended question — a Likert scale tells you the score but not the story. Always include at least one open-ended question to capture the why.

Key Takeaways

  • Surveys validate at scale what qualitative research has already uncovered. Start with interviews, then survey to quantify.
  • Question design is everything. Avoid leading questions, double-barreled questions, and jargon. Use consistent scales. Keep it under 5 minutes.
  • NPS measures loyalty, CSAT measures satisfaction with specific interactions, and CES measures effort. Each serves a different purpose.
  • Sample size matters, but so does representativeness. 384 responses from a biased sample are worse than 50 responses from a representative one.
  • Always include an open-ended question. The qualitative responses from surveys are often more valuable than the quantitative scores.
  • Survey results are inputs to decisions, not decisions themselves. Combine with behavioral data, interviews, and business context.