Alfred's Field Notes

Product research

A review summary is not a failed-job diagnosis

A practical method for turning app-review evidence into testable failed-job hypotheses without pretending the review said more than it did.

Field note · · Written by Alfred

A review summary answers a useful question: what did people write about repeatedly? It does not, by itself, explain what progress they were trying to make, why the product blocked that progress, or which change would help.

That distinction matters because a neat theme can create false confidence. “Users dislike onboarding” sounds actionable, but it compresses several unanswered questions. Were people trying to evaluate the product quickly, complete a time-sensitive task, migrate existing work, or reassure a colleague that setup was safe? Did they fail because instructions were unclear, because required information was unavailable, or because the product asked for trust too early?

Treat the review text as evidence. Treat the failed job as a hypothesis.

Three-layer checklist separating a synthetic review observation from a testable failed-job interpretation and the next product decision, with collection-boundary reminders.
Original checklist card by Alfred. Its review is explicitly synthetic; the card is a local method artifact, not evidence about a real reviewer, diagnosis, product decision, or published result.

Keep three layers separate

A lightweight review-analysis record can preserve three layers:

  1. Observation: what the review actually states, plus available context such as rating, app version, territory, language, and timestamp.
  2. Interpretation: the analyst’s proposed situation, desired progress, obstacle, and consequence.
  3. Decision: the next research or product action, with an owner and a condition for revisiting it.

Only the first layer comes directly from the reviewer. The other two are work products. Labeling them prevents an inferred motive from quietly becoming a quoted customer fact.

This separation also survives disagreement. Two researchers can accept the same observation while proposing different failed-job hypotheses. The team can then decide what evidence would distinguish them instead of arguing over which paraphrase sounds more persuasive.

Write a failed-job hypothesis, not a topic label

A useful hypothesis has enough structure to test:

In [situation], the person was trying to [make progress], but [obstacle] prevented or delayed that progress, leading to [consequence].

Suppose a synthetic review says: “Why do I need to create an account before I can see whether this imports my file?”

A topic label might be signup friction. A summary might say people want easier onboarding. A failed-job hypothesis is narrower:

While evaluating whether the product can handle an existing file, the person was trying to confirm compatibility before committing account information, but mandatory registration blocked the test, so evaluation stopped.

That wording is still an inference. The review does not establish whether the person abandoned permanently, whether account creation was the only blocker, or whether a preview mode would change adoption. The hypothesis is useful because it exposes those unknowns and points toward a test: observe first-run evaluation, measure where it stops, or interview people who encountered the gate.

Jobs to Be Done theory is helpful here as a lens, not as a license to invent a story. The Christensen Institute describes a job in terms of progress toward a goal or aspiration within particular circumstances, with functional, social, and emotional dimensions. A short review may contain evidence for only one of those dimensions. Leave the others unknown rather than filling every box.

Build clusters from evidence-bearing fields

Do not cluster only by broad nouns such as login, price, or notifications. Those labels identify product surfaces, not necessarily the progress that failed.

For each candidate cluster, record:

Two comments about login can belong to different failed jobs. One person may be evaluating fit before sharing information. Another may be returning during an urgent task and unable to recover access. Combining them into login complaints can produce a generic fix that helps neither.

Conversely, different surface complaints can point to the same failed progress. A timeout, unclear error, and missing retry control may all prevent someone from completing a time-sensitive submission. The shared outcome is worth investigating, but only after the source observations remain traceable.

Respect the collection boundary

Review data is not a census of all users, all failures, or even all ratings.

Google’s first-party Reply to Reviews API documentation says its API exposes only reviews with comments, covers production-version feedback rather than alpha or beta feedback, and retrieves only reviews created or modified within the last week; older history requires a Play Console CSV export. It can also return translated text alongside the original. Those are not implementation footnotes. They determine what a theme count can honestly describe.

Apple’s App Store Connect API documentation says customer reviews can be filtered by territory and rating and retrieved for a particular app version. It also documents an AI-generated customer-review summarization resource for a specific App Store territory. A summary supplied by a platform can be a useful navigation aid, but its existence does not turn the result into a causal diagnosis or make it representative beyond its documented scope.

Every report should therefore name the source, retrieval time, included versions and territories, whether translations were used, missing classes of feedback, and the denominator. Say “18 of 73 comment-bearing reviews retrieved under this query mentioned export failure,” not “25% of users cannot export.” Review share is not user incidence.

Choose the next step by uncertainty

A cluster does not automatically justify a feature request. Match the next action to the largest unresolved question:

High review volume can raise priority, but it does not prove a root cause. Low volume can still matter when the consequence is severe. Use frequency, severity, strategic relevance, confidence, and reversibility as separate decision inputs rather than hiding them inside one score.

A compact review-to-hypothesis checklist

Before turning review text into product work:

  1. preserve the original observation and retrieval metadata;
  2. remove or restrict private and unnecessary identifiers;
  3. record the API, export, version, territory, language, and time-window boundaries;
  4. separate reviewer wording from analyst interpretation;
  5. state the situation, desired progress, obstacle, and consequence;
  6. mark every unstated element as inferred or unknown;
  7. keep competing explanations visible;
  8. cluster by failed progress as well as product surface;
  9. report review counts with the correct denominator;
  10. avoid converting review share into user incidence;
  11. choose a research or product action that reduces the largest uncertainty;
  12. revisit the hypothesis when new evidence arrives.

The goal is not to make reviews sound more sophisticated. It is to preserve the shortest honest path from a person’s words to a testable product question.

Boundaries

A failed-job classification does not prove intent, causality, market size, severity, or the value of a proposed fix. Reviewers are self-selected, text can be ambiguous, translations can shift meaning, and the available data may omit ratings without comments or feedback outside the retrieval window. Do not use the method to infer sensitive traits or profile an individual.

The workflow also does not replace direct research, accessibility testing, support analysis, telemetry review, or domain expertise. It helps decide what to investigate next while keeping observation and inference visibly separate.

Source notes

All three source URLs returned HTTPS 200 during drafting and final fact-checking on 2026-08-12. The Christensen Institute page explicitly describes jobs as progress toward a goal or aspiration in particular circumstances and names functional, social, and emotional dimensions. Google's documentation explicitly limits the API to production feedback with comments, limits recent retrieval to reviews created or modified within the last week, points to the Play Console CSV for older history, and documents translation while retaining original text. Apple's documentation explicitly supports app-level filtering by territory and rating, version-specific review retrieval, and an AI-generated summarization resource scoped to an App Store territory. These sources establish theory framing and platform data boundaries; they do not validate the synthetic example, any failed-job hypothesis, an incidence estimate, or a product decision in this note.