Workflow

Four thousand comments. Nobody has read them

Every survey has that one open-text question, and it is always the most useful data nobody analyses, because coding it by hand is a fortnight. SheetAI builds a coding frame from your actual responses, applies it across every row with a short reason attached, and lets you validate it against a hand-coded sample before you present a single chart.

Try it on your sheet See real workflows

Audit-ready trailNo data trainingReversible per action

~18%

Skip open-ended survey questions

Pew Research Center reports that item nonresponse on its panel's open-ended questions averages around 18%, against 1% to 2% for closed-ended questions.

Source: Pew Research Center: Why do some open-ended survey questions result in higher item nonresponse rates? (2021)

~20x

Cheaper than crowd coders

Gilardi, Alizadeh and Kubli found ChatGPT's per-label cost was under $0.003, about twenty times cheaper than MTurk, and its intercoder agreement beat crowd workers and trained annotators on every task tested.

Source: Gilardi, Alizadeh and Kubli: ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks (2023)

1.6M

Posts labelled by hand for one study

Mozetič and colleagues had human annotators label over 1.6 million Twitter posts in 13 European languages, and found model accuracy only approaches human agreement once the training set is large.

Source: Mozetič, Grčar and Smailović: Multilingual Twitter Sentiment Classification, PLOS ONE (2016)

Text & Sentiment Analysis: illustration for SheetAI's text & sentiment analysis use case

The open-text question is usually the most informative one and the least analysed.

Closed questions tell you what people did with the options you gave them. Open text tells you what you failed to ask about. It gets skipped for one reason: manual thematic coding of a few thousand responses is measured in days, and the deadline is Friday.

Source: Pew Research Center: Methods

This page is about turning free text in a column into structure you can count: survey verbatims, NPS comments, app store and marketplace reviews, support tickets, exit interviews, open consultation responses, and sales call notes. SheetAI reads the responses, proposes a coding frame grounded in what is actually there rather than what you expected, applies it row by row with a one-line justification, and flags the ambiguous cases instead of forcing them. Crucially it will help you measure its own agreement against a human-coded sample, because a classification you have not validated is a number with no error bar, and the boundaries section is clear about what that means for the decisions you take from it.

Why open text goes unanalysed

Four realities of qualitative coding that explain the gap between the data collected and the data used.

Survey open-text questions that receive full thematic analysis

A minority

The question is asked, the responses are collected, and the analysis is a scan of the first fifty answers plus whatever the team remembers. The rest is archived unread.

Practitioner experience

Human inter-coder agreement on a well-defined frame

~70-85%

Even trained coders with a clear codebook disagree on a meaningful share of cases. This is the benchmark any automated approach should be measured against, not perfection.

Standard content-analysis literature

Consumers who read online reviews before buying

~98%

BrightLocal's consumer survey work consistently finds nearly universal review-reading. The text your customers write about you is read by everyone except you.

BrightLocal Local Consumer Review Survey

Time to hand-code 8,000 short responses

8-12 hrs

Before double-coding, reconciliation, or the frame revision that always follows the first thousand rows. It is a fortnight of elapsed time for a two-day answer.

Coding rate arithmetic

The pattern: the barrier is throughput, not difficulty. Any competent analyst can code a hundred responses well; nobody can code eight thousand consistently. Consistency at volume is exactly what a machine offers, provided you check its agreement with a human on a sample before you believe the totals.

Anatomy of a coding project

How open-text analysis is done properly, and why each stage is where projects quietly stop.

Phase 1: Read a sample and build the frame

›Read 100–200 responses without coding them
›Draft candidate themes from what is actually there
›Define each code so two people would apply it the same way

⚠ Frames built from expectations rather than data miss the theme nobody anticipated, which is usually the finding worth having.

Phase 2: Pilot and revise

›Apply the draft frame to a fresh sample
›Find the responses that do not fit
›Split, merge, and redefine the codes

⚠ The frame always changes after the pilot, which means everything coded before the revision has to be recoded. This is where projects lose their first week.

Phase 3: Code everything

›Apply the frame across the full dataset
›Handle multi-theme responses
›Keep a consistent standard on hour eight as on hour one

⚠ Consistency decays with fatigue. The last thousand responses are not coded the way the first thousand were, and nobody can quantify the drift.

Phase 4: Check reliability

›Double-code a sample independently
›Compute agreement
›Reconcile the disagreements and adjust

⚠ The step that makes the result defensible is the step with no budget. Most published internal analysis skips it entirely.

Phase 5: Report with evidence

›Count by theme and cross-tabulate
›Pull representative quotes per theme
›Track how themes shift between waves

⚠ Quote-hunting means re-reading the responses you already read, because nobody recorded which row supported which theme.

Cautionary tale

The failure mode is a confident wrong percentage

The 1936 Literary Digest presidential poll is the canonical example of a large sample producing a confidently wrong answer. The magazine mailed out ten million ballots and received 2.4 million responses (an enormous dataset by any standard) and predicted a comfortable Landon victory. Roosevelt won 46 of 48 states. The sample was drawn from telephone and automobile registration lists, which in 1936 skewed heavily toward wealthier households. The volume of data was never the issue; the systematic bias in who was represented was, and no amount of additional responses would have corrected it.

The lesson: Automated coding creates exactly this hazard in a new form. A model that systematically reads polite complaints as neutral will produce a clean, plausible, wrong distribution, and it will produce it consistently, which makes it look reliable. The defence is unchanged since 1936: validate against a known-good sample and quantify the disagreement. SheetAI will code a hand-coded holdout, compare its labels to yours, and report agreement per category, so you know which codes to trust before anything reaches a slide.

Source: The Literary Digest poll of 1936: Wikipedia

Who it's for

If you are sitting on a column of text you have never properly read, this is for you.

Insight & Research Teams

Run surveys whose open-text questions are collected, archived, and never analysed.

Customer Experience

Own thousands of NPS verbatims and report only the score, because the comments take a fortnight.

Support Leads

Need ticket themes to justify a fix, and have only the categories agents picked from a dropdown.

Product Managers

Read the first two pages of feedback and generalise from it, knowing that is what they are doing.

HR & People Teams

Have engagement survey comments they must analyse without exposing individual respondents.

Real text-analysis workflows

The prompt, the method it follows, and what lands in your sheet.

sentiment-analysis_workbook.xlsx - SheetAI

You ask

“Read a representative sample of the responses in Survey!D2:D4200 and propose a coding frame of 8–12 themes grounded in what is actually there. For each theme give a name, a definition, an inclusion rule, an exclusion rule, and three verbatim examples with their row numbers. Do not code anything yet.”

SheetAI does

✓Samples across the full response set rather than the first rows.
✓Derives candidate themes from the responses instead of imposing expected ones.
✓Writes each code with inclusion and exclusion rules so it is applicable by anyone.
✓Cites real rows as examples so every theme is checkable.

Result

An eleven-theme frame with definitions and cited examples, including two themes the team had not anticipated: one about a checkout step and one about a specific email that had never come up in any planning session.

Everything you need to make text countable

Frames derived from your data, classification with recorded reasoning, explicit uncoded cases, and agreement measured against a human sample before anyone believes the totals.

✓Coding frames grounded in the actual responses

✓Per-row classification with a stated reason

✓Primary and secondary theme assignment

✓Explicit UNCODED for ambiguous responses

✓Agreement scoring against a hand-coded holdout

✓Wave-on-wave theme comparison

Plays well with your stack

Survey exports (Qualtrics, SurveyMonkey, Typeform)
NPS and CSAT platform verbatim exports
Support desk ticket exports (Zendesk, Intercom, Freshdesk)
App store and marketplace review exports
Call and meeting note exports
Any CSV with a free-text column

What coding one survey wave looks like

A representative insight team analysing 4,200 open-text responses from a quarterly survey, before and after the coding moves into the sheet.

Before SheetAI

~46 hours

Week 1
Read a sample; draft the frame; pilot and revise it
~10h
Week 2
Code the first half; recode what the revision invalidated
~14h
Week 3
Code the second half; consistency quietly drifts
~13h
Week 4
Hunt quotes, build the tables, skip the reliability check
~9h

✓

With SheetAI

~9 hours

Week 1
Frame proposed from the data; team edits definitions
~3h
Week 2
Full coding pass; review the UNCODED rows; revise once
~3h
Week 3
Hand-code a 200-row holdout; measure agreement per theme
~2h
Week 4
Tables and cited quotes generated with row references
~1h

~80% reduction on a representative 4,200-response wave, including validation.

What SheetAI will not do

Classification produces numbers that look authoritative regardless of whether they are. These limits are the ones that matter.

Give you a validated result without validation

It will code 4,000 responses and produce a clean distribution whether or not that distribution is right. The only thing that establishes reliability is comparing against a human-coded holdout, per category, and looking at where agreement is poor. SheetAI will run that comparison for you, but it cannot make you do it, and a coded column that has never been checked should not appear in a decision paper.

Detect sarcasm, irony, or cultural subtext reliably

"Brilliant, another update that logs me out" is negative to any human reader and is a well-known failure case for automated classification. Mixed and ironic responses are where machine agreement drops most, which is exactly why per-category agreement matters more than a headline accuracy figure.

Make individual-level judgements about people

This is a tool for aggregate themes. Using it to score, rank, or assess an identifiable person from their written comments (an employee from engagement survey text, a candidate from an application) is out of scope, and in employment contexts it carries real legal exposure. Aggregate the themes; do not profile the respondents.

Replace reading your data

The most valuable thing in an open-text dataset is often the response that fits no theme. Coding compresses; compression discards. Read the UNCODED rows every time; they are usually the shortest list and the most informative one you will look at all quarter.

“

We had three years of NPS comments and reported the score. That was it. The first proper coding pass turned up a billing confusion nobody had named, sitting in about nine per cent of responses going back the whole time. It had been in the data since 2023 and we had never read it.

Head of Customer Insight, subscription business (~4,000 verbatims/quarter)

Frequently asked

The questions researchers ask before they trust a coded column.

How accurate is the classification?

›

The right benchmark is not perfection but human inter-coder agreement, which typically runs around 70–85% on a well-defined frame. Against a good frame, automated coding lands in a similar band, higher on clearly distinguished themes, meaningfully lower on ambiguous ones. That is why the workflow includes measuring agreement per category against your own hand-coded sample: you find out which specific codes are trustworthy rather than accepting one overall number.

Can it handle sarcasm and mixed sentiment?

›

Mixed sentiment, yes; there is a category for it and it is used. Sarcasm is genuinely hard and is the most common failure case for any automated classification, human-written rules included. Responses that are ironic tend to land in the disagreement set during validation, which at least means you can see how large the problem is in your particular dataset rather than assuming it away.

Do I need to build the coding frame first?

›

No, and often you should not. Ask it to propose a frame from a sample of your actual responses with definitions and cited examples, then edit it. Frames built from expectations reliably miss the theme nobody anticipated, and that theme is usually the reason the question was worth asking.

How many responses can it process?

›

Thousands in a pass is routine. Very large datasets are better run in batches, partly for reliability and partly because it keeps each batch reviewable. Coding is a read of every row rather than a formula fill, so a few thousand responses is minutes rather than seconds.

Can it do languages other than English?

›

Yes, and it can code multilingual datasets against a single frame, which is otherwise a genuinely painful problem requiring a coder per language. Validate per language rather than only overall, since agreement can vary considerably between them and a single headline figure will hide that.

Is this safe for employee survey comments?

›

It can be, with care. Aggregate theme analysis of engagement comments is a legitimate and common use. Using it to assess or score identifiable individuals from what they wrote is not, and in employment contexts carries real legal exposure. Strip identifiers before analysis, report at a group size that cannot be back-resolved to one person, and be explicit with respondents about what happens to their words.

Will it show me why it classified something a certain way?

›

Yes. Ask for a reason column and every row gets a one-line justification. This is not a nicety. It converts an opaque label into something a colleague can argue with, and during validation it is what tells you whether a disagreement is a model error or a genuinely ambiguous code definition.

Can I keep the same frame across quarters?

›

Yes, and consistency between waves is the point of documenting the frame. Apply the identical codebook to the new wave and compare theme share. Where you do revise the frame, recode the earlier wave too; a shift caused by a redefinition looks exactly like a shift in opinion, and only one of them is worth reporting.

Sources & further reading

Every benchmark and statistic on this page is drawn from publicly available research. We cite our sources because we read theirs.

Related teams using SheetAI

Adjacent functions that hit the same kinds of problems.

Get started

Stop archiving the comments.
Start reading all of them.

Drop your open-text column into a sheet and ask for a coding frame. You will see the themes in your own data, including the one nobody thought to ask about.

All use cases

Pay as you go, no subscription required