Building blocks
Code, model, me
Every tool splits the work three ways, each part used only for what it does best.
Reading documents, comparing versions, checking survey logic, running every path a respondent could take. Code gives the same result every time and costs almost nothing to rerun.
Whether a question suits its audience, whether a finding is stated clearly, whether two differently worded rules mean the same thing. The model works from what code has already pulled out.
The model reviews or does the legwork, and I decide what to act on. Nothing reaches a client without my sign-off.
Philosophy
Get code to 90%, add judgment, keep learning
Getting code to handle every case perfectly is close to impossible: real documents are messy, and chasing the last few cases means an ever-growing pile of brittle rules. Getting code to about 90% is very doable. A model, briefed with what I know about how the work is done, covers the last 10% and double-checks the calls that matter most. That leaves me with only the decisions that need my judgment. And the 90% keeps growing: each project leaves the tools a little better than it found them.
Patterns
Patterns that carried over
I built and refined these tools one stage at a time, across the whole project lifecycle. Again and again, an improvement in one place turned out to be a pattern that worked elsewhere too. A few examples:
-
Read, then check the readingCode→Model
Code reads the document, and a model checks that reading against the original before anything relies on it. A bad reading makes every later check wrong.
Steps 01–02 ↓ -
The funnelCode→Model→Me
Code flags everything, a model says which flags are real and why, and I see only what's left.
Steps 04–06 ↓ -
The learning loopModel→Me→Code
When something goes wrong, the model or I write it down. Once the same kind of problem shows up again, I write a test for it and change the tool so it's caught automatically from then on. The survey's self-check came out of this loop.
Step 03 ↓
Example
Checking a programmed survey against its design
Before a survey goes to the client or into the field, the programmed version has to match the approved questionnaire exactly: every question, answer option, skip rule, and quota. The usual way to check is to put the Word document and the survey platform side by side and scroll through both. That takes one to two hours depending on the survey, and it's easy to miss a detail deep in a long one.
-
01CodeRead the Word document
Code turns the questionnaire into structured data: questions, options, logic notes, quotas.
In the Word document [ASK IF Q2.1 = CURRENTLY PURCHASE] Q2.3 What is your role in purchasing decisions for warehouse shelving? 1 Final decision maker 2 Strong influence 3 Some input 4 Not involved [TERMINATE] [PN: SINGLE SELECT. QUOTA: MIN 40% FINAL DECISION MAKER] -
02ModelCheck the reading
Word documents are messy, so a model checks code's reading against the original and fixes what it got wrong, like a programming note that ended up inside an answer option or a grid whose rows shifted. A bad reading produces mostly false alarms later, so this has to finish first.
Output: the questionnaire as structured data { "id": "Q2.3", "type": "single choice", "display": "Q2.1 = Currently purchase", "options": ["Final decision maker", "Strong influence", "Some input", "Not involved"], "terminate": [Not involved], "quotas": [Final decision maker ≥ 40%] }
-
03CodeRead and check the survey export
The survey platform exports a structured file, so code reads it directly, with no model involved. Before any comparison, code also checks the survey against itself: whether every rule points at a question and answer that exist, whether quotas are tied to live questions, whether every value the survey reads is set somewhere. Each of those checks exists because that mistake once got through. Then it runs every combination of screener answers and reports who gets through. A comparison can't catch a mistake both documents share, so this has to happen on its own.
Output: the survey as structured data { "id": "Q2.3", "type": "single choice", "display": "Q2.1 choice 1 selected", "options": ["Final decision maker", "Strong influence", "Some input", "Not involved"], "terminate": [Some input], "quotas": [] }Checks on the programmed survey ok rules all point at real questions ok quotas all tied to live questions FAIL "segment" read at Q5.2, never set screener paths, Q2.1 and Q2.3: qualify: Final decision maker, Strong influence, Not involved screened out: Some input
-
04CodeLine them up
Code matches every question and option across the two readings and lists the differences. Some are certain, like a missing question or a wrong number of options. Others need judgment, like slightly different wording or logic written as prose in one and as a rule in the other.
Word vs programmed, question by question Q2.3 options ................ match Word 4, same wording Programmed 4, same wording Q2.3 display ....... needs judgment Word Q2.1 = Currently purchase Programmed Q2.1 choice 1 selected Q2.3 terminate ............ differs Word Not involved Programmed Some input Q2.3 quota ................ missing Word Final dec. maker ≥ 40% Programmed (none) -
05ModelSort real problems from noise
A model works through the uncertain differences and separates real errors from harmless ones, with a reason for each. On one real survey, all 15 serious-looking flags turned out to be harmless, and three small real errors surfaced underneath them.
Verdicts Q2.3 terminate Real error Respondents with some input would be screened out, and those not involved would get through. Q2.3 quota Real error Nothing guarantees enough final decision makers in the sample. Q2.3 display Harmless Choice 1 of Q2.1 is "Currently purchase". -
06MeDecide
I get a short checklist of real issues, each with the fix already written. I tick what to fix, or add a note to push back or redirect, and the analyst gets the result. Anything the tools couldn't confirm is listed separately.
What I see Programming review · v1 2 to fix · 1 unconfirmed Q2.3 [ ] Screen-out is on "Some input"; the questionnaire says "Not involved". → Move it to option 4. [ ] Quota missing: final decision makers at least 40%. → Add a minimum-40% quota on option 1. Unconfirmed [ ] "segment" is read at Q5.2 but never set. → Ask the analyst where it should be assigned. Hidden: 1 harmless difference (Q2.3 display logic, same meaning)
| By hand | With the tooling | |
|---|---|---|
| Time | One to two hours of a senior researcher's attention | 5–20 minutes, mostly unattended, plus a few minutes for my decisions |
| Cost | That same senior time | A few dollars of model usage |
| Coverage | Whatever I catch while scrolling | Every question, option, logic rule, and screener path |
| What it misses | Details deep in a long survey | Anything it can't confirm, which goes on my list instead of passing |
The same approach runs through the rest of my process, from drafting the statement of work and reviewing questionnaires to checking soft-launch data and the final report.