piddonkadonk
When I tell people I work in AI quality control, I usually get a polite nod and a quick change of subject. Fair. It sounds like a job title from a sci-fi movie. The day to day is much more familiar. Picture an editor, a teacher and a video game playtester sharing one desk.
Here is the short version. An AI model writes answers. Somebody has to check whether those answers are any good. That somebody is me.
Step one: read the answer. All of it.
A person asks a question, the model responds, and I read the response the way a careful editor would. Slowly. Twice if needed. Skimming is how bad answers sneak through.
Step two: grade it against a rubric
A rubric is a written scoring guide. It lists what a good answer needs to do and how much each part counts. Teachers use rubrics to grade essays. Evaluators use them so two different people can look at the same answer and land on the same score. Without one, grading turns into vibes, and vibes are hard to repeat on a Tuesday.
Step three: check what a score can miss
I look at four things on every answer.
- Tone. Does it sound like a helpful person, or like a terms of service page?
- Clarity. Could a busy reader follow it on the first pass?
- Usefulness. Did it answer the question that was asked, or a nearby question that was easier?
- Trust. Are the facts right? Does it say so when it does not know? A confident wrong answer is the most dangerous kind, because it looks exactly like a right one.
Step four: write test prompts
A prompt is the question or instruction you give the model. Part of the job is writing prompts built to find weak spots. Tricky wording. Missing information. Two instructions that pull in different directions. It is a lot like playtesting a game by running into every wall to see which ones you can clip through.
Step five: explain why a good answer is good
This is the part people do not expect. Choosing the better of two answers is only half of it. I also have to write down the reason in plain language, clearly enough that someone else could apply the same reasoning tomorrow. A note like "it felt better" helps nobody. A useful note reads more like "it answered the question in the first sentence, gave the steps in order, then flagged the one risk."
That is the day. Read, grade, poke, explain, repeat. It is patient work, and I love it. Every clear note helps the next answer land a little better for someone with a question and not much time.
Curious about this kind of work? The core skills are reading closely, writing clearly and being fair. No wizard robe required, though I would wear one.
