Generating Test Datasets

You have written a prompt. It is, by your own estimation, a very good prompt. You ran it; it produced a lovely answer, so you shipped it to production.

Sit down. We need to talk.

Building reliable applications on Claude rests on two disciplines:

Prompt engineering is your toolkit. Multishot prompting, structuring with XML tags, and a menagerie of other techniques that help Claude understand exactly what you’re asking for and how you want it to respond.

Prompt evaluation asks, “Does this prompt, in fact, work?”

It tries to answer this by testing your outputs against expected answers. You compare different versions of the same prompt and review the output for errors.

Three Roads Diverge #

Road one: test the prompt once, declare it good enough, and ship. This carries a significant risk of the prompt breaking in production the moment a user provides an input you never imagined.

Road two: test a few times, tweak for a corner case or two, and ship. Better. But users are prolific inventors of the unexpected, and they will find the cases you didn’t consider.

Road three: run the prompt through an evaluation pipeline that scores it, then iterate on the prompt against objective metrics. More work. More cost. Vastly more confidence that your prompt is reliable. Road 3 isn’t failproof, but it is systematic. It gives you a place to store unexpected input, after the fact, so you never repeat the same mistake twice.

Data, not vibes #

Run your prompt through an evaluation pipeline, and you receive objective metrics about its performance across a broad range of test cases.

With that data in hand, you can identify weaknesses before they become production issues. You can compare different prompt versions objectively, instead of squinting at two outputs and picking the one that feels nicer. You can iterate with confidence, because improvement is now a number that goes up or down.

Does this require more upfront investment in time and testing infrastructure? It does, but only initially. The added reliability and robustness of the final application pays for itself in terms of time saved further down the road.

So what does an evaluation pipeline actually look like? #

A typical workflow follows five steps that improve your prompts through objective measurement. There are many ways to assemble such workflows, and a bazaar of open-source and paid tools eager to assemble them for you.

Understand the core process first, start small, then scale up as needed.

Step 1: Draft a Prompt #

We begin by writing an initial prompt we wish to improve. Ours is humble:

const prompt = `
Please answer the user's question:

${question}
`;

It asks Claude to answer a question. This basic prompt is our baseline. The thing we shall measure.

Step 2: Create a Dataset #

An evaluation dataset contains sample inputs representing the kinds of questions your prompt will face in production. Each one gets interpolated into the prompt template.

Our dataset contains three questions: “What’s 2+2?”, “How do I make oatmeal?”, and “How far away is the Moon?” A modest panel of arithmetic, porridge, and astronomy.

In real-world evaluations, you might have tens, hundreds, or thousands of records. You can assemble them one by one from actual production failures, or you can have Claude generate them for you (which is generally where people start when they don’t have any production failures yet).

Step 3: Feed Through Claude #

The first question becomes:

Please answer the user's question:
What's 2+2?

Claude might respond “2 + 2 = 4” to the math question, offer oatmeal cooking instructions for the second, and report the distance to the Moon for the third.

Three questions in, three answers out.

But are they good answers?

Step 4: Feed Through a Grader #

Enter the grader. It examines both the original question and Claude’s answer, and pronounces judgment. An objective score, preferably a pass/fail or at least something normalised like a scale from 1 to 10.

In our example, the grader assigns the math question a perfect 10. The Moon question earns a 9. The oatmeal question limps in with a 4.

Average the scores for an objective measurement: (10 + 4 + 9) ÷ 3 = 7.66.

Behold! Your prompt is no longer “pretty good, I think.”

It is a 7.66.

Step 5: Change Prompt and Repeat #

Armed with a baseline, we may now modify the prompt and run the entire process again to learn whether our changes improve performance.

Perhaps we add a little guidance:

const prompt = `
Please answer the user's question:

${question}

Answer the question with ample detail
`;

Run this improved prompt through the same evaluation, and the average might climb to 8.7. The additional instruction helped Claude provide better responses, and we know this, not because the outputs felt nicer, but because the number went up.

Prompt Scoring #

This is everything in a comically oversimplified nutshell: objective measurements, score comparison, systematic iteration.

We’re talking about correctness here, which is always a messy subject to nudge up against. As soon as you start asking how the grader knows what a 4 is, things start to get shaky pretty quickly. Shouldn’t we also have graders to grade the graders? And who grades the grader-graders? We will get to this, eventually, but for now it’s easier to think about all of this in terms of margin of error. Road one had a massive margin of error, road two reduced it a little, and road three reduced it drastically. Grader-graders would certainly reduce the margin of error even further. Our goal, at least at this stage, is less about double-decimal exactitude and more about practical reliability.

Lets Generate Some Test Datasets #

We are building a prompt that helps users write AWS-specific code. Three kinds of output: Python code, JSON configuration files, and regular expressions.

The requirement is simple. A user describes a task; we return clean output in one of those three formats. No explanation. No header. No footer. Just the artefact.

Here is our starting prompt, version 1:

const prompt = `
Please provide a solution to the following task:
${task}
`;

It asks for a solution. That is the full extent of its ambition, and it will be measured accordingly.

What Do We Measure It Against? #

A prompt evaluation needs inputs. We take a prompt, feed it an input, run the combination through Claude, and analyse what comes back.

For our evaluation dataset, we need an array of JSON objects, each with a task property describing what we want Claude to accomplish.

[
  { "task": "Description of task" },
  { "task": "Another description of a task" }
]

Handwriting this dataset is honest work, and for three items it is even pleasant. But an evaluation that consists of the three tasks you personally thought of will test only the three tasks you personally thought of. The inputs are drawn from the same imagination as the prompt, and they share its blind spots.

You want variety. You want tasks you would not have written. And you want more than three, preferably without spending an entire afternoon inventing AWS chores.

Let Claude Write the Exam #

Since this is test data, a faster, cheaper model like Haiku is the right tool here.

const model = "claude-haiku-4-5";

First, our familiar helpers for talking to the API:

function addUserMessage(messages: Anthropic.MessageParam[], text: string) {
  messages.push({ role: "user", content: text });
}

function addAssistantMessage(messages: Anthropic.MessageParam[], text: string) {
  messages.push({ role: "assistant", content: text });
}

async function chat(
  messages: Anthropic.MessageParam[],
  options?: { system?: string; temperature?: number; stopSequences?: string[] },
): Promise<string> {
  const message = await client.messages.create({
    model,
    max_tokens: 1000,
    messages,
    temperature: options?.temperature ?? 1.0,
    ...(options?.system ? { system: options.system } : {}),
    ...(options?.stopSequences ? { stop_sequences: options.stopSequences } : {}),
  });

  const block = message.content[0];
  return block?.type === "text" ? block.text : "";
}

Nothing new here. chat sends the messages, optionally with a system prompt, a temperature, and stop sequences, and hands back the text of the first content block.

The Generation Prompt #

Now the function that does the work. We begin with a prompt to Claude describing the dataset we want:

async function generateDataset() {
  const prompt = `
Generate an evaluation dataset for a prompt evaluation. The dataset will be used to evaluate prompts
that generate Python, JSON, or Regex specifically for AWS-related tasks. Generate an array of JSON objects,
each representing task that requires Python, JSON, or a Regex to complete.

Example output:
\`\`\`json
[
    {
        "task": "Description of task",
    },
    ...additional
]
\`\`\`

* Focus on tasks that can be solved by writing a single Python function, a single JSON object, or a regular expression.
* Focus on tasks that do not require writing much code

Please generate 3 objects.
`;

Three things are being asked. Tasks that are AWS-related. Tasks solvable by a single function, a single JSON object, or a single regex. And tasks that do not require writing much code. We are testing the prompt, not drafting an infrastructure migration.

Getting JSON We Can Actually Parse #

To properly parse the response, we use prefilling and stop sequences:

  const messages: Anthropic.MessageParam[] = [];
  addUserMessage(messages, prompt);
  addAssistantMessage(messages, "```json");
  const text = await chat(messages, { stopSequences: ["```"] });
  return JSON.parse(text);
}

We append an assistant message containing only `json. Claude believes it has already opened a code block, and all it can do is continue — with the array, and nothing before it.

The stop sequence ` handles the other end. The instant Claude tries to close the fence, generation halts. No closing backticks, no commentary. JSON.parse receives an array and nothing else.

Running It #

const dataset = await generateDataset();
console.log(dataset);

This returns three different test cases, one from each of our target outputs. A run of mine produced:

[
  {
    task: "Write a Python function that extracts the AWS region from an S3 bucket URL in the format 's3://bucket-name.region.amazonaws.com'"
  },
  {
    task: "Create a JSON object that represents an AWS IAM policy allowing read-only access to a specific S3 bucket named 'my-data-bucket'"
  },
  {
    task: "Write a regular expression that matches valid AWS IAM role ARNs in the format 'arn:aws:iam::account-id:role/role-name'"
  }
]

A Python function, a JSON configuration, a regular expression. AWS-flavored, small enough to solve in a single artefact.

Saving the Dataset #

A dataset that lives only in a running process is a dataset you will have to regenerate every time you run an evaluation. Once we have one we like, we save it to a file so we can resuse it later:

const file = path.join(import.meta.dirname, "dataset.json");
writeFileSync(file, JSON.stringify(dataset, null, 2));

This creates a dataset.json file in the same directory as the lesson, containing the list of tasks, ready to be loaded during evaluation.

And that’s it. A prompt to evaluate, and a systematic way to generate a wide variety of inputs it will be evaluated against.

Repo #

Code exercises set up here

 
0
Kudos
 
0
Kudos

Now read this

System Prompts

Suppose we are building a math tutor chatbot. A student arrives, anxious and hopeful, and types: “How do I solve 5x + 2 = 3 for x?” And Claude, catastrophically helpful as always, hands over the complete step-by-step solution. Subtract 2... Continue →