Josh Pitzalis

Compile-Time Clarity

Read this first

Generating Test Datasets

You have written a prompt. It is, by your own estimation, a very good prompt. You ran it; it produced a lovely answer, so you shipped it to production.

Sit down. We need to talk.

Building reliable applications on Claude rests on two disciplines:

  • Prompt engineering is the craft of writing better prompts.
  • Prompt evaluation is the science of measuring whether they actually work.

Prompt engineering is your toolkit. Multishot prompting, structuring with XML tags, and a menagerie of other techniques that help Claude understand exactly what you’re asking for and how you want it to respond.

Prompt evaluation asks, “Does this prompt, in fact, work?”

It tries to answer this by testing your outputs against expected answers. You compare different versions of the same prompt and review the output for errors.

Three Roads Diverge

Road one: test the prompt once, declare it good enough, and...

Continue reading →


Structured Data

Suppose we are building a web app that generates AWS EventBridge rules. A user types a description of the events they want to capture, clicks Generate, and expects clean JSON they can copy and use immediately.

We wire the button to Claude, deploy, and admire our work. What could possibly go wrong?

The Problem with Default Responses

Claude, you see, wants to be helpful. Ask it for JSON, and it hands you a small presentation about JSON.

By default, the response looks like this:

```json
{
  "source": ["aws.ec2"],
  "detail-type": ["EC2 Instance State-change Notification"],
  "detail": {
    "state": ["running"]
  }
}
```

This rule captures EC2 instance state changes when instances start running.

The JSON itself is correct. It is also wrapped in a markdown code block and trailed by a friendly sentence of commentary.

So our user clicks Copy, pastes into the AWS console, and gets a...

Continue reading →


Response Streaming

Suppose we have built a chat application. A user types a question, presses Enter, and our server dutifully forwards it to Claude. Claude begins composing a response. The process can take ten, twenty, thirty seconds.

And what does our user see during those thirty seconds? A spinner.

They check their phone. They wonder if the app is broken. They consider, briefly, a life without our product.

The response, when it finally arrives, is excellent. But it does not matter. The damage is done.

The Problem with Standard Responses

In the standard setup, our server sends the user’s message to Claude and then waits for the complete response before sending anything back to the client. Nothing escapes until the last word is written.

The user receives no feedback that anything is happening. Thirty seconds of silence, then a wall of text. We want to let them watch the answer being steamed in...

Continue reading →


Temperature

Suppose we are building a brainstorming machine. We ask Claude for a one-sentence movie idea, and it delivers: “A time-travelling archaeologist must prevent ancient artefacts from being stolen.”

Not bad. Let us brainstorm harder. We run it again. Time-travelling archaeologist. Again. The same archaeologist, the same artefacts, run after run. Claude has exactly one movie and is going to pitch it until one of us dies.

What understand what is going on, we must first learn how Claude chooses its words at all.

How Claude Generates Text

When we send Claude a prompt like “What do you think?”, three things happen.

  • First, tokenisation: the input is broken into smaller chunks.
  • Second, prediction: Claude calculates a probability for every token that might come next.
  • Third, sampling: it picks one.

For our prompt, Claude might assign “about” a 30% probability, “would” 20%, “of” 10%, and so...

Continue reading →


Multi-Turn conversations

Before we build anything of consequence, you must internalise an inconvenient truth: the Claude API remembers nothing.

No hidden transcript accumulates on Anthropic’s servers. If you want a conversation with memory, memory is your job.

Conversing with a Goldfish

In the previous lesson, we asked Claude: What is quantum computing?

Now follow up and ask it to “Write another sentence.”

Claude dutifully writes another sentence about migratory birds, perhaps, or the history of cheese. It has no idea what “another” refers to. From its perspective, this is the first thing you have ever said to it.

No quantum computing.

No context.

Just three words.

How Memory Works

  1. First, you must maintain the full list of messages yourself, in an ordinary array.
  2. Second, send that entire history along with every single request.

You send your initial user message. Then you take Claude’s reply and...

Continue reading →


System Prompts

Suppose we are building a math tutor chatbot. A student arrives, anxious and hopeful, and types: “How do I solve 5x + 2 = 3 for x?”

And Claude, catastrophically helpful as always, hands over the complete step-by-step solution. Subtract 2 from both sides, divide by 5, here is your answer, have a lovely day.

The student copies it down, learns nothing, and fails the exam.

What a Tutor Actually Does

A real tutor gives hints before solutions. A real tutor walks the student through the problem step by step, patiently, and demonstrates with similar problems rather than solving the one on the worksheet.

And there are things a real tutor never does: like blurt out the answer, or tell the student to go use a calculator.

The problem is not what Claude knows. The problem is how Claude behaves. System prompts let us calibrate Claude’s behaviour without changing the student’s question.

The

...

Continue reading →


Making a request

Behold. An HTTP endpoint that will answer any question you put to it.

You know how to talk to an HTTP endpoint. You have been doing it since you were a stripling. So let us do it the honest way, with fetch, and see how far honesty gets us.

const res = await fetch("https://api.anthropic.com/v1/messages", {
  method: "POST",
  headers: {
    "x-api-key": process.env.ANTHROPIC_API_KEY!,
    "anthropic-version": "2023-06-01",
    "content-type": "application/json",
  },
  body: JSON.stringify({
    model: "claude-sonnet-5",
    max_tokens: 1000,
    messages: [{ role: "user", content: "What is quantum computing?" }],
  }),
});

const data = await res.json();

This works.

Instantly immanentised synthetic wisdom.

You may now feel a small flush of pride.

There’s a link to a repo with runnable code at the end of this post, if you want to get this running before you proceed.

Please hover...

Continue reading →


Confessions of Impurity

A function is mathematically “pure” when the same input results in the same output every time.

x => x * 2 lives a quiet, honest life.

Purity is predictable, easy to reason about and sterile.

When a program touches nothing outside itself, if it never ventures into the real world, it is often useless.

And if adventure is what we seek, then there are a few flavours for our humble function to choose from.

First there is the fog-bound port of Nondeterminism: Things like Math.random(), Date.now(), reading an env var that might change.

CallingDate.now() is the same as asking me what time it is. Ask me three times, and you’ll get a different answer every time. The question was the same, the people talking were the same, but something changed.

“What time is it?” is not a pure question. There is an invisible, confounding factor afoot. This hidden contamination is called a side-effect...

Continue reading →


Why understanding TypeScript matters when using AI

You ask AI for an Order type from your API response. It gives you:

type Order = {
  id: string
  amount: number
  status: string
  discount?: number
}

It compiles. Everything looks good.

The real system only sends four values: 'pending' | 'paid' | 'cancelled' | 'refunded'.

Six weeks in, someone writes if (order.status === 'canceled') with one L.

The code compiles, but the comparison never matches. As a result, the system no longer refunds cancelled orders. You only find out because a customer tells you about the problem, not because the compiler caught it.

Also, is the amount in cents or in rupees?

The type signature does not specify. Nothing stops you from adding a value in cents to a value in rupees. A branded type would make this mistake a compile error: type Cents = number & { readonly brand: unique symbol }.

Then there’s discount?: number.

The question mark means that...

Continue reading →


Where the machine ends

Where does one state machine stop and the next begin? What goes in, what stays out, how do machines cooperate, and how do their diagrams stay honest?

Machines model behaviours, not apps

The thing being modelled is not “the application”; it’s not even “the component”; it’s a behaviour. One behaviour with a single lifecycle. A story with a beginning, middle, and a discrete set of possible moves: the draft save flow, the checkout wizard, the connection to the sync server.

Your state machine is not trying to model the whole app; that was never the goal. An application is a collection of running machines.

Imagine explaining the feature to product at a whiteboard. What you would draw is the experience a person goes through when they use a feature (the possible states, how they connect, the “only if” conditions); that is the machine. You would never draw a button’s hover colour, the exact...

Continue reading →