I Built an AI Ad Creative Generator and Taught an LLM to Judge It

Getting an AI to write an ad takes about ten minutes. Getting it to write a good ad for one specific brand is a different problem. The copy has to sound like the brand, avoid the words the brand never uses, and promise nothing the product cannot deliver.

Over a couple of weeks I built an AI ad creative generator to chase that second problem. You give it a brand brief and a product photo. It writes the ad copy, generates an image for every variant, and then scores each variant against the brand's own rules with an LLM judge before a human has to look at it.

Then I did the part most demos skip. I labelled 90 variants by hand and measured how often the judge agreed with me. The answer was humbling, useful, and the reason I think this project is worth writing about.

In this post I will walk through how the generator works, how I built and tested the judge, what broke along the way, and how I put it online without handing strangers my API keys. The live app and the code are linked at the end.

Why generating ads is easy and trusting them is hard

Anyone can paste a prompt into a chatbot and get twenty headlines. The trouble starts when you have to ship them.

To test this properly I made up three brands. Velara sells noise cancelling headphones and wants restraint, in the spirit of Apple and Bang & Olufsen. Verdan sells hempseed cooking oil, and its brief forbids any hint of cannabis or CBD. Riverbend Brewing Co. sells an IPA called Tide & Treetop and does not want to sound like every other taproom. Each brand has a brief with a voice, words to use, words to avoid, an offer and a call to action.

Checking twenty variants against a brief like that is manageable. Checking two hundred, every time you change a prompt, is not. Nobody does it consistently, and that is exactly how a banned word or an invented claim slips through.

So the question changed. It stopped being "can an AI make ads?" and became "can I trust a machine to catch the ads that break the rules?" That is an evaluation problem, and it is the part of AI products that gets the least attention.

Quick note on the brands: all three are fictional, with made-up briefs and made-up product photos. I did that so I could experiment freely. The app works with any brief that follows the same JSON structure, real or invented.

How the AI ad creative generator works

The pipeline is simple to describe. A brief and a product photo go in, and scored ad variants come out.

1. The brief. A JSON file with five parts: brand, product, audience, tone and offer. The tone section carries the useful detail: voice words, a "we are" list, a "we are not" list, vocabulary to use, vocabulary to avoid, an example line and a few reference brands.
2. The copy. Claude writes a headline, a short body and a call to action for each variant. I force the output into a Pydantic schema, so I get clean structured data back instead of a wall of text.
3. The image. OpenAI's image model edits the product photo. The instruction is to keep the product, label and composition as they are, and change only the background, lighting and setting to suit the headline.
4. The judge. Every variant gets a pass or fail on five criteria: tone, claims, call to action, length and visual fit.
5. The ranking. Variants are sorted by how many criteria they pass, best first, and every failure comes with a one-sentence critique.

Under the hood it is Python, with FastAPI for the API and Streamlit for the front end. Nothing exotic. The interesting part is not the plumbing, it is step four.

My first batch of AI ads was not good

Before any judge existed, I generated 20 copy variants from a single made-up coffee brand and read every one. Three problems jumped out almost immediately:

1. Every variant ended with the same call to action.
2. The copy kept repeating the brand's mission and values instead of using concrete facts and figures about the product.
3. It skipped most of the offer section and ignored the "we are" and "we are not" lists in the tone brief.

I only spotted this because I read all 20 myself. That works at 20. At 200 it quietly stops working, and the problems slide through. This is the job I wanted the judge to do: read everything, every time, against the brief.

Building the LLM judge: five criteria, all pass or fail

Before writing a single judge prompt, I read three guides: Hamel Husain's Using LLM-as-a-Judge For Evaluation, Eugene Yan's Product Evals in Three Simple Steps, and Anthropic's docs on building evaluations. They agree on a few things, and I followed them closely.

Pass or fail, never 1 to 5. Nobody can act on the difference between a 3 and a 4. A binary verdict tells you which rule got broken.

One judge per criterion. A single prompt that grades tone, claims, CTA, length and visual fit all at once is hard to calibrate, and you can never tell which part is wrong. So I built five small judges instead of one big one.

Use code wherever code can decide. Length is a word count. Half of the claims check is a string match against the brief's avoid list. Code costs nothing and never changes its mind. Only the things code cannot catch go to a model.

Set thresholds from your own data. I did not pick round numbers. Headlines are capped at 12 words because my three briefs' example lines ran 8, 10 and 11 words. CTAs are capped at 6 words because every brief's own CTA was 3. Bodies get two sentences, which is what my copy prompt already asked for.

Here is how the five criteria ended up:

Criterion	What it checks	How it is graded
Tone	Does it sound like the brand's voice and stay away from the "we are not" list?	LLM judge with a worked pass and fail example
Claims	Avoid-listed words, invented numbers or capabilities	Code first, LLM only for what string matching cannot catch
Call to action	Is it consistent with the real offer, with no invented discounts or bonuses?	LLM judge
Length	Headline, body and CTA within the word and sentence limits	Code only
Visual fit	Product and label unchanged and legible, scene fits the brand, on-image text and typography hold up	Vision LLM judge that sees both the original photo and the generated ad

Labelling 90 variants by hand

The judge is only worth trusting if you can measure it, and for that you need ground truth. So I labelled all 90 variants myself, pass or fail on each of the five criteria. That is 450 labels in total.

Why 90 and not 50? At 60 or less variants, a measured failure rate of 12% could honestly be anywhere from about 4% to 20%. That is too wide to conclude anything. Hamel's guide puts the line at around 60 examples, below which the confidence intervals are often too wide to be useful. I went with 30 variants per brand, because each brand breaks in its own way. Verdan carries compliance risk, Riverbend carries cliche risk, and Velara carries jargon risk.

What the numbers said about my LLM judge

After one round of prompt fixes, the judge agreed with my hand labels on 82.0% of the 450 individual calls, up from 80.9%. That headline number hides a lot, so here it is criterion by criterion.

Criterion	Agreement, v1	Agreement, v2 (after one prompt fix)
Tone	78.9%	80.0%
Claims	73.3%	75.6%
Call to action	95.6%	95.6%
Length	98.9%	98.9%
Visual fit	57.8%	60.0%
Overall	80.9%	82.0%

Length and call to action are near perfect, which is no surprise. Length is plain code. The criteria that need judgment are where it gets interesting.

Claims: the judge was punishing the brand's own words

The judge kept failing Velara copy for mentioning "spatial sound" and automatic mode switching, calling them invented capabilities. But both are listed in the product's key features, and "spatial sound" is literally in the brand's list of words to use. The judge had never seen that list. I had only given it the proof points. Of the 18 cases where the judge was stricter than me on Claims, 13 were Velara.

The fix was to show the judge the key features and tell it that a real feature described in different words is not an invented claim.

Not every disagreement was the judge's fault, though. On Riverbend, it caught words from the brand's avoid list, like "extreme", "hoppy bomb" and "hand-crafted", that I had waved through. I was labelling leniently. The judge was enforcing the written bar.

Visual fit: I was right, and the judge had no way to know

Visual fit was the weakest criterion at 57.8%, and the gap was almost entirely one brand. I failed 23 of the 30 Velara variants on visual fit. The judge failed 4.

The Velara brief asks for product-forward ads with very little typography, in the manner of Apple and Bang & Olufsen. The judge was checking product fidelity, the scene, the words on the image and the font. It was not checking how much text there was or how loud it was. A headline in the right font with the right words could pass while burying the product. I was applying a real bar that the judge simply was not looking at.

I did not hard-code a Velara rule. Every brief already lists reference brands, so I fed those into the visual judge and added a restraint check that calibrates against them.

The surprise: agreement went up everywhere and still went down

Here is the finding that taught me the most. Per-criterion agreement improved. But variant-level agreement, meaning "does the judge's fully approved match my fully approved?", fell from 66.7% to 61.1%.

Here is why. Before the Claims fix, the judge was failing some Velara variants for the wrong reason. I had also failed those variants, for tone or visual restraint, so we agreed by accident. After the fix, the judge stopped failing them on Claims, which was correct. But it was still missing what I had failed them for, so those variants flipped from "both fail" to "judge passes, I fail". Cases where the judge was more lenient than me rose from 17 to 21.

The lesson is that a five-gate AND is unforgiving. Fix one gate and the disagreement hiding behind another becomes visible. This is why I report agreement per criterion instead of trusting a single combined number. For the record, the judge passed all five criteria on 48 of 90 variants before the fix and 51 of 90 after, which is 56.7%.

What broke along the way, and what each bug taught me

A fair amount went wrong. These are the ones worth passing on.

The model folded the offer into the call to action

In the pilot run, every Velara CTA came back at 15 words. The model had been handed both the CTA ("Hear the difference") and the offer (10% off plus a free travel case) with nothing keeping them apart, so it merged them into one long line.

The fix was to put the rubric's numbers into the generation prompt: headline of 12 words or fewer, body under 40 words, CTA of 6 words or fewer, plus a line explaining that the offer sells the deal while the CTA tells the reader what to click. CTAs dropped to 3 or 4 words across all three brands, not just Velara. The lesson is simple. Tell the generator the rules you plan to grade it on.

The fonts drifted, and the packaging was the reason

The image model started rendering the headline straight onto the picture, which I had not asked for. I decided to keep it and grade it. But then the typeface changed from one variant to the next, and one landed in plain Times New Roman for no visible reason.

Each image call is stateless, so the model has no memory of the font it used in variant one. My first fix was to tell it to match the typeface on the product label. That was not enough, because Riverbend's can uses a sans serif for the brand name and a serif for the beer name. With two legitimate fonts to copy, the model picked a different one each time.

The fix that held was to point at one unambiguous thing: the brand's own name and logotype. On the re-run, both variants for each brand used the same typeface. It is not a guarantee, since an image model is probabilistic, so typography became a criterion the judge checks rather than a promise I make.

The judge's own output got cut off

My judge crashed twice on its very first variant. The first error looked like a missing safety check. The real cause was stranger. On a pass, the model was writing a full paragraph in the "critique" field, and eventually it hit the token limit mid-sentence.

I had a comment in my Python class saying "leave this empty when passed". The model never sees Python comments. Moving that instruction into the field's description, which becomes part of the schema the model fills in, fixed it. If you want the model to know something, it has to be in the prompt or the schema. Your code comments are only for you.

The judge only knows what you show it

This was the most expensive lesson. After I fixed the Claims judge to see the key features, I ran the live app and it flagged Riverbend's free bottle opener and carabiner as an unsupported claim. That offer lives in the brief's offer section, which the Claims judge had never been given. It cost me a few dollars to find out.

So I audited every judge and made sure each one reads the full slice of the brief it needs. I also wrote tests that capture the exact prompt text and check that the offer's wording is in it. They cost nothing to run, and I confirmed they fail on the old code, which is how I know they really catch the bug.

Code that only worked because it was just me

Two bugs would have hit the first stranger to use the app. The judge looked for the product photo in my own data folder instead of where a visitor's upload actually landed. And output folders were named with a timestamp accurate to the second, so two people generating for the same brand at the same moment would have written into the same folder.

I found both by asking one question about every file path: is this unique to this run, or does it only work because I am the only one testing?

And the usual suspects

A missing .env file. A src layout that needed pip install -e . before Python could find my own package. An image parameter, input_fidelity, that the newer image model flatly rejects. Not glamorous, but each one can eat an evening if you let it.

Putting an AI app online without paying for strangers' API calls

I had one rule for this project: it is finished when a stranger can open a link, upload a product photo and a brief, and get scored ad variants back. But every generation costs real money, and I was not going to hand the internet my API keys.

I weighed four options:

• A shared key with a hard monthly cap. Both Anthropic and OpenAI let you set spend limits that stop calls once you hit them. Safe, but still my money.
• Bring your own key. Visitors paste their own keys. Zero cost and zero risk to me.
• Pay to use. Visitors pay me first. This is real scope creep, with payments and webhooks, and it would have pulled me away from the actual project.
• Stay local and call it future work.

I went with bring your own key. It had a nice side effect on the architecture. I had split the app into a FastAPI backend and a Streamlit front end mostly to keep my keys away from the browser. With visitors supplying their own, there is no key of mine to protect. So I collapsed it into one Streamlit app that calls the pipeline directly. Keys are used in memory for that one run and never written to disk. The FastAPI version still works locally.

For hosting, I started with Hugging Face Spaces. When I went to create the Space, the free tier no longer covered Streamlit, so I switched to Streamlit Community Cloud. It deploys straight from the GitHub repo, which also meant no second repository and no extra config file.

If you want to run it yourself, a 20-variant batch costs around $5 or less in API usage and can take up to 30 to 40 minutes, often less. The images are the expensive and slow part.

What I would do next, and what this project does not prove

First, the honest limits. I tested three made-up brands with 90 variants and one labeller, and I labelled leniently. Someone else would give different labels and the numbers would move. The same Claude model writes the copy and judges it, while Anthropic's own docs suggest grading with a different model than the one that generated the output. I have not tested that yet. And the numbers in this post describe the judge before the last round of fixes, including the offer fix and the prompt audit. I have not paid to re-measure them.

Here is what I would do next:

• Use a real brand guideline. A formal brief with logo placement, colours, fonts and text sizes gives the judge firmer rules to enforce and gives the generator fewer places to improvise. I expect the scores to be more reliable with it.
• Add a brand-in-the-corner mode. Image models are unreliable at exact typography, however carefully you prompt them. A more realistic setup is to generate clean product photography with the brand mark in a fixed corner and no text at all, then add the headline afterwards in a template tool or with a designer, where the type can be pixel exact.
• Check the text on the image with code. Read the on-image text with OCR and run it through the same avoid-list and length checks I already use for the copy.
• Offer a hosted version with my own API and strict limits, once I know how I want to price it.

Frequently asked questions

What is an LLM judge?

An LLM judge is a language model that grades another model's output against a rubric, usually with a simple pass or fail. It is useful when the quality you care about needs judgment that code cannot provide, like brand tone. It is only worth trusting once you have measured it against human labels.

How many examples do you need to validate an LLM judge?

Hamel Husain's guide says that below about 60 labelled examples the confidence intervals are often too wide to support a useful conclusion. I used 90 variants, 30 per brand, with five labels each. That is workable for a prototype, but still small.

How much does it cost to generate and judge AI ad variants?

In my setup, a 20-variant batch costs around $5 or less in API usage and can take up to 30 to 40 minutes, often less. Most of that is image generation. Running the judge across all 90 variants cost about a dollar.

Do I need my own API keys to use the live demo?

Yes. You paste in your own Anthropic and OpenAI keys. They are used in memory for your session and are not written to disk.

Can I use my own brand brief and product photo?

Yes. Use any brief, real or made up, as long as it follows the same JSON structure as the sample briefs in the repo's data folder. A more detailed brief gives the judge better constraints to work with.

Is the code open source?

Yes. It is on GitHub under the MIT license.

The takeaway

The generator was the easy part. The judge, and the hand labelling I did to test it, is where I learned something. Hamel Husain argues that the real value of building a judge is that it forces you to look closely at your data. Having done it, I agree. Reading 90 variants against a brief taught me more about what "on brand" actually means than any prompt did.

If you are building something with LLMs, three habits would have saved me a lot of time. Label some outputs by hand before you write the judge. Give every judge the whole slice of context it needs. And report agreement per criterion, not as one number.

Try it or read more

• Try the live demo (you will need your own Anthropic and OpenAI API keys)
• Read the code on GitHub
• Read the full case study
• Want something like this for your own brand or team? Get in touch

Sources and further reading

• Using LLM-as-a-Judge For Evaluation: A Complete Guide by Hamel Husain
• Product Evals in Three Simple Steps by Eugene Yan
• Define success criteria and build evaluations from the Claude Platform docs