AI Ad Creative Generator Case Study: Building and Validating an LLM Judge

Generative AI can produce hundreds of ad variants in minutes, but nobody can check hundreds of variants against a brand brief by hand. This AI ad creative generator case study covers how I built a tool that creates ad copy and images from a brand brief and a product photo, then scores every variant against the brand's own rules with an LLM judge, and how I measured whether that judge could be trusted.

Project snapshot

Detail	Summary
Project	Ad Creative Generator: a brand brief and a product photo go in, scored ad variants come out
Type	Personal project
Role	Solo: product thinking, evaluation design, hand labelling, engineering and deployment
Timeline	About two weeks
Stack	Python, Claude API (copy and judging), OpenAI image API (gpt-image-2), Pydantic, FastAPI, Streamlit, pandas, pytest, Docker, GitHub Actions
Dataset	3 fictional brands, 90 ad variants, 450 hand-labelled pass or fail judgments
Outcome	A live public app, and a judge that agrees with my labels on 82.0% of calls
Links	Live demo, GitHub repo, Blog post

At a glance

• 90 variants generated across 3 brands, each hand-labelled on 5 criteria
• 82.0% overall judge-to-human agreement after one round of prompt fixes, up from 80.9%
• $5 or less in API cost per 20-variant batch
• 0 shared API keys on the public app, because visitors bring their own

The challenge: AI makes ad variants cheap, and reviewing them expensive

When a model can write and illustrate ad variants in seconds, the bottleneck moves from creating to checking. Every variant has to match the brand's voice, avoid its banned words, stick to claims the product can actually support, and stay consistent with the real offer. Miss one and you ship something off brand, or worse, something that makes a promise the company cannot keep.

My first test made the problem concrete. I generated 20 copy variants from a single made-up coffee brand brief and read them all. Three problems appeared straight away:

1. Every variant used the same call to action.
2. The copy kept repeating the brand's mission and core values instead of concrete product facts and figures.
3. It skipped most of the offer section and ignored the "we are" and "we are not" lists in the tone brief.

Reading 20 variants is fine. Reading 200 after every prompt change is not, and the failures that matter most are the quiet ones. The real question for this project was not "can AI make ads?" It was "can a machine reliably catch the ads that break the rules, and how would I know?"

Goals and constraints

The project had one definition of done, written down at the start: a stranger can open the live link, upload a product photo and a brief, and get scored ad variants back, and the README reports how often the judge agrees with my own labels.

The constraints shaped almost every decision after that:

• Solo build, about two weeks. I was also learning the tooling as I went.
• No exposed keys and no uncapped spend. A public app that calls paid APIs is an invitation to run up someone's (my) bill.
• Measurable quality. The judge and I had to use the same five criteria, so agreement could be calculated rather than guessed.
• Strict scope rules. Two passes at most on the UI and the judge prompt. Fix only the two biggest bugs before launch. Everything else goes on a "later" list.

The approach: design the evaluation before scaling the generator

I started with the evaluation rather than the volume. Before generating a large dataset, I wrote a five-criterion rubric (tone, claims, call to action, length and visual fit) and built the judge around the best practice I found in three guides on LLM evaluation: Hamel Husain's guide to LLM-as-a-judge, Eugene Yan's Product Evals in Three Simple Steps, and Anthropic's documentation on building evaluations.

The key decisions, and the reasoning behind each:

Decision	Why
Pass or fail on every criterion, never a 1 to 5 score	A 3 versus a 4 is not actionable and hides which brand rule was broken
One judge per criterion instead of one big judge	Each is easier to calibrate, and a failure points to a specific rule
Code first, LLM second	Length and avoid-list checks are free, deterministic and never disagree with themselves. Only judgment calls go to a model
Thresholds taken from the briefs themselves	Headline of 12 words or fewer, CTA of 6 or fewer and a two-sentence body, based on the example lines and CTAs in the three briefs
30 variants per brand, 90 in total	Anything less than 60 variants or less, a measured failure rate is too uncertain to draw conclusions. Each brand also fails in its own way
Hand labels as ground truth	The judge is measured against my labels, not trusted on faith
OpenAI image API instead of ComfyUI	Easier to integrate and faster to deliver, with higher quality images for less effort
gpt-image-2 at a fixed high quality setting	Visual fit depends on printed label text staying legible, and "auto" quality could change between runs and confuse the results
Visitors bring their own API keys	No spend and no key exposure for me, and it simplified the architecture
Streamlit Community Cloud for hosting	Free for public apps and it deploys straight from GitHub

The dataset itself used three fictional brands with made-up briefs and product photos: Riverbend Brewing Co. (an IPA called Tide & Treetop), Verdan (hempseed cooking oil) and Velara (noise cancelling headphones). Each brand carries a different risk. Velara needs restraint, Verdan has compliance limits around cannabis references, and Riverbend must avoid taproom cliches.

The solution: how the Ad Creative Generator works

A request moves through six steps.

1. Brief and photo in. The brand brief is a JSON file with five sections: brand, product, audience, tone and offer. The brief also names its own product photo, so the two always travel together.
2. Copy generation. One batched Claude call returns every variant as structured data, validated with Pydantic. The prompt carries the rubric's own limits, so the generator knows the rules it will be graded on. The code also checks whether the response was cut off, before any money is spent on images.
3. Image generation. For each variant, OpenAI's image model edits the product photo. The instruction is to keep the product, label and composition unchanged and adjust only the background, lighting and setting. Every run saves into its own folder, with a manifest and a results file.
4. Judging. Five separate judges grade each variant. Length is pure code. Claims runs an avoid-list string match first and calls a model only when the code finds nothing. Tone uses a worked pass and fail example. Call to action is checked against the real offer. Visual fit uses a vision model that sees both the original product photo and the generated ad, so it can verify that the product is unchanged.
5. Scoring and ranking. The score is the number of criteria passed, from 0 to 5. Variants are sorted best first, and each failed criterion carries a one-sentence critique.
6. Delivery. A Streamlit app lets a visitor upload a photo, paste a brief, choose how many variants to generate (capped at 10) and read the ranked results with a pass or fail mark per criterion.

Engineering choices worth noting

• Clients are passed in, not created inside. Every function that calls an API takes its client as an argument. That is what made bring-your-own-key easy, and it also made the code testable.
• Free tests for the plumbing. The test suite mocks the models, so it checks the scoring, the sorting, the judge's output parsing and the contents of each judge prompt without calling any API or spending anything. Real quality can only be measured with real, paid runs, and I kept those deliberate.
• FastAPI, Docker and CI from my project template. The same starting setup for every project means no rebuilding per project. The FastAPI endpoint still works locally.

How I measured quality: 450 hand labels against the LLM judge

A judge is only useful if you know how far to trust it. So I treated it like any other classifier and measured it against ground truth.

The labelling. I labelled all 90 variants pass or fail on each of the five criteria, which gives 450 labels. My first attempt was a single overall pass or fail per variant. That made per-criterion agreement impossible to calculate, so I relabelled everything on all five criteria. It was the right call.

The comparison. For each criterion, I compared the judge's verdict with mine and counted two different kinds of disagreement: the judge being too strict (I passed, it failed) and the judge being too lenient (it passed, I failed). They need different fixes, so a single blended percentage would have hidden the story.

Why not just one number. Raw agreement can mislead when one outcome dominates. If only 5% of outputs fail, a judge that always says pass still scores 95% while catching nothing. Looking at each criterion, and at both error types, guards against that.

The improvement loop. I read every disagreement, made one round of prompt improvements, re-ran the judge on all 90 variants and measured again. One deliberate pass, in line with my rule of two passes at most.

A caveat on my own labels. I labelled leniently on some criteria, particularly tone and claims. I kept my labels as ground truth anyway, because the disagreement analysis is what shows whether my gut or the written rubric was right. In several cases the judge was enforcing the rubric correctly and I had waved a variant through.

Results: the judge is reliable on rules and weaker on taste

After one round of prompt fixes, the judge agreed with my hand labels on 82.0% of the 450 individual calls, up from 80.9%. It was near perfect on the objective criteria and weakest on visual fit.

Criterion	Agreement, v1	Agreement, v2 (after one prompt fix)
Tone	78.9%	80.0%
Claims	73.3%	75.6%
Call to action	95.6%	95.6%
Length	98.9%	98.9%
Visual fit	57.8%	60.0%
Overall	80.9%	82.0%

What the numbers showed

• Objective checks were reliable. Length reached 98.9% and call to action 95.6%. Rules a machine can verify were enforced almost perfectly.
• The Claims judge was missing context. Of the 18 cases where it was stricter than me, 13 were Velara. It was failing copy for features that were in the brief, because it had only been shown the proof points. Adding the key features was the main fix.
• Visual fit exposed a gap in the judge, not in my labels. I failed 23 of 30 Velara variants because the brief wants product-forward ads with minimal typography. The judge failed 4, because it had no check for restraint. I added one, calibrated on each brief's reference brands.
• Agreement rose on every criterion and still fell at the variant level. The share of variants where the judge's "fully approved" matched mine dropped from 66.7% to 61.1%. Cases where the judge was more lenient than me rose from 17 to 21. Fixing one gate in a five-gate AND exposed disagreement on another.
• Overall pass rate. The judge passed all five criteria on 48 of 90 variants before the fix and 51 of 90 after, or 56.7%.

Generation quality improved too

Once I put the rubric's limits into the generation prompt, the Velara call to action fell from 15 words to 3 or 4, and the same fix tightened the other two brands. In the six-variant re-run, every call to action was 3 to 4 words and every body was between 21 and 32 words.

Cost and time

• A 20-variant batch costs around $5 or less in API usage and can take up to 30 to 40 minutes, often less.
• Judging all 90 variants cost roughly $1. Image generation is the main expense.
• The public app costs me nothing to run, because visitors use their own keys.

What this means in practice. For a marketing team, the judge works well as a first-pass filter. It enforces word limits, banned terms and offer consistency almost perfectly, and it ranks variants so a person reviews the best ones first. For subjective calls like tone and visual restraint, a human should still make the final decision. It is a triage tool, not an approver.

Problems we hit and how each was fixed

The table covers the issues that changed the design. Each one was found by running the pipeline on real data, usually in a small pilot of two variants per brand before spending on the full set.

Problem	Root cause	Fix
Velara calls to action ran to 15 words	The prompt gave the model both the CTA and the offer, with nothing keeping them separate	Put the rubric's numeric limits into the generation prompt and state that the offer sells the deal while the CTA tells the reader what to click
Inconsistent on-image fonts, including a stray Times New Roman	Each image call is stateless, and product labels with two typefaces gave the model two valid fonts to copy	Anchor the font to the brand's own name and logotype, and add typography as a visual-fit check, since an image model cannot be fully controlled
Judge output cut off mid-sentence	On a pass, the model wrote long critiques. The "leave this empty" instruction lived in a code comment the model never sees	Move the instruction into the schema field description and raise the token limit as a safety margin
Claims judge flagged real features and offers as invented	It was never shown the key features, and later was never shown the offer section	Give every judge the full slice of the brief it needs, audit all of them, and add tests that check the prompt text
Visual fit too lenient on Velara	No check for restraint, so a loud headline could pass	Add a restraint check calibrated on each brief's reference brands
Two runs in the same second could share an output folder	Folder names used a timestamp accurate to the second	Add a random suffix and write images straight into each run's own folder
Image request rejected	The newer image model does not accept the input-fidelity parameter	Remove it, and check product fidelity visually in the pilot
Planned free host no longer covered Streamlit	Hosting terms had changed when I went to deploy	Switch to Streamlit Community Cloud

Lessons learned

1. A judge can only grade what it can see. The biggest Claims errors came from not showing the judge the key features, and later the offer. Before blaming the model, read the exact prompt it received.
2. Measure per criterion and per error type. A single combined agreement figure would have hidden the fact that variant-level agreement dropped while every individual criterion improved.
3. Give the generator the same rules as the grader. Adding the rubric's limits to the generation prompt fixed the call to action problem at the source, which is cheaper than catching it afterwards.
4. Know when the tool is the wrong tool. Image models are unreliable at exact typography, and no amount of prompt wording fixes that completely. The honest answer is to change the architecture, not keep rewording the prompt.
5. Test the plumbing for free, and spend money only on quality. Mocked tests caught real bugs at no cost. Paid runs were kept for the one thing only they can tell you: whether the AI output is good.
6. Scope rules protect a small project. Agreeing in advance on two passes at most, two bugs before launch and a clear definition of done is what kept this a shipped project instead of an endless one.

Limitations and what comes next

Limitations

• Small, synthetic dataset. Three fictional brands, 90 variants and a single labeller, who labelled leniently. A different labeller would give different numbers, and 90 labels still leave a margin of roughly 6 to 10 points on any single rate.
• The same model family writes and grades. Claude generates the copy and Claude judges it. Anthropic's evaluation docs suggest using a different model for grading than the one that produced the output, which I have not tested yet.
• The numbers describe an earlier judge. The 82.0% figure was measured before the later offer fix and the full prompt audit. I have not paid to re-measure.
• Image typography is probabilistic. The fix reduced font drift but cannot guarantee it.

Next steps

• Real brand guidelines. A formal brief with logo placement, colours, fonts and text sizes would give the judge firmer constraints and the generator less room to improvise.
• A brand-in-the-corner mode. Generate clean product photography with the brand mark in a fixed corner and no text, and add the headline afterwards in a template tool or with a designer, where type can be pixel exact. This separates the job image models do well from the one they do badly.
• OCR on the generated image. Read the on-image text and run it through the same avoid-list and length checks as the copy.
• More realistic failures. Eugene Yan suggests generating some outputs with weaker models to get organic failures, instead of relying on a strong model to produce them.
• A hosted version with my own API and strict limits, once I decide how I want to offer it.

See it in action

The Ad Creative Generator is live and the code is open source. You can bring your own brand brief and product photo, real or made up, as long as the brief follows the JSON structure of the samples in the repo.

• Try the live demo (you will need your own Anthropic and OpenAI API keys)
• Read the code on GitHub
• Read the story behind the build in the blog post

If your team is producing generative creatives and needs a way to measure whether it stays on brand, I would be glad to talk. Get in touch.