AI Design Agents Are Clumsy. Context Is How You Fix That.

AI design agents are getting better every month. They are also still clumsy. In my experience they miss more than they hit, and after using them daily I think I finally understand why. It is not the model. It is the context.

Key takeaways

• AI design agents fail most often because they start from nothing, not because the underlying model is weak.
• Today you can hand an agent some text or a screenshot, but not your full creative context.
• The fix is a context layer: a curated visual library of references an agent can actually learn from.
• This mirrors how designers have always worked. Study what you admire, internalize the decisions behind it, then make it your own.
• I am building that context layer as a tool called Hippo (feedhippo.io).

Why AI design agents miss more than they hit

The honest answer is that most people hand a design agent nothing to start from. If you open a blank prompt and expect a finished interface, you are either not a designer, or you enjoy gambling. Agents are only as good as the context you give them, and right now most of us give them almost none.

My time is limited. Between gratefully raising my daughter, our three dogs, and a partnership with the love of my life, the hours I spend with an agent need to count. I hate burning that time re-prompting until something is right. I want the agent to get close enough that when I pick the work back up, I am refining judgment calls instead of fixing what should have been right the first time.

The problem is structural. Right now you can pass an agent markdown or plain text, or you can pass it a screenshot, and it will try to mirror what it sees. It is usually one or the other. Outside of doing it manually, over and over, there is no real method to give a design agent its full context. I do not mean to sound naive. People like TJ Pitre, founder of Southleft and now leading design systems and AI strategy at Figma, are constantly pushing what an agent can do when it is grounded in a design system. But what happens when the thing you are building is not a design system, and is not rooted in code as its final form? How do you build something genuinely unique with AI and design tools?

Stop trying to one-shot your design

One-shot prompting is a party trick. It is great for grabbing attention on a feed, and it is mostly useless for real design work. We have already watched this play out in writing, in image generation, and in code. If you expect to drop a single prompt and pull a finished thing straight out of your head, good for you, but that is rarely how good design actually gets made.

The bar is not mind reading. We can all agree an agent is not going to nail what is in your head on the first try. The bar is reducing the fumbling. I do not need an agent to be me. I need it to get close enough, consistently, so that my time goes to taste and decisions rather than cleanup.

Context is the next frontier, not a better model

We have built the execution layer. The context layer is what is missing. There is even a name for it now: context engineering, which Andrej Karpathy describes as the delicate art and science of filling the context window with just the right information for the next step. Since he and Shopify's Tobi Lutke put language to it in mid-2025, U.S. search interest in the phrase climbed from roughly 90 searches a month to a peak near 14,800 within two months, then settled into the thousands (Google Ads search volume via DataForSEO, 2026). I do not think that spike is a coincidence. As models converge on similar capabilities, the differentiator stops being the model and becomes the context and the judgment you bring to it.

This is not only a design idea. Jack Dorsey has described a version of Block where AI handles coordination while humans sit at the edge supplying intuition, cultural context, and judgment, what he calls facilitating a more context-rich decision. I have heard a related idea floated on the All-In podcast: that we will each carry a kind of metaphorical briefcase of individualized context from one model to the next, and that briefcase is what actually powers the work once the models all start to feel the same. I think that is right, and I wanted to adapt it for the creative field. I want to bring context to AI in many different ways so it understands what I am thinking, instead of me having to describe it from scratch every single time.

Designers already know how to do this. It is called stealing like an artist

Austin Kleon's Steal Like an Artist (2012) is, underneath, a book about context. When I think back over the last ten years of designing, the lesson always comes back to it. Take the things you admire, study them, analyze them, even replicate them so you understand the decisions that were made and why. Then take that learning and turn it into your own, with your own spin on it.

Boil that down to its core and it is a context-gathering loop. Gather, study, internalize, remix. The trouble is we have never had a clean way to hand that loop to an agent. So instead of running it manually, over and over, I decided to build something that would speed it up.

The fix: a visual context layer

So I built one. Hippo is a visual library for context, and the name comes from one of my favorite childhood games, Hungry Hungry Hippo. I wanted to be able to save and upload anything that catches my eye, the way that hippo just inhales whatever is in front of it.

Hippo lets me organize anything I see on the web, whether that is text, imagery, or specifically UI elements, into one library. On top of that I built the utility piece I have always felt was missing as a designer: I can bring that whole library, or any selection from it, straight into Figma with a single click. Hippo is in beta and will be for a little while before I start implementing the AI features, but it is already one of my favorite tools to use, because it is how I am building the library I will feed my design agent more and more.

How to work with a design agent today

Until a context layer is everywhere, you can capture most of the benefit manually. This is the workflow I use.

• Build a reference library before you prompt. Collect the screens, patterns, and styles you admire so you have something real to start from.
• Feed curated context, not adjectives. Show the agent three precise references instead of ten vague words.
• Separate execution from judgment. Let the agent run the first pass, and keep the decisions for yourself.
• Reuse your context. A library compounds. Every save makes the next prompt sharper than the last.

Frequently asked questions

What is an AI design agent?

An AI design agent is a tool that turns prompts, text, or reference images into design work such as UI layouts, components, or full screens. Unlike a single image generator, an agent can take direction, iterate, and work inside design environments, but its output quality depends heavily on the context it is given.

Why do AI design agents give poor results?

The most common reason is missing context. When you hand an agent a short prompt or a single screenshot, it has to guess at your taste, references, and intent. Without a curated library of what you actually like, the agent fills the gaps with generic patterns, which is why the output so often misses.

What is context engineering for design?

Context engineering is the practice of assembling and structuring the information an AI uses to make decisions. For design, that means feeding an agent the references, UI patterns, and visual language you want it to draw from, so its output reflects your judgment instead of an average of the internet.

Can AI design agents replace designers?

No. Agents are strong at execution and weak at judgment. As models converge, the lasting advantage belongs to the person supplying the context and making the calls. Design agents change how designers work, but the taste, the references, and the final decisions still come from a human.

Where this goes next

Models will keep getting better, and they will keep converging. When they do, the thing that separates real design from generated noise will be the context you bring and the judgment to use it. That is the bet I am making with Hippo. If you want to follow along or try the beta, it lives at feedhippo.io.

---

Written by Jon Sorrentino, a product designer with over ten years of experience and the founder of Call The Design Guy. Jon writes about design, AI, and the craft of building things people actually want to use.