Chroma AI

Sept 15, 2024

I was a bit late to the AI game—while others were teaching robots backflips, I was just happy my fridge’s water filter worked. But better late than never! This weekend, I finally dove into AI models, prompt engineering, and AI code editing!

🤖 Actually, I'm not new to AI—I've been designing with it for a while.

In 2002, I applied for a PhD at Zhejiang University, where they explored using Artificial Intelligence expert systems for design and CAD tasks, such as creating a poster given a topic like “Beers on beaches.” Today, we call it “generative AI.” (P.S. I did not get into the program.) I began working with AI as a UX designer at GE Healthcare in 2015, where I designed interfaces for AI-driven cancer detection in X-rays. At 3M in 2017, I designed user experiences for food lab equipment enhanced by TensorFlow models, outperforming traditional OpenCV algorithms.

In this post, I’ll share my experience as a beginner building an AI-powered app and offer insights from a designer’s perspective on collaborating with AI.

Thoughts on AI

“Will AI replace UX designers?” I’m not sure if AI is just a buzzword or something we should really take seriously now. Hearing Lex Fridman on his podcast talk about his fear of being replaced—and his use of an AI-powered IDE to write code—made me curious. I wanted to find out for myself.

The challenges we face in designing and building products today include:
- Design, like software engineering, is still labor-intensive and involves a lot of repetitive “drudgery.”
- We often find ourselves relearning solutions to problems that have already been solved, and this occurs at the personal, team, and organizational levels.
- The tools we use can define our roles, restricting our focus on the higher-level tasks that truly matter.
- Designers often spend too much time on minor details. While craftsmanship is important, it can distract us from big-picture, creative thinking—and it's not scalable.
- While automation can be beneficial, it often requires a level of effort that exceeds typical product design and development, defeating the purpose of automation. Additionally, it doesn’t always tackle the daily 'drudgery' that each person faces.
- There are no standard heuristics for evaluating the quality of a design or system, particularly in the context of their applications.

The Big Idea

The next stage is going to be computers as “agents.”

More Projects

— Steve Jobs, 1984

Can AI build a design system? And what would that workflow look like, from initial requirements to the final product?

So why am I thinking about this?

Having helped build several design systems and watched their rise in popularity since the mid-2010s, I can’t help but remain a bit skeptical:

Unintended Consequences: Every company, and often teams within the same company, seem to reinvent the wheel. Most design systems start with a button, following nearly identical requirements and specs.

Repetitive Work: Despite similar needs, building these systems requires at least one designer and one developer. These teams end up doing the same work as everyone else who builds a design system.

Lack of Higher-Level Abstraction: Most design systems are based on UI components, layouts, colors, and themes. But businesses speak in terms of broader concepts like "high-tech," "gen-Z," or workflows like "user authentication" and "checkout."

To test this idea, I built a small app over the weekend. The concept was simple: a designer provides a keyword for a color theme, and the app generates color palettes based on that keyword.

Here are my takeaways:

“The future is now.” It’s not just a catchy phrase—it’s happening. AI is breaking down silos and boosting productivity for developers and designers alike. To collaborate effectively with AI, designers need to adopt a product manager’s mindset: focus on the big picture and know when to let AI take the reins.

You can check out the project on GitHub.

Thanks for reading! If you're curious about how I collaborated with AI, keep going for the details.

How LLMs help build this app

Generate & edit code

Brainstorm ideas

Copywriting & editing

Takeaways

“The future is now,” and it's not just a cool slogan—it's the real deal that's flipping the script for both developers and designers. AI can break down silos, significantly boosting efficiency for both. To collaborate effectively with AI, designers need to adopt a product management mindset, focus on the bigger picture, and know when to delegate tasks to AI.
- Human intuition and expertise remain crucial for direction-setting, effective planning, and divide-and-conquer between humans and AI.
- Success requires understanding both the overall vision and specific details. Human craftsmanship will not become obsolete.
- Currently, LLM-generated UI is quick and good enough, making it ideal for demos or prototypes. Check out these color palette component UIs:
  - Human-designed: Created by me using traditional methods.
  - AI-generated: Created by me using Cursor (and Claude-3.5-sonnet).
  My initial draft took several tries and likely more time, while Cursor's AI-generated version was its first attempt. It's a clear demonstration of how AI is changing the game in design.
- In the future, LLMs will be able to generate the entire design system, or having its own taste and style.

Humans drive the Project, not AI. AI still falls short of human intuition and situational awareness:
- Take a moment to consider the bigger picture and plan your approach before asking AI.
- Give clear and specific instructions.
- Break down complex tasks into smaller, manageable steps.
- Look at the output with a critical eye.