Recouping after a rich week at AI Engineer World Fair, we’re still energized by all the insights and conversations from all the talks and conversations with hundreds of people over three days. We’ve been to all the AI Engineer events so far, and this felt like the best one yet — particularly because the vast majority of the 3000+ people at the event are all past the exploration stage now and into building seriously with AI.

Two trends stood out:

1. Everyone is building some kind of “agent” (or multi-step workflow).

2. Evaluation is still the biggest bottleneck in AI engineering.

Complexity is exploding (MCP! A2A!), and everyone is trying to figure our how to manage quality and optimize production systems. We were excited by all the interest in what we’re building at Freeplay.

The event featured over 200 talks, and they’ll all be up on YouTube soon. If you’re curious to dig in now, the first ones are up — check out the keynotes and MCP talks from the Anthropic team here.

Jeremy from our team joined our friends Chris Hernandez and Eliza Cabrera for two talks at the conference that will be particularly interesting for product teams building AI features and agents. We’ll post the videos as soon as we have them, and we’ll post a quick recap plus the full slides for now.

The Build-Operate Divide: Bridging Product Vision and AI Operational Reality

Chris and Jeremy’s talk focused on the operational workflow involved to run a successful AI product in production. Chris manages a team of domain experts who spend their time measuring system quality, reviewing AI system outputs, curating datasets, improving evals, and running experiments. Jeremy’s worked closely dozens of people and teams like Chris’ who use Freeplay to run these workflows.

Some key takeaways:

1. Too many companies fail to deliver high quality AI products because they haven’t planned for these operational demands. They ship an AI feature but don’t have the people in place to run the process. You can’t just “set it and forget it” with AI! Models change constantly, customer behavior can be surprising, and every time you make a change to your system, output quality can change in unexpected ways.

2. Scaling AI products still requires a lot of manual effort by way of human in the loop for labeling, evaluation and prompt engineering. Building good systems and tools can help limit the effort required, but at least today in June 2025, no production systems can hit ideal quality bars without regular human review and ongoing experimentation. Automated evals speed this process up, but they don’t replace it.

3. Existing ops and quality teams and be repurposed into AI quality teams. They already have the domain expertise, and it’s possible to teach people the technical skills required to get good at prompt engineering and AI evaluations. Chris and his team are great examples of people who have learned these skills, and act like the “AI Quality Lead” role we’ve described in the past.

Check out the full slides here.

And if you’re really interested in these topics, we highly recommend Hamel Husain and Shreya Shankar’s upcoming course: AI Evals for Engineers & PMs. Lots of good deep dives and advice!

Build Dynamic Products and Stop the AI Sideshow

Eliza and Jeremy dove into what it really takes to move past flashy demos and bolted-on chatbots, and start to fold AI deeply into existing, revenue-generating product suites.

Drawing on recent customer roll-outs and hard-won lessons from Eliza’s experience at Workday and across the Freeplay customer community, they laid out practical suggestions for product teams that want AI features to feel indispensable (rather than optional add-ons) inside a broader product experience.

Some key takeaways:

1. Customers want AI that’s woven into the core experience—not bolted on. Chatbots and sparkly side panels were novel in 2023, but in 2025 end-users expect AI to be present where they already work: enriching search, shortening workflows, pre-populating suggestions, etc. If an AI feature lives in its own corner of the UI and isn’t part of core workflows, usage often drops quickly after the first week. At the same time, thoughtful integration of AI can boost engagement metrics because value is delivered in the exact context where the problem already arises for customers.

2. “AI strategy” must be integral to overall product strategy. Treating AI as a parallel R&D track leads to conflicting priorities, duplicated UX patterns, and unclear success metrics. Leading AI teams meanwhile start roadmap discussions by asking, “What core job is the customer hiring our product to do, and how can AI make that job faster, cheaper, or more delightful?” From there, ideas for how to incorporate AI fall naturally out of the same OKRs that guide the rest of the product.

3. Adopt a crawl-walk-run approach to ramp up, and to de-risk. Eliza showed how teams she advises begin with an internal-only “crawl” phase: building small, opinionated workflows that prove value with AI. In the “walk” phase they expose power users to more complex AI-augmented features. Only after success metrics stabilize do they “run” by expanding to harder to build agents and personalization features, multi-modal, etc. Each stage generates learnings and organizational know-how for the next stage, preventing the “big-bang” attempts and failures that plagued early AI projects.

Full slides here.

If you've not been to AI Engineer, check it out! We'll update this post with the full talk recordings soon.