Improve Your AI Product Faster with Freeplay’s New Human Review Features
A few months ago, we wrote about why human review is crucial for AI product development, and how more teams are investing in dedicated data review and labeling efforts to support the quality of their AI products.
Simply put: The best AI teams look at lots of row-level data to develop understanding of their systems’ performance, and to build intuition about how to improve.
We now see teams with 30 or 40 people using Freeplay to coordinate data reviews at scale for a single AI feature or agent, and they’ve been asking for better tools to manage their process. And not just to manage the review and labeling aspects — also the knowledge sharing and distribution of learnings at the end. Any labeling effort needs to lead to closing the loop and taking action.
That’s why we’ve add a few key things to Freeplay to manage reviews at scale and to make it easy to share learnings to a wider team. Whether you’re a small team or a big one, these will help!
Review Queues add simple project management. Define data that needs review, assign tasks to various team members, set due dates, and track statuses
Integrated Insights make it easy to share learnings with a simple link. A short written summary and interactive charts for all label values and eval scores make it easy for stakeholders to digest learnings.
Granular user roles and private projects protect even sensitive data. New user roles for analysts and contractors make it easy to limit who can do what in Freeplay, and new “private projects” protect data so that only people with approval can see row-level data and prompts.
One of our customers from a large financial institution had this to say after using these new features for a couple weeks:
“Freeplay’s Reviews feature has been a game-changer for operationalizing LLMs within our prompt ops workflow. Analysts can now efficiently review interactions, surface insights, and refine prompts—all in a seamless, closed-loop system. Our insights now directly inform stakeholders, and we can deploy prompt updates to production without relying on engineering. This has accelerated our iteration cycle and made continuous improvement a reality."
Here’s a quick video that shows the basic workflow, and can go deeper in our docs here. Read on for more perspective on how Review Queues can help your team.
The Evolution of Human Review
As we highlighted in our previous post, looking at lots of row-level data is one of the biggest levers for improving AI product quality. Industry leaders are investing heavily in this approach by:
Dedicating full-time roles to AI product quality
Building teams of specialized data analysts and reviewers with the right domain expertise
Making data review a core part of their ongoing development process
Importantly, we see most of these efforts form in house when it comes to AI product development. Data curation and labeling for general foundation models may make sense to outsource, but it’s often the case that interpreting quality for LLM outputs in a targeted B2B product are too nuanced to outsource to others.
As these internal data ops efforts scale, teams face new challenges:
How do you coordinate review work across growing teams?
How do you ensure consistency in review processes?
How do you track progress and demonstrate ROI from these efforts?
How do you turn review findings into actionable improvements?
Review Queues address these challenges head-on by bringing integrated, enterprise-grade workflow management to the human review process.
How Review Queues Work In Freeplay
Structured Review Workflows
Review Queues helps you organize review efforts into discrete, manageable chunks:
Use Freeplay’s integrated search and filtering features to create bounded sets of completions for review
Assign work to specific team members or groups and set deadlines
Track progress with review status and queue-level analytics
Filter and prioritize review items based on your needs — e.g. to inspect data with poor scores, or find all the examples of a given intent in an agent product
Most importantly, Review Queues creates a clear output – every queue generates an insights report that helps communicate findings and drive action.
Turn Review Findings into Action
Review Queues aren’t just about organizing work – it's ultimately about driving improvements to your AI products.
As these efforts formalize, we see more teams producing things like weekly quality reports for engineering teams, or pulling quality data into email summaries for executive review. Whenever people spend time to review and label lots of data, they want those learnings to be shared.
Some of the ways Review Queues help feed that learning loop:
Insights Reports: Auto-generated reports surface patterns and trends in your reviewed data, updated in real-time as work progresses — no need to build separate pivot tables and charts!
Dataset Creation: Convert reviewed completions into datasets for testing and fine-tuning
Support Analysis by All Stakeholders: Once data is labeled, anyone can drill down to row-level data to investigate specific issues
Cross-Team Collaboration: Share findings easily between analysts, engineers, and product managers — all in one spot, without the need to ship spreadsheets around and copy/paste data
The Integrated Human Review Solution
Review Queues integrates seamlessly with Freeplay's existing features to provide a complete and deeply integrated solution for human review:
Observability: Create Review Queues from observed production data
Monitoring: Use auto-evals to catch issues and help flag things for human review
Data Labeling: Apply and manage custom labels
Evaluations: Review and correct LLM judge scores as part of a Review Queue
Testing: Convert reviewed data into datasets for better batch testing
Collaboration: Share insights across your organization
Getting Started
Review Queues are available today for all Freeplay customers. You'll find them under the new Reviews tab in any project.
We're excited to see how teams use Review Queues to improve their AI products. Let us know what you think!
Categories
Product
Industry
Authors
Jeremy Silva
