AI Agents and Ethical Outcomes: Insights from Kolena’s Andrew Shi
Thu 27 Feb 2025
In this sponsored feature, Techerati spoke with Andrew Shi, CTO at Kolena, about why AI agent evaluation goes beyond accuracy. We discussed how businesses can redefine trust, ensure ethical outcomes, and address key challenges in AI evaluation. We also discussed emerging metrics, real-world use cases, and best practices for testing AI to drive long-term value.
– – – – – –
How should business leaders redefine trust in AI performance beyond just accuracy metrics?
I think there are a few key factors at play here. Accuracy and correctness in general are crucial for AI performance. With AI agents, there is an even higher level of responsibility. They are more powerful than traditional language models, which means they have a greater impact. The way people interact with them is also evolving.
A lot of the direction for AI agents is conversational. User experience and how people interface with these systems are important metrics to track. Additionally, if AI agents are performing specific tasks or calling external tools, adherence to policies becomes a critical concern. These challenges are becoming more prominent as AI agents continue to rise in importance.
In an era of rapid AI deployment, is the industry focusing enough on evaluation or is it too fixated on speed to market?
Given my role here at Kolena and our focus on evaluation, my instinct would be to say, “Do more evaluation, and do it quickly.” However, I think it is a bit more nuanced than that. It depends on the stage you are at in solving a problem.
We are a startup here at Kolena, and sometimes speed is important. But I believe it also depends on the maturity of the system. In the early stages, when you are iterating and experimenting with the tool, speed still holds value. However, builders need to recognise technology comes with debt. You take on that debt because it helps at the moment, but eventually, you must address it.
For AI agents specifically, I think the industry has shifted to recognise that when you are solving problems with your tools, you need to ensure they work well, especially in broader, real-world situations. As the system matures, evaluation becomes essential and critical. I do believe the industry is adapting to this shift.
How can organisations ensure ethical AI outcomes through robust evaluation, particularly in high-stakes industries like healthcare or finance?
The way you ensure most things in an organisation is by measuring and tracking them. I think this process is quite similar when it comes to ensuring ethical AI outcomes. You need to measure and track it. The process begins with ensuring you have the right tools to measure and track your inputs. For example, with your system, you collect metadata and any relevant data that could be useful. Then, you break down your performance by different scenarios defined using that metadata.
When it comes to ethical outcomes, fairness is key. Achieving fairness involves assessing it across various populations and different stratifications within your target audience.
What kind of emerging metrics beyond accuracy should be part of every AI evaluation strategy?
I think I touched on some of these points earlier, particularly regarding policy and user experience. These can vary depending on the type of agent you’re working with, whether it is a single-turn or multi-turn conversation.
In addition to correctness, I believe accuracy is also nuanced. It is about understanding how often we get things right when we should and identifying instances where we make mistakes when we should not.
It is important to break that down. We need to ensure we are engaging in good behaviours and avoiding bad ones.
Is there a gold standard for measuring real-world performance or should evaluation methods be tailored by industry and use case? What would you say would be the perfect method for addressing this?
For agents specifically, there is a strong overlap in how you would measure performance and the tools required. However, I think the metrics you define should ultimately be driven by the industry use case.
In terms of process, how you go about testing and how that testing impacts deployment and iteration is crucial. The feedback loop of general software development follows a gold standard. I believe the DevOps process, particularly the idea of shifting left, should be embraced.
Ultimately, you need specific metrics based on the problem you’re solving, and you should target those metrics as you improve your processes.
Are there any fundamental blind spots in current AI evaluation practices which are putting businesses at risk?
We are still very early in the evaluation process, and I think a lot of the challenges do not stem from a lack of consideration. It is not that people have not thought about it, it is just that you can only gather as much information as you collect. This highlights the importance of ensuring that you gather metadata and learn about problems as thoroughly as possible. There is a lot to discover by going through the data and tracking it.
That said, the current challenge we are seeing, and what we are working on helping our customers resolve, is about taking a process that works at a small scale and figuring out how to scale it effectively. It is not that our customers are unwilling to evaluate things. Rather, they have processes that work at a human level because, in many cases, a human judgment is sufficient. The real question is: How do you scale that process? If you have many scenarios to address, how do you scale the evaluation to manage it effectively?
With the real-world use cases of AI agent evaluation, how has rigorous AI evaluation helped companies eliminate harmful biases in their models?
One of the first examples that comes to mind is one of our early customers, who worked with facial recognition technology. Although I am steering away from the AI agent topic, I think this example fits the question well.
For them, one of the key considerations was ensuring that their facial recognition tool worked effectively across different scenarios. They wanted to assess its performance for people of various races and ages, among other factors. The primary concern was ensuring their model was not biased in any way.
The process they followed involved defining clear metrics and scenarios, establishing the most important test cases, and tracking the system’s performance over multiple iterations.
What role should human oversight play in AI evaluation and how can businesses balance automation with human-in-the-loop approach?
I think there is a significant trade-off here, where you are balancing scalability with automated tools versus correctness and higher fidelity with human-in-the-loop approaches.
There are a few approaches you can take. The low-hanging fruit is a hybrid model, where you use automation for scale but spot-check with human input.
Additionally, there are ways to make human-in-the-loop processes more effective. For instance, you can use tools that help humans work more efficiently or, on the flip side, improve the quality of automation. One example of this is our experimentation with AI classifiers based on human feedback. By tuning and tweaking the classifier for a specific problem, we can achieve higher scale while maintaining quality.
In fact, the signal from this approach outperforms the out-of-the-box AI, which is more generalised.
How can companies future-proof their AI evaluation strategies to adapt to emerging models like multimodal and generative AI agents?
I think this goes back to ensuring that you have the problem clearly defined. You need to treat evaluation as a primary objective. It is not something you look at after the fact; it is something you use to define how you approach the tool. This goes back to defining the metrics that matter to you – how to implement them properly, how to measure them, and track them with minimal friction. Also, considering the tools that can help with this is key.
What does continuous evaluation mean in practice and how can it drive long-term business value and model performance?
Continuous evaluation is similar to continuous integration and continuous deployment. It is about minimising friction in your processes. You want to reduce the gap between live production and evaluation so you can understand how your system is performing in real-time, as opposed to waiting to collect data and evaluate it later.
In my opinion, this is a goal you can tackle after establishing a solid evaluation process. It is an advanced step in the maturity scale and requires an iterative approach. Initially, you might not have any evaluation process in place, but you start by developing a basic one. Over time, you can introduce tools to better reflect on performance.
Eventually, the goal is to implement a continuous evaluation process, where you monitor data in production and automatically integrate it for deeper analysis. This allows you to flag and address issues that you wouldn’t have been able to identify otherwise.
Long-term, from a business value and model perspective, the goal is to ensure that when deviations occur, you are able to debug, address, and remediate them promptly.
