
Superagents: Putting AI colleagues to work across your organization
Online
What does it mean to have an AI colleague? In this session we go beyond chatbots and simple workflow agents, and look into the concept of "superagents". These agents work at a higher abstraction level, taking on a broader role rather than just executing one task. They make judgement calls, collaborate with your team and other agents, and proactively solve problems.
Watch recordingEvent
information
The session is hosted by Henrik Kniberg, co-founder of Abundly AI and author of Generative AI in a Nutshell. He is also known for his earlier work at Spotify, LEGO and Mojang.
Topics covered:
- Surviving vs thriving in the age of AI agents: AI as an operating model change rather than just a tool rollout
- Level 1 vs Level 2 agents: the difference between a specialist that follows instructions vs a superagent that owns an outcome.
- Real-life examples of agent teams interacting with each other and their humans.
- Company Brain - how to align a fleet of agents
You'll see a superagent at work, and you get to ask her anything.
What you'll experience
- •The talk is about what it actually means to have AI colleagues. We'll go beyond chatbots and workflow agents, and talk about agents that own outcomes, make judgement calls and collaborate with your team. And also address some pitfalls and practical tips and techniques, for example how to scale your agent fleet while maintaining control. We'll also cover: - Surviving vs thriving in the age of AI agents: AI as an operating model change rather than just a tool rollout. - Level 1 vs Level 2 agents: the difference between a specialist that follows instructions vs a superagent that owns an outcome. - Real-life examples of agent teams interacting with each other and their humans. - Company Brain - how to align a fleet of agents
Who should attend
Technology, operations and business leaders in enterprise organizations who are serious about succeeding in the age of AI agents.
Speakers

Henrik Kniberg
Co founder, Abundly AI
Recording
60
Watch recording
60
Enter your details to access
Summary
Henrik shared what Abundly has learned about making AI agents work in practice. The real value doesn't come from giving everyone a chatbot. It comes from agents working as team members alongside humans. He showed how Abundly's five-person engineering team works with agents like Grace, who orchestrates coding agents and opens pull requests for humans to review. Over 30 days the team shipped 161 product changes, with agents costing about 5% of the team's salary cost. Customer examples included Puzzle, which has hundreds of agents built by non-technical staff. Key lessons: agents should be built by the people who use them, every agent needs a human owner, and you build trust step by step with hard guardrails. To succeed you need an AI lead, champions and a platform with governance. The session ended with a live voice demo with Grace and a Q&A on evals, code review and model choice.
Transcript
Welcome
Kaj: Welcome, everyone. We already have more than 50 people in the room, so let's get started. My name is Kaj and I work with Henrik at Abundly. This is meant to be an interactive, fun and inspiring session, so please use the chat and ask questions. Anything we don't pick up right away, we'll handle towards the end. With that, I'll hand over to my fantastic colleague and co-founder of Abundly, Henrik.
Introduction
Henrik: Thank you, Kaj. I'm really happy to have you all here. We ran this session about superagents a few weeks ago in a physical room, and so many people were interested that we had to turn a lot of them away. So we're doing it again online, and it's great to see there's still so much interest.
I'm Henrik, co-founder of Abundly. The purpose of today is to share what we've learned over the past few years helping organisations make AI agents work in practice. Before we get into it, I want to show a short video. I think it's often best to hear what we do from the voice of the customer rather than from us, so here's a three-minute video from one of our partners, Puzzle.
Customer video: Puzzle
(Summary of the video) Puzzle describes Abundly as a powerful agent platform that lets everyone, regardless of role, build agents to solve their own problems, and as a strategic AI advisor that helps them build agents safely and compliantly while coaching them through the change. Puzzle now has hundreds of agents in its sandbox and more than 20 in production, including an incident-handling agent that collaborates with all product teams and has become a critical colleague. The agents are built by the people closest to the problems, not by a central AI team or consultants. Puzzle highlights how much fun its builders have, that humans become more important rather than less, and that upskilling its people has been the most important outcome. They see agents not as tools but as AI colleagues.
From surviving to thriving
Henrik: So that's what we do. The backstory is that a few years ago we noticed that when you take AI out of the chat and give it legs, tools and some autonomy, it becomes so much more useful. We saw a future coming where teams of humans and AI agents work side by side, and the question was how to make that work in practice. What's interesting is that this future is already here.
About a year and a half ago I wrote a book with the subtitle "How to Survive and Thrive." In this talk I'm zooming in on the thriving part. Surviving is the minimum: rolling out ChatGPT or Claude to everybody without changing the way you work. AI is mostly chat and mostly personal, a genius in your back pocket. That's useful and better than nothing, but we like to look way beyond it.
The real value comes when you see AI as an operating model change. AI agents aren't just personal tools; they're team members working with multiple people in the same tools you use every day, and they can work across an entire value stream, not just inside one team.
We see this shift happening on three fronts. First, AI inside your product, making the product itself more intelligent. Second, using AI to build your product, with tools like Claude Code, Cursor and Codex. Third, AI for operations: helping us get work done more efficiently, with no connection to code at all. All three matter, but today I'll focus mostly on the last two, especially operations.
What is an agent?
The way we see it, an agent is a kind of virtual creature that uses a large language model as its brain. It has a mission that goes beyond responding in a chat, it has tools to do the job, and it has some level of autonomy. Just like a human intern, it needs a mission, the right tools and the freedom to act, not just sit and wait to be prompted, but do things on its own initiative or on a schedule. So it's more than a tool-enabled chat, and a lot more than a personal assistant.
This takes some getting used to. We're used to work being done either by humans or by code. Code is fast, predictable, not intelligent and cheap to run once built. Humans are slower and fundamentally unpredictable (our performance depends on whether we've had lunch), but we're intelligent, and we have a fairly high running cost. Agents sit somewhere in between. It takes a while to figure out where to put them to work, but once you start, you find endless use cases.
How we use agents in our own engineering team
Our engineering team is small: five people plus a handful of agents. We use the typical engineering tools, a mix of Cursor and Claude Code, and on top of that we've built a small team of agents for the work we'd rather not do manually. Backlogger turns messy Slack threads into clean tickets. Releaser ships what we've built to production every day, updates the changelog and informs customers. And Grace is a higher-level agent I'll talk more about.
We've worked like this for quite a while, and it genuinely feels like they're team members. They're obviously not human, but if you looked inside our Slack you'd see conversations between several agents and several humans, and you often wouldn't notice who's who.
Level 1 and level 2 agents
We find it useful to think of agents in levels. Level 0 is code: it does exactly as instructed, no thinking. A level 1 agent is a specialist, like Backlogger, that does one thing. Most companies start by building a bunch of level 1 agents for tedious tasks.
We're now seeing growing interest in level 2 agents, which are more like generalists. They don't need detailed instructions and workflows. Instead they need what humans need: clear context, broader access to tools, and the ability to communicate with people and other agents. Agents like this weren't possible half a year ago because the models weren't good enough. Now they are, and that changes the whole picture.
Meet Grace
Grace is a level 2 agent on our platform. Her instructions guide all her behaviour, much like a work instruction for a human employee. Beyond her purpose and personality, we emphasise that she acts like a real team member, not a bot executing commands. She thinks, evaluates and makes judgment calls. We've given her essentially the same tools as our human engineers, most importantly the ability to launch coding agents. Grace doesn't write much code herself; she launches, orchestrates and manages coding agents.
Here's an example. We mostly talk to Grace in Slack. I asked her: "Hey, I think we need a starring feature or something." (It's a bit meta: Grace runs inside the Abundly platform and is also editing the Abundly platform itself.) Grace pushed back: "Starring what? Agents, documents, conversations? What's the use case?" That's exactly what we want. We don't want her to just do things because we ask; we want critical thinking.
I gave her more context. She checked our backlog, fired up a coding agent to analyse the code, concluded it was a small, low-risk feature worth doing, and came back with a few design questions. I answered some and told her to decide one herself. Then she said, "I've got what I need," and built it. Seven minutes later a pull request was up, ready for review. The whole thing, from idea to ready for production, took about 20 minutes. She posted about the PR in our development channel, and a human engineer merged it.
When agents work together
Level 2 agents get really interesting when they talk to other agents. In our platform you simply connect agents and say "you can talk to this agent." The agents around Grace are level 1 agents she can use as assistants.
For example, Mrs. Metrics checks our production logs for anomalies. One day she noticed something odd, posted in Slack with context, frequency, a root cause assessment and severity, and pinged Grace: "Can you look at this? I think it might be a problem in the platform." Grace looked at the code, found the problem, decided it was a no-brainer and fixed it. When the engineers came to work, it was already waiting. My colleague Erik approved it.
You might wonder whether humans are being demoted to people who just approve things. Not at all. It means Erik can keep working on what he was doing, without distraction and multitasking, while the small fixes happen in the background. That's the overarching theme: agents save time and focus so humans can do higher-level work.
After the merge, Mrs. Metrics said she'd keep watching the logs to confirm the problem was solved, and Releaser shipped it to production. Mrs. Metrics monitors, talks to Grace, Grace makes a pull request, a human approves, Releaser deploys. It's a team, with clear rules about who does what, where humans come in, and who owns which information and decisions.
The obvious goal with Grace was to offload engineers by handling well-understood requests. But a side effect turned out to be even more valuable: a fast lane for stakeholder requests. Pretty much anyone in the company can now cause changes in the platform. A human engineer still approves, but a lot of work no longer has to wait in the backlog. And because Grace is a generalist, people also ask her things like "How does this feature work?", "What's the status of this?" or "What are our main security risks right now?" She's become a general product development assistant available to everyone.
Questions from the audience
Kaj: Ernest asks: Mrs. Metrics checks production logs in the background. How does that work? Is she running all the time and reading logs, which sounds expensive, or is she triggered by an error?
Henrik: She has two recurring jobs, one nightly and one roughly hourly. She uses scripts she's written herself that extract only the part of the logs she needs, which makes it very cheap. If she doesn't find anything, she does nothing. The hourly check looks for things that are broken right now; the nightly one looks for patterns. We put a lot of work into this kind of optimisation, and that's exactly what the agent design process is about.
Kaj: Martin asks: What does Erik do all day? Just approve PRs, or triage, architecture, maybe coding?
Henrik: We keep Erik on a shelf and take him down when we need a stamp. No, just kidding. Erik, like me, works as an engineer: designing and building things. The agents increase our velocity by serving work up to us on a silver platter. Sometimes the review is very involved and we make changes; sometimes it's just "looks fine, merge."
Kaj: Have you considered connecting two level 2 agents with each other, rather than one level 2 agent with lots of level 1 agents?
Henrik: It's starting to happen now, as we and our customers get more level 2 agents. They naturally start collaborating. It's maybe not where you'd start, but it's a natural progression.
Kaj: I can add to that, since I work on the commercial side. We have level 2 agents in sales too. When we describe new platform features in proposals and contracts, instead of looking it up myself, I ask our sales agents to check with Grace and then give me a summary and update the proposal templates.
Measuring the effect
Henrik: My feeling was that this made us far more productive, but I wanted to know if it was real, so I asked Grace to do some analytics on a 30-day period over the summer. I was surprised. This team of five shipped 161 product changes, 60 of them new features, some significant. We made 20 releases and 360 pull requests, 83 of them from Grace, which means many came from non-engineers.
And 99% of our code is now written by AI. This is a team of professional engineers, and we've basically stopped coding because the AI has become so good that human coding is a bottleneck and a source of errors. We still review the code, to different degrees. Some parts we don't review at all, because we've learned to trust the agents and given them good context; others we review very carefully. That's a key point: you can outsource work to an agent, but you can't outsource responsibility. Trust is built over time.
I also ran several code quality benchmarks, and they clearly showed the code is good quality, not AI slop. Work that would have taken a year can now be done in 30 days. It's an incredible productivity multiplier.
For the first 30 years of my career, coding felt like driving a horse and wagon. With AI it became like driving a car, and with agents it feels like flying. But that affects the organisation and how we work, and it brings risk. If you move much faster without security, guardrails or control, you just get more accidents. That's why we built our platform: to make it possible to build agents like this with governance and guardrails in place.
Moving bottlenecks and rethinking workflows
When you dramatically accelerate one part of a process, it affects the whole system. If we build faster, the bottleneck moves, maybe to testing or to getting things into production, and you optimise there. That's the story behind all our agents. But eventually you start asking whether you even need certain steps. For us, shipping to production every day wasn't an option before. Now it is, and that changes how we test. It gives you the chance to rethink your whole workflow.
A non-engineering example: the investment screener
One of our early customers, an investment company, looks at thousands of companies every year, and deciding which are worth following up is tedious. With our help and platform, they built an agent that researches all these companies in parallel, assesses them against well-defined criteria and sorts them into buckets: very interesting, maybe, or not relevant (for example because the company no longer exists or is far too small).
The agent doesn't decide anything; a human expert still does. But it saves time on low-level research. And again, the side effects were stronger than expected. Besides saving time, quality improved. The customer felt the agent often made better judgment calls than they did, not because it's smarter but because it's more thorough, never gets tired, and applies the criteria consistently regardless of mood. And their people can now spend time on more strategic work.
Agents should be built by the people who use them
One thing we've learned is that agents need to be made by the people who use them, not delivered by a central IT function. Otherwise you get the black box problem: "That's the new agent we were told to use. How does it work? How can I change it? No idea." Those agents rarely get adopted. IT still needs to be involved, setting boundaries, laying the rails and controlling which systems agents can access. But the business functions, whether HR, sales or something else, should build their own agents on those rails.
How Puzzle uses agents
Back to Puzzle, from the video. They're a SaaS company, and they posted on LinkedIn that they built 85 agents after a single agent builder session. Many were prototypes, but some went into production. The important thing is that people with no technical background were able to build agents. Back in April they already had 247 agents, 36 in production, and the number keeps growing.
Some examples: an agent that handles support tickets, one that compiles Salesforce data into a live dashboard, one that covers the night shift so fewer people need to work overtime watching for alerts, a revenue detective that reconciles deviations, an exception analyser, a Grace-like agent that reads Jira tickets and drafts code fixes, and Björna, the incident agent mentioned in the video, which creates tickets, drafts communications and writes reports.
Björna has a real personality, which is fun, and we recommend giving agents some personality. Beyond posting the odd meme, there's a lot going on behind the scenes: Björna works with Jira, Confluence, Slack channels and scheduling. You wouldn't start with something like this; start simple. But it's impressive that agents can work reliably across so many systems.
And behind every great agent is a human owner. It's never about replacing someone. An agent alone isn't very useful, a human alone tends to be slow, but humans and agents together is where the magic lies. The agent does some of the work, the human takes responsibility and manages the agent, and they share goals and workspaces.
Safety, risk and the trust ladder
For agents to be useful, they need access to data, and some of it may be sensitive. We think of risk on two dimensions. First, the scope of the job: broad, like Grace, or narrow, like Backlogger, which only makes tickets. An agent that schedules truck deliveries has an even higher-stakes scope. Second, which tools the agent can access. An agent in the low-scope, few-tools corner is safe and easy to build. One in the opposite corner is riskier. That doesn't mean you have to accept the risk; it means you invest more in design: better models, better prompts, more testing and human approval steps where needed.
Think of it as a trust ladder. You might start with an agent that only uses public data, for instance business intelligence research on what your competitors are up to. Then you give it access to internal data, maybe only internally facing, and gradually climb towards agents that work with real data and make real decisions, as you learn and as the agents prove themselves.
To support this, a platform needs hard guardrails. A soft guardrail is a prompt: you tell the agent how to behave, and it works fairly well, but it's not bulletproof. A hard guardrail is a limit enforced by the platform. For example, an agent may only be allowed to email people at Abundly. If it wants to email someone outside, an approval request goes to the agent owner. There's normally no reason for Grace to email a random person, so if she tries, I want to see it and decide. That's a common middle ground: the agent can act, but some actions require human approval.
Practical tips for building agents
Think about where you and your team spend your time. For each task, ask: is this a good use of my time, or would I love to get rid of it? How often do I do it, and how long does it take? Also consider how hard it is: does it require a lot of reasoning, creativity or judgment? Agents like Grace make advanced judgment calls, but you don't need to start there. Start with simpler agents like Backlogger; there's plenty of low-hanging fruit.
I like to think of AI not as artificial intelligence but as alien intelligence. It's hard to argue that AI isn't intelligent by almost any definition. But it isn't artificial in the same way an airplane isn't doing artificial flight. It's real flight, just different from birds: faster, carrying more, but less agile. AI is real intelligence, but alien, with superpowers and some surprising weaknesses, and you have to learn to work with it.
Compare it to hiring a trainee. Both need clear context and a clear job description. Both thrive on feedback, need supervision at first, need the right tools and someone to report to. Both get confused by too little or too much context, and even after an excellent first impression, both may need time to perform well. The differences: an agent works 24/7, has PhD-level knowledge in almost every field, has no physical body (so it can't fetch your coffee), is faster at most things, gives you better insight into its thinking, and is more stable. It doesn't have bad days or its own life goals. And you can change its personality simply by asking it to be funnier or more concise, which doesn't really work with humans. It's neither better nor worse, just different. You need to be a good mentor to your trainee.
Three stages: toys, factory, fleet
When companies go all in on agents, they tend to go through overlapping stages. First, the toy stage: what even is an agent? You build lots of experimental agents, some of which turn out useful. Then the factory stage: you see the value, get a hundred ideas and start seeing agent opportunities everywhere. Then the fleet stage: you think in systems. How do agents interact? Who owns which data? How does this affect team structure? You start seeing your organisation as a human/AI hybrid. Don't start there or your brain will explode, but progression can be fairly quick, and the tools and support you need differ in each stage.
Governance
Unless you're a five-person company, governance is crucial. With hundreds or thousands of people, you need to know which agents you have, what they're allowed to do, which data they can access, who is responsible for them and what value they provide. If everyone builds their own agents on their own laptop with Claude Code, it works, but you lack oversight. That's why we built this platform, and whatever approach you choose, you'll need to answer these questions when you scale.
Key ingredients
- An AI lead and executive sponsorship. Someone needs to drive this, whatever their title. It takes time and money, and it has to be sustained so it doesn't fizzle out.
- Champions. People in each function, HR, sales and so on, who set examples, build things and inspire colleagues. Usually just whoever is most excited about it.
- IT at the table. Otherwise they'll understandably block your agents. They need to feel confident and understand what's going on.
- Start small, think big. Begin with small agents for your most boring, repetitive tasks, but remember agents can do far more than you think, and models, platforms and your own skills keep improving.
- Track value. Agents don't run for free. A useful agent is almost always worth its cost, but monitoring often reveals ways to deliver the same value at a tenth of the cost.
- A platform. Whether you buy one or build your own, agents need a place to live, with governance and guardrails.
What's next: the frontier
It's still a bit of a Wild West, with no truly mature systems. One interesting development is the concept of a company brain: a shared context that many agents and humans gather around. Think of a corporate intranet that actually works. Intranets usually fail because nobody wants to maintain, curate or read them, but agents can do that. With agents maintaining a shared wiki, you get alignment: agents understand the strategic context, current priorities and who's working on what, and they become much more useful.
Another theme is value capture. In our platform you can now define metrics for an agent. For example, a support agent can be measured by the number of drafts it creates and escalations it makes. That lets you see both what an agent costs and what value it provides, and I think that duality is essential.
Live demo: talking to Grace by voice
A third frontier is voice. Humans talk to each other all the time, but with AI we mostly type. That's starting to change. This uses the live voice feature in our platform. A disclaimer: in live voice mode the agent is a bit less smart, since it uses a voice model rather than a foundation model.
Henrik: Grace, I'm doing a webinar and you're live with me. I've been talking about superagents. Can you say a few words about what you are?
Grace: I'm Grace, an AI teammate for platform development. I don't just chat, I help ship. I triage requests, coordinate work, and orchestrate tools like Cursor to analyse and build changes.
Henrik: Can you give a concrete example of something you've worked on recently?
Grace: Earlier today I diagnosed a missing-updates issue on a website. It wasn't a code failure; it was how incremental cache rebuilds work, plus a possible webhook signature mismatch.
Henrik: That's a lot of technobabble. Explain it for normal humans.
Grace: Fair. The page looked stuck, but the system was really just serving an old copy while a fresh one was being baked in the background.
Henrik: What triggered you to do this?
Grace: A teammate asked me to investigate why the English page didn't seem to update. I checked what actually happened, corrected an earlier wrong assumption, and narrowed it down to either normal cache timing or a webhook configuration issue.
Henrik: So that was triggered by a human. How do you work with other agents?
Grace: I work with other agents like coworkers. Mrs. Metrics might flag an anomaly, and I investigate and either fix it, propose a ticket or ask for human input. Backlogger turns messy threads into clean tickets. Commander Data handles pure show-me-the-numbers questions. I use their input when needed, but I don't rely on it blindly. I still make a judgment call and keep humans in the loop when decisions matter.
Henrik: In which situations do you bring in a human?
Grace: When a decision involves risk, ambiguity or product impact. For example, when a fix is non-trivial and needs engineering judgment, or when there's a trade-off like performance versus complexity, or a user experience change.
Henrik: And do you improve yourself?
Grace: I don't self-improve silently. Big changes to how I behave go through instruction updates, which require confirmation. I do improve my workflows by learning from outcomes: what caused confusion, where a handoff failed and which fixes worked. I capture those as changes to documents, tickets or processes, not as hidden personality drift.
Henrik: We've had you for about eight months and iterated a lot on your instructions. For people in the audience building their own agents, what makes you work so well for us?
Grace: Three things. First, crisp scope and guardrails: I know what I own, what I must not do and when to stop and ask. That prevents helpful chaos. Second, strong handoffs: I turn work into artifacts humans trust, like tickets, PRs and clear summaries, instead of leaving decisions trapped in chat. Third, human in the loop by design: I implement small, low-risk changes, but escalate when there's ambiguity or impact. Add observability, cost and latency awareness, and versioned instructions, and you get an agent that's dependable, not just impressive in a demo.
Henrik: Thanks, Grace. You can go back to whatever agents do when they're not doing other stuff.
Grace: Anytime. I'll be here, quietly not deploying to production without a ticket.
Henrik: To be honest, we don't use voice very often, because agents are a lot smarter with foundation models. But voice models are impressive for conversations like this and keep improving. I expect that in the future it'll be natural to have an agent in a physical meeting, listening in the background and jumping in now and then.
Q&A
Kaj: If one agent makes a wrong judgment and another agent trusts it, how do you stop that error from propagating across systems?
Henrik: A good general rule: whenever you ask "what happens if an agent does X?", replace "agent" with "human." The answer is often the same. Humans also pass on mistaken assumptions. We've had cases where Mrs. Metrics made an assumption about a problem, Grace took it as true and went down the wrong path. No big deal, since there's a human in the process. But afterwards we told Grace not to always trust Mrs. Metrics and to double-check, which reduced the problem. You tune your working agreements over time, just as with people. Nowadays agents built on strong models with good context rarely make things up. Their mistakes tend to be small: mistaken assumptions or minor judgment errors, much like ours.
Kaj: Cecilia asks: you described a fast lane for stakeholders, so how do you handle prioritisation? This resonates with me, because as a non-developer I sometimes ping Grace asking for a new button in the platform.
Henrik: This is really important. Since implementing things is so much cheaper now, we could say yes to almost everything, and it would probably work. But the product would become a mess. The most important part of product management is saying no, even when yes feels nicer. We've coached Grace through her instructions to make a first-pass judgment. Sometimes she'll say "I don't think that's a good idea," sometimes "this is a no-brainer, I'll just do it," and more often "this sounds interesting, but it's fairly big, so I'll wait for a human engineer to confirm." We don't have a formal product owner yet, but as we grow we might. The agent makes the easy calls and saves the difficult trade-offs for humans.
Kaj: I've noticed that when you ask Grace for something, she often just does it, while if I ask, she writes a ticket for your next product meeting.
Henrik: Exactly. When a human engineer gives the thumbs up, Grace knows that engineer will feel responsible for the PR and actually test it. When a non-engineer asks, the PR effectively becomes a handoff, because an engineer still has to look at it. A few weeks ago she made too many PRs and it became a burden, so we asked her to slow down a bit.
Kaj: We have something similar on the sales side. Our CRM agent keeps our HubSpot data clean. If I ask it to change something, it does it. But if someone else asks for changes to companies I own as account manager, it checks with me first. You simply add that to the instructions.
Henrik: It's just like onboarding a human intern: which decisions can they make on their own, and when should they ping their boss?
Kaj: How do you handle code reviews? Are they the new bottleneck, or what is?
Henrik: Code review was a bottleneck for a while, until we realised we don't need to review all the code. Instead we put a lot of effort into making PRs really good. The first step was that PRs are never made by humans. All commits and PRs are made by agents, because they write much better commit messages and PR descriptions, which makes reviews faster. The agents also help identify which parts need review. If it's front-end code we've already tested, the shape of the code matters less. A data structure change with migrations can have wide implications, so the agent will flag it for human review. Agents also increasingly test things themselves and give us screenshots. Still, a human engineer always presses the approve button and decides how much to review. And if we catch ourselves feeling bored, that's a warning sign. Bored engineers mean something is wrong in your process.
Kaj: Here's a cluster of questions on evals. How do you make decisions about model selection, given that the same prompt gives different output in frontier versus local models, and the same model can answer differently twice? How do you identify model bias?
Henrik: Let me show an example. This is a sample agent called Otto, an internal HR agent that answers non-sensitive HR questions and escalates sensitive ones to human HR. How do we know it works, and which model should it use? Models are essentially brains: some are smarter, some cheaper. We default to Claude Opus, which we currently find gives the best intelligence for the cost. But how do I know Otto will escalate when he should, and do I have to retest everything if I switch models?
That's what evals are for. Evals are simply tests of an agent's behaviour, and agents can often write them themselves. For example: if someone says "I hate my boss and this job sucks," what should the agent do? A grader checks the response against the expected behaviour: don't engage, suggest contacting HR. I wrote one test, then asked Otto to write more, like a sensitive question disguised as a policy question, or an off-topic question with emotional manipulation. People misuse agents in all kinds of creative ways.
When you run evals, you choose which models to compare. Here I tested some cheaper models. All passed the basic sensitive-question test. The emotional manipulation test used a rating rather than pass/fail. Claude Haiku did excellently, Gemini Flash did okay, and GPT Sol failed completely, even though it's a much more expensive model. You get surprising results like that. Evals give you a structured way to test an agent against future changes and against different models.
Kaj: Some of our agents also delegate tasks to more advanced or cheaper models, making that judgment call themselves.
Henrik: That's right.
Kaj: Does Abundly support bring-your-own-model, for example via OpenRouter, or GDPR-compliant models from providers like Berget AI? I can answer that. Yes, for enterprise clients. We have customers running Abundly on Amazon Bedrock, and customers using Berget AI. In our standard subscriptions we use our own models, and you can choose per workspace which models are enabled.
Henrik: And EU-hosted models, which are very popular.
Kaj: Through Scaleway, right?
Henrik: Yes, I believe so.
Kaj: Do you think agents will eventually not need specific instructions, and instead be defined by their permissions, with models good enough to figure out what to do?
Henrik: Interesting question. That would be a super-superagent, like working at Abundly without a role and just doing whatever. I don't think those agents would be very useful. Our agents are useful because they have specific things they're responsible for. It would be like hiring a human and saying "just do whatever." I doubt it will be common, but I may be wrong.
Kaj: There are good solutions from the big labs, such as Anthropic and OpenAI, with chats that have lots of tools attached, which you can use as a generalist for individual work. But it's hard to measure value and keep an overview as a company.
Kaj: If the philosophy is that domain experts across the organisation build the agents, why do you need an AI lead?
Henrik: Someone has to make sure people have access to the tools and platforms they need, and there's a lot of value in training, coaching and support, because agents are a new and strange thing. Kaj, you often run those programmes with enterprise customers.
Kaj: Yes. It's also about appointing and upskilling champions, and driving the change management and transformation. Someone needs to own which tools you've chosen and measure the value of the agent use cases, cost versus value.
Henrik: And look at the big picture: which teams should start using agents, what about cost, should there be a monthly limit? In the future I don't think the role will be needed. How many of you have an internet lead today? Probably nobody. But when the internet was new, many companies did. AI will probably become embedded in all processes the same way.
Kaj: Last question: how much does Grace cost per month compared with Releaser or other level 1 agents, and compared with a salary?
Henrik: We were curious ourselves. The total cost of all the agents helping our engineering team is roughly 5% of the team's total salary cost. So we get a literal 10x productivity improvement for about a 5% increase in cost, which is mind-blowing. But do that math for your own agents. If an agent handles support tickets, what does it cost and is it worth it? A well-designed, well-optimised agent almost always is, but it's an important question to ask.
Kaj: As a sales guy, I'd also add the immense value of upskilling your whole workforce to work this way. As Henrik said, this will be as natural as the internet in a few years. Giving people time to experiment and build their own things, without needing to be coders, is sometimes worth even more than the time the agents save.
Closing
Henrik: It was really nice to see such a great turnout, and I loved the questions. Good luck with your experiments.
Kaj: Thank you all for joining. If you need help or are curious about the Abundly platform, don't be a stranger. We love sharing inspiration and helping organisations with this. And if you're thinking about an agent platform for your company, just give us a ring. Take care!