Superagents: putting AI colleagues to work across your organization

[00:00:00] We have files also on files, yes.

Does this integrate with any platform that we use with Can integrate with any platform. Yeah, it's just a matter of configuration. You want to talk to an API, give it an endpoint and an API key. The platform has ways of managing secrets in a safe way. Um, so yeah, API integration becomes surprisingly simple.

The hard part is getting approval from your, you know, IT or whatever. So that, that's the part. Yeah. First suggestion to the company, great. So you work with these companies-

current things because I didn't increase the pushback you tried to do To not get the spread of we build those things because we build all those things. Yeah. So yeah, the question was about how do I avoid agents that just do stuff, I guess. And humans. You, you-- I, I think this is a great idea of actually testing that early in the product.

Yeah. You know, one of the things you learn when you start working with this weird intern is they come with a pre-baked personality a little bit, [00:01:00] depending a bit on which model you pick. And the pre-baked personality is, "I wanna help." Right? I remember the first time I gave an agent access to email, send email to people.

This was about-- this was two years ago, so they, they're better since then. But the agent was like looking for excuses to send emails. It's like, "I can send emails. I wanna send... Here, I'll email you about this, or I'll send an email about this. I'll send you one every day." It just loves sending emails. So same thing, if you create an agent, you give it access to, to coding, uh, system backlog, it's by default gonna be very enthusiastic.

It wanna help you. But that's easy to fix. You just tell it what you want. So it's all about giving feedback, what do you want it to do, and then testing-

Um, what is the value that Eric, the human, brings in a PR review, knowing that, um- Wait, wait, wait, wait. Start again. Yeah. Uh, what[00:02:00]

is the value that Eric, the human, your colleague- Yeah ... brings in doing a PR pull request review, knowing how complex pull requests are made by in the past? Right. So what is the value of, of the human Eric reviewing a pull request? I think that's a really, really interesting question. And as a backdrop to that, I'm just gonna show this picture.

Um, I mean, the whole, like, why are we humans are still around question is interesting. Yeah. Um, I'll, I'll use this as backdrop. It might be a little surprising, but we find that all the humans are still needed. Just what we spend time on is a little different. So in the past, like, I, um, this is the picture I wanted to talk about.

I was-- me and my whole team were very surprised when this data came up, that AI is writing all our code. We didn't really realize that because we're all engineers. We like-- we thought we liked to write code. Uh, but then when I asked people to self-assess, like, out, out of this EPR, I found a bunch of changes.

How much did you actually look at? And it turns out that about thirty percent was carefully reviewed. We looked at every, every line. Forty-five percent was just skimmed. Like, "Ah, it looks okay. I don't see anything that looks really ugly or messy." And twenty-four [00:03:00] percent was, "I'm not even gonna look at it.

I'm gonna trust it." So the value of the human here is making that judgment call, because no AI can take responsibility for your code base, at least not yet. Uh, it's still humans that have ownership. So you need to decide, do we care about the code base or not? Sometimes it's okay to not care about it. If you're building a small standalone system of some sort, you might not care about the code.

It works. I've tested it. I don't give a damn. I don't even know what language it's written in. That's fine. But for a platform, and in our case, a business-critical platform, we care a lot about the code. Uh, however, we've learned where the AI is really good at doing stuff and where we need a bit of oversight, just like learning to work with a colleague.

So, so, so the value is really in, in our confidence that there's not dumb stuff ending up in the code. Maybe Eric looked at that and just scrolled through it and said, "You know what? This is probably fine. Whatever." He-- and he might even be wrong. There might be a mistake in there somewhere, but it is a judgment call, and we want that to be a, a human-

Do you find the stuff that you described today works just as well for like [00:04:00] brownfield projects as well as green? And are there specific things there that you need beforehand if you have a big monster that you're trying to tackle with then introducing agents? Sorry, I caught about half of that. Uh, yeah.

I'm wondering if there are any specific tips when it comes to working with brownfield projects where you might have some quality there, some- Mm. Existing products ... rubbish as well. Yeah, existing products as well. Right. So brownfield, I like that term. So yeah, work with existing products rather than new ones.

Um-

In general, I would say it works, but you need to start maybe carefully. So let's say you have a mega huge code base, and you have a massive mess over here, right? You're gonna need to build up a context for the agents to understand what's in there, bui-build a map. So I'd probably start with that. Use AI to build a map of your system, and then use AI to build parts of it, and then you're gonna have a lot of human oversight at the beginning.

Maybe in the beginning, the only thing AI is doing is fixing minor bugs, little things. And then over time, as you build up a better and better context, [00:05:00] then your agents will understand the code base better, then you can give it bigger jobs. But in general, I get very surprised at how quickly an agent can understand a messy code base because they are trained to read code, and they are better at it than humans.

They're faster at it than humans. So within seconds, it can read a whole thing and go, "Yep, I get it." And the nice thing is it'll not only be able to work with it but also be able to help clean it up. But yeah, you'll need to do it with, with careful because, yeah, in the beginning, if messy code base, high stakes, there will need to be more human oversight Yeah.

Um, can you elaborate a little bit on that thinking of tasks that are automatic? It could either be my personal workflow-

Yeah ... like shaping public code or something, or what-- or it could be on an organizational level, like operating that way the workflow is to see

then[00:06:00]

your work is super agents. Yeah. So where, where do you find that distinction between the- The- ... level one agent help for me and externalizing it to an organizational level to super agent? Really interesting question. I don't have a good, clear answer, but I have some observations. Mm-hmm. That, um, we find agents, we find are most-- the most useful ones are the ones that work for the team, not just one person.

But we do have some agents that work with just one person. Example, Kai here. You, uh, you wanna say something about your agent or?

Uh, yeah. So, so, uh, I work with sales, and there's a lot of stuff to do in the CRM, all the admin that is not about meeting people, but actually logging meetings, following up, writing proposals, all of that has to be done. So just like in the dev team, we first had a releaser and later Grace, we started building an agent that just works towards HubSpot, just works with the CRM, just logs meetings, and that's it.

Creates deals. All of that is now handled by, by that agent. But on top of that, we now have agents that talk to that agent that can help us reaching out to new clients, that can set our whole sales strategy, that develop new, new agreements, all of those things. So you kind of start building with one agent that only helps me log meetings.[00:07:00]

Then I tell all of my colleagues that als-also work with sales and also those that don't work in sales, "Hey, whenever you have a meeting, give this agent a ping on Slack." And then from that, it gets so much context and has so much info from a CRM that I can then sit together with another team and set a wholesale strategy based on real data.

So you kind of develop the agent capabilities and then escalate it higher up in the organization. Yeah. And a clarification on that. Th-those are not just your agents. Yeah. 'Cause we are several people in the company that rely on that. And so they have become not-- Or

if you compare the survive and thrive picture- Yeah ... these are for sure the thriving part because

we are several people that interact with and get support for- Yeah ... and pushed by those, those agents. A-and most processes in an organization are not just one person doing one thing. Most processes cut across a team or across several teams. So if you find those workflows that work across several teams, then you will notice, like, okay, someone owns this agent and is responsible for this process, but the interaction of-- in this process happens across the organization.

So I actually have a kind of tattle tale thing in my HubSpot agent. So whenever someone else, [00:08:00] because I'm a bit of a control freak, whenever someone else does big changes to the CRM data, I get a notification about it. So I can just always be in charge, but someone else can access it. I, I, yeah, I can even-- I have a personal agent called Moneypenny, which helps me a little bit with my email.

So it sends me a, a morning kind of Mr. President report, just here's a high level, here's what you should care about most. These are just archived. These can be later. And, uh, the next step for that would probably be to say, if it finds an important client inquiry, wh-why, why wait for me? Why don't just post on Slack, you know?

So he can see- So agents that start personal quite quickly will notice that, oh, wait a second, I can make this more useful

What, what are the main practical differences between the level one and level two agents at HipChat? So practical differences between level one and two, there is no formal difference. It's just a word we made up. Uh, it's just more a feeling. But I would say the main distinguishing factor is that a level one agent has fairly defined task steps.

You do this thing, right? While, uh, level two agent doesn't, it has more a [00:09:00] high-level work areas. For example, Grace, there is no list of this is exactly how you should handle PR. You're an engineer on the team, be a good engineer. Here are our tools. So if someone asks her to, to make a PR, she'll kind of be able to make a judgment call.

So it's a little different when you're prompting or instructing a level one agent, it's quite a lot of focus on workflows, like these are the steps. You often don't need to make it very detailed 'cause they're quite smart on their own, but just, you know, here, here are the steps. While for level two agents, it's less about workflow steps and more about giving a broad context and giving feedback.

They're also fundamentally less predictable, which is by design. Um, so it's, it's a, it's a trade-off. I accept that Grace is a, is a little bit less predictable than the Backlogger, but it also is what makes her more useful. Are you using the same underlying en-en-engine to, to power them up? Same exact engine.

It's just what, it's just what the instructions are- Mm. -and what the tools are. So Grace has more wide, wide instructions and a wide range of tools. Backlogger is a little more kind of, uh, um, boxed in to what it can do. And we [00:10:00] might remove the Backlogger. I think Backlogger is redundant now because of Grace.

So over time, maybe level two agents will eat level one agents. But there are also some situations where Mrs. Metrics does a great job. Why should Grace do that job? It's better if Grace talks to Mrs. Metrics. So this is the kind of stuff you think about when you start moving towards a fleet model. It's how do we, you know, personal agents versus team agents.

Should this agent talk to that one directly? Should it talk on Slack and ping me or ping another agent? Should it wait for approval? Should it-- And that's just part of the evolution of your fleet. Yes. Um, I wonder what does Grace cost a month or for example, or Bjoern and or Mrs. Prince, what is the, the cost approximately like exactly?

So let's guess. Um, we have a credit system which is loosely connected to, uh, dollar depending a little bit on your account. But if we were to guess wildly, let's say, let's say Grace uses about twenty k credits per month-- per day, [00:11:00] roughly. Um-

Thousand, six thousand seconds among

So what we find in general is, um, when an agent does a really good job, then the cost is usually, you know, very low compared with if you, if you, if you do it manually. Um, but it's something you do need to look at, right? 'Cause sometimes-- Well, typically, what we find is when you make an agent that does the job, even a simple job like backorder, right?

They can sometimes waste a lot of credits. They do things inefficiently. They're doing all these API calls. It's lost data. So they, they do the job, it costs a lot, but then we have these skills in our platform to optimize. We tell the agent, "Hey, can you make yourself cheaper?" And we have all these tricks.

So it's gonna go, "Oh, I can write a trick for this instead. I can do this. I can optimize that. I can cache this," and then boom, it's, it's ten times cheaper. So the, the way we think about it is make your agent useful first. [00:12:00] Don't even think about cost. Make, make an agent people love to use, then think about cost.

And we've so far, I don't think ever had a situation where someone was like, "Oh, this agent is really useful, but it's too expensive and we can't use it." It's more like, "This agent is really useful. It's really expensive. Let's, let's, let's optimize it." But yeah, we're really, really efficient. When it comes to this, I did that math a little bit and concluded that the total cost for engineers equates to roughly about maybe two to five percent of their salary.

So we're, we're-- essentially our engineers are maybe five percent more expensive than otherwise would have been, and we get a ten X return on improvement. So the math is like really simple

You say something on how you work with, uh, like data access and then compliance. So say you want to build a financial agent and this access only financial data, but what everyone in the company should have access to- Yeah ... financial

data. But anyone can call the agent. So how do you-- how do you- The question is about how we handle data access, right? Yeah. [00:13:00] Um, yeah, really important kind of crucial question. There's, there's different levels of ways of thinking about it. First is what data do we even wanna give the agent access to, right?

And how do we do that? There's gonna be an API of some sort. How can we scope that API to the data the agent needs? Quite often we find that APIs are a little bit too crude. We can't give it the scope we really, really need. So you, you either give it a too wide scope or, or too narrow scope. So often there needs to be some little bit of layers between.

Usually, sometimes, uh, a, a customer will build their own in-between layer, uh, and sometimes they'll just in our platform have an agent write a script that accesses an intermediary. So that way, uh, this API lets us do all these things with data, but the agent only needs these things. So therefore, we can write a script that kind of essentially becomes like a, a tool wrapper.

And the third thing is sometimes moving data. So we have like, we have a, a, a, an agent called, um, Data-- Dr. Data, what was it called again? I forgot. Command Data. Command Data. We use it all the time to ask how people use our platform, but we don't want that [00:14:00] agent to have access to sensitive customer information.

So we have, uh, like a, a process where that data gets moved and kind of anonymized and just so that, that, that, that pile of data is safe by nature because we just take the data it needs, and then we give the agent full access to that data. But this question is one of the fundamental kind of design questions.

When you go beyond the simple agents that do non-sensitive things and you wanna make agents that work with sensitive stuff, then this is the exact question you kind of need to, need, need to, need to deal with. But the nice thing is all the tools are there, so it's really just a matter of deciding. If, if I may add to that, there's also on the kind of system level in the platform, you need access controls.

So per default, anyone who can access an agent would be able to access data the agent can access. So in the platform, you can create teams and sub-teams and give access levels to those. You can store secrets on a agent level, team level, or workspace level. So if you have an agent that's in the finance team that only finance colleagues should be able to access data from, then put it in a team where only finance colleagues can access the agents, and don't connect it to [00:15:00] other agents in other teams that then could, like, query it.

So it's stuff like that. And then you need a platform where that actually is built into the, the toolkit. So here's an example just in our own internal workspace. These are a bunch of secrets that are managed b-by the platform. And for each secret, you can control which people have access to it, which agents have access to it, and to what level.

And that's a really important part of it.

How are we with time, by the way? Are we gonna get kicked out of here or what's the deal on my, like- I think we're out. What? We're out. We're out. Okay. We'll probably stop here. So, but I hope you enjoyed this. I'm really glad you came.