I got an email today. Here’s what it said (emails and names omitted to protect the innocent).
Offering sliding pricing for nonprofits and schools is a thoughtful way to make fractional CIO work reachable.
Fractional CIOs tell us education and nonprofit clients decide slowly, and knowing when to check in without pushing is the hard part. Curious how you approach that.
We are building a tool for small consulting teams that catches exactly that moment.
It keeps track of who owes the team a reply and nudges the right person when a follow-up is due.
We are pre-launch, and we’d rather build around real workflows than guesses.
Would you be open to answering 5 quick questions? All tick-box, 45 seconds tops.
If yes, just reply “sure” and I will send the form over, plus a priority pass for 30% off for life at launch, if the tool turns out to be a fit for you.
To be clear, when I saw this I immediately read it as “oh look - another AI generated annoying email.” The tells are all over the text. Every AI model has a writing style and it seems that it’s awfully hard for those models to shake their styles loose. An initial relatable appeal. Vague claims. “This without that” or “This rather than that” patterns that belie a programmed fear of making mistakes. Even words like “tick-box.” In the US we call these checkboxes. But the team that wrote this LLM isn’t in the US, and the accent comes through. This message is thick with it. It’s clearly AI generated spam.
For some reason, though, this one has been stuck in my head all morning (and into the afternoon). It’s problematic on so many levels that I need to write it all down as a thought dump. So it shall be a blog.
The AI Loop
Here we have a company (or an individual vibe coding a tool - who knows?) using AI to scrape my website and leverage what it learns to tailor an email directly to me. A fine and certainly not uncommon use of AI - but there’s likely no human reading this message before it goes out. It’s a tool building and sending a message.
So what happens if I, on the receiving end, employ an AI tool to respond to every email like this that I get? Have we then created a dynamic where two AI agents are bouncing back and forth with each other, answering each other’s questions? Clarifying a scope of work? Does my AI agent cut a deal with their AI agent? Do they negotiate pricing with each other? Do they send the pricing and whatever agreement they come up with to an AI lawyer to hammer out the legal questions? Do they then sign the agreements and get to work with each other? At what point do I find out? Does it matter if I find out? Does the economy begin to turn on machines cutting deals with other machines? It gives me a chuckle. But also the creeps.
We all get these emails every day. I totally get it. There are a zillion little companies out there. Lots of individuals and partners striking out on their own armed with a laptop and a Claude subscription. With a well written prompt they can fire off 10,000 emails in minutes - each message designed for the specific recipient based on what the agent thinks it knows (and what it hallucinated or inferred through its training). People need to make money to live, and in support of that aim there’s now a cacophony of email that we people need to wade through to discern what’s actually a genuine message and what’s spam. Or I suppose we could just ask our own AI tools to review and sort all our emails for us.
The Governance Issue
First we get past the absurdity of the image laid out above - Artificially Intelligent underlings doing our bidding with each other while we humans hang on the beach or in our backyard or on our boat. (I don’t have a boat, and probably should learn to swim before I get one, but I digress.)
Then we consider what this all means - or at least suggests - when we talk about AI governance. When I first think governance, I think about ethics. And law. And public policy. And the greater good. And reining in technology so that it’s aligned with ethical, legal, and moral principles and audited by humans who can interpret behaviors in the context of the best interest of humans writ large. But plenty of folks don’t care much about pesky ethics or legality.
What I haven’t thought about until now is the automating of interactions between, or among large numbers of, agents who are empowered to make commitments on our behalf to which we humans are obliged. In the example above, say my agent responds to this email agreeing to answer the questions but only in exchange for entirely free access to this budding new product forever. Then their agent determines that it’s an acceptable deal and agrees. Then my agent answers their questionnaire. And their agent hands me free access for life - far more than the human behind this note ever contemplated.
That’s a governance issue that crosses boundaries of the ethical, legal, and moral - and steps right into potentially making a mess of our business. I suspect that as these interactions start playing out, the folks who aren’t driven by the ethical or legal will be driven by their own bottom line. And for that reason alone, clear governance around your agentic tools will become mandatory. May we all discover this before any real damage is done.
A Broken Premise
Aside from the fact that, at its core, this message is describing Customer (or Constituent) Relationship Management software - which has existed for decades and is in no way new - there’s a greater premise issue. Nowhere in that theoretical exchange between two systems has the premise of the original question been challenged. To be clear: it’s way off base. Knowing when to check in with my clients is in no way difficult. I have a relationship with my clients. A personal one. It’s why it’s so nice being a small firm. I know the names of their spouses, what’s worrying them, and whether they take lunch at noon or one or never. It’s easy for me to glance into their environment from the outside and provide human perspective to help them navigate hard challenges. It’s core to the value I bring. But systems can’t yet do that. They need rules that say “if the following conditions are true then it’s time to check in.”
The only conditions that matter (to me) are centered on ongoing knowledge of the people, their projects, and their feelings. I’ve yet to find an Agent that can nail the feelings part. And when I do, it’s time to retire my shingle and loose the boat from its mooring.