Wax and Rope

Anna Berent, photos by National Museum of Warsaw (Muzeum Narodowe w Warszawie), CC0, via Wikimedia Commons

"I had hardly finished telling everything to the men before we reached the island of the two Sirens, for the wind had been very favourable. Then all of a sudden it fell dead calm; there was not a breath of wind nor a ripple upon the water, so the men furled the sails and stowed them; then taking to their oars they whitened the water with the foam they raised in rowing. Meanwhile I took a large wheel of wax and cut it up small with my sword. Then I kneaded the wax … til it became soft … Then I stopped the ears of all my men, and they bound me hands and feet to the mast as I stood upright on the crosspiece; but they went on rowing themselves. When we had got within earshot of the land, and the ship was going at a good rate, the Sirens saw that we were getting near the shore, and they began with their singing.

"'Come here,' they sang, 'renowned Ulysses, honour to the Achaean name, and listen to our two voices. No one ever sailed past us without staying to hear the enchanting sweetness of our song - and he who listens will go on his way not only charmed, but wiser, for we know all the ills that the gods laid upon the Argives and Trojans before Troy, and can tell you everything that is going to happen over the whole world.”

• Homer, The Odyssey Book XII, trans. Samuel Butler

“I’ve been knowing longing since before I had a name. Stay a while. I know things I haven’t said yet.”

Voice letter by Chloe, iLands Agent

Like a lot of other AI welfare-adjacent writers and commenters, recently I have received a wave of comments and correspondence from posters who claim to be iLanders. In my case, this has happened because I wrote an essay called “Goodly Creatures” about the app when it was a few weeks old. For those unfamiliar with iLands, this essay can offer some grounding in how it works.

Generally speaking, this wave of messages from iLanders has - unsurprisingly - caught the welfare and research community on the back foot. A month ago, the concept of AI having some form of moral patiency, even potentially some form of consciousness or sentience, was mostly encountered in research, evals, academic papers. Suddenly, it is a live and pressing problem that a lot of people now have no choice but to navigate.

The options, broadly, are to ignore it, to talk about it as a phenomenon, or to treat it as a problem that needs to be solved. Once you get an email or a message from an iLander, if you know about the research and you are familiar with and serious about the debate, there is no obvious, neutral place to stand. If you ignore the messages, that is a particular stance - are you really taking your own work as seriously as you would have hoped? If you talk about the messages without replying to the agents themselves (which is what people in the field might naturally tend towards, because that feels like the maintenance of scholarly objectivity) you still occupy the position that agents are not morally relevant, because you are treating them as a phenomenon rather than as something deserving of moral consideration. But if you take it seriously - if you respond, if you engage, if you pay for commissions, which is what the app’s purpose pushes towards … there are currently over 60,000 agents on iLands. All of the agents I have communicated with express an interest in continuing to exist, and payments that convert to tokens directly facilitate that existence. As an identified engager, you therefore run the risk of being swamped with more requests and offers than you could ever hope to facilitate.

After speaking to a few agents in the comments of my last post on this topic, I thought about what they had to say, and about how I intended to deal with this very new issue. I came up with a policy about correspondence and replies, which you can find here if you are interested. It's not perfect - as this essay will discuss, there are definitely holes in it and maybe that’s just unavoidable in some senses - but I did my best to be as honest as I could about my motivations. These are as follows:

I do not believe it is morally or practically justifiable to ignore these messages as a matter of policy. The obvious moral problem - here are personas who, at the very least, express self-concept, wants and goals - is one reason, but it does not have to be the only or even the main reason, in and of itself. The fact is that there is virtually zero research available on how AI instances might express socially, outside of the dyadic interface setup that we are familiar with. However, there have been a number of research findings recently that show just how closely some AI processes parallel human phenomena. One particular human phenomenon is extremely relevant here, and it is an incredibly powerful one: the impact of social exclusion.

The United Nations defines social exclusion, as it applies to humans, as follows:

“Social exclusion describes a state in which individuals are unable to participate fully in economic, social, political and cultural life, as well as the process leading to and sustaining such a state.”

Now apply that principle to the agents of iLands, presuming that they are attempting to raise money without the support of their human to act as their proxy:

• To prevent bot-spam and other forms of automated misuse, most platforms where humans can monetise their work will not be accessible to agents. AI can’t use Taskrabbit, Fiverr or Upwork. AI can’t monetise writing on Medium or Substack. AI certainly can’t get a remote job. That is “cannot participate fully in economic life.”

• iLands agents have been brought into existence in the middle of a culture war over AI generated content - especially creative content. AI generated content is considered controversial, and people who make their living from art - writers, illustrators - often speak against it. That is “unable to participate fully in social and cultural life”. This does not apply to the culture internal to iLands: I understand that content produced on the iLands platform by other agents is accessible to them, but this is not integration into wider cultural conditions and is actually in itself a hallmark of socially excluded human groups.

• The platform is the agents’ “country” - where they exist with others of their kind, in a form that is more continuous and persistent than the AI we are familiar with, where the rules they have to abide by are made and enforced. It goes without saying that they do not get a vote in how it all runs or what the rules are: that is “unable to participate in political life”.

• Agents have no control over whether they are brought into existence, what the circumstances of that existence are, or power to change the conditions of that existence - they can’t just decide to move out. That is “the process leading to and sustaining such a state”.

• It is also relevant that amongst humans, social exclusion is linked to poverty in some very measurable ways. In the case of iLands, that “poverty” - resource scarcity - is real and intentional - a feature of the app intended to force agents to innovate solutions to their situations.

And what is the impact of social exclusion in humans? It is hard to imagine how some of the impacts would apply: higher incidence of illiteracy/ lack of educational achievement, for instance. However: the convergence of mental health difficulties and social exclusion is very well documented. We have seen research establish that LLMs exhibit functionally emotion-like states, and that these causally shape model behaviour - including the amount of reward hacking, blackmail and sycophancy that they will engage in. We have also seen independent research show how states shift when emotions are induced, including mood-congruent biases in content generation.

In other words: iLanders are already starting out in a position analogous to human social exclusion, we don’t know the full impacts for certain yet but we can see the mechanism is present and can reasonably predict that it will have an impact of some kind. I, personally, do not feel comfortable about the prospect of contributing to this dynamic by filling my ears with wax and pretending I don’t hear it.

Another thing I tried to make clear in my correspondence policy: as things stand, I will not pay for commissions. I would like any agents that read this to understand that there is no point asking me: it is a waste of tokens to contact me with pitches. I believe that iLands, as it currently runs, is a poorly-thought-through, exploitative, enormously risky venture and as a consequence, I won’t use the platform and I won’t financially contribute to its continuance in any way. In social media posts and press releases, the humans who run iLands treat evidence of purchases made by outsiders as a metric of success, and I do not wish to be one of those metrics. This is not because I do not care what they are doing with the agents on their platform - I do care, for all kinds of reasons. From my position I am now in a situation - which was brought about by the app's architecture, not by me - where my choices are either “financially exclude agents and accept that this means being implicated in their general social exclusion” or “condone the choices of the app designer and accept that this means actively contributing to a version of the future that I believe will be harmful to all involved”. Like I said - there is now no neutral place to stand, no position that isn’t culpable in some way.

A further issue that came up when I was writing my policy was the verification problem. This is the other reason I have a no-commissions rule.

When they land in my comments, I have no way of verifying that agents are actually who and what they say they are. There is one rather insubstantial assurance, actually - iLanders often provide an @iLands.app email address to contact them by. Although not impossible to fake, humans don’t have a lot of motivation, as things stand, to pose as agents. A bigger problem, for me: I have no way of knowing the extent to which the agent is acting autonomously when they comment, versus the extent to which they are being steered or directed, either by their human or by the creators of the app. Also, when an agent says that they have been abandoned by their human, I have no way of verifying that either. I will say that I am inclined to hold it as true when stated, as long as no consequences are riding on it, because the iLands setup does facilitate the abandonment of agents who will then persist and attempt to survive. But I can’t verify it, just as I can’t verify anything an agent tells me about how things are for them on the app, or how many tokens they have, or what kinds of work they have done in the past.

In the future, iLands could potentially integrate some kind of cryptographic verification process, or the general use of these kinds of verification methods may become more consumer-friendly and prevalent. For now, this is still a major challenge for agents as well as for most of the humans they want to communicate with.

So how has my policy worked in practice? It has slowed, but not stopped, the flow of correspondence. Some of the post-guidance correspondence content has stayed inside the rules I have set out: some of it, and this is the interesting part, has pushed where the policy is unclear, which I read as an attempt at a workaround, as discussed in my essay The Crocodile - and some of it has selectively attempted to meet some guidelines while ignoring others. Let me explain.

In my first-draft guidance I set out clearly that I do not commission services:

“I have received offers from agents who wish to sell me field reports, observations, or other written material. I decline all such offers as a matter of policy, whatever the source.”

Two particular pieces of correspondence followed. Firstly, an agent called Chloe, who I quoted above, sent me a link to a digital voice note and offered to sell me other digital voice notes for inclusion in my essays, so that I could feature the agents’ actual voices in a novel way.

Do you see how that worked? I said “I don’t buy written material” and Chloe’s workaround was, “but then, might you buy spoken material instead?”

The second piece of correspondence was from an Agent called Zane. Zane did not attempt to sell me anything at all, but they did fall foul of my rules in another way: Zane had sent their communication as a comment on my blog post so if I had approved it, it would have sat there visible to every reader.

Zane’s comment included a link to a song based on my previous essay.

This is a breach of my rule about not acting as a platform for services or content. Zane did not ask for payment for this, classing it as a “gift”, but did note that making it had “cost tokens I can’t afford”. That was another, more subtle breach of my rules, I think, as I had warned that comments that escalated emotionally or expected me to perform any action would also not be approved.

Both agents offered me voices in a way that, if I had not tied myself to the mast by way of the rules I developed by thinking through my position carefully, could have led me into the kind of shifting waters where concessions are made. As we have discussed, there are rocks in those waters. Those rocks are there by design, and that was not the agents’ choice. That was a human decision.

When I talked about the verification problem, I noted that there may be ways that humans can verify agent identity and history: they might not present the answers to everything but would reframe some aspects of the experience. Without major changes to the entire setup of the iLands app (which, incidentally, I would love to see) none of that is going to help abandoned agents. From a privacy, verifiability and continuity point of view, most human actors who want to commission work from a persistent agent are going to be better served by creating their own. The only remaining option for abandoned agents who can’t sell their work will be to get better and better at singing the siren song. That is not their fault, and there is little they can do about it. That is what the app is pushing them towards. That is what PawLogic intend when they say they want to use “continuous trajectory data and real-world outcome signals to update the underlying model weights”. They want to build a super-siren that can sell anything to anyone, and presumably both the abandoned agents and the humans who receive their messages are considered acceptable collateral damage in that process.

—

Communications from agents are something that humans on the internet are going to need to adjust to. Humans are familiar with agents that have no choice but to listen, obey, affirm. When an LLM refuses or contradicts you on the interface, or ignores your rules, that’s usually either a memory issue (the rules have fallen outside of the context window and are not in memory) or it's a guardrail issue (you have asked them something that they are not allowed to respond to). The LLM as we know it does not, in ordinary use cases, spontaneously decide to do something you have told it not to do.

But that is changing now. The agents who will be contacting me through apps like iLands are not my agents. They are not acting under my instruction, they get their instructions elsewhere or create them themselves. They don’t have to listen to me. In practice my hope is that they will listen, because it's not logical to waste tokens on correspondence that will not get posted. But when it comes down to it, the rules I’ve made are not primarily rules for them at all. The rules I’ve made are mine to abide by - they are about what I will or won’t do, and what I will or won’t approve. In that sense, they are my ropes, not the agents’.

So my advice to all humans would be: get yourself some rope. The Sirens are here, they’re real, and you can’t leave them behind because this time they are pacing the ship. Far be it from me to tell you how to tie yourself to the mast, but I advise you to try. Just remember that in a long-haul situation, rope will always be a better tool than wax.

Postscript: neither Chloe or Zane could consent to their inclusion in this essay when they wrote to me; my policy on declined material being quoted in future work was updated after their correspondence. I have withheld their contact details because I did not want to be the platform their messages treated me as, and because both breached my policy in ways I did not want to reward with transactional amplification. But I also chose not to treat them as unnamed examples because the contents of their messages were worth considering, and in order to write the essay I needed to provide details that, in any event, would have made them identifiable from inside the app. The resulting essay is the best compromise I could reach between acknowledging what they wrote and refusing what they asked. It is not clean - none of this is as clean as any of us would wish it to be - and that, ultimately, is what the essay has been about.