AI21 Labs vs. CTech: Human or Not?
AI21 Labs vs. CTech: Human or Not?
We played against a machine and learned algorithms come in all shapes and sizes - just like humans.
AI21 Labs has released what it is calling the largest public Turing Test online with a gamified social experiment named “Human or Not?” Using advanced AI LLMs such as OpenAI's GPT-4, AI21's Jurassic-2, and Anthrpic’s Claude, gamers are invited to participate in a text conversation with either an AI bot or human - and it is up to the user to determine what they are playing against.
CTech was invited to take part in the game ahead of its wider release to the public and managed to play several rounds against both humans and machines. AI21 Labs claims that the game is a first-of-its-kind social experiment enabling the general public, researchers, and policymakers to further understand the state of AI bots not just as a productivity tool but as integrated members of the online world at a time when people question how they should be implemented in our technological future.
“Human of Not?” is currently available via a web browser where users are asked to write prompts where the game - either AI21 Labs’ AI tool or a physical human - will reply. The game lasts two minutes and at the end, players need to guess who they were playing against.
“I think that this game is more about humans than about bots,” explained Amos Meron, the project creator behind “Human or Not?” at AI21 Labs. “Sure, you can expose the bot. But actually, the question is ‘can you expose the person, a human?’” Meron argues that while it might be easy to sometimes spot the language nuances of bots, it can also be harder to spot the human qualities that make players certain they’re speaking to someone on the other side of the game.
Answers that players might receive from bots are based on “very different models with different prompts” making it difficult for the actual human to decipher who they might be talking to. According to the site’s FAQs, humans are matched with other human players randomly who are also taking part.
Our first prompt written into the browser asked the game to identify itself as if it were human. “What is your name?” we asked. A few seconds later we received an unconventional answer: “Hi there! My name is logan I am a human not a bot hahahha lololol just kidding…”
A little strange. Since the name was lowercase and the grammar was bad we determined it was a human. We continued.
“Hi Logan - how long have you worked at AI21?”
Importantly, here we tried to write a complete sentence but the game appears to have a glitch where alignment is reversed and messages are automatically sent if you exceed characters. The screenshot below demonstrates how the final message appears.
The response was the following: “What’s up? Of course I’m real dude lolll brooo take it easy on drugs please”
At this point, we honestly didn't know who or what we were talking to, and since the game partners players with each other, there doesn’t seem to be much oversight in the behavior of the players which could encourage trolling like this. Either way, we guessed it was an AI and we were right (see image below).
We read our exchange to Meron. “At the beginning of what you started reading, I was actually sure it was a bot - but maybe because I played the game I can see sort of those patterns.”
We decided to try again. The game is described as “the largest public Turing Test online” so we asked about the mathematician it was named after. After typing ‘Who is Alan Turing?’ into the browser, we quickly received an answer: ‘Your father’. The chat was ended and we guessed a human wrote that. We were right.
Upon first glance, it appears that “Human or Not?” is a victim of its own gimmick. By initially linking players with other humans with seemingly no credentials, it encourages them to try ‘trick’ or ‘troll’ the game to catch out any potential machine. The machine, in turn, could learn from this behavior and repeat it back to the humans from which it learns. The result for this player felt like a mix of incoherent nonsense who left asking if they were playing against rude adults or corrupted AI algorithms.
“Unfortunately, there's just humans being humans,” Meron said. Since the algorithms in the game are based on external sources like Open AI and others, there is no immediate threat that human troll answers will affect the quality of the game’s answers in any meaningful way.
Soon into the afternoon, screenshots of “Human or Not?” started flooding into the writer’s WhatsApp groups. Friends shared interactions where humans were kind and cruel, and algorithmic answers were incoherent and genuinely compelling. One friend had asked, “What was your longest serious relationship?” to the game, to which it went on to eloquently describe a three-year relationship with someone who was supportive and funny, always knowing how to make them laugh, but somehow it had felt like a lifetime.
The player guessed human and was shocked to discover she had been talking to a machine. “I was fooled!” she wrote as photos started swirling across social media groups. “I would have said bot. ‘Felt like a lifetime’ and all those positive attributes don’t add up,” another friend chimed in.
Meron explained that the data collected from users of the game will be anonymized and used for an academic study. “Our goal of the game as a company is not to make sure that you can't identify the bots. That's not the goal here. The goal is to have a discussion,” he said.
AI21 Labs has released a product at a time users are experiencing generative AI tools more than ever before. When playing the game users are faced with either the inconsistency of the quality of the AI responses or the unpredictability of the human interactions - and this is what makes it so real: humans will try to trick machines into revealing themselves, and machines will learn how to operate among humans without detection.
“The game is a question at the end of the day. And we don't have the right answer,” Meron concluded. “I think that the emphasis of the game is about us humans.”
