Artificial intelligence programs that analyze and produce text are transforming how we read and learn. To parse writing, AI models sleuth through textual clues, such as word choices, to see their connections. But what happens when those clues are deliberately vague and confusing? I tried to answer this question when I challenged AI developers to solve the nearly century-old Cain’s Jawbone, a murder-mystery puzzle book from 1934.
The book arrived in my life as mysteriously as a literary sleuth could wish. One October afternoon in 2022, a random package from Amazon was dropped on my doorstep with no accompanying note or return address. I had never heard of the book inside, but a Google search told me that Cain’s Jawbone is both a murder mystery and a brain-teasing puzzle. The book was purposely published with all its pages out of order; to crack the case, the reader must first reorder the pages, and then name the six murderers and their victims.
The writer of this fiendish plot was (surprise surprise) a puzzle expert. Edward Mathers worked as a crossword compiler for The Observer newspaper under the pseudonym Torquemada. He published Cain’s Jawbone at the height of the so-called golden age of detective fiction, but only two people managed to solve it before the book went out of print. In 2019, John Mitchinson, the co-founder of publishing platform Unbound Publishing, came across a copy of the story and its solution at a literary museum in the U.K. Mitchinson decided to reprint the 100-page puzzle. “I said, ‘Well, this is amazing. It’s a detective novel, so how difficult could it be to put in order?’” he recalls.
The answer, it turns out, is “very, very difficult.” In the past few years, only four more people have solved the puzzle. Then the book went viral thanks to a couple of TikTokers who tried to reorder the pages using a colorful “murder wall.” Its new popularity spurred Mitchinson to print additional copies on top of the initial 5,000-copy run.
When my copy of Cain’s Jawbone appeared, instead of designating wall space for the pages, my husband and I spread them out on our guest bed. As we pored over the flowery and deliberately vague language one dimly lit evening, I suggested using an AI algorithm to solve the novel.
Because I’m not a software expert, I started looking for an AI company willing to tackle this puzzle. But most AIs are not trained specifically to reorder book pages, or to analyze the linguistic quirks of 1930s English. Finally, I connected with Zindi, an Africa-based company that hosts AI competitions in which 50,000 data scientists use algorithms to solve puzzles and win prizes. Zindi was interested in hosting the competition, and with Unbound’s blessing, I created the 2022 Cain’s Jawbone Murder Mystery Competition; we digitized the 90-year-old book and challenged the world to use natural language processing (NLP) algorithms to reorder the pages.
NLP algorithms, such as the famous ChatGPT, try to understand the information within a text by comparing its context and language to the training data it receives. Such algorithms can analyze never-before-seen text by transforming each word into a “token” and then analyzing how each token fits into the complete work. This helps AI algorithms to analyze texts, whether literature or scientific reports, quickly and effectively. I nobly resisted using AI to crack the case of who sent me this intriguing book, instead texting friends and posting on Instagram to uncover the culprit.
For our competition, participants started with an existing NLP model called BERT, developed by Google and available in an open-source library, where it can be modified for specific uses. “These models are … trained on just gobs of the data that the model creators can get their hands on and then are refined to follow a certain set of instructions,” says Jonathan May, a research associate professor of computer science at the University of Southern California. In order to refine their models for this particular use, we gave participants Agatha Christie’s first mystery novel, The Mysterious Affair at Styles, to use as training data, because that story was written during the same time period as Cain’s Jawbone and contains similar language, as well as demonstrating the context clues of a classic mystery.
AI has had a long history with writing novels, including murder mysteries. In 1973, computer scientist Sheldon Klein proposed the Automatic Novel Writer, which he claimed could produce 2,100-word murder mystery stories in less than 20 seconds. Since then, programmers and engineers have improved the output of these models using more data. “In a way, a murder mystery is easy,” says Mike Sharples, an emeritus professor of educational technology at the Institute of Educational Technology at the Open University, England. “There is a standard plot structure to it: find the body, the sleuth comes, you’ve got a red herring, and so on.” This plot structure is not only helpful to authors dashing off a quick story but could also help AI language programs trying to put the mixed-up pages of those stories back into the right order—in theory.
Unfortunately, Cain’s Jawbone creates the ultimate challenge for language-analyzing algorithms: the story is not only completely out of order, but also designed to stymie readers. For instance, the language is highly stylized—Mitchinson describes it as “a postmodernist poem”— and deliberately vague, in order to make ordering the pages as difficult as possible. Plus, the story abounds in false clues, such as fake names for some characters and misleading names for others, all of which might confuse AI models as well as human solvers. As a result, none of the AI developers managed to crack the puzzle—although some of them made a little headway.
M.G. Ferreira, an econometrician from South Africa, was one of the AI competition winners, with the highest score of 42 percent. That means his program correctly ordered 42 out of the 100 pages. “NLP does have some comprehension to it, like knowing that thunder and rain go together,” Ferreira says. “But the problem here is that the book is trying to throw you off with false clues. It breaks NLP comprehension.” In order to solve the puzzle, he explains, the AI needs a human to step in, look at the context and identify which ideas go together. “Going in that direction, eventually we will be able to solve the whole thing. But by that time the NLP will be such a small part and the human overlay will be such a big part that I’d call it machine-assisted,” he adds.
The murder mystery competition revealed that current AI language programs may be capable of impressive feats, but they won’t be going toe to toe with Poirot any time soon. These models are bad at analyzing things without context, which could cause issues for researchers who hope to use NLPs to analyze ancient languages. Because there are few historical records on some long-gone civilizations, the lack of context makes it difficult for AI to learn how to translate their lost languages.
At least this experience helped me solve one puzzle: I tracked down the person who sent me the book and set me off on this quest to solve it. The culprit turned out to be one of my elementary school friends, a person who doesn’t have social media but does have a penchant for murder mysteries—just like me.
This is an opinion and analysis article, and the views expressed by the author or authors are not necessarily those of Scientific American.