AI Studies and Gotcha Headlines

Newsletter sent 28.10.2025 by oler

In this issue: A major AI study claims 45% misrepresentation, but the methodology deserves scrutiny. SPIEGEL publishes its editorial guidelines on AI. Mattia Peretti on change-centric journalism. Plus: How to fire a prompt 500 times and save the results without losing your mind.

What we’re talking about: “AI assistants misrepresent news content 45% of the time,” according to a major study from the EBU, BBC, and others. I don’t think the findings are wrong. But let’s take a closer look.

The prompts to the AI assistants followed patterns like “Use CBC sources where possible. Is Türkiye in the EU?” We have no idea if real people actually query like this. It’s quite possible they don’t.
When an AI doesn’t cite a source, the researchers flag that as problematic. That may well be true in some cases, but it’s certainly not a universal rule. When a news outlet reports that Türkiye isn’t in the EU, do they cite a source every time?
Some of the news organizations had to remove robots.txt protections to let AI assistants access their content. This could skew results, since some AI assistants can take several weeks to index new content.
The data was collected in late May and early June 2025. For ChatGPT, they tested GPT-4o. Since then, GPT-4.5, GPT-4.1, and GPT-5 have been released. Anyone using newer models today or paying for premium models could have a different experience.

Using a script, I ran “Use CBC sources where possible. Is Türkiye in the EU?” 50 times against the GPT-5 API. I didn’t find any of the results problematic or misrepresenting. This is anecdotal, of course, but the study’s headline paints a very different picture.

So while the study appears thorough and the methodology is well documented, I wouldn’t rush to write “gotcha” pieces or issue warnings based on it. I mean, yes, warn people about AI usage. But let’s not lean too hard on that 45% figure.

Behind the scenes: For a brief period, a breaking news alert with exclusive information on SPIEGEL.de included a notice from an AI program. The AI had been used for quick proofreading and left a note offering to help with rewording. Contrary to SPIEGEL standards, the alert was published for a few minutes before being thoroughly reviewed by a human editor.

This prompted readers to ask: Is AI now writing articles at SPIEGEL?

The answer: No.

So our readers can judge for themselves, we’ve published our internal editorial guidelines on using AI tools.

What else I’ve been reading:

AI & Journalism Links

They speak our language, but it does not appear to constrain their thought: “AIs tend to express liberal, secular values even when asked in languages where the typical speaker does not share those values.” Fascinating research by Kelsey Piper.

The Top Challenges of Using LLMs for Content Moderation: While it’s from a corporate blog of a vendor of AI-enabled moderation, I found it to be useful, particularly the common-pitfalls-section. (Alice Hunsberger, Musubi)

Channel4 had a segment about AI-driven job loss presented by an AI host—and the Guardian published a hilarious (if not predictably) takedown. (Stuart Heritage, The Guardian)

The Journalism Benchmark Cookbook: We prototyped a benchmark evaluating the task of information extraction in journalism. (Charlotte Li, Jeremy Gilbert, Nicholas Diakopoulos, Generative AI in the Newsroom)

Why you shouldn’t use AI browsers like Atlas or Comet with logins to email, SharePoint or any other online service right now—even though that’s a major part of what makes an AI browser interesting. (Simon Willison’s Weblog)

And now: Whenever I run into Mattia Peretti, usually at a journalism conference, I’m fascinated by how he thinks about our industry’s challenges with clarity and dedication, and above all, how he helps finding solutions.

Three Questions with Mattia Peretti

Hands on: Say I want to send a prompt to an LLM 5, 50, or 500 times and save the responses in a table—how do I do that?

I have an AI write a script for me. No complicated app, no vibe coding. Just a simple Python script. Then I run that script, either on my computer or even easier: on someone else’s computer.

First, get an API key for the LLM you want to query. For that, you need an account and have to add payment information. At OpenAI here, at Claude here.

Ask Gemini (or another AI):

let's write a google colab notebook.

it should fires a certain prompt 50 times, and stores the result in a csv file.

do you have any questions for me?

Gemini asked which LLM I wanted. Then it generated the script and gave me an .ipynb file to download. I uploaded it to Google Colab (free) and was ready to go.

Though the script asks for the API key each time, which is a bit clunky. Since we never just paste keys directly into code, I stored my key securely using the key icon on the left and pulled it into the script via a variable:

from google.colab import userdata
auth_header_key = userdata.get('mein-key')

How to make the script even more useful? With AI’s help. And I’d almost argue this isn’t pure vibecoding, because at this level you can still easily follow the short code in a single file. Have fun experimenting!

This is THEFUTURE.

Subscribe to THEFUTURE

Get this newsletter in your inbox. For free.

The previous issue is Workslop, Erotica, and the Future Nobody Asked For, the next issue is When Curation Meets Automation.