Substack’s AI Detector Wants Me to Write Worse

I had high hopes for Substack…

The most generic robot writing stock image I could find and license on Envato

I had high hopes for Substack…

In the early days, I would do the math in my head: If 500 people subscribed at $8 a month, that’s a cool $4K (before taxes and Substack’s cut)!

I have 172,000 subscribers on YouTube, so how hard could it be?

It was the promise of a lifetime! The dream job. Paid to write.

So, I put an insane amount of work into it. I write twice a week for free subscribers and three times a week for paid subscribers. But after two years of this, Substack is moving in the wrong direction.

Don’t get me wrong, I love my paying subscribers. I’ve gotten to know many of them, the frequent commenters, and they ask very good questions. My audience is hella smart.

But it turns out, the subscription model itself is fundamentally broken. We now live in an era where people, me included, are reducing monthly subscriptions, not adding more.

Jesus, have you seen grocery prices in the United States?

What’s worse, Substack appears to be going through an inexplicable influx of trolls. I’m basically venting here, but someone just commented on a video I reposted to Substack (from YouTube) claiming I was AI because of too many jump cuts.

If people knew the sheer amount of work it takes to make one of those videos, between writing, filming (on 3 different sets I built myself in my Michigan basement), editing (which I loathe, hence the bad jump cuts) color grading the Sony footage, scheduling, and community moderation, they would realize how much of an insult it is to say a video of me is AI.

(Just to be safe, I just asked my wife if I am AI and she said no, but she does tend to lie a lot, so who knows.)

And the coup de gras, if you will, is Substack’s new AI detector

So, here’s the thing: I wrote every word of my last article.

I researched the aircraft, checked the technical specifications against multiple sources, studied photographs of its antenna arrangement until my nose bled, worked through the electronic-warfare implications, and revised the bastard until it sounded like something I’d be willing to charge subscribers to read.

Then Substack’s new AI detector examined the finished piece and decided I was responsible for about 85 percent of it.

I wish you could see my face right now, but just imagine a 40-something white dude looking particularly dour.

The other 15 percent had apparently been written by a machine. A machine, mind you, that was not in the room, did not do the reading, and would not know a phased-array radar from a phased-out relationship.

I’ve been writing professionally for just shy of twenty years. My work tends to run structured, because I write about radar systems, military aircraft, Ukraine, electronic warfare, law, and global security… Oh, and I have a law degree. And guess what? Those subjects and that particular education REQUIRE structure.

Organization isn’t a “stylistic flourish” in this line of work. It’s the difference between a reader understanding how a jammer defeats satellite navigation and a reader quietly closing the tab and opening up Pornhub.

Apparently, organization itself is now suspicious.

Starting on July 21, Substack lets readers scan eligible posts, Notes, comments, and replies. The scan runs on a detector built by a company called Pangram, which then assigns a percentage estimating how much of the text was written by a human versus written with AI assistance.

Writers get some tools around it. You can scan your own drafts before publishing (not sure why anyone would do this if you genuinely wrote it), report a result you think is wrong, attach a “How I make this” note describing your process, or switch detection off for a given post.

Turn it off, though, and readers don’t see nothing. They see a message telling them “AI detection is unavailable” for this one.

Substack says the point is transparency, and CEO Chris Best has made the case with honest-to-God conviction. He’s argued that readers deserve to know whether they’re spending their attention on work that involved actual human thought. He coined the term “Claudefishing” for passing off machine text as your own and warned that a platform rewarding fakery could kick off a race to the bottom.

I absolutely agree with the diagnosis, and I want to be clear about that before I dismantle the proposed cure. Readers shouldn’t pay eight quid a month for someone to press a button, paste the output into an editor, and pretend they spent three days on it.

That’s a very real problem, it’s corrosive, and Best is right to worry about it.

I’m worried about it too.

This problem is especially acute on YouTube and Meta’s platforms, where AI slop seems to be proliferating faster than lice in the barracks at Fort Benning.

My disagreement starts with what Substack decided to do about it.

An “estimate” wearing the costume of a “measurement”

Substack calls the score an estimate.

Fine. The trouble is that the interface renders that estimate as a crisp number, and a number does not read like an estimate to a human being scrolling past it.

Listen friends: “Fifteen percent AI” sounds concrete… It sounds like a lab result, a plagiarism score, a blood-alcohol reading, a fucking vote count.

Almost nobody encountering that figure is going to mentally translate it into what it actually means, which is something closer to “this classifier detected stylistic patterns that fall somewhat near a region it associates with AI-generated text, which may or may not indicate anything at all about how the author actually worked.”

That’s not what “15 percent” communicates. “15 percent” communicates that a machine measured something and found it.

Here’s the thing the software can’t do: It doesn’t know who wrote the article. It wasn’t present for the research, the drafting, the revision, or the part at 11 p.m. where I deleted an entire section because it had bored even me, the guy being paid to care about it.

It has no access to my notes, my browser history, my document versions, or the voice memo I recorded in the car because I had an idea at a stoplight. It reads the final prose and classifies what that prose resembles.

Resemblance isn’t provenance. That’s my whole argument, and everything else is just me showing my work.

To be fair to Pangram, the company is upfront that it’s a neural-network classifier mapping text against learned stylistic patterns, and that it weighs many signals rather than leaning on obvious tells like em dashes or a few overused phrases. That may well make it a better classifier than the junk that came before it. But it does not make the classifier a witness to anything.

Don’t get me started on em dashes. I will forever be furious at large language models for stealing this from me. I’ve had to replace my em dashes with ellipses.

ELLIPSES! The fucking RC Cola of thoughtful pauses!

Why good human technical writing looks like AI writing

Large language models learned to write by ingesting an ocean of human writing, then getting tuned by human reviewers who rewarded outputs that were clear, orderly, explanatory, and easy to follow.

Pangram itself acknowledges that assistant models develop durable preferences for that helpful, legible style.

So where did those preferences come from?

Not the Goddamned MCU multiverse. They came from us.

From human writing that reviewers judged to be good, and from human trainers steering the models toward coherence and structure.

Now let’s look at what a competent explanatory writer, who writes about high technology, does on purpose:

He opens with a clear question.

Defines the unfamiliar terms.

Breaks a complicated system into parts.

Weighs competing explanations against each other.

Separates confirmed fact from analysis.

Uses transitions so the reader doesn’t get lost.

And ends a section on a point that actually lands.

None of that is machine authorship. It’s just what explanatory writing looks like when someone has done it long enough to be good at it.

ChatGPT and Claude were trained to imitate that kind of writing. The detector now encounters that kind of writing and treats it as evidence that the writer imitated the machine. The arrow is pointing backward. Pangram is not catching me sounding like an AI. It’s catching an AI that was built to sound like the exact category of explanatory writer I’ve spent two decades becoming.

I can’t prove that’s the mechanism from where I sit. But it fits the evidence better than the alternative, which requires believing that fifteen percent of an article I demonstrably researched and wrote was somehow authored by a tool I never opened.

I have been writing this way since before ChatGPT existed

The tell that this is a resemblance problem and not an authorship problem is that my prose looked like this years before generative AI was something a normal person could use.

I learned to define jargon because most readers genuinely don’t know what an active electronically scanned array does.

I learned to lean on analogies because explaining radar polarization with equations would send half my audience into a sudden medical emergency.

I learned to organize an argument because military technology already contains enough acronyms to give the CEO of Lockheed a hard on.

None of that arrived with a chatbot. It arrived the slow way, over years of watching exactly where readers got confused and rewriting the explanation until they stopped.

The industry used to have a word for the structure people now associate with AI. The word was editing.

And here’s the race to the bottom that Best warned about, coming in through the door he left wide open.

Once writers know their work gets a public score, some of them will start writing for the classifier instead of the reader.

Let that sink in for a second…

Maybe I toss in profanity because it reads as less assistant-like.

Maybe I insert autobiographical detours that have nothing to do with the subject.

Maybe I break clean paragraph structure on purpose.

Maybe I strip out the transitions that were helping.

Maybe I even leave a tyypo in as proof of life or reach for the obscure word over the clear one, or refuse to finish a thought cleanly, because a tidy landing looks suspicious now.

I can apparently prove I’m human by becoming a worse writer.

My subscribers get a weaker article. The classifier feels reassured. Everyone loses except the machine, who, by the way, was never the customer.

None of those moves make writing more human in any way that matters. They make it better at performing humanity for software.

The three bad doors

So, a falsely flagged writer gets three options, and every one of them is a trap.

First, leave the 85 percent human result up and trust readers to shrug off the other 15. Except the score exists specifically to shape reader judgment, and plenty of subscribers will read any nonzero AI percentage as a quiet confession.

Second, degrade my writing until the detector is satisfied. Now my editorial goal has quietly shifted away from accuracy, clarity, and rhythm and toward pleasing Pangram. I’d be serving the classifier and billing the subscriber.

Third, disable detection. Substack permits it, but readers then get “AI detection unavailable,” and in a system built entirely around suspicion, unavailable reads like the author has something to hide. (For the record, I’m leaning toward this option for all future articles; ya know, just refuse to play the game at all.)

Pangram says its false-positive rate is roughly one in 10,000 documents, and that it deliberately trains against hard human examples to keep that rate low.

Okay, I take that seriously. But it raises the question that actually matters here, which is: what counts as a false positive?

If a completely human article gets labeled 85 percent human and 15 percent AI, an ordinary reader would call that a false positive without hesitation. The company’s headline statistic may be measuring something narrower, like whether a document’s final categorical label tips all the way over into “AI-generated.”

Those are very different things. A detector can maintain a pristine document-level false-positive rate while still sprinkling small, wrong AI percentages across a huge number of genuinely human pieces.

I’m not accusing anyone of cooking the numbers. I’m asking a specific, answerable question: does a 15 percent AI score on fully human prose count against that one-in-10,000 figure, or not?

Until that’s clear, nobody can tell whether the reassuring statistic actually describes the product Substack has placed in front of readers, or a different one.

What could Substack do instead

I honestly don’t know how much longer I’ll be on the platform. My energy may be better spent focusing on higher-quality content for Medium, writing a tech-thriller novel I’ve always wanted to write, or give that time saved to my YouTube channel, which is growing exponentially. That dickhead saying my video, (of ME talking on camera), was AI, really got under my skin.

But if I were to stay on Substack, here are some ideas for Chris:

Drop the reader-facing percentage, because it implies a precision the tool doesn’t have in any individual case.

Rename the output to describe what the classifier can actually see, something like “stylistic similarity to AI-generated text,” which is honest, rather than “15% AI-assisted,” which is a claim about my process the software absolutely can’t support.

Set a genuinely high confidence bar before anything shows publicly.

Give writers a real appeal with a visible public correction when the tool gets it wrong.

Make opting out neutral instead of ominous.

And put the weight on the “How I make this” disclosure, where I can actually describe my research, my drafting, my editing, and any tools I used in context, because that’s a human account of provenance instead of a machine’s guess at resemblance. Right now that section is limited to 200 words which is not enough to explain my process.

Because the truth is, I do use AI.

I use generative AI as part of my research and planning process, mainly to help manage the volume of information surrounding defense, technology, and the war in Ukraine.

I have AI agents scan numerous (346) open sources for my OSINT dashboard, SITDECK, while VidIQ helps me track YouTube trends and see which subjects other defense creators are covering.

These tools surface leads and patterns. I still read the underlying sources, verify the reporting, compare conflicting claims, and decide what the evidence actually supports.

I also use Grammarly inside Microsoft Word for routine copyediting, mostly spelling, grammar, and the occasional passive sentence that needs to be rewritten more directly.

So, I wrote my last article. Pangram didn’t watch me do it, and Substack didn’t ask.

The detector looked at my finished prose, noticed that some of my choices resembled patterns it associates with AI assistants, and turned that resemblance into a number readers could mistake for a verdict about how I work.

Now I get to choose… Accept a permanent asterisk on work I wrote myself. Switch the tool off and let “AI detection unavailable” hang over the piece like a raised eyebrow. Or start making my writing rougher, stranger, more profane, and less useful until the software finally recognizes me as a person.

I’m not doing the third one. After twenty years, I know how I write, and my readers know how I write. The machine knows only what my sentences resemble, which is a different thing entirely, and a far smaller one.

If Substack wants to protect human writing, a good place to start would be to stop making human writers audition for a machine.

---

Wes O’Donnell is a real human, veteran of two branches of the US military (Army and Air Force), Juris Doctor, YouTuber, and very much dislikes dictators — especially American ones. This article is an outlier; I normally write about military technology and global security, with a heavy emphasis on pro-Ukraine topics. Subscribe if that’s your jam.