Why Pangram will win

Brittle Tools

About a year ago, I wrote and submitted a paper evaluating the performance degradation of AI detectors under various prompting strategies. It got rejected, probably because it wasn’t a very surprising result; the related work section was filled with papers finding mediocre performance of AI detectors and the results concurred with a prevailing view that the detectors were, at best, brittle tools that failed on non-typical results.

Recently, that seems to have changed with the rise of a new generation of model-based detectors like Pangram. The detectors have gotten measurably better in non-adversarial setups¹ as the paradigm shifted from statistics/ensemble based detection to trained classifier models. Epoch AI recently found that Pangram was capable of accurately categorising all 297 AI generated and 495 human written essays of similar lengths and on similar topics.

More impressively the detectors have become very robust to adversarial prompting methods including ‘humanizers’ like StealthGPT which aim to generate AI generated text indistinguishable to classifiers. The CEO of Pangram recently explained in an interview with Hard Fork how they are able to do this.

Largely what we do is, we train against it [humanizers] and so we’re always picking up new humanizers and training against them for our new model

So at the highest level, they generate outputs from the humanizers, label them as AI in a dataset and train their own models to distinguish StealthGPT outputs from human written text. Later in the interview, he elaborates on the specifics of how he thinks Pangram can detect outputs from the frontier labs.

A lot of what we’re detecting is a lot more subtle than what you might pick up as the Claude writing style so I think it might get better at trying to imitate a voice and doing well at it but it would still have signals that Pangram picks up on and I think kind of what we see is that these frontier labs are really focused on climbing capabilities … they’re applying preferences to these models. They’re saying instead of this model predicting like average next token prediction of any writer anywhere this model prefers to write correct code and correct math and I think these preferences are largely what Pangram is able to pick up on

What he’s saying here is that writing in the most human way is not necessarily writing the best way or even the most useful way to humans. Afterall, the line of code that a human is likely to write might not be the best possible line of code. For a long time, AI generated outputs were getting less distinguishable from human text, as we improved them by training language models to imitate human written data. As the labs increasingly shift towards building systems through post-training RL (on verifiable rewards and/or human feedback) rather than supervised pre-training (imitative learning on human-created data), the models will reach superhuman output quality as they appear to be doing in some domains like cyber security.

Importantly, quality and humanness aren’t orthogonal properties that can be both be true at the same time, they directly contradict. As long as we assume that human outputs are not the best possible outputs then a maximally intelligent model cannot also be maximally human sounding. Making better outputs will require that those outputs be less human. Therefore, I don’t think the frontier labs will focus much, if at all, on making human-sounding outputs in the future, the frontier models outputs will just be too good to be mistaken as human.

Illustrative diagram

That said, the long term enemy for Pangram might come from non-frontier labs because there are still other reasons why being able to generate human sounding text would be useful. It’s possible that a true superintelligence wouldn’t be interesting to talk to for humans and we will want conversational models more on ‘on our level’. It’s also possible that humans will prefer conversations with systems they perceive to be human specifically, not just around our general level of intelligence. Platforms could also impose restrictions on AI-generated content in a way that content farms or enthusiasts would want to circumvent.

These are example components of a distinct market segment with a shared incentive for models trained specifically to produce human-sounding text, like a foundation model StealthGPT². In turn, if deceiving humans that a text is human written is considered illegal in the future³ or if content platforms want to enforce no-AI policies, there could be a market for AI detection as well.

While the ASI will march onwards, away from human-sounding writing, there could still be an imitation game going on between generator models that try to sound like humans and discriminator models that try to distinguish human written and AI generated outputs (text or otherwise).

Distributed GANs

Architecture Diagram of SeqGAN

The problem for the detector companies is actually the same as the problem for the generators, the other guys can train on your outputs to improve their models. If you know a platform is going to use a detector to scan for AI generated content due to their human-only policy then you need to train your generator on that detection to produce human-sounding content. In other words: the outputs of the detector are a reward signal for the generator and vice versa.

This hypothetical future is actually a well researched generative AI framework known as Generative Adversarial Networks (GANs). There’s essentially a competition between a classifier and a generator to produce outputs indistinguishable from the existing data; traditionally the point of a GAN setup is to make a robust imitative generator but here these are distinct organizations working against each other with different goals. The basic principle is the same, two models (or sets of models from different organisations) compete and train on each others outputs to improve.

In fact both sides will likely be forced to participate in this arrangement; Pangram has just demonstrated that using this kind of setup reliably detects AI outputs and researchers from UCL have separately demonstrated that their AuthorMist paraphraser (similar to StealthGPT) can trained with RL to produce undetectable outputs as well⁴. Choosing not to train on your opponent’s outputs is going to leave your tool useless for your respective customers.

AuthorMist system architecture from this recent paper.

This might seem like an equal matchup but research on GANs on their equilibriums tells us who is really the favourite: detectors have a big advantage that stems from asymmetric data flow. The discriminator gets to use a rich high dimensional piece of data and map it down to a single binary classification⁵ while the generator has to learn how to produce the many numbers that represent good synthetic data from a single 1/0 input. So internally the discriminator has the easier job but that’s really just the beginning.

For text (as well as many other data modalities) this problem is compounded by the stochastic nature of generation. This means that the language model reads the previous text, predicts the next word in a long chain. Since only the entire sequence is graded with a binary yes or no in reinforcement learning, there’s a fundamental problem of credit assignment throughout the stochastic trace. If the generator writes two paragraphs but gets caught as AI writing, it doesn’t know which token triggered the detection or how each token influenced it.

When trained against each other, the discriminators can use (faster) pretraining instead of RL since there’s no random sampling in detection. This allows them to react faster and not have to worry about other common issues in RL like reward sparsity. In fact these issues go so far that it’s common practice with textual GANs to handicap discriminators or give generators extra information. You need to level the playing field to produce productive learning dynamics. One survey explains:

RL-based approaches for training GANs on discrete sequences perform gradient policy update via REINFORCE (Williams, 1992) to bypass the generator differentiation problem. Nevertheless, RL training presents its own challenges difficult to deal with, such as the large action space, reward sparsity, the credit assignment problem and large variance for gradient estimation (Maddison et al., 2016), (Zhang et al., 2017b). Indeed, RL algorithms applied to dynamic environments with sparse reward are very unstable and the credit assignment problem through discrete computation makes it difficult to pass gradient information to the generator (Che et al., 2017).

It’s also possible to make the discriminator much smaller than the generator (levelling the playing field with fewer resources) but in our setting, given comparable computational resources, we should expect the detectors to be able to correctly classify the vast majority of generator outputs.

Counterpoints and Complications

In spite of the strong advantage held by the detectors in this case, there are some complicating factors.

Firstly, AI architectures could evolve to generate text in different ways and even in existing research there are different detection strategies that might prove to outperform current methods. For example, everything I’ve covered in this essay has been a zero-shot detection meaning that the model is trained on many other pieces of text but has to tell without any context whether a given piece of text is human or AI generated. This might be an unfair assumption in practice; while people might not carry around counterexample human essays, AI detectors could evolve into agents that pull or even generate relevant human-written and AI-written examples to feed into an multi-shot detector. This kind of strategy has previously been demonstrated to outperform zero-shot methods even when both the defender and attacker have access to the same context⁶ and therefore it could tip the balance further to the defenders favour.

Secondly, there exists a broad set of tools to make AI generation weirder. False negatives (AI generation that slips under the radar) are likely to continue being much more acceptable than false positives (false accusations). This means if there is a narrowly distinguishable typical AI writing style versus a typical human writing style, an AI generator with high enough temperature that outputs very unusual text might still be classified as a human, just because the cost of a false positive is much greater than the cost of a false negative. Producing unusual outputs might reduce output quality in some way but this is still a potential mechanism to overwhelm the censors trying to detect output .

Finally, price and implementation cost might also be a relevant practical factor here. Typically the discriminator models in GANs and these detector setups are much smaller than the generator and are therefore much cheaper to run. This currently doesn’t seem to be true however as Pangram seems to have comparable detection costs to high end generation costs⁷. If detection continues to be broadly more expensive than generation then it might make AI detection economically infeasible for many applications⁸.

While these are real possibilities, the scope of their impact is limited to edge cases that can’t cancel out what is generally true: the classification task is fundamentally simpler. Classifiers have more input data to make a decision on but with a much more narrowly defined output. A generator has a 1000 chances to predict a token that gives itself away, the detector has much more data to learn from on an RL signal than the generator which will have to figure out by itself where in the long output it went wrong. There is a reason why machine learning started almost exclusively with classification tasks.

We might remember these early days of AI as the weird time where there was genuine ambiguity around what was generated and what wasn’t before the detector technology caught up. On the flip side, the ambiguity we see now is kind of incidental; the frontier labs aren’t trying to fool you, beating a target that isn’t fighting back is not particularly difficult. In fact right now some expert humans are as good at AI detection as Pangram is!

Unfortunately, this can’t last forever, as detectors and generators race forward, the future will leave us with maximally human models that make us question the nature of our own intelligence more than ASI ever could.