Exploring Artificial Intelligence in Detecting Online Material: Effective Techniques and Common Pitfalls

Exploring Artificial Intelligence in Detecting Online Material: Effective Techniques and Common Pitfalls

AI content detection is not always accurate! Your detection tool may sometimes flag human-written content as AI-generated. Here’s why this happens and what you can do about it.

How Do AI Detectors Work?

Before we understand why AI detectors fail, let’s lay some groundwork. AI content detection is all about finding patterns.

Why? Simply because when a human writes, they blend random thoughts into meaningful sentences. There is no set pattern. Some sentences can be too long to read, and some can be short.

This is the exact opposite of how an AI thinks and writes. There is minimal randomness, and the text is very structured. There may also be repetition of ideas or words. And the choice of words itself may be too robotic to read.

AI content detectors take all this into account. They look for such patterns to distinguish between human-written and AI-generated content.

To do this, four concepts come into play.

They Apply Classifiers

A classifier is an algorithm that categorizes text into different classes based on factors like usage, grammar, style, and tone.

For example, a text with a bland tone, poor grammar, and repetitive writing style is more likely to be categorized as AI-generated.

They Use Embeddings

In AI-content detection, Embeddings are numerical representations of words and their relationships with each other. They are expressed as vectors in high dimensional space, each with a unique code.

These codes help computers understand how each word relates to one another and the context of their usage. The underlying machine learning model is constantly trained to determine which codes are common for AI-generated text and which are not.

They Look at Perplexity

Perplexity is a characteristic of text that defines the degree of randomness in a piece of writing. Humans write with very high perplexity. AI does not.

For example, think about the possible endings of this sentence: “I went to watch Oppenheimer yesterday, and it was _____.”

If you answered something expected like “spectacular,” “outstanding,” “remarkable,” “impressive,” or “captivating,” I’m sorry, but you might be a robot. However, you have a good taste in movies!

Jokes apart, a human is more likely to complete the sentence with something more conversational or based on personal experience. Something like “totally crazy” or “not what I expected it to be.” After all, a human can expect something from a movie. AI obviously cannot. If it does claim to, the underlying language model is probably hallucinating (making up claims on the spot without factual evidence) or lacking protective guardrails (output structuring and quality control).

They Check for Burstiness

We have already talked about how humans write unpredictably. And how some sentences can be long and some can be short. Burstiness is another text characteristic that defines this.

AI-written text is usually made of sentences similar in length and structure (low burstiness). Here’s an example of some text generated by ChatGPT. Notice the monotonous structure and comparable length of both the sentences:

“Text burstiness, also known as word burstiness or term burstiness, is a concept in natural language processing and text analysis that refers to the non-uniform distribution of words or terms in a given text or document. In other words, it describes the phenomenon where certain words or terms appear more frequently in a specific context or document than would be expected based on a random or uniform distribution.”

Human text is the opposite (like this article). It has a healthy mix of long and short sentences with just enough creativity to break patterns. And steers clear of dull structures (high burstiness).

AI detectors use a combination of these four concepts to spot AI-written content. So, the science is there. But is it sound?

Is AI Detection Accurate?

Sadly, AI detection is not 100% accurate. Not yet, anyway. It is just a probability game.

And that is why running any content through an AI detector returns a confidence level, never an accuracy level. For example, if your AI detector gives you a score of 70%, it means that it is 70% confident that the content is AI-generated, and 30% confident that it’s human-written.

Now, imagine this. I show you ten chocolates and tell you seven are dark and three are white. Now I ask you to choose one randomly and tell me the flavor you got without opening the wrapper. Can you answer this? Of course not! The premise itself is setting you up for failure. And that is exactly what is happening with AI detectors. With only confidence levels and probabilities to fall back on, they are bound to be wrong sooner or later.

Why Do AI Content Detectors Fail?

There are many reasons why AI content detection is becoming increasingly difficult.

1. AI content generators are outpacing them: Models like ChatGPT 4 (and even the free version ) are getting really good at writing human-like content. They use just the right classifiers, embeddings, perplexity, and burstiness. They have analyzed crazy amounts of human-generated content to find the sweet spot between proper grammar usage and choice of vocabulary.
2. Your AI detection tool is just not good enough: Just like AI generators, even AI detectors need to be trained on massive amounts of data. Otherwise, they cannot classify human and AI-generated content accurately.
3. Bias often creeps into training data: When an AI makes systematically incorrect decisions for specific use cases, it is known as a bias. And this is a serious issue. They exist because all training data comes from humans. Humans have biases, even if they are unaware of them.
4. New AI content generation strategies are making things worse: AI pro writers and bloggers are constantly developing new strategies to trick AI detectors. For example, they have figured out specific prompts to make ChatGPT write content that is more likely to go undetected. There is even a dedicated plugin now to humanize ChatGPT text !

What Can You Do About It?

Your best bet is to learn how to spot AI content yourself.

Is it easy? Not really. But it is certainly possible. With some practice, you can train your eye to look for the following:

• Repetitive words and phrases, especially focused on possible target keywords. Even the structure may seem too uniform. For example, “I love cats because cats are cute. Cats have soft fur and warm purrs. I can’t imagine my life without cats.”
• Generic and robotic tone with zero to minimal creativity. For example, “Welcome to our website. We offer a wide range of products and services. We cater to our customers’ needs. Our team works hard to provide the best quality and satisfaction to our clients.”
• Surface-level depth of key ideas with no real insight or practical learnings based on real-life experiences. For example, “You should always stay positive and never give up. Positivity brings success and happiness in life. It is a good habit.”
• Factual mistakes and outdated information. AI content generators are known to “hallucinate” occasionally and make something up on the spot without any real foundation. For example, “According to a recent study in 2002, the Earth is flat, and the sun revolves around it.”
• Logical inconsistencies and errors that are just embarrassing to read. For example, “John was having his lunch at night when the morning mail came.”
• A general sense of lifelessness in the content.

The future of AI generators Vs. AI detectors is genuinely unpredictable. There’s no telling who will eventually win the race. For now, it is best to take the manual route and work on developing this intuitive skill.