6 Types of Duplicate Content According to Google's Patent

Okay, so this paper called Detecting Query-Specific Duplicate Tokens is a patent by Google. And basically what this paper is all about is that it outlines the pattern of ways in which Google can use to detect duplicated documents. And then it knows how to find the unoriginal content or duplicated content and then use the best mechanics to display only relevant documents on the search engine result page.

How Google Process Search Results to Filter Out Duplicate Content

Below are the steps search engines like google can detect duplicate contents and only show users relevant results when they make a query on the search page.

Step One: User Enters a Query

So the first step, of course, is when the client goes into the search engine and puts in the query. Let's say they put in a search term called "best coffee shops near me."

Search Engine Analyzes the Query

And then what the search engine will do is it will take in this result, take in this term, and then try to check its own reposition. And then take this term into its repository and then try to see which tokens are most relevant to this term.

Grouping Relevant Documents

After, it tries to group the documents relevant to this term. Let's say somebody searches about "best coffee shops near me" in the UK or in New York, it, it tends to group them based on the query.

Extraction of Relevant Snippets from Documents

The next thing for it to do is that it looks into each document and then looks at the most relevant part of the document that best serves the query to which the person or the user has re-positioned. This extraction of important pieces of text is called snippets.

Compression and Grouping Relevant Snippets

It then moves into the fifth stage, which is going to compress the document. So this relevant part, which it has drawn out from the stage 4, it tries to group them. And the reason why it's grouping them, let's say there is a document called "document A" and it has this "best coffee shops near me" information which is relevant to the query. Document B also has this "best coffee shops near me" information which is relevant to the query.

Filtering Out Duplicate Content

After it has gotten this snippet, which is the relevant permission it has drawn from each document, it then tries to see which one has the most similarity or which one leaves gaps. So based on these categorization of those relevant snippets, it tries to compare which are identical or similar. So whichever is identical and not authentic on its own, it tends to filter them out. This duplicate can be of different types (more on that below).

Prevention of Duplicate Results for Improved User Experience

Then in the next step, after it has filtering out the duplicates, the search engine ranks the relevant documents in the search result page. And the reason why the system does all these (from stage one to 6) is because of user experience - to prevents the case where they need to go through ten links on Google which most of them seem alike.

Types of Duplicated Content According to Google Patent

So, this patent mentions six different types of duplicates, though it didn't attribute a defined term used to explain each duplicate. It used diagrams, like from figure 1 to figure 6, so six different figures. So, I mentioned each figure, and then sought to explain each duplicate denoted by a figure. And then I try to give them a better name so you can easily understand what each duplicate means, even at first glance.

Figure 1: An Expanded Duplicate

For Fig. 1, I called name it "Expanded Duplicate." This diagram contains two documents - the first document and the second document. So, as you can see (figure 1 in the image attached), the second document contains or encapsulates every bit and part of the first document. Moreso, you will notice that the writer of the second document went out of their way to include "a bit extra information" of what the first document does not.

Figure 2: A Partial Overlapping Duplicates

For the second figure, I call this a "Partial Overlap Duplicate." When you check figure 2 in the image, you will see that both documents in that type of duplicate have similar content - but it doesn't end there. Even as each has similar content, the writers for both documents went out of their way to research to provide information that is unique to both documents ensuring that each piece of info covers a particular aspect or topic that the other (writer) does not.

Figure 3: A Minimal Common Duplicate

Moving on to Fig. 3, I call this a "Minimal Common Duplicate." In this case, the two different documents have some information in common. Document A and Document B each contain sections that are common to both, but it's not necessarily the entirety of either document. This means a large part of each document is unique, with only a few differentiating common elements, which is why I call it a "Minimal Common Duplicate."

Figure 4: An Embedded Duplicate

For Fig. 4, the first document contains all the content of the second document and a significant amount of additional information. So, for figure 4, you can see that document A entirely consumes document B. And document B, which is consumed by document A, is below 50% size of document A. So, to use a real life scenario to describe this, imagine a big publisher like Yahoo News or BBC News, and then they publish a particular news story, and then a very small publisher website or a new publisher website just goes on that page, and they just copy half part of that document and they publish it. So, in essence, the first document or document A, which is from BBC, totally encapsulates the smaller version of information because the second or small publisher only picked few exact information from the document A here, which is the document from BBC. This is why I call it an "Embedded Duplicate."

Figure 5: A Contextual Duplicate

For Fig. 5, I call this a "Contextual Duplicate." Here, document A and document B both have a little bit of information about Company X. In real-life context, imagine a news story from the BBC (Document A) about a release of product about Company X and the news story from the small publisher (Document b), in this case, the small publisher doesn't entirely take a chunk about company X product release from the BBC's article, but only a very little part. Let's say the small publisher takes a small precise information about the product release from the news story from the BBC, and then they try to develop that with other information that they can get online or from the staffs of company X. So, in the very end, when this comes up in the search results, you'll see that probably the title or even a small part of the body might be the same thing. For this can be in form of qoutes.

Figure 6: An Embedded Contextual Duplicate

Finally, for Fig. 6, which is the last kind of duplicate, I will call it an "Embedded Contextual Duplicate." Looking at the earlier diagram, you can see CompanyX was also used for illustration, but in a slightly different way. I also explain this using a real-life scenario that covers CompanyX (precisely its biography) and how its founder came to be. The Document B could be written by a BBC journalist who focused solely on the history of CompanyX.. As for Document A, it was written and published by a writer at Wikipedia detailing not only how CompanyX came about but also how its CEO grew from the grassroots to founding the company.

Complete Patent Document: Detecting Query-Specific Duplicate Tokens