Even though many argue otherwise, the legal framework governing web scraping lacks clarity, and some experts suggest this dispute might already be obsolete.
The conflict over web scraping has recently escalated.
Last December, Google revealed its intent to pursue legal action against SerpApi, a web scraping firm whose API enables clients’ scrapers to emulate human search behavior. Google asserts that SerpApi’s utility “bypasses security protocols” safeguarding its search outcomes, thereby supplying the substantial data demands of numerous AI large language models, frequently without the express consent or awareness of the respective website owners.
SerpApi has now mounted a defense. On February 20, the company submitted a motion to a California court seeking to dismiss the lawsuit. SerpApi contends that Google is trying to “misuse the Digital Millennium Copyright Act” to hinder other entities from engaging in the very activity Google itself has practiced: extensive web scraping.
In a press release, SerpApi asserted: “Google believes it has proprietorship over the internet. This is the underlying message of its lawsuit against SerpApi, a silent assumption now loudly proclaimed. The fundamental issue is that the internet is not owned by any single entity, a principle unequivocally supported by law.”
Legal Landscape Remains Unclear
Nonetheless, contrary to SerpApi’s assertions, the legal situation is far from unambiguous.
“Although bare facts (such as the sky being blue) generally cannot be copyrighted, judicial rulings have frequently determined that collections of these facts can be copyrighted to a degree. For instance, an encyclopedia or a phone directory could hold some copyright protection (e.g., concerning their specific organization), even if their core content consists of fundamental facts,” explained intellectual property (IP) attorney Kirk Sigmon.
“Put differently, it remains uncertain whether Google possesses a copyright over the search results, summaries, or data it produces and presents. Should the court interpret the work in this way, SerpApi will encounter a considerably more challenging legal challenge,” he observed.
Nevertheless, certain commentators believe that the contention between Google and SerpApi is already somewhat anachronistic.
“In my view, this litigation appears behind the times. The practices of crawling and scraping have evolved significantly beyond their state a few years prior,” stated Martin Jeffrey, who founded the AI search optimization consultancy Harton Works. “Since October, we’ve observed a substantial surge in search traffic from China, being rerouted via Singapore in an effort to obscure its true source. Additionally, the frequency of AppleBot instances has risen dramatically. The sector has progressed beyond the relevance of SerpApi.”
Furthermore, he pointed out, “these occurrences originating from China might impact [corporate] websites built on WordPress or those with poor maintenance. Such organizations could discover their intellectual property or formerly private intranet communications now being incorporated into AI language models.”
Google is not unique in its extensive scraping of training data for AI, he elaborated; other AI companies are employing alternative methods for data acquisition. “Both Anthropic and OpenAI previously engaged in significant scraping activities,” he mentioned, “but this practice has shifted over the past year. While ChatGPT continues to depend considerably on scraping, its reliance is now diminishing. Moreover, we are observing a substantial decrease in Anthropic’s data gathering; although Claude’s exact methodology isn’t entirely transparent, it appears they are no longer scraping entire websites but instead targeting specific pages.”
Despite this, IP attorney Sigmon remarked that predicting the outcome of the court case remains premature. “Broadly speaking, even with the internet’s long existence, there’s a noticeable lack of robust legal precedent regarding web scraping, especially in the manner it’s conducted today,” he stated. “SerpApi’s defense could potentially assist the court in exploring these complex subtleties, but I would not readily label it an assured victory.”
