How to Scrape Google Search Results Without Building Your Own Scraper

So you want to scrape Google search results.

At first, it sounds simple.

Send a request.

Download the HTML.

Parse the titles, links, snippets, and rankings.

Save everything into a database.

For a quick test, this can work.

But once you need the workflow to run every day, across many keywords, locations, or result types, the simple scraper starts turning into a maintenance project.

That is usually when developers start looking for a different approach.

Instead of building and maintaining your own Google scraper, you can use a SERP API.

In this article, we’ll look at how to collect Google search results in JSON without managing the scraping infrastructure yourself.

The workflow looks like this:

Search query → SERP API → structured JSON → your application

Why scraping Google directly gets hard

A basic scraper may work for a few test queries.

But production scraping is different.

Google search results are not just a static list of links. A result page can include:

organic results
paid ads
local packs
maps results
shopping results
images
videos
news results
People Also Ask
related searches
AI-generated results
knowledge panels

The layout can change depending on the query, country, language, device, and location.

That means your scraper needs to handle more than HTML parsing.

You may also need to deal with:

blocked requests
CAPTCHA
proxy rotation
location targeting
browser automation
parsing failures
retry logic
logging
monitoring
data normalization

If your real goal is to build an SEO tool, AI agent, competitor tracker, or reporting workflow, maintaining all of this may not be the best use of engineering time.

You probably do not want to own the scraping infrastructure.

You want the data.

What a SERP API does

A SERP API collects search engine results and returns them in a structured format, usually JSON and sometimes HTML.

Instead of scraping Google directly, you send an API request with parameters like:

search query
search engine
country
language
location
device
output format

Then you get back structured data.

A simplified response may look like this:

{
  "query": "best project management software",
  "organic_results": [
    {
      "position": 1,
      "title": "Best Project Management Software Tools",
      "link": "https://example.com",
      "snippet": "Compare project management platforms, pricing, and features..."
    },
    {
      "position": 2,
      "title": "Top Project Management Apps for Teams",
      "link": "https://example.org",
      "snippet": "A list of project management tools for startups and agencies..."
    }
  ]
}

This is much easier to use than raw HTML.

You can store it in a database, display it in a dashboard, send it into an AI workflow, or export it to CSV.

What we are building

We’ll build a small Python script that:

Sends a Google search query to a SERP API
Gets JSON results back
Extracts organic result fields
Saves the results to a CSV file

Example query:

best email marketing software

Example output:

position,title,link,snippet
1,Best Email Marketing Software Tools,https://example.com,Compare platforms and pricing...
2,Top Email Marketing Services,https://example.org,A guide for small businesses...

This is a good starting point for:

SEO rank tracking
competitor monitoring
AI agent search context
market research
content research
automated reporting
search result analysis

Install dependencies

We only need two packages:

pip install requests python-dotenv

requests is used to call the API.

python-dotenv is used to load environment variables from a .env file.

Create a .env file

Create a file named .env:

SERP_API_KEY=your_api_key_here
SERP_API_URL=https://your-serp-api-endpoint.example.com/search

Do not hardcode your API key directly in your script.

Basic Python request

Create a file named google_serp.py.

import os
import requests
from dotenv import load_dotenv


load_dotenv()

SERP_API_KEY = os.getenv("SERP_API_KEY")
SERP_API_URL = os.getenv("SERP_API_URL")


def search_google(query, location="United States", language="en"):
    if not SERP_API_KEY:
        raise ValueError("Missing SERP_API_KEY environment variable")

    if not SERP_API_URL:
        raise ValueError("Missing SERP_API_URL environment variable")

    params = {
        "api_key": SERP_API_KEY,
        "engine": "google",
        "q": query,
        "location": location,
        "language": language,
        "output": "json",
    }

    response = requests.get(SERP_API_URL, params=params, timeout=30)
    response.raise_for_status()

    return response.json()


if __name__ == "__main__":
    data = search_google("best email marketing software")
    print(data)

Run it:

python google_serp.py

At this point, you should receive a JSON response from your SERP API provider.

The exact endpoint and parameter names may vary depending on the API you use. Some providers may use gl, hl, country, locale, or other parameter names. Always check the provider documentation and adjust the request.

Extract organic results

Most SERP APIs return organic results in a list.

The key may be called:

organic_results
organic
results

To keep the script flexible, we can write a helper function.

def get_organic_results(data):
    possible_keys = [
        "organic_results",
        "organic",
        "results",
    ]

    for key in possible_keys:
        value = data.get(key)
        if isinstance(value, list):
            return value

    return []

Now we can normalize the fields.

def normalize_result(item):
    return {
        "position": item.get("position") or item.get("rank"),
        "title": item.get("title"),
        "link": item.get("link") or item.get("url"),
        "snippet": item.get("snippet") or item.get("description"),
    }

This makes the output easier to save and reuse.

Print the top results

Update the script:

import os
import requests
from dotenv import load_dotenv


load_dotenv()

SERP_API_KEY = os.getenv("SERP_API_KEY")
SERP_API_URL = os.getenv("SERP_API_URL")


def search_google(query, location="United States", language="en"):
    if not SERP_API_KEY:
        raise ValueError("Missing SERP_API_KEY environment variable")

    if not SERP_API_URL:
        raise ValueError("Missing SERP_API_URL environment variable")

    params = {
        "api_key": SERP_API_KEY,
        "engine": "google",
        "q": query,
        "location": location,
        "language": language,
        "output": "json",
    }

    response = requests.get(SERP_API_URL, params=params, timeout=30)
    response.raise_for_status()

    return response.json()


def get_organic_results(data):
    possible_keys = [
        "organic_results",
        "organic",
        "results",
    ]

    for key in possible_keys:
        value = data.get(key)
        if isinstance(value, list):
            return value

    return []


def normalize_result(item):
    return {
        "position": item.get("position") or item.get("rank"),
        "title": item.get("title"),
        "link": item.get("link") or item.get("url"),
        "snippet": item.get("snippet") or item.get("description"),
    }


if __name__ == "__main__":
    data = search_google("best email marketing software")
    organic_items = get_organic_results(data)

    results = [normalize_result(item) for item in organic_items]

    for result in results[:10]:
        print(result["position"])
        print(result["title"])
        print(result["link"])
        print(result["snippet"])
        print("---")

Example output:

1
Best Email Marketing Software Tools
https://example.com
Compare email marketing platforms, pricing, and features...
---
2
Top Email Marketing Services for Small Businesses
https://example.org
A guide to email marketing tools for startups and growing teams...
---

Now we have clean search result fields instead of raw HTML.

Save results to CSV

For SEO and research workflows, CSV is often enough for a first version.

Add this helper function:

import csv


def save_to_csv(rows, filename="serp_results.csv"):
    fieldnames = [
        "position",
        "title",
        "link",
        "snippet",
    ]

    with open(filename, mode="w", newline="", encoding="utf-8") as file:
        writer = csv.DictWriter(file, fieldnames=fieldnames)
        writer.writeheader()

        for row in rows:
            writer.writerow(row)

Now call it after parsing the results:

save_to_csv(results, "google_results.csv")
print(f"Saved {len(results)} results to google_results.csv")

Full script:

import os
import csv
import requests
from dotenv import load_dotenv


load_dotenv()

SERP_API_KEY = os.getenv("SERP_API_KEY")
SERP_API_URL = os.getenv("SERP_API_URL")


def search_google(query, location="United States", language="en"):
    if not SERP_API_KEY:
        raise ValueError("Missing SERP_API_KEY environment variable")

    if not SERP_API_URL:
        raise ValueError("Missing SERP_API_URL environment variable")

    params = {
        "api_key": SERP_API_KEY,
        "engine": "google",
        "q": query,
        "location": location,
        "language": language,
        "output": "json",
    }

    response = requests.get(SERP_API_URL, params=params, timeout=30)
    response.raise_for_status()

    return response.json()


def get_organic_results(data):
    possible_keys = [
        "organic_results",
        "organic",
        "results",
    ]

    for key in possible_keys:
        value = data.get(key)
        if isinstance(value, list):
            return value

    return []


def normalize_result(item):
    return {
        "position": item.get("position") or item.get("rank"),
        "title": item.get("title"),
        "link": item.get("link") or item.get("url"),
        "snippet": item.get("snippet") or item.get("description"),
    }


def save_to_csv(rows, filename="serp_results.csv"):
    fieldnames = [
        "position",
        "title",
        "link",
        "snippet",
    ]

    with open(filename, mode="w", newline="", encoding="utf-8") as file:
        writer = csv.DictWriter(file, fieldnames=fieldnames)
        writer.writeheader()

        for row in rows:
            writer.writerow(row)


if __name__ == "__main__":
    query = "best email marketing software"

    data = search_google(query)
    organic_items = get_organic_results(data)

    results = [normalize_result(item) for item in organic_items]

    save_to_csv(results, "google_results.csv")

    print(f"Saved {len(results)} results to google_results.csv")

Search multiple keywords

A real SEO or AI workflow usually needs more than one query.

Let’s add multiple keywords:

keywords = [
    "best email marketing software",
    "email marketing tools for small business",
    "mailchimp alternatives",
]

all_rows = []

for keyword in keywords:
    data = search_google(keyword)
    organic_items = get_organic_results(data)

    for item in organic_items:
        row = normalize_result(item)
        row["query"] = keyword
        all_rows.append(row)

If you add the query field, update the CSV field names:

fieldnames = [
    "query",
    "position",
    "title",
    "link",
    "snippet",
]

Now each result is connected to the keyword that produced it.

Search by location

Google results are not the same everywhere.

The same keyword can produce different results depending on country, city, language, and device.

For example:

best CRM software

may not show the same results in the United States, Germany, or Singapore.

A local keyword like:

dentist near me

can change completely from one city to another.

That is why location parameters matter.

Example:

data = search_google(
    query="best CRM software",
    location="United Kingdom",
    language="en",
)

For local SEO, you may want to loop through several cities:

locations = [
    "New York, United States",
    "London, United Kingdom",
    "Singapore",
]

query = "best digital marketing agency"

all_rows = []

for location in locations:
    data = search_google(query=query, location=location)
    organic_items = get_organic_results(data)

    for item in organic_items:
        row = normalize_result(item)
        row["query"] = query
        row["location"] = location
        all_rows.append(row)

This can be the start of a simple location-based rank tracking workflow.

Using SERP data in an AI agent

SERP data is also useful for AI agents.

Instead of asking an LLM to answer from static knowledge, you can give it fresh search context.

A simple workflow:

User task → Google search query → SERP API → JSON results → LLM prompt

You can turn organic results into a prompt like this:

def build_search_context(results, max_results=5):
    blocks = []

    for result in results[:max_results]:
        block = f"""
Position: {result.get("position")}
Title: {result.get("title")}
URL: {result.get("link")}
Snippet: {result.get("snippet")}
""".strip()

        blocks.append(block)

    return "\n\n".join(blocks)

Then build an agent prompt:

def build_agent_prompt(user_task, search_context):
    return f"""
You are a research assistant.

Use the search results below to answer the user's task.
Do not invent sources.
If the search results are not enough, say what is missing.

User task:
{user_task}

Search results:
{search_context}

Write a concise answer with:
- key findings
- important sources
- source URLs
""".strip()

Usage:

user_task = "Find the top competitors in email marketing software."

data = search_google("best email marketing software")
organic_items = get_organic_results(data)
results = [normalize_result(item) for item in organic_items]

search_context = build_search_context(results)
prompt = build_agent_prompt(user_task, search_context)

print(prompt)

At this point, you can send the prompt to your LLM of choice.

The key idea is simple:

Your agent gets fresh search context without having to browse or parse raw HTML itself.

What to check before choosing a SERP API

Before choosing a provider, test it with your real queries.

Do not only run one easy demo search.

Try different types of queries:

best email marketing software
mailchimp alternatives
coffee shop in Austin
best laptops under 1000
latest AI search tools

Then check:

Does the API return clean JSON?
Are titles, links, snippets, and positions complete?
Can you request HTML if needed?
Does geo-targeting work?
Does it support the search engines you need?
Does it include local results, ads, shopping, news, or other SERP blocks?
Are failed requests billed?
How much cleanup does your code need?

If you are comparing providers such as SerpApi, SearchAPI, Bright Data, Serper, or Talordata, send the same real queries to each one and compare the actual response body.

The best API is usually the one that gives your application usable data with the least extra work.

When this approach is useful

Using a SERP API instead of building your own scraper is useful for:

SEO rank tracking
local SEO monitoring
competitor research
brand monitoring
market research
e-commerce intelligence
AI agent search context
content research
automated reporting
search result analysis

For one-time experiments, a small scraper may be enough.

For production workflows, a SERP API can save a lot of maintenance work.

Final thoughts

Scraping Google search results sounds simple at first.

But maintaining a reliable scraper means dealing with changing layouts, blocked requests, CAPTCHA, proxy logic, location targeting, retries, monitoring, and parser updates.

If your real goal is to use search data, not maintain scraping infrastructure, a SERP API is often the easier path.

You send a search query.

You get structured JSON.

You save it, analyze it, or pass it into an AI workflow.

If you want to test this kind of workflow, Talordata is one option worth comparing. It supports structured SERP data, JSON / HTML response formats, geo-targeted results, and search workflows for SEO monitoring, AI agents, competitor tracking, and market research.

Talordata also offers 1,000 free API requests after signup, which is enough to test real search queries before choosing a provider.