Scrape Google Search Results
Build a SERP monitoring script that tracks keyword rankings and extracts organic results using the Autom API.
Overview
In this playbook you will build a script that queries Google Search and extracts structured organic results — positions, titles, URLs, and snippets — for any keyword. A typical use case is rank tracking: run this on a schedule to monitor where your pages appear for target keywords.
The endpoint returns up to 10 organic results per page and supports pagination, country (gl) and language (hl) targeting.
Prerequisites
- An Autom API key — get one at app.autom.dev
- Install dependencies for your language:
pip install requestsSteps
Make your first search request
Call GET /v1/google/search with the q parameter and your API key in the x-api-key header.
import requests
API_KEY = "YOUR_API_KEY"
response = requests.get(
"https://api.autom.dev/v1/google/search",
headers={"x-api-key": API_KEY},
params={"query": "best python web scraping libraries", "gl": "us", "hl": "en"},
)
data = response.json()
print(data)Parse the organic results
The response contains an organic_results array. Each entry has position, title, link, snippet, domain, and source.
for result in data["organic_results"]:
print(f"[{result['position']}] {result['title']}")
print(f" URL: {result['link']}")
print(f" {result.get('snippet', '')}")
print()Handle pagination
Use the page parameter to fetch subsequent pages. The response includes a pagination object with has_next_page and next page number.
import requests
API_KEY = "YOUR_API_KEY"
QUERY = "best python web scraping libraries"
def fetch_all_results(query: str, max_pages: int = 3) -> list:
all_results = []
page = 1
while page <= max_pages:
response = requests.get(
"https://api.autom.dev/v1/google/search",
headers={"x-api-key": API_KEY},
params={"query": query, "gl": "us", "hl": "en", "page": page},
)
data = response.json()
all_results.extend(data.get("organic_results", []))
if not data.get("pagination", {}).get("has_next_page"):
break
page += 1
return all_results
results = fetch_all_results(QUERY)
for r in results:
print(f"[{r['position']}] {r['title']} — {r['link']}")Use the gl parameter to target a specific country (e.g. fr for France, de for Germany) and hl for the language of results. This is essential for accurate local rank tracking.