How to Do Keyword Research at Scale with a SERP API

Traditional keyword research tools — Ahrefs, Semrush, Moz — are excellent products, but they come at a price. Subscriptions start at $100 per month and quickly climb to $400 or more for plans with meaningful API access and data export capabilities. For independent developers, small agencies, and startups, those costs are significant. More importantly, those tools give you access to their data, not raw Google data processed the way you want it.

2026 update: To scale this approach to thousands of pages, read our programmatic SEO playbook.

A SERP API changes the equation entirely. Instead of paying for a software subscription that locks you into someone else's interface and data model, you query search engines directly and build your own analysis pipeline. At $0.00003 per search with Serpent API — the $0.03-per-1,000 Scale tier — you can research 100,000 keywords for $3. You can build exactly the reports you need, export data in any format, and integrate your keyword research directly into your own tools and workflows.

This guide walks through the complete process: how to set up a keyword research pipeline with a SERP API, how to extract and interpret the signals that matter, and how to build a keyword database that gives you genuine competitive insight.

What SERP Data Reveals About Keywords

The search engine results page contains far more information than just a list of ten blue links. Every element on a SERP is a signal about the keyword behind it. When you learn to read these signals systematically, you get a remarkably complete picture of keyword opportunity without needing a dedicated keyword tool.

Search intent signals. The types of results Google shows for a keyword tell you what Google thinks users are looking for. A SERP dominated by product pages and shopping ads signals transactional intent — users want to buy. A SERP full of how-to articles and Wikipedia entries signals informational intent — users want to learn. Understanding intent is essential for knowing what type of content to create to rank for a keyword.

Competition level proxies. By examining the domain authority of the sites ranking in the top ten positions, you can build a proxy measure of keyword difficulty. If positions one through five are all occupied by major publications with millions of backlinks, a newer site has a harder path to ranking. If the top results include smaller, less authoritative sites, there may be a genuine opportunity.

SERP feature presence. Google enriches search results with dozens of special features: featured snippets, People Also Ask boxes, Knowledge Panels, local packs, shopping carousels, and more. The presence of these features tells you how competitive and how enriched the content opportunity is for a given keyword. A keyword with a featured snippet that you can target is a different opportunity than one without.

Related searches for keyword expansion. The "Related searches" section at the bottom of a Google SERP is a direct window into how Google categorizes and clusters related queries. These related searches are gold for keyword research — they represent real user queries that are semantically connected to your seed keyword, giving you a structured starting point for expanding your keyword universe.

Ad presence as commercial intent signal. The number of paid ads appearing for a keyword is one of the clearest signals of commercial intent and keyword value. Keywords with four or more ads at the top of the SERP are being actively bid on by advertisers, which means they convert to revenue for someone. That is a strong signal that there is money in the niche.

Setting Up Your Keyword Research Pipeline

Getting started requires only a Serpent API key and a few lines of Python. The core functions you need are a single-keyword research function and a batch function for processing multiple keywords efficiently.

import requests
import time

API_KEY = "YOUR_API_KEY"

def research_keyword(keyword, country="us"):
    """Get SERP data for keyword research."""
    response = requests.get(
        "https://apiserpent.com/api/search",
        params={"q": keyword, "num": 10, "country": country, "apiKey": API_KEY}
    )
    return response.json()

def bulk_research(keywords, delay=0.5):
    """Research multiple keywords with rate limiting."""
    results = {}
    for kw in keywords:
        results[kw] = research_keyword(kw)
        time.sleep(delay)  # Respect rate limits
    return results

The bulk_research function includes a configurable delay between requests. This is good practice regardless of your API's rate limit policy — spreading requests over time gives you more consistent response times and avoids thundering-herd problems when processing large keyword lists.

For a typical keyword research workflow, you will start with a list of seed keywords relevant to your niche, run them through the pipeline, and then use the results to expand your keyword universe and prioritize targets. Let us walk through each of those steps.

To research a list of seed keywords, you might start like this:

seed_keywords = [
    "SERP API",
    "keyword rank tracker",
    "SEO automation python",
    "search engine results API",
    "google search API"
]

# Research all seed keywords
data = bulk_research(seed_keywords)

# Process results for each keyword
for keyword, serp_data in data.items():
    organic = serp_data.get("results", {}).get("organic", [])
    print(f"{keyword}: {len(organic)} organic results found")

This gives you a starting point. From here, you layer in the more sophisticated analysis techniques described in the following sections.

Extracting Related Searches for Keyword Discovery

One of the most powerful features of SERP data for keyword research is the related searches array. Google's related searches represent algorithmically determined clusters of queries that users who searched for your seed keyword also searched for. This is a direct map of how Google thinks about the topic space around your keyword.

def expand_keywords(seed_keyword):
    """Use related searches to find keyword variations."""
    data = research_keyword(seed_keyword)
    related = data.get("results", {}).get("relatedSearches", [])
    return [item["query"] for item in related]

# Starting with "SERP API"
expansions = expand_keywords("SERP API")
print(expansions)
# ['serp api free', 'serp api python', 'serp api pricing', ...]

The real power comes from running this recursively. You start with a seed keyword, extract its related searches, then research each of those as new seed keywords and extract their related searches. After two or three rounds of expansion, you typically have hundreds of keyword variations that you would never have thought of manually.

def recursive_expand(seed_keyword, depth=2, seen=None):
    """Recursively expand keywords using related searches."""
    if seen is None:
        seen = set()

    if seed_keyword in seen or depth == 0:
        return []

    seen.add(seed_keyword)
    expanded = expand_keywords(seed_keyword)

    all_keywords = list(expanded)
    for kw in expanded:
        if kw not in seen:
            time.sleep(0.3)  # Rate limiting
            all_keywords.extend(recursive_expand(kw, depth - 1, seen))

    return list(set(all_keywords))

# Full expansion from a single seed
all_keywords = recursive_expand("SERP API", depth=2)
print(f"Discovered {len(all_keywords)} keyword variations")

This approach can discover keyword clusters that established tools miss, because you are working directly from Google's own related-query data rather than a third-party database that may be months or years out of date.

Analyzing SERP Features for Intent Classification

Once you have a list of keywords to analyze, the next step is classifying each one by search intent. This determines what type of content you need to create to compete, and it helps you prioritize keywords that align with your content strategy.

SERP features are the most reliable intent signal available. Here is a function that classifies keyword intent based on what features appear in the results:

def analyze_intent(serp_data):
    """Determine keyword intent from SERP features."""
    results = serp_data.get("results", {})
    ads = results.get("ads", {})
    paa = results.get("peopleAlsoAsk", [])

    has_ads = (ads.get("top", []) or ads.get("bottom", [])) is not None
    commercial_score = len(ads.get("top", [])) + len(ads.get("bottom", []))

    if commercial_score >= 3:
        intent = "commercial"
    elif len(paa) >= 3:
        intent = "informational"
    else:
        intent = "mixed"

    return intent

You can extend this classification to detect other intent signals. The presence of a featured snippet suggests the keyword has a clear question-answer format that rewards comprehensive, structured content. The presence of a local pack indicates location-based intent. A shopping carousel indicates product-buying intent even for non-branded keywords.

def detailed_serp_analysis(keyword):
    """Comprehensive SERP feature analysis for a keyword."""
    data = research_keyword(keyword)
    results = data.get("results", {})

    analysis = {
        "keyword": keyword,
        "intent": analyze_intent(data),
        "organic_count": len(results.get("organic", [])),
        "has_featured_snippet": bool(results.get("featuredSnippet")),
        "has_knowledge_panel": bool(results.get("knowledgePanel")),
        "paa_count": len(results.get("peopleAlsoAsk", [])),
        "related_searches_count": len(results.get("relatedSearches", [])),
        "top_ads_count": len(results.get("ads", {}).get("top", [])),
        "bottom_ads_count": len(results.get("ads", {}).get("bottom", []))
    }
    return analysis

Running this analysis across your keyword list gives you a structured dataset you can sort, filter, and prioritize. Keywords with high commercial scores and featured snippet opportunities are often the highest-value targets for content investment.

Tracking Keyword Difficulty

Without access to backlink data for every URL in the organic results, you cannot compute a true keyword difficulty score in the way that tools like Ahrefs or Moz do. However, you can build a useful proxy by examining which domains are ranking in the top ten positions.

The idea is straightforward: certain domains are known to be highly authoritative — major publications, Wikipedia, established industry players. When these domains dominate the top ten for a keyword, it signals that the keyword is difficult to rank for. When smaller, less established domains appear in the top positions, it suggests an opening.

HIGH_AUTHORITY_DOMAINS = {
    "wikipedia.org", "reddit.com", "amazon.com", "youtube.com",
    "forbes.com", "nytimes.com", "techcrunch.com", "hubspot.com",
    "moz.com", "ahrefs.com", "semrush.com", "neilpatel.com"
}

def estimate_difficulty(serp_data):
    """Estimate keyword difficulty from top-ranking domains."""
    organic = serp_data.get("results", {}).get("organic", [])
    if not organic:
        return None

    high_auth_count = 0
    for result in organic[:10]:
        url = result.get("link", "")
        domain = url.split("/")[2] if "//" in url else ""
        # Check if domain or parent domain is high authority
        for auth_domain in HIGH_AUTHORITY_DOMAINS:
            if auth_domain in domain:
                high_auth_count += 1
                break

    # Score from 0-100 based on proportion of high-authority domains
    difficulty_score = (high_auth_count / min(len(organic), 10)) * 100

    if difficulty_score >= 70:
        label = "Hard"
    elif difficulty_score >= 40:
        label = "Medium"
    else:
        label = "Easy"

    return {"score": round(difficulty_score), "label": label}

This is not a replacement for proper backlink-based difficulty scores, but it is a useful first-pass filter that costs fractions of a cent per keyword to compute. You can quickly identify the easy-win opportunities in your keyword list before investing in deeper analysis.

Building a Keyword Database

Once you have all of this analysis logic in place, the final step is storing your results in a queryable format. A SQLite database is perfect for this use case — it is file-based, requires no server, and is more than capable of handling millions of keyword records.

import sqlite3
import csv
from datetime import datetime

def create_keyword_db(db_path="keywords.db"):
    """Initialize the keyword research database."""
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()
    cursor.execute("""
        CREATE TABLE IF NOT EXISTS keywords (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            keyword TEXT UNIQUE NOT NULL,
            intent TEXT,
            difficulty_score INTEGER,
            difficulty_label TEXT,
            has_featured_snippet BOOLEAN,
            paa_count INTEGER,
            top_ads_count INTEGER,
            related_searches TEXT,
            organic_count INTEGER,
            researched_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
    """)
    conn.commit()
    return conn

def save_keyword_data(conn, keyword, analysis, difficulty):
    """Save keyword analysis results to the database."""
    cursor = conn.cursor()
    cursor.execute("""
        INSERT OR REPLACE INTO keywords
        (keyword, intent, difficulty_score, difficulty_label,
         has_featured_snippet, paa_count, top_ads_count,
         organic_count, researched_at)
        VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
    """, (
        keyword,
        analysis.get("intent"),
        difficulty.get("score") if difficulty else None,
        difficulty.get("label") if difficulty else None,
        analysis.get("has_featured_snippet"),
        analysis.get("paa_count"),
        analysis.get("top_ads_count"),
        analysis.get("organic_count"),
        datetime.now().isoformat()
    ))
    conn.commit()

def export_to_csv(conn, output_path="keywords_export.csv"):
    """Export the keyword database to a CSV file."""
    cursor = conn.cursor()
    cursor.execute("SELECT * FROM keywords ORDER BY difficulty_score ASC")
    rows = cursor.fetchall()
    columns = [desc[0] for desc in cursor.description]

    with open(output_path, "w", newline="") as f:
        writer = csv.writer(f)
        writer.writerow(columns)
        writer.writerows(rows)

    print(f"Exported {len(rows)} keywords to {output_path}")

With this database in place, you have a complete keyword research system. You can query it to find all easy-difficulty commercial-intent keywords, export filtered lists for content planning, or join it with your own ranking data to find gaps between the keywords you are researching and the ones you are already targeting.

The total cost to research 1,000 keywords with this pipeline? About $0.03 at Serpent API's Scale-tier pricing. The same research through a traditional keyword tool would consume a meaningful portion of a monthly subscription. For detailed guidance on extending this pipeline into a full rank-tracking system, see our guide on building a keyword rank tracker. And if you want to automate regular reporting from this data, our guide on automating SEO reports covers the full workflow.