SEO Content Gap Analysis & Automation

01 Challenge

Researching competitor content by hand, structuring it and turning it into your own content strategy is tedious – and classic scrapers fail on modern, JavaScript-heavy sites.

02 Approach

A Python pipeline loads competitor pages via undetected-chromedriver, lets a local LLM locate the main content and derives a summary, keywords and a new article draft from it.

03 Result

From URL to finished Markdown draft in a single pass – fully local, with no API costs and full data sovereignty.

Stack

Python 3undetected-chromedriverBeautifulSoup4LM Studiohashlib

In competitive markets, data intelligence is the key to digital visibility. This project implements an automated pipeline for Competitive Intelligence and Content Gap Analysis: the system analyzes high-performing competitor pages, identifies semantic structures and keywords, and derives data-driven content strategies from them.

Local Large Language Models (LLMs) make the process scalable and cost-efficient. Manual research is eliminated – marketing teams can focus on strategy and quality assurance.

Pipeline stages

5

Download to article draft

Pass

1

URL → Markdown draft

Characters extracted

4,033

Example: golf blog

API costs

€0

local LLM

Business Value & Application

• Competitive Intelligence: Automated extraction and structuring of competitor data from complex web environments.
• Market Analysis: Semantic analysis to uncover untapped keyword potential and topic clusters.
• Automated Content Drafting: Content briefs and drafts based on proven performance metrics (Skyscraper method).
• Data Privacy & Cost Efficiency: Fully local data processing – no API costs, full data sovereignty.

Architecture

1. 01Download & cacheRender page, store locally under MD5 hash of the URLundetected-chromedriver
2. 02ExtractionClean HTML, LLM determines the content selectorBeautifulSoup4 + LLM
3. 03SummaryStructured summary as a sanity checkLLM
4. 04KeywordsRelevant keywords per section of the originalLLM
5. 05New articleOriginal draft built on the keyword structureMarkdown

The AI integration talks via REST API to a local LLM server (e.g. LM Studio), compatible with the OpenAI chat completions format. json handles data exchange, hashlib the caching strategy.

Technical Highlights and Solutions

Dynamic Selector Detection via AI

Problem: Every website has a different structure. A rigid scraper that looks for <div class="content"> often fails.

Solution: The script sends a cleaned, truncated version of the HTML skeleton to the LLM with the task: “Identify the CSS selector that wraps the main article.” The LLM acts as an intelligent parser that semantically understands where the content is – making the tool extremely flexible across different layouts.

Efficient Token Management

Problem: LLMs have a limited context window. Raw HTML is full of noise (JavaScript, CSS, SVG) that wastes tokens and can confuse the model.

Solution: Before analysis, the HTML passes through a strict cleaning phase (clean_html_for_analysis). Tags like <script>, <style>, <nav>, and <footer> are removed; only the structural skeleton of the <body> is sent to the AI. This drastically reduces token count and improves the quality of the responses.

Robust Browser Automation

Problem: Many modern sites block simple requests or standard Selenium bots.

Solution: undetected-chromedriver (a modified version of Selenium) simulates a real Chrome profile and also renders JavaScript-heavy pages. The script creates temporary user profiles to avoid conflicts with running browser instances, and intelligently waits for the page to load before extracting the source code.

Caching

Downloaded pages are stored locally under an MD5 hash of the URL. This avoids unnecessary network requests and speeds up development – repeated test runs need no new request.

A Concrete Example: Analysis of a Golf Blog

Shortened console output from a real run. The goal was to analyze an article about golf etiquette and generate new content based on it.

Downloading page: https://www.adidas.de/blog/golfregeln-fur-die-etikette-auf-dem-grun
Starting undetected-chromedriver...
Saved to cache: page_cache\aabf6112ef2a5db7e571350a68ce733f.html

[Content Extraction] Analyzing HTML structure...
Sending request to LLM...
LLM identified selector: div._article-body_1onw8_209
Extracted 4033 characters.

[Summarization] Generating summary...
SUMMARY:
**Kurzfassung: Golf‑Etikette & Platzregeln**
| Thema | Kernaussagen |
|-------|---------------|
| **Allgemeines** | Golf ist ruhiges Naturspiel... |

[Keyword Extraction] Extracting keywords...
KEYWORDS:
### Einführung in Golfetikette
Keywords: Golfetikette Regeln, Ruhe auf dem Grün, Stressfreies Putten...

[Content Generation] Creating new blog article...
NEW ARTICLE:
# Einführung in Golfetikette
Golf ist mehr als nur ein Sport – es ist eine Kunstform...

Step by Step

1. Download & caching: The script downloads the URL .../golfregeln-fur-die-etikette.... Since the page requires JavaScript, undetected-chromedriver starts up. The result is immediately saved as a hash (aabf61...) – repeated tests need no new request.
2. Intelligent extraction: Instead of me searching manually in the browser inspector, the AI detects the cryptic selector div._article-body_1onw8_209.
3. Summarization: The extracted text (over 4,000 characters) is sent to the local model, which creates a structured summary – a first check that the right content was found.
4. Keyword extraction: The tool analyzes the structure of the original article and extracts relevant keywords per section (e.g. “Stressfreies Putten”, “Golfetikette Regeln”). They form the skeleton of the new article.
5. Content generation: The LLM becomes the copywriter: keywords and structure turn into a completely new blog post (“Einführung in Golfetikette”) that is thematically equivalent but unique in wording.