SEO Content Gap Analysis & Automation

In competitive markets, data intelligence is the key to digital visibility. This project implements an automated pipeline for Competitive Intelligence and Content Gap Analysis. The system systematically analyzes high-performing competitor pages, identifies semantic structures and keywords, and derives data-driven content strategies from them.

By integrating local Large Language Models (LLMs), the entire process becomes scalable and cost-efficient. Manual research efforts are eliminated, allowing marketing teams to focus on strategy and quality assurance.

Business Value & Application

This automation tool optimizes the SEO workflow through technological scaling:

  • Competitive Intelligence: Automated extraction and structuring of competitor data from complex web environments.
  • Market Analysis: Semantic analysis to identify untapped keyword potentials and topic clusters.
  • Automated Content Drafting: Generation of optimized content briefs and drafts based on proven performance metrics (Skyscraper method).
  • Data Privacy & Cost Efficiency: Fully local data processing eliminates API costs and ensures data sovereignty.

Technical Highlights (Key Features)

  • Stealth Scraping: Uses undetected-chromedriver to bypass bot detection mechanisms and load pages like a real browser.
  • Intelligent Content Detection: Instead of hardcoding CSS selectors, the LLM analyzes the HTML structure and dynamically decides where the main text is located.
  • Token Optimization: Cleans HTML of unnecessary noise (scripts, styles) to efficiently utilize the LLM’s context window.
  • Caching System: Stores downloaded pages locally (based on an MD5 hash of the URL) to avoid unnecessary network requests and speed up development.
  • Complete Pipeline: From URL to finished Markdown article draft in a single pass.

Technologies Used

  • Language: Python 3
  • Web Automation: undetected-chromedriver (a modified version of Selenium) for rendering JavaScript-heavy pages.
  • Parsing: BeautifulSoup4 for HTML processing and cleanup.
  • AI Integration: REST API communication with a local LLM server (e.g. LM Studio), compatible with OpenAI chat completions format.
  • Data Processing: hashlib for caching strategies and json for data exchange.

Technical Highlights and Solutions

Dynamic Selector Detection via AI

Problem: Every website has a different structure. A rigid scraper that looks for <div class="content"> often fails.

Solution: The script sends a cleaned, truncated version of the HTML skeleton to the LLM with the task: “Identify the CSS selector that wraps the main article.” The LLM acts here as an intelligent parser that semantically understands where the content is. This makes the tool extremely flexible across different website layouts.

Efficient Token Management

Problem: LLMs have a limited context window. Raw HTML is full of noise (JavaScript, CSS, SVG) that wastes valuable tokens and can confuse the model.

Solution: Before analysis, the HTML passes through a strict cleaning phase (clean_html_for_analysis). Tags like <script>, <style>, <nav>, and <footer> are removed. Only the structural skeleton of the <body> is sent to the AI. This drastically reduces token count and improves the quality of AI responses.

Robust Browser Automation

Problem: Many modern sites block simple requests or standard Selenium bots.

Solution: By using undetected-chromedriver, a real Chrome profile is simulated. The script creates temporary user profiles to avoid conflicts with running browser instances, and intelligently waits for the page to load before extracting the source code.

A Concrete Example: Analysis of a Golf Blog

To illustrate how the tool works, here is a shortened console output from a real run. The goal was to analyze an article about golf etiquette and generate new content based on it.

Downloading page: https://www.adidas.de/blog/golfregeln-fur-die-etikette-auf-dem-grun
Starting undetected-chromedriver...
Saved to cache: page_cacheabf6112ef2a5db7e571350a68ce733f.html

[Content Extraction] Analyzing HTML structure...
Sending request to LLM...
LLM identified selector: div._article-body_1onw8_209
Extracted 4033 characters.

[Summarization] Generating summary...
SUMMARY:
**Kurzfassung: Golf‑Etikette & Platzregeln**
| Thema | Kernaussagen |
|-------|---------------|
| **Allgemeines** | Golf ist ruhiges Naturspiel... |

[Keyword Extraction] Extracting keywords...
KEYWORDS:
### Einführung in Golfetikette
Keywords: Golfetikette Regeln, Ruhe auf dem Grün, Stressfreies Putten...

[Content Generation] Creating new blog article...
NEW ARTICLE:
# Einführung in Golfetikette
Golf ist mehr als nur ein Sport – es ist eine Kunstform...

Step-by-Step Explanation of the Process

  1. Download & Caching: The script downloads the URL .../golfregeln-fur-die-etikette.... Since the page requires JavaScript, undetected-chromedriver starts up. The result is immediately saved as a hash (aabf61...), so no new request is needed during repeated tests.

  2. Intelligent Extraction: This is where the LLM’s strength shines. Instead of me manually searching in the browser inspector, the AI detects the cryptic selector div._article-body_1onw8_209. This is typical for modern frameworks (like React or Vue) that generate class names. A classic scraper would have failed here.

  3. Summarization: The extracted text (over 4,000 characters) is sent to the local model, which creates a structured summary. This serves as a first check to confirm the correct content was found.

  4. Keyword Extraction: The tool analyzes the structure of the original article and extracts relevant keywords for each section (e.g. “Stressfreies Putten”, “Golfetikette Regeln”). These form the skeleton for the new article.

  5. Content Generation: In the final step, the LLM acts as a copywriter. It takes the extracted keywords and structure and writes a completely new blog post (“Einführung in Golfetikette”) that is thematically equivalent but unique in wording.