robots.txt: allow, disallow and blocking AI crawlers
Write robots.txt to manage crawler traffic and block the AI crawlers you mean to block, and know why the file can't hide a page from search.
Oct 5, 2026 · 6 min read
All Posts
Page 2 of 3
Google Trends: what 0-100 interest means and when to pay for volumes
How to read Google Trends' 0-100 interest, compare and region views, and when relative interest is enough versus when you need absolute search volumes and a live results page.
Oct 5, 2026 · 6 min read
Google Search Console: queries, impressions, and when to pull the API
How to read your own site's queries, impressions, clicks and position in Search Console, and which questions need a live results page instead.
Oct 4, 2026 · 7 min read
What is link gap analysis: the you-plus-five pull and match priority
Find the sites linking to your competitors but not to you, sort them by how many competitors they link to, and qualify each row before you pitch.
Oct 4, 2026 · 6 min read
Low-competition keywords: finding the winnable set
What makes a keyword low-competition, why the difficulty score alone is not proof, how to audit page one by hand, and how to build a small, dated shortlist a new site can actually win.
Oct 3, 2026 · 6 min read
llms.txt: the file that makes you eligible, not cited
What llms.txt is, how it differs from robots.txt and sitemap.xml, how to shape and place the file, how to keep it current, and what the adoption numbers say about whether it works yet.
Oct 3, 2026 · 7 min read
Google &num=100 removal: why one request became ten
What the &num=100 parameter did, why Google removed it in September 2025, and how to budget rank tracking and SERP pulls now that every hundred results takes ten page fetches.
Oct 2, 2026 · 6 min read
AI Visibility: Does the AI Mention You?
A cited-or-not playbook for AI visibility: which prompts to test, which engines to check, what to count, and how often to repeat it so the number means something.
Oct 2, 2026 · 6 min read
CTR in SEO: the formula, and when fixed CTR tables mislead
Read CTR as clicks over impressions, see why a fixed by-position table breaks when SERP features crowd the page, and learn when each read fits with dated examples.
Oct 1, 2026 · 7 min read
Keyword research with agents: seed to cluster, then volume
Run keyword research as an agent loop: fan out from a seed through live suggestions, clean and cluster what comes back, and spend paid volume lookups only on the head terms that survive.
Oct 1, 2026 · 6 min read
CPC (cost per click): formula, auction and weighing it with volume
What cost per click means, how the ad auction sets it, the one-line formula, and how keyword researchers use CPC as a value signal next to volume, with the source and month written beside every figure.
Sep 30, 2026 · 6 min read
Google AI Overviews: how the summary box displaces snippets
What Google's AI Overview is, how it is built, how quickly it pushed featured snippets off informational queries, and why its citations are the hardest part of the page to track.
Sep 30, 2026 · 6 min read
Search intent types: reading the why before you enter a niche
The four search intent types, the results-page read that tells you which one a keyword carries, and the questions that turn intent into an enter-or-not call before you write a brief.
Sep 29, 2026 · 7 min read
Google Keyword Planner: when free averages are enough
What Google Keyword Planner shows, how to reach it through a Google Ads account, when its free monthly averages are enough, and when to check the live results page before you commit.
Sep 29, 2026 · 6 min read
What is a SERP? Organic results, ads and features explained
What a Google results page actually contains, how each part earns its place, and which parts a SERP API hands back as structured data for code and AI workflows.
Sep 28, 2026 · 6 min read
MCP server vs direct API call for SEO data: when to use which
When working direct calls are enough for SEO data, when an MCP server earns the switch, and how the top-up question settles it.
Sep 28, 2026 · 6 min read
SERP API vs DIY scraping vs independent index: what each really costs
DIY looks free until infrastructure and maintenance dominate. A three-way total-cost and trade-off guide for choosing DIY scraping, a SERP API, or an independent index.
Sep 27, 2026 · 9 min read
Are DataForSEO backlinks as good as Ahrefs? Live links vs totals
Read 2 trillion vs 35 trillion as live vs historical totals, price an audit at per-row math, and verify what is still live before trusting either number.
Sep 27, 2026 · 8 min read
Keyword Clustering: Grouping by SERP Overlap
Group the terms that deserve one page with the SERP-overlap test: same top pages, same cluster. A portable method from keyword universe to content calendar, with a date on every grouping.
Sep 26, 2026 · 6 min read
Bring your own key for SEO data: pay per query, not per seat
What bring-your-own-key means for SEO data, how pay-per-query differs from seat rental, where the surprise burn hides, and the key hygiene that comes with owning the meter.
Sep 26, 2026 · 6 min read