Technical SEO Checklist 2026: 13 Checks to Get Indexed

A technical SEO checklist for 2026 runs thirteen checks, from HTTP status codes up to AI crawler access: HTTP and HTTPS health, robots.txt directives, index coverage, canonical consolidation, XML sitemaps, URL structure, crawl budget and log files, internal linking, mobile parity, JavaScript rendering, Core Web Vitals, structured data, and AI crawler access. Each one decides whether a crawler reaches a page and whether the index keeps it once it does. Trust and ranking come later, and only for pages that cleared those first gates.

Most teams start with content and assume the technical layer sorts itself out. It does not. A page with a broken canonical tag or a 3.2s LCP never gets the chance to rank, because it never gets fully indexed. The checklist below is ordered foundation first, then standard, then advanced, so you fix the gates that block everything before you spend a day on schema.

One mental model holds the whole list together. A search engine crawls a URL, decides whether to index it, and for JavaScript sites renders it somewhere in between. Every check here intervenes at one of those stages: robots.txt and HTTP status govern the crawl, canonicals and noindex govern the index, and JavaScript rendering sits in the middle where it quietly breaks both. Keep that spine in mind and each section tells you which stage you are repairing.

1. HTTP Status Codes, Redirects, HTTPS and Soft-404s

Start where the crawler starts: the response your server actually returns. Crawl your site with Screaming Frog and open the Response Codes report, then cross-check Google Search Console's Pages report and spot-check individual URLs with curl -I.

You are hunting four things. Broken internal links pointing at 4xx URLs, which leak link equity into dead ends. Soft-404s, where the server returns 200 but the page has no real content, so Google may quietly swap your URL for a better one on the same query. Redirect chains longer than one hop, since every extra hop burns crawl budget and weakens the PageRank that passes through. And 302s used for moves that are actually permanent, which keep Google canonicalizing the old URL and stop link equity from transferring.

The fix is mechanical. Collapse every redirect chain to a single hop (A to C, not A to B to C). Change permanent 302s to 301s. Serve a real 410 for content you deleted on purpose, because 410 drops from the index faster than 404. For HTTPS, confirm every HTTP URL 301-redirects to its HTTPS version and add an HSTS header. This stopped being optional maintenance: Google's Security Blog (October 2025) confirmed Chrome will warn before loading public HTTP sites by default from Chrome 147 in April 2026 for the billion-plus Enhanced Safe Browsing users, and from Chrome 154 in October 2026 for everyone. Stray HTTP URLs become a visible trust problem this year.

2. Robots.txt and Crawl Directives

Robots.txt is the front door. Googlebot reads it before every crawl request, so one wrong line can wall off your whole site. Fetch it with curl -sI https://yourdomain.com/robots.txt and confirm a 200, then open the robots.txt report in Search Console to see the version Google actually cached.

Two failure modes matter most. First, a 5xx on the robots file: Google treats that as a temporary full-disallow and stops crawling, so a flaky robots endpoint is more dangerous than a missing one. For context, the Web Almanac 2024 (HTTP Archive, SEO chapter) found roughly 84% of sites return a clean HTTP 200 for robots.txt, which means a non-200 puts you in a small, broken minority worth investigating. Second, blocking the CSS and JavaScript that Google needs to render the page, which breaks rendering and drops JS-dependent content from the index.

The single most common misuse is treating Disallow as a way to remove a page from Google. It is not. Disallow blocks crawling, not indexing, so a blocked URL that earns an external link can still appear in results as a bare link with no snippet. To actually deindex a page, remove the Disallow, which lets Google crawl and see the tag, and add a noindex directive. Note too that Google ignores Crawl-delay, so anyone relying on it for server load is protecting nothing.

3. Indexability: Noindex, Canonical and GSC Index Coverage

A crawled page is not an indexed page. Run Screaming Frog's Directives tab to list every noindex and X-Robots-Tag, then open Search Console's Pages report and read the "Why pages aren't indexed" breakdown. This is the check most competitors gloss over, because they read the total exclusion count and stop there.

The distinction that matters is between two similar-sounding statuses, and they need opposite fixes.

"Crawled, not indexed" is a content problem. "Discovered, not crawled" is a crawl budget problem. Fix them the same way and you waste your week. Also watch for the trap where a page carries both a noindex and a canonical to another URL: Google ignores the canonical on a noindexed page, so pick one intent.

4. Duplicate Content and Canonical Consolidation

Duplicate handling decides which URL competes and where link equity pools. Crawl with Screaming Frog and check the Canonicals report for self-referencing accuracy, then look in Search Console for "Duplicate, Google chose different canonical than user." That warning means Google overrode your tag, which almost always traces back to a signal conflict.

Google treats rel=canonical as a hint, not a command, and weighs it against your sitemap, internal link counts, and backlinks. So the fix is alignment, not just a tag: point the canonical tag, the sitemap entry, your internal links, and any redirects at the same single URL. When GSC actively reports "Google chose a different canonical than user," diagnose it in order. First check URL Inspection to see which URL Google actually picked, then check which signal it is following: the sitemap entry, the majority of internal links, or external backlinks. Fix that signal to point at your preferred URL before you touch the canonical tag again. Consolidate the obvious duplicates too. Make sure http redirects to https, www and non-www resolve to one host, and trailing-slash variants do not both return 200.

Two specifics for 2026. Pagination no longer uses rel=next/rel=prev, which Google deprecated in 2019, so every paginated page should carry a self-referencing canonical. And as an internal audit heuristic, treat any page that shares the large majority of its wording with another of your pages as a near-duplicate risk: Google will pick one and drop the rest. Programmatic and template pages are the usual offenders, so measure body-content overlap with the boilerplate excluded before you publish at scale.

5. XML Sitemap Health

Your sitemap is the clean list of URLs you want crawled, plus a freshness signal. Fetch it with curl, validate the XML with xmllint --noout sitemap.xml, and check the Sitemaps report in Search Console for parse errors and the submitted-versus-indexed gap.

Respect the hard limit: a single sitemap holds a maximum of 50,000 URLs and 50MB uncompressed (sitemaps.org and Google). Past either ceiling, split into a sitemap index that references child sitemaps by category or content type. Keep the file honest. It should list only canonical URLs that return 200, with no noindex pages, no redirects, and no 404s, because every dirty URL in the sitemap is a crawl instruction Google cannot act on. Make lastmod mean something: update it when the content genuinely changed, not on a CSS tweak, or Google learns to distrust it and crawls the file less often.

When submitted greatly exceeds indexed, do not blame the sitemap first. Filter each URL by its Coverage category. The real cause is usually content quality, a duplicate cluster, or an orphan page with no internal links, and each of those lives in a different section of this checklist.

6. URL Structure, Parameters and Faceted Navigation

Messy URLs multiply the number of pages Google thinks you have. Crawl with Screaming Frog and review the URL and Parameters reports, then check Search Console for a swollen "Discovered, currently not indexed" count, which often signals a URL explosion the crawler cannot keep up with.

Three checks carry the weight. First, fragment routing: a URL like example.com/#/product/123 is invisible to Google, because it never sees the fragment as a server-side URL. Single-page apps must use the History API to serve clean paths like example.com/product/123. Second, low-value parameters. Sort, color, and tracking parameters that do not change the content create duplicate URLs, so point them at the canonical version with a rel=canonical tag. Third, faceted navigation, where filter combinations breed thousands of thin URLs. Canonicalize the combinations that add no unique value, and use robots.txt to disallow the worthless facet permutations outright.

Then standardize the boring details across the whole site: one trailing-slash convention, all-lowercase paths, hyphens instead of underscores (Google reads seo-guide as two words and seo_guide as one), and short keyword-bearing slugs. None of these move rankings on their own, but each variant you leave loose is another duplicate URL splitting your signals.

7. Crawl Budget and Log File Analysis

This is the check that separates a real technical audit from a plugin scan. Search Console coverage tells you what Google says it indexed. Your server logs tell you what Googlebot actually requested, and the two disagree more often than teams expect. Pull the Crawl Stats report in Search Console for the high-level trend, then parse raw access logs with pandas or a tool like Screaming Frog's Log File Analyser for the detail.

The core metric is crawl efficiency: the share of Googlebot requests that returned a 200, calculated as 200-responses divided by total Googlebot requests. As an internal audit heuristic, we flag crawl efficiency below roughly 85% as a sign budget is leaking to low-value URLs, 404s, redirects, and 5xx errors instead of your content. From the same logs, group Googlebot's requests by URL section (/blog/, /product/, /tag/) and look for waste: if tag pages and parameter URLs pull more crawl than your product pages, robots.txt and noindex interventions need to come first. Then sort URLs by daily hit count to catch bot traps, where a single URL gets crawled ten or more times a day because of a session ID, an infinite paginator, or a redirect loop. Verify that the "Googlebot" hits are real with a reverse DNS lookup (host -t PTR <ip> should resolve to googlebot.com), so scrapers spoofing the user-agent do not skew your numbers.

This is the unglamorous discipline behind results, not a growth hack. The same crawl and content rigor drove Pozitif Teknoloji's +225K organic clicks in six months, and it starts with reading the logs instead of guessing.

8. Internal Linking and Site Architecture

Internal links are both the crawler's map and the pipes that move PageRank around your site. Crawl with Screaming Frog and read two reports: Inlinks, to find orphan pages with zero internal links, and Crawl Depth, to find pages buried too far from the homepage.

Fix the structural faults in order. Any page you care about with zero inlinks is orphaned: the crawler cannot discover it through your site and it earns no internal authority, so link to it from a relevant hub. A common rule of thumb keeps important pages within three clicks of the homepage: pages at a click depth of four or more tend to get crawled less often, which delays how fast updates get noticed. Clear every broken internal link (4xx targets), because each one is a dead end that stops a crawl path and blocks equity flow. And keep your critical navigation and hub links as real <a href> elements in the static HTML. Links that only appear after JavaScript runs enter the crawl queue late, if at all, which quietly starves your deeper pages of both discovery and authority.

9. Mobile-First Parity

Google indexes the mobile version of your pages, so anything present only on desktop does not count. Confirm parity with the URL Inspection tool in Search Console, which shows the rendered mobile HTML Googlebot Smartphone actually sees, and back it up with a dual-user-agent crawl in Screaming Frog: crawl once as Googlebot Smartphone, once as Googlebot Desktop, and diff the two.

Check that the same content, the same internal links, and the same structured data appear in the mobile render as on desktop. A heading hidden on mobile does not get indexed. An internal link that only exists on desktop breaks equity flow and can orphan the target. Structured data missing from the mobile HTML makes the page ineligible for the matching rich result. Confirm every template carries a <meta name="viewport" content="width=device-width, initial-scale=1">, because without it mobile rendering breaks and page-experience signals suffer. Parity gaps are almost always template-level, so fix the template once and it propagates across every page built from it.

10. JavaScript Rendering and Render Parity

This is the mechanical bridge to the AI section, so it earns extra attention. Google processes JavaScript in stages: crawl, then a separate render queue, then indexing. On a client-rendered site, the first HTTP response is a near-empty shell and the real content waits in the render queue, which can lag for hours or days. If the raw HTML and the rendered DOM disagree, Google may index a different page than your users see.

Test it directly. Compare the raw HTML against the rendered DOM using Screaming Frog's "rendered page" crawl mode, and confirm against reality with URL Inspection's "View Crawled Page," which shows the HTML Googlebot genuinely rendered. Your H1, meta description, canonical, main body text, and JSON-LD should all be present in the raw HTML, not injected later by script. When they are not, the fix is server-side rendering (SSR) or static site generation (SSG) so the content ships in the first response.

We hit this on our own site. MissionGrowth's marketing site began as a React single-page app, so a crawler that skipped JavaScript received almost nothing. We migrated 20 marketing pages to prerendered static HTML for exactly that reason, and they now return their full text on the first request with no JavaScript executed. You do not have to rebuild your stack to match it, but you do have to serve meaningful HTML without JS, because the next section explains why that gap is far worse for AI engines than it is for Google.

11. Core Web Vitals and Page Experience

Core Web Vitals are a tiebreaker, not a ranking multiplier, and they are measured on real users, not lab tests. Use PageSpeed Insights for field data, the Experience report in Search Console to find "Poor" URL groups, and Chrome DevTools to diagnose a single page. The one rule people forget: a Lighthouse (lab) improvement does not automatically move your field data, which comes from CrUX on a 28-day rolling window, so expect four to eight weeks before a fix shows in Search Console.

The thresholds have not changed for 2026. Google Search Central (updated 2025-12-10) confirms the "good" bar stays at LCP 2.5s or faster, INP under 200ms, and CLS under 0.1, all measured at the 75th percentile of real Chrome users. INP replaced FID as a Core Web Vital in March 2024 (web.dev). TTFB under 800ms (web.dev) is the server-response target that feeds LCP.

One frequent self-inflicted wound: applying loading="lazy" to the hero image. Google confirmed that lazy-loading an above-the-fold image delays its paint and shows up directly in LCP. Set fetchpriority="high" and explicit dimensions on the hero, and reserve lazy-loading for what sits below the fold.

12. Structured Data and Schema

Structured data does not move rankings directly, but valid markup makes your page eligible for rich results, and rich results change click-through at the same rank. Validate every type with Google's Rich Results Test and monitor the Enhancements reports in Search Console for errors and warnings.

Three rules keep schema working. Serve JSON-LD server-side inside the raw HTML, because markup injected by JavaScript (or by Google Tag Manager) enters the render queue and indexes late. Never mark up data that is not visible on the page: a fabricated aggregateRating with no on-page reviews is a spam-policy violation that risks a manual action, and pulling in aggregated reviews from a third-party service is explicitly disallowed. And repeat the markup in your mobile HTML, since mobile-first indexing ignores desktop-only schema.

Schema also earns its place in AI answers. AirOps's "The 2026 State of AI Search" found pages with three or more schema types are about 13% more likely to be cited by AI engines, roughly 61% of cited pages use three or more types, and FAQ schema appears on only 10.5% of cited pages, so it is under-used rather than saturated. The upside compounds with the rest of the checklist. Google's own case study on Saramin reports that fixing crawl errors, consolidating canonicals, and adding JobPosting and Breadcrumb schema doubled organic traffic year over year, lifted signups 93%, and raised conversion 9%. For how the same markup discipline feeds AI citations specifically, see our guide to generative engine optimization.

13. AI Crawler Access: GPTBot, PerplexityBot, Google-Extended and llms.txt

Here the render-parity problem from section 10 becomes a citation problem. Test what an AI crawler sees with a spoofed user-agent: curl -A "GPTBot" https://yourdomain.com/your-page and check whether your key content is in the response. The mechanical fact underneath everything: the major AI crawlers do not execute JavaScript. AppleBot is the exception. Per Vercel's "The rise of the AI crawler" (2024-12-17), December 2024 monthly volumes were Googlebot at 4.5B requests, GPTBot at 569M, ClaudeBot at 370M, AppleBot at 314M, and PerplexityBot at 24.4M. If your content renders client-side, most of the engines feeding AI answers receive an empty shell, the same failure as section 10 but with citations at stake instead of rankings.

The most common misconfiguration is conflating two different decisions. Training crawlers (GPTBot, Google-Extended, ClaudeBot in its training role) feed model weights. Retrieval crawlers (OAI-SearchBot, PerplexityBot) fetch live at query time to build a cited answer. Blocking a retrieval bot kills your citations. Blocking a training bot does not, and may be a deliberate rights decision. Manage them with one blanket rule and you either give away training data you meant to protect or lose the retrieval access you wanted. Two more traps: a nosnippet or max-snippet:0 directive suppresses your page from AI Overviews as well as regular snippets, so check it is intentional; and a WAF or CDN rule (Cloudflare AI bot blocking, for example) can override robots.txt entirely, which means a bot you allowed in robots.txt may still be blocked at the edge.

llms.txt is honest hygiene, not a growth lever. Frame it accordingly. The major AI crawlers rarely fetch it: public llms.txt-adoption monitoring logged only 408 llms.txt fetches across a 500-million AI-bot-request window, Google confirmed in July 2025 it does not use llms.txt as a signal, and top-10k adoption sits around 5.6% in 2026. It is cheap and it will not hurt you. We publish our own llms.txt and a fuller llms-full.txt as a curated plain-text map for AI crawlers, and we built a free llms.txt checker so you can validate yours before you ship. For the full crawler-access playbook and answer-first writing that earns citations, see how to optimize for AI search engines, and for the AI Overviews channel specifically, how to show up in Google AI Overviews.

Your 13-Point Technical SEO Checklist Recap

1. HTTP status, redirects (one hop), HTTPS with HSTS, no soft-404s.
2. Robots.txt returns 200, never blocks CSS/JS, and you use noindex (not Disallow) to deindex.
3. Split "Crawled, not indexed" (quality) from "Discovered, not crawled" (crawl budget).
4. Align canonical tag, sitemap, links, and redirects to one URL.
5. Sitemap under 50,000 URLs and 50MB, canonical 200s only, honest lastmod.
6. Clean URLs, History API routing, canonicalized parameters, disallowed junk facets.
7. Read the logs: watch crawl efficiency, kill bot traps and crawl waste.
8. No orphans, click depth three or less, no broken internal links, links in static HTML.
9. Mobile-first parity for content, links, and schema, with a viewport tag.
10. Render parity: H1, meta, canonical, body, and schema in the raw HTML via SSR or SSG.
11. Core Web Vitals at CrUX p75: LCP 2.5s, INP under 200ms, CLS under 0.1, TTFB under 800ms.
12. Valid server-rendered JSON-LD, no markup for invisible data, mobile parity.
13. AI crawler access decided per bot, retrieval versus training kept separate.

Run the foundation checks (1 through 6) every quarter and after any migration or major deploy. A single bad release can undo three months of cleanup in a week, which is why the log file, not the dashboard, is the last thing you should trust.