Category: Technical SEO

A canonical tag, <link rel="canonical" href="...">, tells search engines which URL is the main version of a page when the same or very similar content is available at several addresses. It consolidates ranking signals into one URL and keeps duplicates out of search results.

Why duplicates happen

• URL parameters for tracking, sorting or filtering, such as ?utm_source= or ?sort=price.
• HTTP and HTTPS, or www and non-www versions of the same page.
• Trailing slash and non-trailing slash versions.
• The same product in several categories of an online store.
• Content syndicated or republished on other websites.

How to use canonical tags

• Self-referencing canonical: every indexable page should point to its own clean URL. It protects you from parameter duplicates you did not foresee.
• Point duplicates to the main version: a filtered category page can point to the unfiltered category.
• Use absolute URLs, including https and the domain.
• Cross-domain canonical: when you republish an article on another site, the copy can point to the original.

A hint, not a command

Google treats the canonical tag as a strong signal, not an order. It also looks at redirects, internal links, sitemaps and HTTPS. If your signals contradict each other, for example internal links point to one version and the canonical to another, Google may choose a different canonical. The URL Inspection tool in Search Console shows which one Google selected.

Common mistakes

• Canonicals pointing to URLs that redirect, return a 404 or are noindex.
• Several canonical tags on the same page with different URLs.
• Pointing every page of a paginated series to page one, which hides the content of the other pages.
• Using canonical and noindex on the same page, which sends contradictory signals.
• Canonical tags added by JavaScript only, which Google may process later or not at all.

Check the canonical, noindex and other indexing signals of any page with our SEO Audit Tool.

Frequently asked questions

Canonical or 301 redirect?

Use a 301 when users do not need the duplicate URL at all. Use a canonical when the duplicate must stay available, such as a filtered view or a tracking URL.

Does a canonical tag pass link value?

Yes. Google consolidates signals from the duplicates into the canonical URL, much like a redirect.

Can I set a canonical in HTTP headers?

Yes. A Link header with rel=”canonical” works for files without HTML, such as PDFs.

An XML sitemap is a file that lists the URLs you want search engines to crawl and index. It does not guarantee indexing, but it helps search engines discover new and updated pages faster, especially on large sites, new sites and sites with weak internal linking.

What a sitemap looks like

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/blog/my-post/</loc>
    <lastmod>2026-10-15</lastmod>
  </url>
</urlset>

Only loc is required. Google uses lastmod when it is accurate and ignores priority and changefreq.

What to include and what to leave out

• Include canonical URLs that return a 200 status and that you want in search results.
• Leave out redirects, 404s, noindex pages, URLs blocked by robots.txt, duplicates with parameters and non-canonical versions.
• A sitemap full of URLs that Google cannot index sends mixed signals and wastes crawl time.

Limits and sitemap indexes

Each sitemap file can contain up to 50,000 URLs and weigh up to 50 MB uncompressed. Larger sites split their URLs into several sitemaps and list them in a sitemap index, which is what WordPress SEO plugins do with sitemap_index.xml: one sitemap for posts, another for pages, another for categories and so on.

Accurate lastmod dates

Update lastmod only when the content of a page really changes. If every page shows today’s date every day, Google learns to ignore the field.

How to submit a sitemap

1. Add a Sitemap: line to your robots.txt so every search engine can find it.
2. Submit it in Google Search Console, under Sitemaps, and in Bing Webmaster Tools.
3. Check the report a few days later: it shows how many URLs were discovered and any errors.

Validate any sitemap and find common problems with our XML Sitemap Validator.

Other sitemap types

• Image sitemaps help Google discover images, useful when images load with JavaScript.
• Video sitemaps add details such as duration and thumbnail.
• News sitemaps are for publishers in Google News and should only include articles from the last two days.

Frequently asked questions

Does a sitemap improve rankings?

No. It helps search engines find and recrawl pages, which can speed up indexing, but it does not make a page rank higher.

Do small sites need a sitemap?

It is not essential if your internal linking is good, but it costs nothing and WordPress creates one automatically.

Why does Search Console say ‘Discovered, currently not indexed’?

Google knows the URL but has not crawled or indexed it yet, often because it considers the page low priority. Improve internal links and content quality.

A redirect sends users and search engines from one URL to another. The status code you choose tells search engines whether the move is permanent or temporary, and that decides which URL stays in the index. Choosing the wrong one rarely breaks a site, but it can keep the wrong URL in Google for months.

The redirect codes

• 301 Moved Permanently: the page has moved for good. Google replaces the old URL with the new one and transfers ranking signals.
• 308 Permanent Redirect: the same as 301, but the browser must keep the request method (for example a form POST). Google treats it like a 301.
• 302 Found: a temporary move. Google keeps the original URL indexed, as long as the redirect looks temporary.
• 307 Temporary Redirect: the temporary version that keeps the request method. Treated like a 302.

When to use each one

• 301 for changed URLs, merged pages, a move to HTTPS, a new domain or a site migration.
• 302 for short-term changes: a page under maintenance, a temporary promotion, an A/B test or a redirect based on the user’s location.
• If you are not sure, ask yourself whether the old URL will ever come back. If not, use a 301.

Do redirects lose PageRank?

Google has stated that 301, 302 and 308 redirects do not lose PageRank. The practical difference is which URL Google shows: with a 301 it is the new one, with a 302 it usually keeps the old one. If a 302 stays in place for a long time, Google may start treating it as permanent.

Redirect chains and loops

A chain is a sequence of redirects, such as http://example.com to https://example.com to https://www.example.com. Each hop adds latency, and Googlebot follows a limited number of hops. A loop is a chain that never ends. Both are common after migrations.

• Redirect every old URL straight to its final destination in one hop.
• Update internal links so they point to final URLs, not to redirected ones.
• After a migration, keep the old redirects for at least a year, and ideally forever if the old URLs have backlinks.

See every hop and its status code with our Redirect Checker, and check the final page with the HTTP Header Checker.

How to create redirects

• WordPress: plugins such as Redirection or the redirect module in Rank Math.
• Apache: rules in the .htaccess file.
• Nginx: return or rewrite directives in the server configuration.
• CDN: redirect rules in Cloudflare or similar services, which run before the request reaches your server.

Frequently asked questions

Is a meta refresh or JavaScript redirect OK for SEO?

Google can follow them, but they are slower and less reliable than server-side 301 redirects. Use HTTP redirects whenever you can.

Should I redirect deleted pages to the home page?

No. Redirect them to the closest relevant page. If there is none, a 404 or 410 is better: Google treats mass redirects to the home page as soft 404s.

How long should I keep 301 redirects?

At least one year after a migration. Keep them longer if the old URLs still receive traffic or backlinks.

robots.txt is a plain text file at the root of your website, at yourdomain.com/robots.txt, that tells crawlers which parts of the site they may visit. It is one of the oldest web standards, formalized as RFC 9309 in 2022, and it is now also the main way to control AI crawlers. A small mistake in this file can hide an entire site from Google, so it is worth getting right.

What robots.txt does and does not do

• It controls crawling: which URLs a bot may request.
• It does not control indexing: a blocked URL can still appear in Google if other pages link to it, usually without a description. To keep a page out of search results, use a noindex tag and let it be crawled.
• It is a public request, not security: anyone can read your robots.txt, and bad bots ignore it. Never use it to hide private content.

The syntax

A robots.txt file is made of groups. Each group starts with one or more User-agent lines and contains Allow and Disallow rules. A Sitemap line can go anywhere.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap_index.xml

• User-agent: the bot the group applies to. * means every bot without a group of its own.
• Disallow: a path prefix the bot should not crawl. An empty Disallow allows everything.
• Allow: an exception inside a disallowed path.
• * in a path matches any characters, and $ marks the end of the URL.
• Paths are case sensitive; user agent names are not.

How Google decides which rule applies

• A bot follows only the most specific group that names it. If GPTBot has its own group, it ignores the * group completely.
• Inside that group, the longest matching rule wins, regardless of order.
• If an Allow and a Disallow rule have the same length, Allow wins.
• If no rule matches, the URL is allowed.

Test any URL against your rules, and see which line decides, with our robots.txt tester and generator.

Examples

A typical WordPress site

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/sitemap_index.xml

An online store

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?orderby=
Disallow: /*?filter_

Sitemap: https://example.com/sitemap_index.xml

Blocking sorting and filter parameters saves crawl budget on large catalogs, but make sure your category pages themselves stay crawlable.

Robots.txt and AI bots

AI companies use separate crawlers for training models and for AI search. You can block one and allow the other. Read should you block AI bots? for ready-made rules, and check what your file allows today with the AI Bot Checker.

Common mistakes

• Leaving Disallow: / from a staging site after launch. It blocks the whole site.
• Blocking CSS and JavaScript, which Google needs to render pages.
• Blocking pages you want to noindex: Google cannot see the noindex tag if it cannot crawl the page.
• Adding a group for one bot and forgetting that it no longer follows the general rules.
• Serving robots.txt with an error. If the file returns a 5xx error for a long time, Google may stop crawling the site. A 404 is fine: it means everything is allowed.

Frequently asked questions

Do I need a robots.txt file?

No. Without one, every crawler can access everything, which is fine for many small sites. It becomes useful when you need to keep bots out of some sections or control AI crawlers.

How long does Google take to see changes?

Google usually refreshes its cached copy of robots.txt within 24 hours.

Where do I edit robots.txt in WordPress?

WordPress generates a virtual robots.txt. SEO plugins such as Rank Math or Yoast let you edit it, unless a physical robots.txt file exists in the root folder, which always takes priority.