Robots.txt: How to Find, Read & Edit Yours (NZ Guide)

One wrong line in a robots.txt file can wipe a website out of Google overnight. It usually happens by accident — a developer pushes a staging site live, forgets to remove a single rule, and three weeks later the rankings are gone.

Most people meet robots.txt for the first time through a scary warning in Search Console, or they never look at it until something breaks. 

That’s a problem, because this tiny text file is one of the few things on your site powerful enough to make every page disappear from search.

This guide covers what robots.txt is, whether you actually need one, how to find and read yours, how to edit it on WordPress, Shopify, Wix and Squarespace, and the mistakes that quietly cost rankings. 

It also answers the question everyone’s asking in 2026: can you use it to block AI crawlers like ChatGPT?

What a robots.txt file is

A robots.txt file is a plain text file in your website’s root that tells search engine crawlers which parts of your site they’re allowed to request. It’s the first thing most bots check when they arrive.

That’s the whole job. 

It sits at one fixed address — yourdomain.co.nz/robots.txt — and any crawler that visits reads it before doing anything else. 

Think of it like the sign at the front door of a building. It tells visitors which rooms are open and which are off-limits. Polite visitors follow it. The sign itself doesn’t lock any doors.

The file is part of the Robots Exclusion Protocol, which has been around since 1994 and was formally standardised as RFC 9309 in 2022. 

One file governs your whole site, and it only works from the root. A robots.txt sitting in a subfolder does nothing.

Understanding this file is a core part of the technical side of SEO — the behind-the-scenes settings that decide whether search engines can read your site properly in the first place.

What robots.txt is actually used for

Robots.txt exists to manage crawler traffic and protect your server from being hammered — not to control what shows up in search results. Confusing those two jobs is where most SEO damage starts.

Here’s what it’s genuinely good for:

• Managing crawl budget. Large sites have a limited number of pages Google will crawl in a given window. Blocking low-value areas (internal search results, filtered product URLs, cart pages) keeps crawlers focused on the pages that matter. For a builder in Hamilton running a 30-page site, this is rarely a concern. For an ecommerce store with 50,000 URL variations, it matters a lot.
• Keeping bots out of pointless areas. Admin folders, thank-you pages, staging directories — places that add nothing to search.
• Pointing crawlers to your sitemap. A single Sitemap: line tells every crawler exactly where your XML sitemap lives, which helps them find your content faster.

What it is not for is hiding pages from Google or protecting private information. We’ll get to why that matters, because it’s the single biggest robots.txt mistake there is.

Does robots.txt actually work?

Robots.txt works — but only because the major crawlers choose to obey it. There’s no technical force behind it. It’s a request, not a lock.

Googlebot, Bingbot and the other mainstream crawlers respect it because it’s in their interest to. 

Scrapers, spam bots and malicious crawlers ignore it completely. So the file is perfect for telling Google “don’t waste time in here,” and useless for keeping bad actors out of anything.

This voluntary nature is becoming a live issue. One 2025 analysis found that 13.26% of AI bot requests ignored robots.txt directives in Q2 2025, up from just 3.3% the previous year. 

The rules still work for the bots that follow them. They just aren’t enforceable.

Do you actually need a robots.txt file?

If you run a small WordPress or Shopify site, you almost certainly already have a working robots.txt and may never need to touch it. You only need a custom one when you’ve got specific areas to block, or you want to declare your sitemap.

This is where plenty of guides oversell the work. Most small NZ business sites run fine on the default file their platform generates. WordPress creates a virtual one automatically. 

Shopify ships with a sensible default. If you’ve got a tidy 20-page site for a dental practice in Auckland, there’s a good chance you never need to open the file at all.

You do need to think about it when:

• You’re running a large site or ecommerce store with thousands of filtered or faceted URLs eating crawl budget.
• You’ve got a staging or development area you don’t want crawled.
• You want to declare your sitemap location for faster discovery.
• You need to keep crawlers out of specific folders (internal search, certain admin paths).

If you have no robots.txt at all, crawlers assume full access and crawl everything. For most small sites that’s completely fine — the file only earns its place when you have something specific to say.

How to find and check your robots.txt

To see any website’s robots.txt, type the domain followed by /robots.txt into your browser. Your own works exactly the same way.

Check your own site

Type your domain and add /robots.txt on the end — for example, https://yourbusiness.co.nz/robots.txt. Whatever loads is the live file crawlers are reading right now. If you get a 404, you don’t have one, which (as covered above) is often fine.

Check a competitor’s

The same trick works on any site, which makes it a quick recon move. Add /robots.txt to a competitor’s domain and you’ll often spot which sections they’re blocking and where their sitemap lives — useful intel that’s sitting in the open.

View it in Google Search Console

Google’s robots.txt report shows you the exact file Google last fetched, when it fetched it, and any errors it found. 

This is the version that actually matters, because it’s what Google sees — not what you think you uploaded. If you ever run a full site audit, checking this report is one of the first things worth doing.

How to read a robots.txt file

Every robots.txt is built from a handful of simple rules. Learn four — User-agent, Disallow, Allow and Sitemap — and you can read almost any file you’ll come across.

Here’s a typical WordPress file with each part labelled:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourbusiness.co.nz/sitemap_index.xml

User-agent names the bot a rule applies to. The asterisk (*) is a wildcard meaning “all bots.” You can also target a specific crawler — User-agent: Googlebot applies only to Google.

Disallow tells the bot not to crawl a path. Disallow: /wp-admin/ blocks the WordPress admin folder. A bare Disallow: / blocks the entire site, which is exactly as dangerous as it sounds.

Allow overrides a Disallow for a specific path. In the example above, the whole /wp-admin/ folder is blocked except for admin-ajax.php, which WordPress needs to function.

Sitemap points crawlers straight to your XML sitemap. It’s optional but worth including on every site.

Two more you’ll see: the * wildcard can match patterns inside a path (Disallow: /*? blocks URLs containing a question mark), the $ symbol marks the end of a URL, and any line starting with # is a comment the bots ignore. That covers about 95% of real-world files.

Robots.txt vs noindex: the mistake that kills rankings

To keep a page out of Google, you need a noindex tag — not a Disallow in robots.txt. 

Block a page in robots.txt and Google can still index the URL without reading it. That’s exactly what the “indexed, though blocked by robots.txt” warning means.

This trips up more people than anything else, so it’s worth slowing down on.

Crawling and indexing are two different things. Crawling is a bot reading the page. Indexing is Google adding the page to its search results. Robots.txt only controls crawling. It says nothing about indexing.

Here’s the trap.

 You want a page kept out of Google, so you add Disallow: /private-page/ to robots.txt. Google obeys — it stops crawling the page. 

But it can still see links pointing to that URL from other pages, so it indexes the URL anyway, often with a useless grey snippet reading “No information is available for this page.” You’ve made it more visible, not less.

It gets worse. If you block the page in robots.txt, Google can’t crawl it — which means it can’t see a noindex tag even if you’ve added one. The block hides the very instruction that would remove the page. The page stays stuck in the index.

The fix is straightforward once you know the rule:

• To stop a page being indexed: use a noindex meta tag in the page’s HTML, and let Google crawl it so it can read that tag. Don’t block it in robots.txt.
• To stop a page being crawled (to save crawl budget): use robots.txt.

If you want a page gone from Google for good, the meta robots tag is the tool. Google stopped supporting noindex inside robots.txt on 1 September 2019, so any old Noindex: lines in a robots.txt file do nothing today. 

Getting this distinction wrong is one of the most common mistakes that quietly tanks otherwise healthy sites.

How to create or edit robots.txt on your platform

You rarely edit robots.txt by hand anymore. On most NZ sites it’s controlled through your CMS or SEO plugin. Here’s exactly where to find it on each platform.

WordPress with Rank Math

In your WordPress dashboard, go to Rank Math → General Settings → Edit robots.txt. 

You’ll see a text box with the current rules. Edit them there, save, and Rank Math writes the file for you. This is the cleanest way to manage it on a WordPress site, and it’s the setup most well-run WordPress SEO builds use.

WordPress with Yoast

If you use Yoast instead, go to Yoast SEO → Tools → File editor. You’ll find a robots.txt editor there with the same kind of text box. Same idea, different plugin.

The virtual vs physical file gotcha

WordPress generates a virtual robots.txt by default — one that exists only when a crawler asks for it. 

The moment you create a physical robots.txt file on your server (or one gets uploaded), it overrides the virtual one and your plugin settings stop applying. 

If your edits in Rank Math or Yoast aren’t showing up, a stray physical file is usually the reason. Check your server’s root folder via FTP or your host’s file manager and delete any robots.txt sitting there.

Shopify

Shopify generates its robots.txt automatically and, for years, you couldn’t touch it. 

Now you can, through a file called robots.txt.liquid. In your Shopify admin, go to Online Store → Themes → Edit code, then add a robots.txt.liquid template. 

This lets you add or remove rules, but Shopify still controls the defaults, so you’re customising rather than starting from scratch. 

Most stores never need to. If you’re running a store, broader Shopify SEO wins usually matter far more than tweaking this file.

Wix and Squarespace

Wix generates a robots.txt automatically and lets you edit it from your SEO settings under Marketing & SEO → SEO Tools → Robots.txt Editor. 

It’s a more locked-down experience than WordPress, which is normal for Wix SEO. Squarespace handles robots.txt entirely on its own and doesn’t let you edit it — the platform manages crawling for you, which is fine for the kind of brochure sites it’s built for.

Copy-paste robots.txt templates

Here are the robots.txt files you’ll actually use, ready to copy. Pick the one that matches your situation and swap in your own domain.

Allow everything (the safe default for most sites):

User-agent: *
Disallow:

Sitemap: https://yourbusiness.co.nz/sitemap_index.xml

An empty Disallow: means “nothing is blocked.” This is the right starting point for the vast majority of NZ business sites.

A sensible WordPress file:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourbusiness.co.nz/sitemap_index.xml

This blocks the admin area while keeping the one file WordPress needs to run, and declares your sitemap.

Block a single folder:

User-agent: *
Disallow: /private-folder/

Sitemap: https://yourbusiness.co.nz/sitemap_index.xml

Block the entire site (staging only — never on a live site):

User-agent: *
Disallow: /

That bare Disallow: / blocks everything. It belongs on development and staging sites only. Push it live by accident and you’ll deindex your whole website, which brings us to the mistakes section.

Whatever file you use, always include the Sitemap: line. It costs nothing and helps crawlers find your content.

Can robots.txt stop AI crawlers?

You can block the major AI crawlers in robots.txt — GPTBot, Google-Extended, ClaudeBot, CCBot and PerplexityBot — but compliance is voluntary, and blocking them can cost you visibility in AI answers. It’s a genuine strategic decision, not a default setting.

This is the part most older guides skip, because most of them were written before AI crawling took off. It’s now one of the most active areas in SEO.

The AI crawlers worth knowing

Each AI company runs its own named bot, and they fall into two camps:

• Training crawlers collect content to train models. These include GPTBot (OpenAI), Google-Extended (Gemini training), ClaudeBot (Anthropic) and CCBot (Common Crawl, which feeds many AI datasets). Blocking these stops your content being used as training data.
• Retrieval crawlers fetch content live to answer a user’s question, such as PerplexityBot and OpenAI’s search bot. Blocking these can stop you being cited in AI answers.

How to block or allow them

You target each bot by name. To block OpenAI’s training crawler:

User-agent: GPTBot
Disallow: /

To block several at once, add a block for each user-agent. To allow them, simply don’t disallow them — the default is open access.

Should you actually block them?

This is the real decision, and there’s no universal right answer. Blocking AI crawlers protects your content from being used without attribution. 

Allowing them keeps you eligible to be cited in ChatGPT, Perplexity and Google’s AI answers — which is fast becoming a genuine traffic and authority source.

The data shows where sentiment is heading. As of mid-2025, around 21% of the top 1,000 websites had GPTBot rules in their robots.txt, and that share has climbed sharply since AI crawling began in 2023.

News publishers in particular are blocking aggressively to protect their work.

For most NZ businesses, though, the maths runs the other way. Being cited by AI is closer to free advertising than theft. 

We let AI crawlers through on our own site, and it’s part of why our SEO company keeps showing up in AI Overviews and ChatGPT answers for SEO queries — visibility that’s getting harder to win through blue links alone. 

If you’re actively working on ranking in AI search, blocking the bots that surface your content is the opposite of what you want.

One honest caveat: robots.txt is voluntary, so a block is a request rather than a guarantee. 

Ignoring it breaks no law — the standard carries no legal force — and some crawlers ignore it anyway. If you genuinely need to keep content out of AI systems, robots.txt alone won’t do it.

Common robots.txt mistakes that tank your SEO

Most robots.txt disasters come down to the same handful of mistakes. Here’s what they look like and how to avoid each.

The accidental site-wide block. Picture a law firm in Wellington launching a redesigned website. 

The developer built it on a staging URL with Disallow: / to keep it out of Google during the build — and when the new site goes live, that one line comes with it. 

Google obediently stops crawling everything, and within weeks the rankings collapse. 

This is the most expensive robots.txt mistake there is, and it happens constantly during site migrations and redesigns. After any launch, the first thing to check is that your live robots.txt doesn’t contain a bare Disallow: /.

Blocking CSS and JavaScript. Some older setups block the folders holding CSS and JS files. 

Google needs those files to render your page the way a visitor sees it. Block them and Google sees a broken, half-loaded page, which can hurt rankings. Let crawlers reach your CSS and JS.

Using robots.txt for security. Your robots.txt is public — anyone can read it at /robots.txt. List Disallow: /admin/ or Disallow: /customer-data/ and you’ve handed every attacker a map straight to the sensitive parts of your site. 

Never use robots.txt to hide anything private. Use proper authentication, password protection, or noindex tags depending on the goal.

Trailing slash and case errors. Paths are case-sensitive and precise. Disallow: /Folder/ and Disallow: /folder/ are two different rules. A misplaced or missing slash can block far more, or far less, than you intended.

Blocking a page you also want noindexed. As covered earlier, blocking a page in robots.txt stops Google seeing its noindex tag, so the page stays in the index. If you want a page gone, let it be crawled and use a meta noindex.

How to test and validate your robots.txt

Before you trust a robots.txt change, test it. Two free tools tell you instantly whether you’ve blocked something you didn’t mean to.

First, Google Search Console’s robots.txt report shows the live file Google has fetched and flags any syntax errors. It’s the authoritative view, because it’s Google’s own.

Second, a third-party validator like TechnicalSEO.com’s robots.txt tester lets you paste a URL and check whether a specific page is blocked, and by which rule. 

This is the fastest way to confirm that a rule you’ve added blocks what you intended and nothing more.

The habit that saves you: test before you publish a change, and test again straight after it goes live. Most catastrophic robots.txt mistakes would have been caught by a 30-second check.

When to handle robots.txt yourself vs hire an SEO

For a small site, editing robots.txt is genuinely a do-it-yourself job. It’s worth bringing in help when pages are already mis-indexed, the site is large or ecommerce, or a migration is involved.

If you run a tidy WordPress site and you just want to add a sitemap line or block your admin folder, do it yourself. Open Rank Math, make the change, test it, done. There’s no need to pay anyone for that.

It’s worth getting help when:

• Pages are already showing the “indexed, though blocked” warning and you’re not sure why.
• You’re running a large or ecommerce site where crawl budget genuinely matters.
• You’re going through a site migration or redesign, where a stray rule can do real damage.
• Something’s already broken and you can’t work out which rule is causing it.

In those cases the cost of a mistake is high, and a second set of eyes pays for itself. 

If you’d rather hand the whole technical side off, that’s what done-for-you SEO services is for. 

And if you just want someone to check whether your setup is holding you back before you commit to anything, a short engagement with an SEO consultant will tell you where you stand.

Robots.txt: FAQs