There is a tiny file sitting at the root of most websites that most small business owners have never heard of. It is called robots.txt, and while it sounds technical, understanding it takes about five minutes. Getting it right can quietly improve your SEO. Getting it wrong can accidentally hide your site from Google entirely.
This guide explains what robots.txt does, why it matters, and how to create or check your own without touching a single line of code.
What Is a Robots.txt File?
A robots.txt file is a plain text file that lives at the very root of your website, meaning it sits at a web address like yourdomain.co.uk/robots.txt. Its job is simple: it tells search engine crawlers (the automated bots that Google, Bing, and others send out to read websites) which pages or sections of your site they are allowed to visit, and which they should skip.
Think of it as a polite set of instructions posted at the front door of your website. Most well-behaved crawlers, including Googlebot, will follow those instructions. It does not lock anything away securely, but it does direct traffic efficiently.
According to Google Search Central, one of the main reasons to use robots.txt is to avoid wasting crawl budget on pages that are not worth indexing, such as admin areas, thank-you pages, or duplicate content. For a small business site, that can make a real difference to how quickly Google discovers and ranks your important pages.
Why Does It Matter for SEO?
Search engines have a limited amount of time and resource they spend crawling any given site. This is sometimes called a crawl budget. If Google is busy crawling your login page, your cart confirmation page, and a dozen other pages that have no search value, it may spend less time indexing the pages that actually matter, your services, your homepage, your blog posts.
A well-configured robots.txt file steers crawlers away from the dead-end pages and points them towards the content you want to rank. It also keeps your site tidy from an SEO standpoint, reducing the chance of duplicate or thin content getting indexed unintentionally.
There is a common mistake worth flagging here: if you block a page in robots.txt, Google cannot crawl it, but that does not always mean it will disappear from search results. If other sites link to a blocked page, Google may still list it with no description. If you want a page fully excluded from search results, you need a noindex meta tag on that page as well. Robots.txt and noindex work together, not as substitutes for each other.
What Does a Basic Robots.txt File Look Like?
Here is the most common setup for a small business website that wants all its public pages crawled normally:
- User-agent: * (this means the rules apply to all crawlers)
- Disallow: (leaving this blank tells crawlers they can access everything)
- Sitemap: https://yourdomain.co.uk/sitemap.xml (point crawlers to your sitemap)
That is it. Three lines. For many small business sites, this is genuinely all you need. It tells every crawler: come on in, look at everything, and here is the map to help you find it all.
If you have sections you want to block, the syntax looks like this:
- User-agent: *
- Disallow: /wp-admin/ (blocks the WordPress admin area)
- Disallow: /thank-you/ (blocks a post-form thank you page)
- Sitemap: https://yourdomain.co.uk/sitemap.xml
Each Disallow line blocks a specific folder or page path. The forward slash before the folder name is important, so double-check you include it.
Pages Worth Blocking for a Typical Small Business Site
Every site is different, but these are the kinds of pages that are generally worth keeping out of search engine indexes:
- Admin and login pages (for example, /wp-admin/ or /login/)
- Cart and checkout pages on e-commerce sites
- Thank-you or confirmation pages after a form submission
- Search results pages generated by an internal site search
- Staging or test versions of your site (these should really have their own robots.txt that blocks everything)
- Any page that is essentially a duplicate of another page
Your main service pages, your homepage, your about page, your blog, your contact page: all of these should be fully accessible to crawlers. If you are a trades business or a service company trying to rank locally, you absolutely want Google crawling every page that describes what you do and where you do it.
How to Create or Edit Your Robots.txt File
If your site is built on WordPress, the easiest way to manage your robots.txt file is through an SEO plugin. Both Yoast SEO and Rank Math have a built-in robots.txt editor under their tools sections. You do not need to touch any files directly. Just open the editor, make your changes, and save.
If you are on a platform like Wix, Squarespace, or Shopify, each has its own way of handling robots.txt. Shopify, for instance, generates one automatically but allows customisation in the theme files. Wix gives you a robots.txt editor in your site settings. Check your platform's help documentation for the specific steps.
If you are comfortable using FTP or your hosting control panel's file manager, you can also create a plain text file called robots.txt and upload it to the root directory of your site. Just make sure it is saved as plain text, not as a Word document or anything else that adds hidden formatting.
How to Check If Your Robots.txt Is Working
Once your file is in place, you can view it yourself by typing your domain followed by /robots.txt into a browser. You should see the plain text of your file.
To test whether specific pages are being blocked correctly, use Google Search Console. Under the Settings section, there is a robots.txt report that shows your current file and flags any errors. There is also a URL Inspection tool that lets you check whether a specific page is accessible to Googlebot.
It is worth checking your robots.txt whenever you launch a new site, after a major redesign, or after switching platforms. Migrations often reset or overwrite robots.txt files, and a misconfigured file after a site move is one of the most common causes of sudden ranking drops.
One Rule to Remember Above All Others
Never, ever launch a live site with Disallow: / in the robots.txt file. This single line tells all crawlers to stay away from your entire site. It is a setting that is often turned on by default during development (to stop Google indexing an unfinished site) and then forgotten when the site goes live.
If your site has been live for a while and you cannot figure out why nothing is ranking, this is one of the first things to check. Go to yourdomain.co.uk/robots.txt and make sure you do not see a blanket Disallow directive blocking everything.
Robots.txt is a small file with a big job. Get the basics right, check it regularly, and you remove one more potential obstacle between your website and the customers searching for what you offer.
