How to Edit Shopify Robots.txt File (Best Tutorial for 2022) – In this article I am going to explain what the Shopify robots.txt file is, where in your online store is located and how to hide a page from search engines.
If you want to learn the basics of SEO for Shopify, make sure you check my other articles:
If you need help with the robots.txt file for your Shopify store, feel free to book our Shopify Support package.
The robots.txt file is where you give instructions to Google and other search engine crawlers about what web pages and resources to crawl and what not.
For example, it’s typical for a robots.txt file to instruct web crawlers not to crawl internal admin or login pages.
From the robots.txt file you can make the following edits:
A web crawler, also called spider or search engine bot, is an automated software with the task of discovering and scanning web pages and resources with the purpose of indexing them on a given search engine.
It’s the automated process by a web crawler to fetch a web page or resource with the purpose of indexing it on a given search engine.
The main web crawler by Google is Googlebot. Googlebot refers to two separate crawlers, a desktop and mobile one, simulating the experience of your web page by a user respectively on desktop and mobile devices.
Although not a web crawler, Google Read Aloud is an important user agent by Google. It enables the reading of web pages by using Text-To-Speech (TTS) technology. This agent is only triggered upon request by a user.
To learn more, here’s the list of all Google crawlers or user agents.
Crawling is the automated process by web crawlers of discovering and scanning your web page or resources, while indexing is the process of adding a web page or resources to a search engine database.
The Shopify robots.txt file is located in the root folder of your primary domain.
In a tweet posted on June 16th, Shopify’s CEO Tobi Lutke confirmed it’s now possible to edit the robots.txt file in Shopify.
User-agent: everyone
Allow: /
Starting today, you have complete control over how search engine bots see your store. #shopifyseohttps://t.co/Hz9Ijj5h1y— Tobi Lutke (@tobi) June 18, 2021
To edit your Shopify robots.txt file:
Please note this is an unsupported customisation and Shopify support won’t help you with troubleshooting any potential issue.
You can’t edit your Shopify robots.txt file directly but You can instruct Google spiders about not crawling one or more page:
theme.liquid
layout file.To hide the search template from Google:
theme.liquid
layout file.<head>
section:
To hide a specific page from Google:
theme.liquid
layout file.<head>
section:
Make sure that you replace page-handle-you-want-to-exclude
with the correct page handle.
In Shopify you call a handle the slug of your URL, which as you probably know is the part of the URL net of the domain eg yourdomain.com/[this is the handle / slug] (source: shopify.dev:
a handle is the object’s title in lowercase with any spaces and special characters replaced by hyphens (-)
To learn more, check the Object Handle documentation on the Shopify Developers page.
To hide multiple pages from Google:
theme.liquid
layout file.<head>
section:
Make sure that you replace page-handle-you-want-to-exclude
with the correct page handle.
To hide dynamic pages from Google:
theme.liquid
layout file.<head>
section:
Make sure that you replace your-text
with the correct keyword you want to exclude from indexing.
Note from the author: the solutions above are not about editing your Shopify robots.txt. They are about adding a meta tag called “robots”. This means that your pages are still being crawled by Google spiders. To hide them from Googlebot and similar, the solution below should replace the editing of your Shopify robots.txt.
To hide Shopify pages from web crawlers like Googlebot:
theme.liquid
layout file.<head>
section:
Make sure that you replace current_tags
with the correct tag you have already assigned to the pages you want to exclude from being crawled.
Note from the author: this doesn’t edit your Shopify Robots.txt file but it’s a temporary solution to a question about crawling raised by a visitor I didn’t have an accurate answer to. It’s likely to change as I get more familiar with the topic.
https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt
https://help.shopify.com/en/manual/promoting-marketing/seo/hide-a-page-from-search-engines
https://www.rmehdee.com/blogs/life-experiences-and-hacks/hiding-multiple-shopify-pages-from-search-engines
https://www.sovrn.com/blog/website-crawling-information/
https://www.cloudflare.com/learning/bots/what-is-a-web-crawler/
https://help.outofthesandbox.com/hc/en-us/articles/115007101508-How-can-I-prevent-certain-content-from-being-indexed-by-search-engines-
Featured image of How to Edit Shopify Robots.txt File (Easy Tutorial) from Storynet.com
Pasquale is Founder at Increasily.com, a H2H marketing agency based in Dublin, Ireland, and owner at print-on-demand ecommerce Mintycase.com. Pasquale has worked in Digital Marketing and Account Management since 2004. He currently lives in Dublin with his wife, stepdaughter and cat.
Hi, I see the same explanation on multiple websites, but nobody explains what the correct syntax is when you want to exclude two or more pages. Or where to put this code in the section (before or after the meta-tags)? Thank you!
Hi Steve, and thanks so much for the feedback. I have just added the piece of code you need to the article. I hope it helps. Please keep me posted and thanks for going through the blog post!
Existing Robort.txt file is blocking collection pages and product pages. There is neither the metatag with robort or no index. How can we resolve the issue ? Need Help.
Hi Sandeep, and thanks for the comment. To look into this I would need to know the URL of your website. Generally speaking, the noindex of a page in Shopify is generally an instruction given in the theme.liquid file of your store, as I explain in the article.
Hi, Can you please explain the afterword process? As I am unable to find the txt file in the code. Thanks
1. From your Shopify admin, go to Online Store > Themes.
2.Find the theme you want to edit and then click Actions > Edit code.
3. Click the theme. liquid layout file.
Hi Rana, and thanks for the comment. What are you trying to do? Are you trying to hide a page from Google? If so, you won’t find the code in your theme.liquid file. You will have to add it as in the example. Hopefully that helps?
Hi Pasquale
Thanks for the great article. Could you please confirm what you mean by page handles? We have a site with lots of detailed filters, leading to thousands of URLs that contain either an + or _ character. The robots.txt file directs Google not to crawl those with the + character, but not those with the _ character and i’d like to ensure that this is handled. So my question is, does the above work with pattern matching?
eg is there a way to block /collections/shoes/colour_red and /collections/shoes/size_8 without at the same time blocking /collections/shoes/?
Note that the above are created dynamically and as i said, there are many thousands.
Secondly, what you’ve provided is how to access the meta robots tags, which manage indexation, not the robots.txt file which manages crawling. In our case i’d like to prevent the crawling of these pages, as well as their indexation, as they are using up crawl budget and preventing legitimate pages from being indexed.
So is there a way to do this, which can only be done using a Disallow directive in the robots.txt file?
Thanks very much
Garry
This is a nice tutorial. Well explanation about edit Shopify robots.txt file. Now I know how to edit the robots.txt file. Thanks for sharing this informative article.
Pasquale Mellone