- Email:[email protected]
- phone: +353 1 960 9526
21 Jul
How to Edit Shopify Robots.txt File – In this article I am going to explain what the Shopify robots.txt file is, where in your online store is located and how to hide a page from search engines.
If you want to learn the basics of SEO for Shopify, make sure you check my other articles:
- Optimise Shopify Page Title and Description
- How to Add Shopify Alt Tags to Theme Images
- How to add Alt Tags to Shopify Product Images
- How to Add Google Analytics to Shopify
- How to Add SEO Headings in Shopify
- How to Optimise Your Shopify Site Structure
- How to Find and Submit Your Shopify Sitemap
- How to Verify Your Shopify Store with Google
- How to Create a Redirect in Shopify
- Duplicate Content and Shopify
- Link Building and Shopify
- How to Compress Images on Shopify
- How to Improve Shopify Page Load Speed
- Optimise Your Shopify Blog Posts
- How to Backup Your Shopify Store
- Enable AMP in Shopify
If you need help with ranking your website on Google, feel free to check our Shopify SEO services.
What is the Robots.txt File?
The robots.txt file is where you give instructions to Google and other search engine crawlers about what web pages and resources to crawl and what not.
For example, it’s typical for a robots.txt file to instruct web crawlers not to crawl internal admin or login pages.
What Edits Can I Make from the Robots.txt File?
From the robots.txt file you can make the following edits:
- allow or disallow certain URLs from being crawled
- add crawl-delay rules for certain crawlers
- add extra sitemap URLs
- block certain crawlers
What is a Web Crawler?
A web crawler, also called spider or search engine bot, is an automated software with the task of discovering and scanning web pages and resources with the purpose of indexing them on a given search engine.
What is the Crawling of a Web Page?
It’s the automated process by a web crawler to fetch a web page or resource with the purpose of indexing it on a given search engine.
What Are Google’s Main Web Crawlers?
The main web crawler by Google is Googlebot. Googlebot refers to two separate crawlers, a desktop and mobile one, simulating the experience of your web page by a user respectively on desktop and mobile devices.
Although not a web crawler, Google Read Aloud is an important user agent by Google. It enables the reading of web pages by using Text-To-Speech (TTS) technology. This agent is only triggered upon request by a user.
To learn more, here’s the list of all Google crawlers or user agents.
Crawling vs Indexing of a Web Page
Crawling is the automated process by web crawlers of discovering and scanning your web page or resources, while indexing is the process of adding a web page or resources to a search engine database.
Where is the Shopify Robots.txt File Located?
The Shopify robots.txt file is located in the root folder of your primary domain.
In a tweet posted on June 16th, Shopify’s CEO Tobi Lutke confirmed it’s now possible to edit the robots.txt file in Shopify.
User-agent: everyone
Allow: /
Starting today, you have complete control over how search engine bots see your store. #shopifyseohttps://t.co/Hz9Ijj5h1y— Tobi Lutke (@tobi) June 18, 2021
How Do I Edit the Shopify Robots.txt File?
To edit your Shopify robots.txt file:
- From your Shopify admin, go to Online Store > Themes.
- Click Actions, and then click Edit Code.
- Click Add a new template, and then select robots.
- Click Create template.
- Make the changes that you want to make to the default template
- Save changes to the robots.txt.liquid file in your published theme
Please note this is an unsupported customisation and Shopify support won’t help you with troubleshooting any potential issue.
How Do I Hide Shopify Pages from Google?
You can’t edit your Shopify robots.txt file directly but You can instruct Google spiders about not crawling one or more page:
- From your Shopify admin, go to Online Store > Themes.
- Find the theme you want to edit, and then click Actions > Edit code.
- Click the
theme.liquidlayout file.
How to Hide Shopify Search Template from Google
To hide the search template from Google:
- From your Shopify admin, go to Online Store > Themes.
- Find the theme you want to edit, and then click Actions > Edit code.
- Click the
theme.liquidlayout file. - Paste the following code in the
<head>section:
How to Hide a Specific Shopify Page from Google
To hide a specific page from Google:
- From your Shopify admin, go to Online Store > Themes.
- Find the theme you want to edit, and then click Actions > Edit code.
- Click the
theme.liquidlayout file. - Paste the following code in the
<head>section:
Make sure that you replace page-handle-you-want-to-exclude with the correct page handle.
- Click Save.
What is a Page Handle?
In Shopify you call a handle the slug of your URL, which as you probably know is the part of the URL net of the domain eg yourdomain.com/[this is the handle / slug] (source: shopify.dev:
a handle is the object’s title in lowercase with any spaces and special characters replaced by hyphens (-)
To learn more, check the Object Handle documentation on the Shopify Developers page.
How to Hide Multiple Shopify Pages from Google
To hide multiple pages from Google:
- From your Shopify admin, go to Online Store > Themes.
- Find the theme you want to edit, and then click Actions > Edit code.
- Click the
theme.liquidlayout file. - Paste the following code in the
<head>section:
Make sure that you replace page-handle-you-want-to-exclude with the correct page handle.
- Click Save.
How to Hide Dynamic Shopify Pages from Google
To hide dynamic pages from Google:
- From your Shopify admin, go to Online Store > Themes.
- Find the theme you want to edit, and then click Actions > Edit code.
- Click the
theme.liquidlayout file. - Paste the following code in the
<head>section:
Make sure that you replace your-text with the correct keyword you want to exclude from indexing.
- Click Save.
How to Block Google from Indexing Your Shopify Pages [VIDEO]
Note from the author: the solutions above are not about editing your Shopify robots.txt. They are about adding a meta tag called “robots”. This means that your pages are still being crawled by Google spiders. To hide them from Googlebot and similar, the solution below should replace the editing of your Shopify robots.txt.
How to Hide Shopify Pages from Web Crawlers
To hide Shopify pages from web crawlers like Googlebot:
- From your Shopify admin, go to Online Store > Themes.
- Find the theme you want to edit, and then click Actions > Edit code.
- Click the
theme.liquidlayout file. - Paste the following code in the
<head>section:
Make sure that you replace current_tags with the correct tag you have already assigned to the pages you want to exclude from being crawled.
- Click Save.
Note from the author: this doesn’t edit your Shopify Robots.txt file but it’s a temporary solution to a question about crawling raised by a visitor I didn’t have an accurate answer to. It’s likely to change as I get more familiar with the topic.
How to Edit Shopify Robots.txt File: Learn More
https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt
https://help.shopify.com/en/manual/promoting-marketing/seo/hide-a-page-from-search-engines
https://www.rmehdee.com/blogs/life-experiences-and-hacks/hiding-multiple-shopify-pages-from-search-engines
https://www.sovrn.com/blog/website-crawling-information/
https://www.cloudflare.com/learning/bots/what-is-a-web-crawler/
https://help.outofthesandbox.com/hc/en-us/articles/115007101508-How-can-I-prevent-certain-content-from-being-indexed-by-search-engines-
Featured image of How to Edit Shopify Robots.txt File (Easy Tutorial) from Storynet.com
Pasquale is Founder at Increasily.com, a H2H marketing agency based in Dublin, Ireland; and Co-Founder at luxury ecommerce marketplace Thesidebe.com. Pasquale has worked in Digital Marketing and Account Management since 2004. He currently lives in Dublin with his wife, stepdaughter and cat.
Related Post
11 Comments
-
ReplySteve
6 December 2020Hi, I see the same explanation on multiple websites, but nobody explains what the correct syntax is when you want to exclude two or more pages. Or where to put this code in the section (before or after the meta-tags)? Thank you!
-
Reply
Pasquale Mellone
7 December 2020Hi Steve, and thanks so much for the feedback. I have just added the piece of code you need to the article. I hope it helps. Please keep me posted and thanks for going through the blog post!
-
Reply
sandeep pangeni
25 February 2021Existing Robort.txt file is blocking collection pages and product pages. There is neither the metatag with robort or no index. How can we resolve the issue ? Need Help.
-
Reply
Pasquale Mellone
25 February 2021Hi Sandeep, and thanks for the comment. To look into this I would need to know the URL of your website. Generally speaking, the noindex of a page in Shopify is generally an instruction given in the theme.liquid file of your store, as I explain in the article.
-
-
Reply
Rana Cheikha
25 February 2021Hi, Can you please explain the afterword process? As I am unable to find the txt file in the code. Thanks
1. From your Shopify admin, go to Online Store > Themes.
2.Find the theme you want to edit and then click Actions > Edit code.
3. Click the theme. liquid layout file.-
Reply
Pasquale Mellone
25 February 2021Hi Rana, and thanks for the comment. What are you trying to do? Are you trying to hide a page from Google? If so, you won’t find the code in your theme.liquid file. You will have to add it as in the example. Hopefully that helps?
-
-
Reply
Garry Viner
11 March 2021Hi Pasquale
Thanks for the great article. Could you please confirm what you mean by page handles? We have a site with lots of detailed filters, leading to thousands of URLs that contain either an + or _ character. The robots.txt file directs Google not to crawl those with the + character, but not those with the _ character and i’d like to ensure that this is handled. So my question is, does the above work with pattern matching?
eg is there a way to block /collections/shoes/colour_red and /collections/shoes/size_8 without at the same time blocking /collections/shoes/?
Note that the above are created dynamically and as i said, there are many thousands.
Secondly, what you’ve provided is how to access the meta robots tags, which manage indexation, not the robots.txt file which manages crawling. In our case i’d like to prevent the crawling of these pages, as well as their indexation, as they are using up crawl budget and preventing legitimate pages from being indexed.
So is there a way to do this, which can only be done using a Disallow directive in the robots.txt file?
Thanks very much
Garry -
-
-
Reply
Mattew Thomas
25 July 2021This is a nice tutorial. Well explanation about edit Shopify robots.txt file. Now I know how to edit the robots.txt file. Thanks for sharing this informative article.
-
Pasquale Mellone