How to Edit Shopify Robots.txt File (Best Tutorial for 2022) – In this article I am going to explain what the Shopify robots.txt file is, where in your online store is located and how to hide a page from search engines.
If you want to learn the basics of SEO for Shopify, make sure you check my other articles:
- Optimise Shopify Page Title and Description
- How to Add Shopify Alt Tags to Theme Images
- How to add Alt Tags to Shopify Product Images
- How to Add Google Analytics to Shopify
- How to Add SEO Headings in Shopify
- How to Optimise Your Shopify Site Structure
- How to Find and Submit Your Shopify Sitemap
- How to Verify Your Shopify Store with Google
- How to Create a Redirect in Shopify
- Duplicate Content and Shopify
- Link Building and Shopify
- How to Compress Images on Shopify
- How to Improve Shopify Page Load Speed
- Optimise Your Shopify Blog Posts
- How to Backup Your Shopify Store
- Enable AMP in Shopify
If you need help with the robots.txt file for your Shopify store, feel free to book our Shopify Support package.
What is the Robots.txt File?
The robots.txt file is where you give instructions to Google and other search engine crawlers about what web pages and resources to crawl and what not.
For example, it’s typical for a robots.txt file to instruct web crawlers not to crawl internal admin or login pages.
What Edits Can I Make from the Robots.txt File?
From the robots.txt file you can make the following edits:
- allow or disallow certain URLs from being crawled
- add crawl-delay rules for certain crawlers
- add extra sitemap URLs
- block certain crawlers
What is a Web Crawler?
A web crawler, also called spider or search engine bot, is an automated software with the task of discovering and scanning web pages and resources with the purpose of indexing them on a given search engine.
What is the Crawling of a Web Page?
It’s the automated process by a web crawler to fetch a web page or resource with the purpose of indexing it on a given search engine.
What Are Google’s Main Web Crawlers?
The main web crawler by Google is Googlebot. Googlebot refers to two separate crawlers, a desktop and mobile one, simulating the experience of your web page by a user respectively on desktop and mobile devices.
Although not a web crawler, Google Read Aloud is an important user agent by Google. It enables the reading of web pages by using Text-To-Speech (TTS) technology. This agent is only triggered upon request by a user.
To learn more, here’s the list of all Google crawlers or user agents.
Crawling vs Indexing of a Web Page
Crawling is the automated process by web crawlers of discovering and scanning your web page or resources, while indexing is the process of adding a web page or resources to a search engine database.
Where is the Shopify Robots.txt File Located?
The Shopify robots.txt file is located in the root folder of your primary domain.
In a tweet posted on June 16th, Shopify’s CEO Tobi Lutke confirmed it’s now possible to edit the robots.txt file in Shopify.
User-agent: everyone
Allow: /
Starting today, you have complete control over how search engine bots see your store. #shopifyseohttps://t.co/Hz9Ijj5h1y— Tobi Lutke (@tobi) June 18, 2021
How Do I Edit the Shopify Robots.txt File?
To edit your Shopify robots.txt file:
- From your Shopify admin, go to Online Store > Themes.
- Click Actions, and then click Edit Code.
- Click Add a new template, and then select robots.
- Click Create template.
- Make the changes that you want to make to the default template
- Save changes to the robots.txt.liquid file in your published theme
Please note this is an unsupported customisation and Shopify support won’t help you with troubleshooting any potential issue.
How Do I Hide Shopify Pages from Google?
You can’t edit your Shopify robots.txt file directly but You can instruct Google spiders about not crawling one or more page:
- From your Shopify admin, go to Online Store > Themes.
- Find the theme you want to edit, and then click Actions > Edit code.
- Click the
theme.liquidlayout file.
How to Hide Shopify Search Template from Google
To hide the search template from Google:
- From your Shopify admin, go to Online Store > Themes.
- Find the theme you want to edit, and then click Actions > Edit code.
- Click the
theme.liquidlayout file. - Paste the following code in the
<head>section:
How to Hide a Specific Shopify Page from Google
To hide a specific page from Google:
- From your Shopify admin, go to Online Store > Themes.
- Find the theme you want to edit, and then click Actions > Edit code.
- Click the
theme.liquidlayout file. - Paste the following code in the
<head>section:
Make sure that you replace page-handle-you-want-to-exclude with the correct page handle.
- Click Save.
What is a Page Handle?
In Shopify you call a handle the slug of your URL, which as you probably know is the part of the URL net of the domain eg yourdomain.com/[this is the handle / slug] (source: shopify.dev:
a handle is the object’s title in lowercase with any spaces and special characters replaced by hyphens (-)
To learn more, check the Object Handle documentation on the Shopify Developers page.
How to Hide Multiple Shopify Pages from Google
To hide multiple pages from Google:
- From your Shopify admin, go to Online Store > Themes.
- Find the theme you want to edit, and then click Actions > Edit code.
- Click the
theme.liquidlayout file. - Paste the following code in the
<head>section:
Make sure that you replace page-handle-you-want-to-exclude with the correct page handle.
- Click Save.
How to Hide Dynamic Shopify Pages from Google
To hide dynamic pages from Google:
- From your Shopify admin, go to Online Store > Themes.
- Find the theme you want to edit, and then click Actions > Edit code.
- Click the
theme.liquidlayout file. - Paste the following code in the
<head>section:
Make sure that you replace your-text with the correct keyword you want to exclude from indexing.
- Click Save.
How to Block Google from Indexing Your Shopify Pages [VIDEO]
Note from the author: the solutions above are not about editing your Shopify robots.txt. They are about adding a meta tag called “robots”. This means that your pages are still being crawled by Google spiders. To hide them from Googlebot and similar, the solution below should replace the editing of your Shopify robots.txt.
How to Hide Shopify Pages from Web Crawlers
To hide Shopify pages from web crawlers like Googlebot:
- From your Shopify admin, go to Online Store > Themes.
- Find the theme you want to edit, and then click Actions > Edit code.
- Click the
theme.liquidlayout file. - Paste the following code in the
<head>section:
Make sure that you replace current_tags with the correct tag you have already assigned to the pages you want to exclude from being crawled.
- Click Save.
Note from the author: this doesn’t edit your Shopify Robots.txt file but it’s a temporary solution to a question about crawling raised by a visitor I didn’t have an accurate answer to. It’s likely to change as I get more familiar with the topic.
How to Edit Shopify Robots.txt File: Learn More
https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt
https://help.shopify.com/en/manual/promoting-marketing/seo/hide-a-page-from-search-engines
https://www.rmehdee.com/blogs/life-experiences-and-hacks/hiding-multiple-shopify-pages-from-search-engines
https://www.sovrn.com/blog/website-crawling-information/
https://www.cloudflare.com/learning/bots/what-is-a-web-crawler/
https://help.outofthesandbox.com/hc/en-us/articles/115007101508-How-can-I-prevent-certain-content-from-being-indexed-by-search-engines-
Featured image of How to Edit Shopify Robots.txt File (Easy Tutorial) from Storynet.com
