Over 10 years we helping companies reach their financial and branding goals. Onum is a values-driven SEO agency dedicated.

CONTACTS
Shopify

How to Edit Shopify Robots.txt File (Best Tutorial for 2022)

how to edit shopify robots.txt file

How to Edit Shopify Robots.txt File (Best Tutorial for 2022) – In this article I am going to explain what the Shopify robots.txt file is, where in your online store is located and how to hide a page from search engines.

If you want to learn the basics of SEO for Shopify, make sure you check my other articles:

If you need help with the robots.txt file for your Shopify store, feel free to book our Shopify Support package.

What is the Robots.txt File?

The robots.txt file is where you give instructions to Google and other search engine crawlers about what web pages and resources to crawl and what not.

For example, it’s typical for a robots.txt file to instruct web crawlers not to crawl internal admin or login pages.

What Edits Can I Make from the Robots.txt File?

From the robots.txt file you can make the following edits:

  • allow or disallow certain URLs from being crawled
  • add crawl-delay rules for certain crawlers
  • add extra sitemap URLs
  • block certain crawlers

What is a Web Crawler?

A web crawler, also called spider or search engine bot, is an automated software with the task of discovering and scanning web pages and resources with the purpose of indexing them on a given search engine.

What is the Crawling of a Web Page?

It’s the automated process by a web crawler to fetch a web page or resource with the purpose of indexing it on a given search engine.

What Are Google’s Main Web Crawlers?

The main web crawler by Google is Googlebot. Googlebot refers to two separate crawlers, a desktop and mobile one, simulating the experience of your web page by a user respectively on desktop and mobile devices.

Although not a web crawler, Google Read Aloud is an important user agent by Google. It enables the reading of web pages by using Text-To-Speech (TTS) technology. This agent is only triggered upon request by a user.

To learn more, here’s the list of all Google crawlers or user agents.

Crawling vs Indexing of a Web Page

Crawling is the automated process by web crawlers of discovering and scanning your web page or resources, while indexing is the process of adding a web page or resources to a search engine database.

Where is the Shopify Robots.txt File Located?

The Shopify robots.txt file is located in the root folder of your primary domain.

In a tweet posted on June 16th, Shopify’s CEO Tobi Lutke confirmed it’s now possible to edit the robots.txt file in Shopify.

How Do I Edit the Shopify Robots.txt File?

To edit your Shopify robots.txt file:

  1. From your Shopify admin, go to Online Store > Themes.
  2. Click Actions, and then click Edit Code.
  3. Click Add a new template, and then select robots.
  4. Click Create template.
  5. Make the changes that you want to make to the default template
  6. Save changes to the robots.txt.liquid file in your published theme

Please note this is an unsupported customisation and Shopify support won’t help you with troubleshooting any potential issue.

 

How Do I Hide Shopify Pages from Google?

You can’t edit your Shopify robots.txt file directly but You can instruct Google spiders about not crawling one or more page:

  • From your Shopify admin, go to Online Store > Themes.
  • Find the theme you want to edit, and then click Actions > Edit code.
  • Click the theme.liquid layout file.

How to Hide Shopify Search Template from Google

To hide the search template from Google:

  • From your Shopify admin, go to Online Store > Themes.
  • Find the theme you want to edit, and then click Actions > Edit code.
  • Click the theme.liquid layout file.
  • Paste the following code in the <head> section:

How to Hide a Specific Shopify Page from Google

To hide a specific page from Google:

  • From your Shopify admin, go to Online Store > Themes.
  • Find the theme you want to edit, and then click Actions > Edit code.
  • Click the theme.liquid layout file.
  • Paste the following code in the <head> section:

exclude specific page

Make sure that you replace page-handle-you-want-to-exclude with the correct page handle.

  • Click Save.

What is a Page Handle?

In Shopify you call a handle the slug of your URL, which as you probably know is the part of the URL net of the domain eg yourdomain.com/[this is the handle / slug] (source: shopify.dev:

a handle is the object’s title in lowercase with any spaces and special characters replaced by hyphens (-)

To learn more, check the Object Handle documentation on the Shopify Developers page.

How to Hide Multiple Shopify Pages from Google

To hide multiple pages from Google:

  • From your Shopify admin, go to Online Store > Themes.
  • Find the theme you want to edit, and then click Actions > Edit code.
  • Click the theme.liquid layout file.
  • Paste the following code in the <head> section:

exclude multiple pages

Make sure that you replace page-handle-you-want-to-exclude with the correct page handle.

  • Click Save.

How to Hide Dynamic Shopify Pages from Google

To hide dynamic pages from Google:

  • From your Shopify admin, go to Online Store > Themes.
  • Find the theme you want to edit, and then click Actions > Edit code.
  • Click the theme.liquid layout file.
  • Paste the following code in the <head> section:

exclude dynamic pages

Make sure that you replace your-text with the correct keyword you want to exclude from indexing.

  • Click Save.

How to Block Google from Indexing Your Shopify Pages [VIDEO]

Note from the author: the solutions above are not about editing your Shopify robots.txt. They are about adding a meta tag called “robots”. This means that your pages are still being crawled by Google spiders. To hide them from Googlebot and similar, the solution below should replace the editing of your Shopify robots.txt.

How to Hide Shopify Pages from Web Crawlers

To hide Shopify pages from web crawlers like Googlebot:

  • From your Shopify admin, go to Online Store > Themes.
  • Find the theme you want to edit, and then click Actions > Edit code.
  • Click the theme.liquid layout file.
  • Paste the following code in the <head> section:

robots tags

Make sure that you replace current_tags with the correct tag you have already assigned to the pages you want to exclude from being crawled.

  • Click Save.

Note from the author: this doesn’t edit your Shopify Robots.txt file but it’s a temporary solution to a question about crawling raised by a visitor I didn’t have an accurate answer to. It’s likely to change as I get more familiar with the topic.

How to Edit Shopify Robots.txt File: Learn More

https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt
https://help.shopify.com/en/manual/promoting-marketing/seo/hide-a-page-from-search-engines
https://www.rmehdee.com/blogs/life-experiences-and-hacks/hiding-multiple-shopify-pages-from-search-engines
https://www.sovrn.com/blog/website-crawling-information/
https://www.cloudflare.com/learning/bots/what-is-a-web-crawler/
https://help.outofthesandbox.com/hc/en-us/articles/115007101508-How-can-I-prevent-certain-content-from-being-indexed-by-search-engines-

Featured image of How to Edit Shopify Robots.txt File (Easy Tutorial) from Storynet.com

shopify seo starter pack ad

Author

Pasquale Mellone

Pasquale is Founder at Increasily.com, a H2H marketing agency based in Dublin, Ireland, and owner at print-on-demand ecommerce Mintycase.com. Pasquale has worked in Digital Marketing and Account Management since 2004. He currently lives in Dublin with his wife, stepdaughter and cat.

Comments (11)

  1. Steve
    6 December 2020

    Hi, I see the same explanation on multiple websites, but nobody explains what the correct syntax is when you want to exclude two or more pages. Or where to put this code in the section (before or after the meta-tags)? Thank you!

  2. Pasquale Mellone
    7 December 2020

    Hi Steve, and thanks so much for the feedback. I have just added the piece of code you need to the article. I hope it helps. Please keep me posted and thanks for going through the blog post!

  3. sandeep pangeni
    25 February 2021

    Existing Robort.txt file is blocking collection pages and product pages. There is neither the metatag with robort or no index. How can we resolve the issue ? Need Help.

    • Pasquale Mellone
      25 February 2021

      Hi Sandeep, and thanks for the comment. To look into this I would need to know the URL of your website. Generally speaking, the noindex of a page in Shopify is generally an instruction given in the theme.liquid file of your store, as I explain in the article.

  4. Rana Cheikha
    25 February 2021

    Hi, Can you please explain the afterword process? As I am unable to find the txt file in the code. Thanks

    1. From your Shopify admin, go to Online Store > Themes.
    2.Find the theme you want to edit and then click Actions > Edit code.
    3. Click the theme. liquid layout file.

    • Pasquale Mellone
      25 February 2021

      Hi Rana, and thanks for the comment. What are you trying to do? Are you trying to hide a page from Google? If so, you won’t find the code in your theme.liquid file. You will have to add it as in the example. Hopefully that helps?

  5. Garry Viner
    11 March 2021

    Hi Pasquale

    Thanks for the great article. Could you please confirm what you mean by page handles? We have a site with lots of detailed filters, leading to thousands of URLs that contain either an + or _ character. The robots.txt file directs Google not to crawl those with the + character, but not those with the _ character and i’d like to ensure that this is handled. So my question is, does the above work with pattern matching?

    eg is there a way to block /collections/shoes/colour_red and /collections/shoes/size_8 without at the same time blocking /collections/shoes/?

    Note that the above are created dynamically and as i said, there are many thousands.

    Secondly, what you’ve provided is how to access the meta robots tags, which manage indexation, not the robots.txt file which manages crawling. In our case i’d like to prevent the crawling of these pages, as well as their indexation, as they are using up crawl budget and preventing legitimate pages from being indexed.

    So is there a way to do this, which can only be done using a Disallow directive in the robots.txt file?

    Thanks very much
    Garry

  6. Detec
    3 April 2021

    Yes, It is very helpful for me. now I solved my shopify issue.

    • Pasquale Mellone
      5 April 2021

      Thanks for the feedback. I am glad you found it helpful!

  7. Mattew Thomas
    25 July 2021

    This is a nice tutorial. Well explanation about edit Shopify robots.txt file. Now I know how to edit the robots.txt file. Thanks for sharing this informative article.

    • Pasquale Mellone
      26 July 2021

      Thanks Mattew

Comments are closed.