how to edit shopify robots.txt file

How to Edit Shopify Robots.txt File – In this article I am going to explain what the Shopify robots.txt file is, where in your online store is located and how to hide a page from search engines.

If you want to learn the basics of SEO for Shopify, make sure you check my other articles:

If you need help with ranking your website on Google, feel free to check our Shopify SEO services.

What is the Robots.txt File?

The robots.txt file is where you give instructions to Google and other search engine crawlers about what web pages and resources to crawl and what not.

See also  Great Digital Marketing Notes You Can't Miss, May 17-21 2021

For example, it’s typical for a robots.txt file to instruct web crawlers not to crawl internal admin or login pages.

What Edits Can I Make from the Robots.txt File?

From the robots.txt file you can make the following edits:

  • allow or disallow certain URLs from being crawled
  • add crawl-delay rules for certain crawlers
  • add extra sitemap URLs
  • block certain crawlers

What is a Web Crawler?

A web crawler, also called spider or search engine bot, is an automated software with the task of discovering and scanning web pages and resources with the purpose of indexing them on a given search engine.

What is the Crawling of a Web Page?

It’s the automated process by a web crawler to fetch a web page or resource with the purpose of indexing it on a given search engine.

What Are Google’s Main Web Crawlers?

The main web crawler by Google is Googlebot. Googlebot refers to two separate crawlers, a desktop and mobile one, simulating the experience of your web page by a user respectively on desktop and mobile devices.

Although not a web crawler, Google Read Aloud is an important user agent by Google. It enables the reading of web pages by using Text-To-Speech (TTS) technology. This agent is only triggered upon request by a user.

To learn more, here’s the list of all Google crawlers or user agents.

Crawling vs Indexing of a Web Page

Crawling is the automated process by web crawlers of discovering and scanning your web page or resources, while indexing is the process of adding a web page or resources to a search engine database.

Where is the Shopify Robots.txt File Located?

The Shopify robots.txt file is located in the root folder of your primary domain.

In a tweet posted on June 16th, Shopify’s CEO Tobi Lutke confirmed it’s now possible to edit the robots.txt file in Shopify.

How Do I Edit the Shopify Robots.txt File?

To edit your Shopify robots.txt file:

  1. From your Shopify admin, go to Online Store > Themes.
  2. Click Actions, and then click Edit Code.
  3. Click Add a new template, and then select robots.
  4. Click Create template.
  5. Make the changes that you want to make to the default template
  6. Save changes to the robots.txt.liquid file in your published theme
See also  Digital Marketing Strategy Weekly, 3rd - 7th May 2021

Please note this is an unsupported customisation and Shopify support won’t help you with troubleshooting any potential issue.

 

How Do I Hide Shopify Pages from Google?

You can’t edit your Shopify robots.txt file directly but You can instruct Google spiders about not crawling one or more page:

  • From your Shopify admin, go to Online Store > Themes.
  • Find the theme you want to edit, and then click Actions > Edit code.
  • Click the theme.liquid layout file.

How to Hide Shopify Search Template from Google

To hide the search template from Google:

  • From your Shopify admin, go to Online Store > Themes.
  • Find the theme you want to edit, and then click Actions > Edit code.
  • Click the theme.liquid layout file.
  • Paste the following code in the <head> section:

How to Edit Shopify Robots txt File 2021

How to Hide a Specific Shopify Page from Google

To hide a specific page from Google:

  • From your Shopify admin, go to Online Store > Themes.
  • Find the theme you want to edit, and then click Actions > Edit code.
  • Click the theme.liquid layout file.
  • Paste the following code in the <head> section:

exclude specific page

Make sure that you replace page-handle-you-want-to-exclude with the correct page handle.

  • Click Save.

What is a Page Handle?

In Shopify you call a handle the slug of your URL, which as you probably know is the part of the URL net of the domain eg yourdomain.com/[this is the handle / slug] (source: shopify.dev:

a handle is the object’s title in lowercase with any spaces and special characters replaced by hyphens (-)

To learn more, check the Object Handle documentation on the Shopify Developers page.

How to Hide Multiple Shopify Pages from Google

To hide multiple pages from Google:

  • From your Shopify admin, go to Online Store > Themes.
  • Find the theme you want to edit, and then click Actions > Edit code.
  • Click the theme.liquid layout file.
  • Paste the following code in the <head> section:
See also  July 2021 Google Core Update (with a Useful Timeline)

exclude multiple pages

Make sure that you replace page-handle-you-want-to-exclude with the correct page handle.

  • Click Save.

How to Hide Dynamic Shopify Pages from Google

To hide dynamic pages from Google:

  • From your Shopify admin, go to Online Store > Themes.
  • Find the theme you want to edit, and then click Actions > Edit code.
  • Click the theme.liquid layout file.
  • Paste the following code in the <head> section:

exclude dynamic pages

Make sure that you replace your-text with the correct keyword you want to exclude from indexing.

  • Click Save.

How to Block Google from Indexing Your Shopify Pages [VIDEO]

Note from the author: the solutions above are not about editing your Shopify robots.txt. They are about adding a meta tag called “robots”. This means that your pages are still being crawled by Google spiders. To hide them from Googlebot and similar, the solution below should replace the editing of your Shopify robots.txt.

How to Hide Shopify Pages from Web Crawlers

To hide Shopify pages from web crawlers like Googlebot:

  • From your Shopify admin, go to Online Store > Themes.
  • Find the theme you want to edit, and then click Actions > Edit code.
  • Click the theme.liquid layout file.
  • Paste the following code in the <head> section:

robots tags

Make sure that you replace current_tags with the correct tag you have already assigned to the pages you want to exclude from being crawled.

  • Click Save.

Note from the author: this doesn’t edit your Shopify Robots.txt file but it’s a temporary solution to a question about crawling raised by a visitor I didn’t have an accurate answer to. It’s likely to change as I get more familiar with the topic.

How to Edit Shopify Robots.txt File: Learn More

https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt
https://help.shopify.com/en/manual/promoting-marketing/seo/hide-a-page-from-search-engines
https://www.rmehdee.com/blogs/life-experiences-and-hacks/hiding-multiple-shopify-pages-from-search-engines
https://www.sovrn.com/blog/website-crawling-information/
https://www.cloudflare.com/learning/bots/what-is-a-web-crawler/
https://help.outofthesandbox.com/hc/en-us/articles/115007101508-How-can-I-prevent-certain-content-from-being-indexed-by-search-engines-

Featured image of How to Edit Shopify Robots.txt File (Easy Tutorial) from Storynet.com

shopify seo starter pack ad

close
Oval@3x 2

Don’t miss our weekly posts!

We don’t spam! Read our privacy policy for more info.

Related Post

11 Comments

Leave a Comment

Recent Posts

trading online voucher august 2021 mobile phone showing 2021 calendar
august 2021 marketing ideas
july 2021 google core update - google logo made with M&Ms
marketing personalisation - woman holding a pen about to write on a notebook
two hands holding - inbound marketing