How to Setup robots.txt on Blogger

How to Setup robots.txt on Blogger

Setup robots.txt, we can set the visibility of our articles on search engines & determine whether the article will be indexed by search engines or not

How to Setup robots.txt on Blogger

Blogger allows custom robots.txt, this is very useful because we can set the visibility of our articles on search engines, we can determine whether the article will be indexed by search engines or not.

  1. Go to the blogger dashboard draft.blogger.com
  2. .
  3. Menu settings Crawlers and indexing
  4. Now Enable custom robots.txt
  5. Click on Custom robots.txt then put the robots.txt as mention below.

By default, every blog that uses the Blogger platform will have a robots.txt as follows:


User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Allow: /

Sitemap: https://www.merobloggingtips.com/sitemap.xml
Sitemap: https://www.merobloggingtips.com/sitemap-pages.xml

And has the following explanations:

Mediapartners-Google is a robot from Google Adsense, leave it as is because if you mistakenly change that than the ads served will not fit with your content.

The next line is for all the robots and marked with an asterisk (*). On the default configuration, it is clear that the label of our blog is not indexed Disallow: /search.

Keep in mind that a slash (/) is as your homepage, so for example if you want the label to get indexed, do not just fill up with a slash like this Disallow: / because that would be you do not allow the robot tracing your blog, but it should like the example below:


User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow:
Allow: /

Sitemap: https://www.merobloggingtips.com/sitemap.xml
Sitemap: https://www.merobloggingtips.com/sitemap-pages.xml

With the configuration as above then all of the articles and the label will be indexed. And to block a robot for particular page (I take the example of my Contact Us page) you can simply write as follows:


User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /p/contact-us.html
Allow: /

Sitemap: https://www.merobloggingtips.com/sitemap.xml
Sitemap: https://www.merobloggingtips.com/sitemap-pages.xml


To resolve the pagination problems on blogspot after we remove the Disallow: /search than we can use the following configuration to block the pagination page:


User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search?updated-min=
Disallow: /search?updated-max=
Disallow: /search/label/*?updated-min=
Disallow: /search/label/*?updated-max=
Allow: /

Sitemap: https://www.merobloggingtips.com/sitemap.xml
Sitemap: https://www.merobloggingtips.com/sitemap-pages.xml

After the changes, make sure everything is fit like what we want by visiting www.example.com/robots.txt. Replace the Example.com with your domain name.

Warning! Use with caution. Incorrect use of these features can result in your blog being ignored by search engines.

Mero Blogging Tips

Mero Blogging Tips Share Blogger Tutorials, HTML, JavaScript, CSS, Widget, SEO, jQuery, Web Tool and General Helpful Stuff for Newbie Bloggers and Readers.

Post a Comment

First of all, thank you for taking the time to read my blog. It's much appreciated! If you would like to leave a comment, please do, I'd love to hear what you think!. Suggestions and/or questions are always welcome, either post them in the comment form.

However, comments are always reviewed and it may take some time to appear. Also, Comments without NAME will not be published. Always keep in mind "URL without nofollow tag" will consider as a spam.

Previous Post Next Post

Published Blogger Posts
40+

Years of Envolving
3+

Contact Form