ROBOTS.TXT: Optimizing it to Boost SEO in 2024 | Blogger

Today, we're going to talk about one of the most important element of search engine optimization (SEO) for your website or blog and it is - robots.txt file. This file is the first item that any search engine crawling bot interacts with when it visits your website. So, it's important that you create a custom robots.txt file and optimize it for better SEO and to crawling your content structure fast.

ROBOTS.TXT —

So you also want to take your blogging skills to the next level and become a professional blogger. If so, then it's important to educate yourself on some technical parts of search engine optimization (SEO). Most of the time new bloggers work on the platforms like Blogger or Blogspot to publish their content and configure a custom robots.txt file on these platforms - and it is a required step in optimizing your blog for search engine listings and then rankings.

In simple, it's a text document that instructs search engine crawling bots, also called as spiders or crawlers used to guide every search engine about which part of your website to crawl and which ones to bypass. By optimizing your custom robots.txt file correctly, you can ensure that search engines like Google, Yahoo, or Bing start to index only the most relevant and important pages from your site.

Well, this is a brief info about how robots.txt guides you to increased traffic and better SERP visibility for your site pages. But how do you set up and edit custom robots.txt file for your blog? And it's a fairly simple process that anyone can learn. But, it's required to get it correct and optimized, or else your content may not be indexed correctly by search engines. Luckily, there are some quick steps you can follow to ensure that your site pages and its structure is optimized for search engines and set up for success.

What is a robots.txt file?

Before we go into deep on how to create a custom robots.txt file, let's first understand what robots.txt file actually is.

A robots.txt file is a text file found in the root directory of your every website. Its primary objective is to instruct search engine crawling bots on which pages or structure of your website to crawl and which ones to skip.

By using this file, you can confirm that search engines only indexing the pages that you want them to and bypass indexing any pages that may be duplicated, unrelated, or toxic to your blog's ranking and overall SEO.

Why is a custom robots.txt file important for Blogger blogs?

Now that you understood what a robots.txt file is, let's check why robots.txt file necessary for better SEO.

By creating a custom robots.txt file, you can allow search engine bots to understand your website structure and hierarchy. This can help enhance your visibility in search engine results pages (SERPs) by confirming that only the targetted pages are indexed and shown to organic visitors driven from search engines only.

Default Robots.txt file

A default robots.txt file is a standard file that most websites have in their root directory, which instructs web robots or crawlers which pages or sections of the website should not be crawled or indexed by search engines. The default settings in a Robots.txt file can vary depending on the website's content management system (CMS) and the web server configuration.

In general, however, a default Robots.txt file usually allows all robots to crawl all parts of the website. The file is usually named "robots.txt," and it serves as a guide to search engine bots on how to access and crawl the website's pages.

What does a robots.txt example look like?

For example, a simple Robots.txt file look like this:

User-agent: *
Disallow:

This is a very simple Robots.txt file that instructs all web crawler robots (indicated by the User-agent: * directive) that they should not access or index any page, directory, or section of the website. (indicated by the Disallow: directive). This rule is often used when a website is under development, or when the owner wants to prevent search engines from indexing their content.

Custom Robots.txt for Blogger

As you know, Blogger/Blogspot is a free blogging platform given by Google, and until recently, bloggers were not able to edit the Robots.txt file directly for their blogs. But, Blogger now allows users to set a custom Robots.txt file for their blog, giving them better control over how search engine bots and other web crawlers fetch their content.

A typical Robots.txt file for a Blogger/Blogspot blog might look like this:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /

Sitemap: https://www.yourblogname.com/sitemap.xml

Analysing Robots.txt

Let's break down each section:

Here are the basic directives of a Custom Robots.txt file with examples of each:

1. /

   The forward slash (/) is commonly used in robots.txt files to indicate the root directory, it does not function as a wildcard character.

   
   

2. *

   The asterisk (*) is the most commonly used wildcard character in robots.txt files and can represent any string of characters.

   For example, the directive "Disallow: /wp-admin/" would block any URLs that contain "/wp-admin/" in the path.

   The asterisk serves as a wildcard character to match any string of characters that follows "/wp-admin/".

   Similarly, the directive "Disallow: /.pdf" would block all URLs that end with the .pdf extension, as the asterisk acts as a wildcard for any string of characters that precede the .pdf extension.

   
   

3. User-agent:

   This directive tells the web crawler or robot to which the following directives apply.

   Example: User-agent: Googlebot

   This line specifies that the following directives apply only to the Googlebot crawler.

   
   

4. User-agent: *

   This line defines which user agents, or search engine bots, the following directives apply to. In this case, the asterisk (*) indicates that these directives apply to all user agents.

   
   

5. Disallow: 

   This directive instructs the web crawler or robot not to crawl or index user-defined pages or sections of a website.

   Example: Disallow: /private/

   This tells the web crawler not to crawl any pages or sections that are within the '/private/' directory of your website.

   
   

6. Allow: 

   This directive instructs the web crawler or robot to crawl and index user-defined pages or sections of a website, even if they would normally be disallowed by other rules.

   Example: Allow: /public/

   This tells the web crawler to crawl any pages or sections that are within the '/public/' directory of your website.

   
   

7. Disallow: /search 

   This directive instructs search engine bots to not crawl any pages containing "/search" in the URL. This is because the search results pages on a blog can often be low

   
   

8. User-agent: Mediapartners-Google 
   Disallow:
   

   The above 2 lines of code is advising the search engine crawler user-agent called "Mediapartners-Google" not to crawl any pages on the website. This user-agent is used by Google AdSense to resolve the content to serve relevant ads. By disallowing this user-agent from crawling any pages and content, it is showing that the website owner does not want ads to be shown on his site.

   It's important to note that this rule only applies to the "Mediapartners-Google" agent and not to other search engine crawlers. Other user-agents will still be able to crawl and index the pages and content on the website unless other rules have been set up to disallow them as well.

   
   

9. Sitemap:

   This directive identifies the location of the XML sitemap of that website, which contains information about the pages that should be crawled and indexed by search engines.

   Example: Sitemap: https://www.yourblogname.com/sitemap.xml

   This tells the web crawler where to find the XML sitemap for your website.

   
   

10. /b:

    This directive identifies to avoid your preview post crawling of that website, which contains only meta information about the pages that indexed by search engines but not previews.

    
    

11. Crawl-delay: 

    This directive instructs the web crawler or robot to delay crawling your website for a specified amount of time. This is useful if your website is experiencing performance issues due to high traffic or limited server resources.

    Example: Crawl-delay: 10

    This tells the web crawler to wait for 10 seconds before crawling your website.

    
    

It's important to use wildcard characters very carefully in your robots.txt file to avoid accidentally blocking important content from search engines.

By using these directives in your Custom Robots.txt file, you can have more control over which pages and sections of your website are crawled and indexed by search engines.

Best Custom Robots.txt File for Blogger/Blogspot

Every new blogger used to ask how to create the perfect ROBOTS.TXT File for SEO? By default, the Robots.txt file for a Blogger blog allows search engines to crawl the archive pages, which can result in duplicate content issues and potentially harm the blog's search engine rankings.

To manage this problem, you can adjust the Robots.txt file to disallow search engines from crawling the archive section. To optimize the default robots.txt file for better SEO on a Blogger blog, we can fix the issue of duplicate content by blocking search engine bots from crawling the archive section.

To do this, we can add the following directive to the file:

To disallow all URLs starting with "/20" in the robots.txt file, you can add the following rule:

User-agent: *
Disallow: /20

This will disallow all URLs starting with "/20", such as "/2018/", "/2019/blog-post/", "/2020/category/news/", etc. However, it will still allow URLs like "/2023/about-us/", "/2023/contact/", etc.

If we only use the Disallow: /20* rule in our robots.txt file, it will block crawling of all URLs that start with "20" such as "/2019/05/my-post.html", "/2020/01/my-post.html", etc.

To allow the crawling of individual post URLs, we can add an Allow rule for the /*.html section of the blog. This will allow search engine bots to crawl all URLs that end with ".html", which typically includes individual post URLs.

/search*

Including "/search*" in the robots.txt file will prevent the crawling of any pages with URLs that contain "/search" such as search result pages, label pages, and archive pages. This can be useful for bloggers who want to avoid duplicate content issues and ensure that search engines are only indexing their most important pages. However, it's important to be careful when using disallow rules like this as they can directly block important pages from being crawled and indexed.

Here's an example of how you can modify the Robots.txt file for a Blogger blog to optimize it for SEO:

User-agent: Mediapartners-Google
Disallow:

#below lines control all search engines bots, and
#blocks all search links, archieve and
#allow indexing all blog posts and pages.

User-agent: *
Disallow: /search*
Disallow: /b
Disallow: /20*
Allow: /*.html

#sitemaps for your blogger blog
Sitemap: https://www.yourblogname.com/sitemap.xml
Sitemap: https://www.yourblogname.com/sitemap-pages.xml
Sitemap: https://www.yourblogname.com/feeds/posts/default?orderby=updated

How to create a custom robots.txt file for your Blogger blog?

Add Custom Robots.txt File on Blogger/Blogspot - Adding a custom robots.txt file on your Blogger/Blogspot blog is a simple process. Here are the step-by-step instructions to help you get started:

1. Log in to your Blogger account and go to your dashboard.
2. Click on the "Settings" option from the left-hand menu.
3. From the drop-down menu, select "Search Preferences".
4. Under the "Crawlers and indexing" section, click on the "Custom robots.txt" option.
5. Click on the "Edit" button.
6. Select "Yes" to enable custom robots.txt content.
7. Enter your custom robots.txt content in the text field.
8. After making changes, click on the "Save Changes" button.

Once you have added your customized robots.txt file, make sure to test it thoroughly to confirm that it's working correctly. You can use Google's robots.txt testing tool to check if your file is valid and if all the desired pages are being crawled.

It's also a good idea to monitor your blog's performance in search consol for traffic and search rankings to see if the changes you've made are having a positive impact on your blog's SEO.

Video: How to Add Custom Robots.txt in Blogger Blog

Frequently Asked Questions

Please take a moment to read through our FAQ section for quick answers to common questions.

Bottom Line

It's important to note that a default Robots.txt file may not be the best option for every website. Depending on the website's content and purpose, it may be necessary to add specific instructions to the Robots.txt file to prevent certain pages or directories from being crawled.

Overall, a Robots.txt file is an important tool for webmasters to control how robots access their website, and it's worth taking the time to customize it to suit the specific needs of the site.

Always test your robots.txt file and monitor your site's search performance to ensure that you are not blocking search engine crawlers from fetching your quality content.