Looking at robots.txt for SEO optimisation

What is robots.txt?

Robots.txt is a text file that contains rules to control the behaviour of search engines as they crawl your site, known as the 'robots directive'. You can create using any text editor software such as Notepad.



Each website created may include a robot.txt file located at the root level of the server. It may also be located at domain.com/robots.txt. This file is what search engines initially look for when they start to identify and crawl your website. As a robot directive, it explains to the search engine what it can and cannot do on the website. The file can either communicate to all search engines or selectively isolate and put independent requirements on individual search engines.

What should you exclude in your robots.txt file?

The robot.txt file may exclude areas in the site that you don’t want the search engine to discover, crawl, or particularly index. Without the robot.txt file, the results will be published in the search engine results page, allowing the general public to identify and access sections of the site that may not be appropriate. These sections may be directories of programming elements, a secure or admin section on the site, or components of your content management systems, depending on what you’re using, that can be blocked from the robots so they don’t crawl and index those elements.

Robots.txt Example:


It is important to be aware that some useful sections of the site could be inappropriately blocked. A common example is the /images directory. On average, five to seven percent of search results may be initiated through an image-based search with a user clicking through to the actual website. When the images directory is blocked, search engines may not be able to discover, identify, and list within their own directories of image search capabilities those images that are related to your business. For many businesses, this could benefit and support Search Engine Optimisation activities.

Sitemaps Location



Another thing you can see in a robot.txt file is a pointer to the location of your Sitemap. A Sitemap is a digital directory written in extensible mark-up language (XML) format which lists all the pages of your website that you wish to have indexed by the search engine. Putting a pointer within your robot.txt file to this location allows search engines to automatically discover your Sitemap and additional Sitemap files you have included, such as a geo Sitemap, and an images Sitemap. Supporting search engines in discovering these is beneficial for your site. For this reason, consider having a pointer to either a master Sitemap file or individual Sitemap files for your website.

The robots directive not only blocks certain robots’ actions as well as the pages to be crawled and indexed. It also determines speed. If the site has any particular issues in hosting, for example, where a crawl may slow down the actual site’s performance and affect the users’ experience, you can actually put speed controls to determine how fast the robot approaches and works its way through your website.

Google Webmaster Tools (GWT) and Bing Webmaster Tools (BWT) allows you to remove individual page files through single requests. The crawling speed may also be supported and addressed within GWT and BWT apart from being listed in the robot.txt file. You can even monitor the traffic, speed, and the amount of data used by the search engines over a period of time to determine if the performance relates to any issues or concerns for your website.

HTML Meta Directives

If you want to specify rules per HTML page, you can do this using HTML meta directives. This informs the search engines a specific action for a certain page.

Google, Bing, and Yahoo have implemented a number of HTML Meta directives including the following:

• NOINDEX META Tag – This tells a crawler not to index a certain page.
• NOFOLLOW META Tag – This tells a crawler not to follow a link going to another content on a certain page.
• NOSNIPPET META Tag – This tells a crawler not to display snippets in the search results for a certain page.
• NOARCHIVE META Tag – This tells a crawler not to show a cached link for a certain page.
• NOODP META Tag – This tells a crawler not to use a title and snippet from ODP (Open Directory Project) for a certain page.

Having your content indexed by major search engines can be a frustrating and time-consuming experience, but it can be done. By executing the techniques mentioned above, you can get confidential content removed quickly and prevent it from showing on search results pages. 

Search Group, a local Perth SEO company helps keep your business visible online through effective SEO and Internet marketing strategies. Visit www.searchroup.com.au for details about our services.