Looking at robots.txt for SEO optimisation

What is robots.txt?

Robots.txt is a text file that contains rules to control the behaviour of search engines as they crawl your site, known as the 'robots directive'. You can create using any text editor software such as Notepad.



Each website created may include a robot.txt file located at the root level of the server. It may also be located at domain.com/robots.txt. This file is what search engines initially look for when they start to identify and crawl your website. As a robot directive, it explains to the search engine what it can and cannot do on the website. The file can either communicate to all search engines or selectively isolate and put independent requirements on individual search engines.

What should you exclude in your robots.txt file?

The robot.txt file may exclude areas in the site that you don’t want the search engine to discover, crawl, or particularly index. Without the robot.txt file, the results will be published in the search engine results page, allowing the general public to identify and access sections of the site that may not be appropriate. These sections may be directories of programming elements, a secure or admin section on the site, or components of your content management systems, depending on what you’re using, that can be blocked from the robots so they don’t crawl and index those elements.

Robots.txt Example:


It is important to be aware that some useful sections of the site could be inappropriately blocked. A common example is the /images directory. On average, five to seven percent of search results may be initiated through an image-based search with a user clicking through to the actual website. When the images directory is blocked, search engines may not be able to discover, identify, and list within their own directories of image search capabilities those images that are related to your business. For many businesses, this could benefit and support Search Engine Optimisation activities.

Sitemaps Location



Another thing you can see in a robot.txt file is a pointer to the location of your Sitemap. A Sitemap is a digital directory written in extensible mark-up language (XML) format which lists all the pages of your website that you wish to have indexed by the search engine. Putting a pointer within your robot.txt file to this location allows search engines to automatically discover your Sitemap and additional Sitemap files you have included, such as a geo Sitemap, and an images Sitemap. Supporting search engines in discovering these is beneficial for your site. For this reason, consider having a pointer to either a master Sitemap file or individual Sitemap files for your website.

The robots directive not only blocks certain robots’ actions as well as the pages to be crawled and indexed. It also determines speed. If the site has any particular issues in hosting, for example, where a crawl may slow down the actual site’s performance and affect the users’ experience, you can actually put speed controls to determine how fast the robot approaches and works its way through your website.

Google Webmaster Tools (GWT) and Bing Webmaster Tools (BWT) allows you to remove individual page files through single requests. The crawling speed may also be supported and addressed within GWT and BWT apart from being listed in the robot.txt file. You can even monitor the traffic, speed, and the amount of data used by the search engines over a period of time to determine if the performance relates to any issues or concerns for your website.

HTML Meta Directives

If you want to specify rules per HTML page, you can do this using HTML meta directives. This informs the search engines a specific action for a certain page.

Google, Bing, and Yahoo have implemented a number of HTML Meta directives including the following:

• NOINDEX META Tag – This tells a crawler not to index a certain page.
• NOFOLLOW META Tag – This tells a crawler not to follow a link going to another content on a certain page.
• NOSNIPPET META Tag – This tells a crawler not to display snippets in the search results for a certain page.
• NOARCHIVE META Tag – This tells a crawler not to show a cached link for a certain page.
• NOODP META Tag – This tells a crawler not to use a title and snippet from ODP (Open Directory Project) for a certain page.

Having your content indexed by major search engines can be a frustrating and time-consuming experience, but it can be done. By executing the techniques mentioned above, you can get confidential content removed quickly and prevent it from showing on search results pages. 

Search Group, a local Perth SEO company helps keep your business visible online through effective SEO and Internet marketing strategies. Visit www.searchroup.com.au for details about our services.

What SEO do I need in 2015?

It’s the middle of 2015. Have you begun your SEO campaign? Despite the predictions you might have read in the past year, Search Engine Optimisation (SEO) is still alive and kicking, and in fact, remains as one of the best ways to boost your traffic, get more from your website and raise returns for your business.

What is SEO?

Let’s do a short recap of what SEO is all about. It is the process of identifying the keyword phrases that potential customers use in search engines to find products and services. SEO helps ensure that your website faces little to no obstacles in capturing the maximum possible volume of relevant incoming traffic.

SEO revolves around the idea that by giving your website a boost, you are providing your business with distinct advantages in an Internet-driven business environment. It helps get your business found more easily. Through this, you gain more leads, better chances at conversions and greater potential to make sales, among others.

Organic traffic is a key element in search engine optimisation. It is the type of traffic that is derived from natural search engine indexes, as opposed to traffic bought through search engine marketing (SEM) campaigns such as Google Adwords. To achieve organic traffic, pages in your website have to be optimised so that they conform to search engine algorithms and as such, get ranked in the search engine results pages.

Needless to say, algorithms can vary from one search engine to another. Google’s ranking criteria may differ from that used by Bing. SEO seeks to optimise your website for any algorithm so that you are guaranteed a good ranking in any search engine. Additionally, search engine algorithm changes periodically—keeping up with the changes and ensuring the best results every time is part of the aims of SEO.

SEO for 2015

A lot of things have changed for SEO over the last couple of years alone, thanks to shifting demands and Google’s changing algorithms. That, however, does not necessarily mean that it is too late to engage in SEO. Here are a few pointers to remember for SEO this year:

·         Google’s goals remain unchanged.

Google may have gone through many algorithm shifts but their purpose has always been to provide users with the best search results. The search giant has always considered user experience a priority and as such, has consistently rolled out algorithm updates that moved towards accurate, informative search results in websites that are easy to use.

That said, your SEO campaign must have user experience as its foremost goal as well. Put yourself in the users’ shoes and imagine what would make your search easier and much more fruitful. Will you benefit from a website that loads quickly? If so, then ensure that your own site loads fast. Would you, as a user, benefit from having more pictures in a site? If so, then add more images to your own site. Obtain the assistance of experts to ensure that the steps you take are in line with current search engine standards, but putting the interest of your site visitors first is a great way to start.

·         Never underestimate content.

As it turns out, one of the best ways to provide optimal user experience is to make information readily available through content.  Blog posts and other forms of web content are now considered as key elements of most content marketing campaigns, and are religiously integrated into many businesses marketing strategies.

To maximise results, make sure that your content is high-quality. Readability and comprehensibility are huge factors for users. They can affect how they see you as a potential resource and how often they return to your site for your content. Boost your web content with SEO strategies to increase their effectiveness and their visibility.

Moreover, content is a great way to spread your links and strengthen your backlink portfolio. Users searching for resources will be more than happy to link back to your content and your website if your posts are informative enough. This is just another reason to invest in well-created content.

·         Go mobile.

Google reported that a large percentage of search queries nowadays are sent via mobile devices. Consequently, the search giant rolled out a mobile-friendly update last April. The update rewarded websites that have adopted a mobile-responsive design.

As more and more users surf the Internet through their mobile devices, it is within reason that your SEO strategy this year should prioritise mobile optimisation. A good start would be ensuring that your website is mobile-responsive. This would mean that users would be able to use it conveniently and comfortably even through the small screens of their mobile devices.

That said, you should be careful about potential errors, especially as the technology is relatively new. This should, however, not discourage you from taking the path to mobile-responsiveness as, with adequate preparation, the benefits far outweigh the potential risks.

The year 2015 is not too late for you to get started on your SEO campaign.  The Internet offers plenty of insights into the many steps you can take, as well as the many ways you can enhance your online marketing campaign. Get started now with Vorian Agency – a Google Partner, Bings Ads Professional company and a Hootsuite Solutions Partner.

Search Group - Perth SEO company - www.SearchGroup.com.au

Hi all,

Just an update to let you all know that I am now working as a Search Engine Optimization specialist with Perth SEO company Search Group.

Search Group is a leading Perth SEO Company specialising in website search engine optimisation (SEO), Search Engine Marketing (SEM), and online marketing activities with a large and growing portfolio of Australian clients.

Address: Suite 90, City West Business Centre, 102 Railway Street, West Perth, WA 6005

T: 08 9278 8899 
F: 08 9278 8890
E: matt@searchgroup.com.au
W: http://www.searchgroup.com.au/

So if you would like a free introductory SEO training session, website appraisal and review or just simply to discuss your search optimisation requirements to improve the visibility of your website on Google, Yahoo! and Bing then please drop me an email.