Looking at robots.txt for SEO optimisation

What is robots.txt?

Robots.txt is a text file that contains rules to control the behaviour of search engines as they crawl your site, known as the 'robots directive'. You can create using any text editor software such as Notepad.



Each website created may include a robot.txt file located at the root level of the server. It may also be located at domain.com/robots.txt. This file is what search engines initially look for when they start to identify and crawl your website. As a robot directive, it explains to the search engine what it can and cannot do on the website. The file can either communicate to all search engines or selectively isolate and put independent requirements on individual search engines.

What should you exclude in your robots.txt file?

The robot.txt file may exclude areas in the site that you don’t want the search engine to discover, crawl, or particularly index. Without the robot.txt file, the results will be published in the search engine results page, allowing the general public to identify and access sections of the site that may not be appropriate. These sections may be directories of programming elements, a secure or admin section on the site, or components of your content management systems, depending on what you’re using, that can be blocked from the robots so they don’t crawl and index those elements.

Robots.txt Example:


It is important to be aware that some useful sections of the site could be inappropriately blocked. A common example is the /images directory. On average, five to seven percent of search results may be initiated through an image-based search with a user clicking through to the actual website. When the images directory is blocked, search engines may not be able to discover, identify, and list within their own directories of image search capabilities those images that are related to your business. For many businesses, this could benefit and support Search Engine Optimisation activities.

Sitemaps Location



Another thing you can see in a robot.txt file is a pointer to the location of your Sitemap. A Sitemap is a digital directory written in extensible mark-up language (XML) format which lists all the pages of your website that you wish to have indexed by the search engine. Putting a pointer within your robot.txt file to this location allows search engines to automatically discover your Sitemap and additional Sitemap files you have included, such as a geo Sitemap, and an images Sitemap. Supporting search engines in discovering these is beneficial for your site. For this reason, consider having a pointer to either a master Sitemap file or individual Sitemap files for your website.

The robots directive not only blocks certain robots’ actions as well as the pages to be crawled and indexed. It also determines speed. If the site has any particular issues in hosting, for example, where a crawl may slow down the actual site’s performance and affect the users’ experience, you can actually put speed controls to determine how fast the robot approaches and works its way through your website.

Google Webmaster Tools (GWT) and Bing Webmaster Tools (BWT) allows you to remove individual page files through single requests. The crawling speed may also be supported and addressed within GWT and BWT apart from being listed in the robot.txt file. You can even monitor the traffic, speed, and the amount of data used by the search engines over a period of time to determine if the performance relates to any issues or concerns for your website.

HTML Meta Directives

If you want to specify rules per HTML page, you can do this using HTML meta directives. This informs the search engines a specific action for a certain page.

Google, Bing, and Yahoo have implemented a number of HTML Meta directives including the following:

• NOINDEX META Tag – This tells a crawler not to index a certain page.
• NOFOLLOW META Tag – This tells a crawler not to follow a link going to another content on a certain page.
• NOSNIPPET META Tag – This tells a crawler not to display snippets in the search results for a certain page.
• NOARCHIVE META Tag – This tells a crawler not to show a cached link for a certain page.
• NOODP META Tag – This tells a crawler not to use a title and snippet from ODP (Open Directory Project) for a certain page.

Having your content indexed by major search engines can be a frustrating and time-consuming experience, but it can be done. By executing the techniques mentioned above, you can get confidential content removed quickly and prevent it from showing on search results pages. 

Search Group, a local Perth SEO company helps keep your business visible online through effective SEO and Internet marketing strategies. Visit www.searchroup.com.au for details about our services.

Search engines: How do they work?

If you have spent enough time online, you must have heard of the likes of Google, Bing, Yahoo Search and probably AOL. They are all search engines used for one of the most popular activities in the Internet—search. They are more than just digitised encyclopaedias; most of them come integrated with an extensive array of functionalities and features and cast an influence that almost literally affects the entirety of the Internet’s surface.

If your business can’t be found within the Search Engine, you don’t get the traffic…

When we talk about “Organic” or “Natural” Search, these are the results that appear in the main body of the SERPs (Search Engine Results Page), that aren’t paid for, and are determined by Google (and the other search engines in their own platforms) based on an algorithm that considers the content of your website and matches this to a consumer’s search query.

Did you know?

• 81% of internet users use search engines to find a website.
• 73% of all online transactions begin with a search engine.
• 87% of users only look at the first page of results.
• 85% of clicks are organic clicks

The Search Engine

Search Engines are online tools that scour the Internet to provide users with the information they need. The resources they allow users access to are virtually limitless, explaining their immense popularity and making them invaluable tools for any Internet surfer.    

To explain the scope of the Search Engine’s influence on modern living using a few words is practically impossible. The way they place information within any individual’s reach has affected the way we do business, communicate, study, eat, and conduct ourselves, among others. Their influence and power grows as the number of their users increase, and as Google has proven over the last few years, any changes can have dramatic effects on the Internet as a whole.

How it works

Search engines use automated robots, otherwise known as “spiders,” “bots,” “crawlers” or “indexers” to find content in websites. As the Internet is a figurative jungle made up of vast troves of information, search engines use links as pathways or guides. They follow these links in order to “spider” the sites and “index” the information in the site.

Webmasters also may also use .xml and .txt feeds as site maps of the content they want search engines to index in the directory. Ideally, these lists contain each of the URLs in the site. All indexed information is then gathered and stored in huge servers for easy retrieval at a later time. When a search is performed, search engines take indexed information matching the search query and present them as search results.

When using the robot method, search engines look at a number of factors to determine how deeply and frequently they will index your website. Some of these factors are:

• The uniqueness of the content in your site
• The uniqueness of the content in a page as compared to other pages in the site
• The uniqueness of the content in a pages versus all other pages in the Internet

To ensure that search engines find your content and index them, you need to keep an eye on one key factor—your links. Search engines determine the quality and the quantity of inbound links to your site. As a rule, it is best to ensure that your links come from high-quality sites. This will encourage search engines to index your site more frequently. It will also lead them to consider you as a high-quality source yourself and give you a better ranking in the results pages.

Obscure or “deep” search makes up a significant percentage all searches performed via search engines. As such, it is important that you pair great links with high-quality copy. Many website owners tend to think this means suffusing their web content with keywords. This may be a mistake as the practice tends to sacrifice readability. One thing you need to remember is that search engines today are focused on providing positive user experience and as such, they favour content that are easy for users to digest and understand. So when you write your copy, make sure to write for both humans and search engines.

Needless to say, the workings of a search engine may not be as simple as this post connotes, but this gives you a good idea. The extent of search engines’ influence over the net may be complex in itself, too, but remember that ultimately, they are tools for humans to use. Your focus therefore, should be the humans using them just as much as the spiders themselves.

Interested in a deeper understanding of Search Marketing for your business? Come along to our free marketing seminars at Vorian Agency. As a Google Partner and Bing Ads Professional Company, we also offer a comprehensive range of Search Marketing services. www.vorianagency.com.au

Introducing you and your website to RSS

RSS which stands for Really Simple Syndication, is a great way for your online business to keep in touch with your website visitors once they have been to your website and keep your website 'sticky'... meaning that once you have had the visitor to your site, you can encourage them to receive ongoing information about your company, news, information, new deals or services and keep your clients engaged.

It is actually very easy to implement RSS into your site! By including the following piece of one line HTML code in the head section of your webpages you'll be all set:
<link rel="alternate" type="application/rss+xml" title="RSS" href="ENTER_RSS_URL">

By doing this visitors to your website will then see within their web browser the RSS icon glow orange and become active - meaning that they can then subscribe directly to your feed, which is like book marking your specials or info page so that they will continue to receive this information from you each time you post out an RSS update.

So RSS is a bit like having an email newsletter subscription service for your website, but only better.

Because RSS was created a few years back as an alternative method to email delivery to get around some of the problems related to email marketing. RSS is automatically opt-in, as the user is choosing to subscribe. There is no spam filters that block your message getting through, nor delays in delivery that can happen with email - RSS is immediate, so a great way for speed to market of your business information or new deals.

A lot of people will actually have RSS available in their email clients like Outlook 2007 and above. So they can subscribe and follow your business information from the comfort of their email application. Once you have RSS set up, you can also provide your clients with a long list of other marketing tools that take advantage of RSS, and which can receive and display RSS. So you could actually create and provide downloadable Google Gadgets, Yahoo! Widgets, iPhone & iPad applications, even Screen Savers that receive and show your company information via RSS. These are some great brand building and marketing tools to help disseminate a brand awareness campaign and attract more eyeballs to your products and services. If you would like some further information about any of these just drop me an email.

So why am I harping on about RSS for your website?

Quite simple really as I'm all about driving more traffic to your online business, and retaining client interaction as well as pointing out ways to create better leverage with Search Engines. We're here to work for you and improve your online business.

RSS feeds can be crawled by Search Engines, they can even be submitted to Search Engines, along with a long list of RSS aggregator websites or online directories. This generates more exposure, and a better interaction with Search Engines encouraging them to find all your content, and particularly any new content you create and get this added into their Search Directories so your site is found more.

And here is a simple tip to actually manage and update your website's RSS feed...

Use Twitter! Yup, signup for your free Twitter account for your business at twitter.com and subsequently Twitter provides a RSS feed of your tweets. This RSS address can be put into that one line of code I provided you with at the top of this email and you are away. This means that you are using Twitter and getting your business information and website links out into Twitter and being seen there as well, and having it generated at the same time into your webpage. Bit like two for the price of... nothing!

Did you know that Twitter has a content relationship deal with Google - meaning, that Google trawls through all the data generated on Twitter. So when you tweet a new deal, a special, a new page added to your site or information about your business it will be found by Google and added very quickly into Google's own search index. This is one of the key reasons I recommend the use of Twitter for your business!

Okay, so a bit of information for you to digest this morning. I hope it makes sense. Of course if you have any further questions shoot me through an email.