Looking at robots.txt for SEO optimisation

What is robots.txt?

Robots.txt is a text file that contains rules to control the behaviour of search engines as they crawl your site, known as the 'robots directive'. You can create using any text editor software such as Notepad.



Each website created may include a robot.txt file located at the root level of the server. It may also be located at domain.com/robots.txt. This file is what search engines initially look for when they start to identify and crawl your website. As a robot directive, it explains to the search engine what it can and cannot do on the website. The file can either communicate to all search engines or selectively isolate and put independent requirements on individual search engines.

What should you exclude in your robots.txt file?

The robot.txt file may exclude areas in the site that you don’t want the search engine to discover, crawl, or particularly index. Without the robot.txt file, the results will be published in the search engine results page, allowing the general public to identify and access sections of the site that may not be appropriate. These sections may be directories of programming elements, a secure or admin section on the site, or components of your content management systems, depending on what you’re using, that can be blocked from the robots so they don’t crawl and index those elements.

Robots.txt Example:


It is important to be aware that some useful sections of the site could be inappropriately blocked. A common example is the /images directory. On average, five to seven percent of search results may be initiated through an image-based search with a user clicking through to the actual website. When the images directory is blocked, search engines may not be able to discover, identify, and list within their own directories of image search capabilities those images that are related to your business. For many businesses, this could benefit and support Search Engine Optimisation activities.

Sitemaps Location



Another thing you can see in a robot.txt file is a pointer to the location of your Sitemap. A Sitemap is a digital directory written in extensible mark-up language (XML) format which lists all the pages of your website that you wish to have indexed by the search engine. Putting a pointer within your robot.txt file to this location allows search engines to automatically discover your Sitemap and additional Sitemap files you have included, such as a geo Sitemap, and an images Sitemap. Supporting search engines in discovering these is beneficial for your site. For this reason, consider having a pointer to either a master Sitemap file or individual Sitemap files for your website.

The robots directive not only blocks certain robots’ actions as well as the pages to be crawled and indexed. It also determines speed. If the site has any particular issues in hosting, for example, where a crawl may slow down the actual site’s performance and affect the users’ experience, you can actually put speed controls to determine how fast the robot approaches and works its way through your website.

Google Webmaster Tools (GWT) and Bing Webmaster Tools (BWT) allows you to remove individual page files through single requests. The crawling speed may also be supported and addressed within GWT and BWT apart from being listed in the robot.txt file. You can even monitor the traffic, speed, and the amount of data used by the search engines over a period of time to determine if the performance relates to any issues or concerns for your website.

HTML Meta Directives

If you want to specify rules per HTML page, you can do this using HTML meta directives. This informs the search engines a specific action for a certain page.

Google, Bing, and Yahoo have implemented a number of HTML Meta directives including the following:

• NOINDEX META Tag – This tells a crawler not to index a certain page.
• NOFOLLOW META Tag – This tells a crawler not to follow a link going to another content on a certain page.
• NOSNIPPET META Tag – This tells a crawler not to display snippets in the search results for a certain page.
• NOARCHIVE META Tag – This tells a crawler not to show a cached link for a certain page.
• NOODP META Tag – This tells a crawler not to use a title and snippet from ODP (Open Directory Project) for a certain page.

Having your content indexed by major search engines can be a frustrating and time-consuming experience, but it can be done. By executing the techniques mentioned above, you can get confidential content removed quickly and prevent it from showing on search results pages. 

Search Group, a local Perth SEO company helps keep your business visible online through effective SEO and Internet marketing strategies. Visit www.searchroup.com.au for details about our services.

New Client Branding Case Study: GoCrisis

Vorian Agency, a West Perth based digital marketing company, is a certified Google Partner, Bing Ads Professional company, Hootsuite Solutions Partner and MailChimp Expert company.

We have recently undertaken a complete branding exercise for new client GoCrisis in preparation for their inaugral conference to be held at The Duxton Hotel Perth this Thursday 8th October, 2015.

The conference 'Crisis Management Leaders Summit' features speakers including; the Crisis Director Malaysia Airlines, Curtin University, Computer Emergency Response Team, Former Queensland Premier, GoCrisis, Air China, Vorian Agency's own General Manager Matt Lynch and many more who will share their personal experiences, discuss the latest research and trends on the risks that businesses face, the impact that social media and the real-time news cycle have on crisis and emergency management, leadership in crisis and what we can learn from recent world events and how to apply these lessons to corporate crisis response. The conference website, speakers list and agenda can be seen here. Follow the conversation online around the conference #gocrisis #vorianagency

Vorian Agency's diverse service range has contributed to GoCrisis with;

• New Logo Design, and Style Guide
• Responsive Website Designs for both the Company and Event
• Custom Event Booking Payment System
• Responsive MailChimp eNewsletter template
• Search Engine Optimisation
• 16-page glossy Training Brochure printed and .pdf for web
• 2-page glossy Conference Agenda printed flyer
• 3 different pull up conference banners

Vorian Agency provides customised marketing solutions including; search engine optimisation (SEO), pay-per-click advertising (PPC), social media marketing (SMM), web design and development, branding, print and graphic design, as well as video and audio production services that are aimed at promoting your brand’s products and services through a variety of results oriented strategies.

Find out how Vorian Agency can help your business today. Call 1300 100 333 now and arrange a no obligation free meeting with our friendly marketing team, and receive a free website evaluation report. Or email info@vorian.com.au and learn about the extensive range of services that Vorian Agency have to offer. Please visit our website: vorianagency.com.au

Did you miss out on our recent Google Partners Event? Here is a recording of the livestream with Google so you can find out more about 'Micro-Moments' in marketing:
https://www.youtube.com/watch?v=uWMYkhI7qtk

Crisis Management Leaders Summit, Perth 2015

Crisis Management Leaders Summit, Perth 2015

Perth 8th October 2015
#gocrisis #vorianagency
http://www.perth2015.gocrisis.com/

Business and government are experiencing an unprecedented increase in risk and an exposure to crisis; globalisation of supply chains and customer base, climate change, population growth, large demographic shifts, new technologies, an explosion of regulatory requirements – more than ever, it seems that the need for preparing for the unexpected is necessary. We are faced with a surge in the occurrence of product recalls, unexpected fatalities, natural disasters, data loss, disrupted supply and, as a consequence, the expectation from those affected by crisis – customers, the media, public, employees, shareholders – are more demanding.

The Crisis Management Leaders Summit focuses on the current topics concerning today’s crisis realities and how businesses and governments are coping. It offers delegates a unique opportunity to look at crisis management from different viewpoints and to learn cross-industry lessons and best practice.

With contributions from the Crisis Director Malaysia Airlines, Curtin University, Computer Emergency Response Team, Former Queensland Premier, GoCrisis, Air China and many more, our speakers will share their personal experiences, discuss the latest research and trends on the risks that businesses face, the impact that social media and the real-time news cycle have on crisis and emergency management, leadership in crisis and what we can learn from recent world events and how to apply these lessons to corporate crisis response. We will look at global organisations in crisis and learn from their failures and successes.

Find out how to Manage Social Media in Crisis, I am presenting on this topic at the Crisis Management Leaders Summit, Perth 8th Oct, and you can find out what is in my own #EDC (Every Day Carry) Marketing Tool-kit.

The Scout Motto is 'Be Prepared'

"Be Prepared... the meaning of the motto is that a scout must prepare himself by previous thinking out and practising how to act on any accident or emergency so that he is never taken by surprise."
Robert Baden-Powell 

How prepared is your business?...

Google My Business Photography - Can you be found?

• 81% of internet users use search engines to find a website.
• 73% of all online transactions begin with a search engine.

Google Map Locations for businesses, the old ‘Google Places’ is now known as Google My Business. But it is now more that just a management console for your business location details… The Dashboard integrates: Location, Google+, Insights for your Business, Review Management, Google Analytics for your website, YouTube, Adwords Express, and even the ability to start a Hangout video call.

Vorian Agency is a Digital Marketing Agency based in West Perth, and we are Google Partners, Bing Ads Professionals, a Hootsuite Solutions Partner as well a MailChimp Expert company.

We're here to assist businesses of all shapes and sizes to be found online. Apart from a range of SEO and Marketing packages we also have a Google Certified Photographer who can photograph the interactive 360-degree Virtual Tours for your business for Google Maps or to place on your website or social media channel. We also have professional photographers to update your social media profile pictures, take product shots or images of your business.

Interested to see what a 360-degree Virtual Tour looks like for your business? Click here to view Vorian Agency and visit us virtually. Then call us on 1300 100 333 or email info@vorian.com.au to book your own photo shoot.

Using Canonical Meta Tags to address Duplicate Content site issues

If you are conscious about your website’s ranking in the search engine results, then one of the last things you want to find out is that you have several instances of duplicate content in your site. In case you do find an accumulation of duplicate content, don’t worry. You can always use the canonical meta tag as an effective means to address the issue.

What is the canonical tag?

The canonical meta tag is placed in the HTML header of a web page. It tells search engines that the page is a copy of the URL indicated in the tag and that it should be treated as such. This prevents search engines from imposing penalties for duplicate content on your page because of the existence of said copies. This is the canonical tag:

<link rel="canonical" href="URL here" />

“Wait,” you say. “I don’t create duplicate content. I don’t need the canonical tag.”

Indeed. Any responsible website owner knows that creating duplicates is one of the easiest ways to drop out of the rankings. As such, many go to great lengths if only to ensure that every page is populated by unique content. Duplicates, however, do not necessarily have to be created by you. Here are some of the instances by which duplicate content may be created without your knowledge—or consent:

·         The system may create session ID URLs in lieu of cookies or other URL trackers if they are disabled in your system.

·         Different URL cases are usually redirected to the same location but at times, they may be recognised by search engines as duplicates of each other.

·         E-commerce pages often make multiple URLs for the same page, especially if there are multiple available options for a single product. They also tend to use country-specific URLs to accommodate different currencies but these may be identified as duplicates of a single page.

·         The mobile version of a page is also prone to getting mislabelled as a duplicate.

·         Different mirrors of secured pages are often crawled separately and as such, may be read as duplicate content.

The canonical tag can be used to prevent these instances from having a detrimental impact on your website. While the process itself of applying the tag is straightforward, there are a number of ways that you can get it wrong.

1.       Do not have more than one canonical tag in a single page. A single canonical tag at the <head> section of every page is enough. Having more or putting the tag in the wrong place will lead the crawlers to ignore the tag(s).

2.       Do not use the canonical tag for multiple pages. In cases when you really must deal with multiple pages, you can use the prev and next tags instead of the canonical tag. You can also redirect crawlers to index a “View All” page instead. This page collates all the results together for easy review.

3.       Do not use your homepage as your preferred site. A canonical tag tells search engines to crawl the indicated URL instead of the page with the tag. Use this opportunity to have search engines reference the other pages in your site.

4.       Do not use canonical tags for featured content. There is a good chance that these pages will be ignored, making you lose some of the traffic for your features.

5.       Do not use 301 redirects and canonical tags interchangeably. While they do have similarities, 301 redirects are useful for when you have performed changes on your site’s structure. A 301 effectively redirects traffic, too. Canonical tags redirect crawlers, not traffic, and are great only for addressing duplicate content.

For all its apparent benefits, the canonical tag is effective only when you use it correctly. Keep in mind the proper mechanics before implementing it in your pages. You can look for strategies on the Internet on how you can use it or which other tags and techniques you can pair it with to give you better results.

Vorian Agency’s dedicated SEO – Search Engine Optimisation division is Search Group. Vorian Agency is a Google Partner, Bing Ads Professional, Hootsuite Solutions Company and a MailChimp expert company.  We provide free seminars on SEO and marketing best practices for businesses in Perth, Western Australia. We will help you get acquainted with the tools you can use to help you reach your marketing goals. Contact us now on 1300 100 333, visit us at www.vorianagency.com.au or email info@vorian.com.au to get started.

Search engines: How do they work?

If you have spent enough time online, you must have heard of the likes of Google, Bing, Yahoo Search and probably AOL. They are all search engines used for one of the most popular activities in the Internet—search. They are more than just digitised encyclopaedias; most of them come integrated with an extensive array of functionalities and features and cast an influence that almost literally affects the entirety of the Internet’s surface.

If your business can’t be found within the Search Engine, you don’t get the traffic…

When we talk about “Organic” or “Natural” Search, these are the results that appear in the main body of the SERPs (Search Engine Results Page), that aren’t paid for, and are determined by Google (and the other search engines in their own platforms) based on an algorithm that considers the content of your website and matches this to a consumer’s search query.

Did you know?

• 81% of internet users use search engines to find a website.
• 73% of all online transactions begin with a search engine.
• 87% of users only look at the first page of results.
• 85% of clicks are organic clicks

The Search Engine

Search Engines are online tools that scour the Internet to provide users with the information they need. The resources they allow users access to are virtually limitless, explaining their immense popularity and making them invaluable tools for any Internet surfer.    

To explain the scope of the Search Engine’s influence on modern living using a few words is practically impossible. The way they place information within any individual’s reach has affected the way we do business, communicate, study, eat, and conduct ourselves, among others. Their influence and power grows as the number of their users increase, and as Google has proven over the last few years, any changes can have dramatic effects on the Internet as a whole.

How it works

Search engines use automated robots, otherwise known as “spiders,” “bots,” “crawlers” or “indexers” to find content in websites. As the Internet is a figurative jungle made up of vast troves of information, search engines use links as pathways or guides. They follow these links in order to “spider” the sites and “index” the information in the site.

Webmasters also may also use .xml and .txt feeds as site maps of the content they want search engines to index in the directory. Ideally, these lists contain each of the URLs in the site. All indexed information is then gathered and stored in huge servers for easy retrieval at a later time. When a search is performed, search engines take indexed information matching the search query and present them as search results.

When using the robot method, search engines look at a number of factors to determine how deeply and frequently they will index your website. Some of these factors are:

• The uniqueness of the content in your site
• The uniqueness of the content in a page as compared to other pages in the site
• The uniqueness of the content in a pages versus all other pages in the Internet

To ensure that search engines find your content and index them, you need to keep an eye on one key factor—your links. Search engines determine the quality and the quantity of inbound links to your site. As a rule, it is best to ensure that your links come from high-quality sites. This will encourage search engines to index your site more frequently. It will also lead them to consider you as a high-quality source yourself and give you a better ranking in the results pages.

Obscure or “deep” search makes up a significant percentage all searches performed via search engines. As such, it is important that you pair great links with high-quality copy. Many website owners tend to think this means suffusing their web content with keywords. This may be a mistake as the practice tends to sacrifice readability. One thing you need to remember is that search engines today are focused on providing positive user experience and as such, they favour content that are easy for users to digest and understand. So when you write your copy, make sure to write for both humans and search engines.

Needless to say, the workings of a search engine may not be as simple as this post connotes, but this gives you a good idea. The extent of search engines’ influence over the net may be complex in itself, too, but remember that ultimately, they are tools for humans to use. Your focus therefore, should be the humans using them just as much as the spiders themselves.

Interested in a deeper understanding of Search Marketing for your business? Come along to our free marketing seminars at Vorian Agency. As a Google Partner and Bing Ads Professional Company, we also offer a comprehensive range of Search Marketing services. www.vorianagency.com.au