How to Attract Search Engines

Search Engine Optimization is a huge buzz-topic among webmasters. The premise is not really hard to grasp. To attract search engines to your site you need the following two things:

1. A Good Sitemap
2. A Robots.txt file

Some search engines, like Google, require that you also have any empty verification file in your file system which they will provide for uploading.

Sitemaps

Sitemaps are a listing of all of the pages on your website. All. of. them. Of course, that really means all of the useful ones. The more you list, the more you likely you are to be indexed in searches.

Sitemaps follow a specific protocol (0.9 to be exact) and are of the form:

< ?xml version='1.0' encoding='UTF-8' ?>

http://mysite.com/index.html

http://mysite.com/about.html

....

Sitemaps are a starting place for indexing, so the is no reason to have one. Sitemaps can also reveal specific links to a spider/bot that it may have missed due to a linking error. Remember spiders/bots crawl from one page reference to another.

If you have an RSS feed, you may also use that as a sitemap.

Robots.txt

The robots.txt file is a text file (that must be named robots.txt) that tells spiders/bots where they can and cannot go. It follows the form:

Allow: /
Disallow: /members.php
Disallow: /member-folder/
Disallow: /user-info

User-agent: *

It is always recommended that you include the root directory and then block specific pages or private directories. If you want to remove a page from a search engine you should include it in your ‘Disallow’ list. If you change domains or move information from one subdomain to another, it is always good to make your robot’s.txt file reflect these changes.
 

Source : tutorialized.com