-XML sitemap
-RSS sitemap
2 Robots.TXT-With the help of robots.txt webmaster can easily allow or disallow the content of web page to the crawler.
Example Of robots.txt
#Robots.txt www.example.com/robots.txt
User-agent: *
Disallow:
# Don’t allow spambot to crawl any pages
User-agent: spambot
disallow: /
3 Meta Robots-An Example of Meta Robots
<head>
<title>The Best Webpage on the Internet</title>
<meta name=”ROBOTS” content=”NOINDEX, NOFOLLOW”>
</head>
In above example “NOINDEX, NOFOLLOW” tells robots not to include the given webpage and links in their indexes.
4. Rel=”Nofollow”
An Example of nofollow
<a href=”http://www.example.com” title=“Example” rel=”nofollow”>Example Link</a>
simply “nofollow” tells search engines not to follow the link, but still some search engines still follow them for discovering new pages.
5. Rel=”canonical”
When two or more copies of the exact same content appear on your website under different URLs.
<head>
<title>The Best Webpage on the Internet</title>
<link rel=”canonical” href=”http://www.example.com”>
</head>