Robots and Sitemaps for Search Engines


    The search engines have billions of web pages to crawl. This task is not only daunting, but takes a lot of time. The result to us is, our website changes could take weeks to appear in the search engine results!

Use the sitemap.xml file to help solve this problem.

Really Great People
    Most search engines give priority to websites using a sitemap for search engines, including search engines you didn't even realize existed. They can simply look over your sitemap, then, only look (or crawl) the webpages that you indicate have changed. It saves Google a ton of time, and if we all did it, we'd get listed a lot sooner.

    The sitemap.xml file can save you a bunch of time too. Essentially, it is a list of files on your site. Each file is dated at the last update. This way, the large search engines can simply read your Sitemap.xml file to see what else to look at, rather than every single thing individually.

    Here is a sample of the sitemap.xml file. The highlighted section is repeated for every web page in your web site (with a different address). Most of the parameters should be self explanatory, but just in case, you can always go to for more details.

sitemap.xml - Notepad
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="">

    <loc> </loc>





    Once you have built your sitemap.xml file, how would anyone know? The simple answer is the robots.txt file.

    Virtually every search engine looks for the robots.txt file. It's job is to tell the search engines what to look at and what to ignore. It's quite simple to add your new sitemap.xml file to the list of things to look at.

robots.txt - Notepad
# robots.txt for

User-agent: *
Allow: /

Disallow: /outlinks/ # Outgoing Links

Permanent Search Engine Submission

    Once we have the Sitemap.xml and robots.txt file created and installed on our site (root directory), then we can submit web site to search engine about their existance. Both Google and Yahoo allow site feeds. For your security, you need to sign up for a free account and install a verification code on your website.

Google Webmaster Resources

Under Site Configuration, select Sitemaps. Also, look at Robots.txt.
Bing/Yahoo - Submit

Select 'Submit Site Feed'.

    Once you take a look at these Webmaster Tools, you'll realize they have a ton of good information. It does take a few days to accumulate enough information to show you anything.

    Search Engine algorithms aren't simple. If you've built in good site content, you'll often discover you rank well for phrases you didn't actually write. They often use synonyms and misspellings. Webmaster tools show your search ranking for various phrases quite easily. How cool is that?