sitemap.xml
The search engines have billions of web pages to crawl. This task is not only daunting, but takes a lot of time. The result to us is, our website changes could take weeks to appear in the search engine results!
Most search engines give priority to websites using a sitemap for search engines, including search engines you didn't even realize existed. They can simply look over your sitemap, then, only look (or crawl) the webpages that you indicate have changed. It saves Google a ton of time, and if we all did it, we'd get listed a lot sooner.
The sitemap.xml file can save you a bunch of time too. Essentially, it is a list of files on your site. Each file is dated at the last update. This way, the large search engines can simply read your Sitemap.xml file to see what else to look at, rather than every single thing individually.
Here is a sample of the sitemap.xml file. The highlighted section is repeated for every web page in your web site (with a different address). Most of the parameters should be self explanatory, but just in case, you can always go to sitemaps.org for more details.
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>http://makemyownwebpage.com/robots-and-sitemaps-search-engines </loc>
<lastmod>2024-05-17</lastmod>
<changefreq>weekly</changefreq>
<priority>0.8</priority>
</url>
...
...
...
</urlset>
robots.txt
Once you have built your sitemap.xml file, how would anyone know? The simple answer is the robots.txt file.
Virtually every search engine looks for the robots.txt file. It's job is to tell the search engines what to look at and what to ignore. It's quite simple to add your new sitemap.xml file to the list of things to look at.
User-agent: *
Allow: /
Disallow: /outlinks/ # Outgoing Links
Sitemap: http://makemyownwebpage.com/Sitemap.xml
Permanent Search Engine Submission
Once we have the Sitemap.xml and robots.txt file created and installed on our site (root directory), then we can submit web site to search engine about their existance. Both Google and Yahoo allow site feeds. For your security, you need to sign up for a free account and install a verification code on your website.
Search Engine algorithms aren't simple. If you've built in good site content, you'll often discover you rank well for phrases you didn't actually write. They often use synonyms and misspellings. Webmaster tools show your search ranking for various phrases quite easily. How cool is that?