This tutorial will cover the two essential files for SEO, sitemap and robots.txt, their purpose, and how to create them.
- Sitemap: A sitemap is an XML file that lists all the pages on your website. It helps search engines find and crawl all the pages on your site, including pages that might be difficult to find using traditional crawling methods.
Example of Sitemap:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://www.example.com/</loc>
<lastmod>2022-02-14T10:01:00+00:00</lastmod>
<priority>1.00</priority>
</url>
<url>
<loc>https://www.example.com/about</loc>
<lastmod>2022-02-14T10:02:00+00:00</lastmod>
<priority>0.80</priority>
</url>
<url>
<loc>https://www.example.com/contact</loc>
<lastmod>2022-02-14T10:03:00+00:00</lastmod>
<priority>0.80</priority>
</url>
<url>
<loc>https://www.example.com/products</loc>
<lastmod>2022-02-14T10:04:00+00:00</lastmod>
<priority>0.70</priority>
</url>
</urlset>
In this example, the sitemap lists four pages on a website, including their URLs, the last time they were modified, and their priority.
- Robots.txt: A robots.txt file is a text file that tells search engines which pages or sections of your site to crawl or not crawl. You can use it to restrict access to certain pages, such as those that contain sensitive information or duplicate content.
Example of Robots.txt:
User-agent: *
Disallow: /admin/
Disallow: /login.php
Disallow: /search/
Sitemap: https://www.example.com/sitemap.xml
In this example, the robots.txt file tells search engines not to crawl the /admin/ and /search/ directories or the login.php page. It also includes a reference to the sitemap file to help search engines find it.
Creating a sitemap and robots.txt file is essential for SEO because it helps search engines crawl and index your website more efficiently. Keep these files updated as you add or remove pages from your site. You can use tools like Yoast SEO or Google Search Console to create and manage these files.