Ask ten Indian small business owners what robots.txt does, and nine will guess wrong. That is not a criticism. These three files and tags sit quietly in the background of every website, and most developers set them once and forget them. But robots.txt, your XML sitemap and your canonical tags decide how Google sees your site. Get them wrong and your best pages stay invisible. Get them right and you give every rupee you spend on content and ads a better chance to work.
Here is what each one actually does, in plain language, and the mistakes we fix most often on Indian business websites.
Robots.txt: the signboard, not the lock
Robots.txt is a small text file at yourdomain.com/robots.txt. It tells search engine crawlers which parts of your site they may visit. Two things surprise people. First, it is public; anyone can read it. Second, it is a request, not a lock. Well-behaved crawlers like Google follow it, but it does not hide a page from someone who has the link, and it does not remove a page from Google by itself.
The most common mistake we see: a developer blocks the whole site with "Disallow: /" during development and nobody removes it at launch. The site then wonders why Google shows nothing for months. The second most common: blocking CSS and JS files, which stops Google from rendering your pages properly and quietly hurts rankings.
One more robots.txt myth worth killing: adding a page to robots.txt does not keep it out of Google. If another site links to that page, Google can still index the address and show it without a description. To keep a page out of search results, the correct tool is a "noindex" meta tag on the page itself, combined with allowing the crawl so Google can see that tag.
XML sitemap: your site's index for Google
A sitemap is a simple list of the pages you want indexed, usually at yourdomain.com/sitemap.xml. Google can find pages without one, but a sitemap tells it what exists, what changed and what matters. For a new website, or one that publishes blogs regularly, it is the difference between Google discovering your new page this week or next month.
The mistakes: sitemaps that still list old deleted pages (which wastes Google's crawl budget on 404s), sitemaps that are never regenerated after new pages go live, and sitemaps that were never submitted in Google Search Console. Submitting takes five minutes and also unlocks the reports that show you exactly how Google sees your site.
Canonical tags: one page, one address
The same page often exists at several addresses: with www and without, with http and https, with tracking parameters, or as a filter page on an e-commerce site. Google sees these as duplicate pages and splits their ranking power. The canonical tag, a single line in the page head, tells Google which address is the real one. All the ranking value then flows to that one URL.
The classic Indian e-commerce mistake: every product filter combination creates a new URL, and none of them have canonicals, so Google indexes thousands of near-identical pages and the actual product pages rank poorly. Another: the canonical points to the www version while the site redirects everything to non-www, sending Google in circles.
A ten-minute self-check
- Open yourdomain.com/robots.txt. It should exist and should not say "Disallow: /" on its own.
- Open yourdomain.com/sitemap.xml. Your newest page should be listed. If it is not, your sitemap is not being regenerated.
- View source on your homepage and search for "canonical". The address there should match the address you actually want ranking, www or non-www, https.
- Log into Google Search Console and check the Pages report. "Discovered, currently not indexed" in large numbers usually points at one of these three files.
The bottom line
None of this is glamorous, and none of it needs a big budget. It needs one careful hour. These three files are the plumbing of SEO: invisible when right, expensive when wrong.
At VAIR IT Technologies, technical SEO checks like these are part of every website we build and every SEO engagement we run for businesses across Delhi NCR and beyond. If you want us to audit your site's foundations and fix what is holding it back, book a free consultation with our team.
