Setting up sitemap.xml and robots.txt: so the search engine knows what you have

Not necessarily. The most common defect in our sample looks like this: the address returns a code of 200, and the response body is zero bytes. The browser shows an empty page, the owner sees that "the file is there", the search engine does not receive any address. Six sites we measured gave the map in exactly this condition. You need to look at the size of the body and the number of records, and not at the response code.

Needed on a large catalog. The map is a straightforward list of what you think is worth checking out, with dates of change; without it, the bot depends on how deep it will go on the links, and it may not reach the deep pages of the directory for months. For a small site with a good link, Google directly says that a map may not be necessary, and we do not argue with this.

Works if the address is explicitly declared in robots.txt. One of the measured wholesale catalogs was saved precisely by this: the standard addresses gave garbage, and the working map lay on its own path and was registered in robots. But this is a trap for audit tools and for humans, so we transfer the card to a standard address. Another detail of the same level: the addresses inside the map must be on the same host and protocol as the map itself - a map from a subdomain or from an old http bot simply won't count.

The format limits are 50 thousand addresses and 50 MB in uncompressed form. In practice, it is better to split earlier: in the measured store of the tool, the map was generated with one file in 34.2 seconds, and the bot has a chance not to wait. Our recommendation is a separate subcard for each type of page: products, categories, informational. Then the traversal report shows exactly which part of the directory was dropped, not "something wrong with the map". In the dental supply store, it is done: an index of four files, 1 + 21 + 96 + 5 = 123 addresses.

Service sections that have nothing to do in the search: shopping cart, checkout, office, internal search, admin. Be careful with filters and parameters - the prohibition in robots does not remove the page from the index, it only prohibits traversal, and the page may remain in the output without a description. There are other mechanics at work to clean up the issue, and we won't pretend that robots can do it.

Because the map does not oblige the search engine to index. It's a "here's what we've got" message, not a command, and Google clearly documents it. A product card without a description, without a unique text, and without demand may remain outside the index, no matter how much you submit it. The real number of indexed pages is visible only in your Search Console — 13 thousand addresses in the map do not mean 13 thousand pages in the index.

No, this is the expected course of events. When duplicates and service addresses disappear from the map, the index counter drops first - simply because the extra stops getting there. The weight is not lost, it stops being scattered between several addresses of the same product. We say this in advance, because without an explanation the fall in the report looks like an accident.

Free verification - 1-2 days and only the website address. More than two days, if you need to regenerate the map and fix the robots; up to ten, if you have to divide the map into submaps, clean it of service addresses and compile it with language versions. From you - access to CMS or hosting, access to Search Console and a list of sections that should not be in the search.