Analyzing server logs for SEO: what the search bot is actually doing on your site

Detailing and coverage. Google's report groups requests by file type and destination and shows a summary with a delay of several days: it shows that the bypass has failed, but not where exactly. In the log, each request is a separate line — address, response code, time, size, agent. Therefore, the question "why did it bypass" the report closes at the level of the trend, and the magazine - at the level of specific addresses that began to give 301 or 500 of such and such a number. The second difference: Search Console only shows Google bots. Bing and parsers that go by someone else's name do not exist in it. We start with the report anyway - it is free and gives a reference for three months ago.

From checking whether they are being conducted at all. On some hostings, the logs are disabled by default, on some they are stored for a day and are deleted. We do this check for free in one or two days. If there are no logs, some of the conclusions can be obtained by other methods: Search Console crawl statistics plus an external sitemap crawl with a check of response codes. That's how all three cases on this page are done — and that's why each one has a paragraph about what this method didn't show. It is a weaker instrument and we call it weaker.

The agent line is faked with a single line of code, so we don't take it seriously. The check goes through a reverse DNS query to the address and a direct query back: the address must resolve to the official name of the search engine, and this name must resolve back to the same address. One return request is not enough, because the record is under the control of the owner of the address. Separately, there is a technical detail on which independent attempts stumble: if a network service is in front of the site, the address of the service itself, and not of the person who came, is in the address field in the log. As long as the real address isn't slipped through the header and read from it, the check will honestly show zero real bots on the entire array.

The most frequent finding is one: the detour goes to the wrong place. Requests are spent on service addresses, on filter combinations, on pages that have not been around for a long time, and on transitions. How large-scale this story is, you can see even from the outside: in the measured lingerie store, out of a sample of 150 random sitemap addresses, 50 returned 301, and on the scale of the directory, this is about 37 thousand addresses, for each of which the bot pays a request. The second most frequent find is a section that is not in the magazine at all: the pages are advertised and linked, and the bot has not reached them even once in a month. The third is the nighttime response time, which is twice as bad as the daytime response time, and a simultaneous drop in the bypass frequency.

At least a week to see daily cycles. The moon gives a picture that can already be believed, especially when the catalog is updated unevenly. Less than a week is guesswork backed by numbers, and that's not how we work. A tip to do today regardless of whether we do anything together: enable 30 days of log storage with rotation and archiving. It costs nothing but a few gigabytes of disk space. The worst situation is when the traffic dropped yesterday, and the magazines live for a day: there is no way to establish the reason.

Sometimes it does, and that's the most valuable scenario. The log shows the date when the bot stopped entering the section; it can be seen that part of the addresses began to give an error; it can be seen that the response time has increased, and the frequency of traversal has followed it. But the line is hard: the log explains bot behavior, not human behavior. If traffic has dropped due to a season, a change in demand, or a new competitor in publishing, it will not be in the magazines. Therefore, we tackle this issue only in conjunction with analytics and Search Console.

This is the most common advice and the easiest way to make things worse. Blocking according to the frequency of requests hits the real search engine first - it just happens more often than all others. Blocking by agent string doesn't work against those who forge that string, which is exactly the ones you wanted to stop. Our recommendation: first look in the log and see who took how much, then limit by address - by the origin of the request, with a list of allowed search engines. And check separately if your load protection is blocking any of them already.

Compressed archive for the required period, in parts. Text logs compress well: a monthly file of a few gigabytes usually turns into hundreds of megabytes. If it is inconvenient to transfer, we take access for reading and take the necessary things ourselves. We do not need personal data: the address of a human visitor can be anonymized or truncated before transmission, the conclusions about bots are not affected. What should be kept unchanged is the request address, response code, time, size, and agent string.

7-18 working days from the moment we receive the logs: 1-2 days for verification and receipt, 1-3 days for format analysis and screening of foreign traffic, 2-5 crawl map, 1-3 sitemap and search console reconciliation, 2-5 report and conversation. On hand - a written document with numbers, samples of raw lines for each output and a list of tasks, divided into what is done by the setup in an hour and what needs a developer. Implementation is not included in this work.