Technical SEO

Indexing of a large online store catalog

Indexing a large catalog of an online store is work on ensuring that each of the tens of thousands of pages is accessible to the bot, is not duplicated, and is rendered quickly. It begins not with a site map, but with a count: how many items are actually in the catalog and how many of them the search engine receives as a list. These two numbers rarely match. At the gear store we measured, the discrepancy between the sitemap and the standard search was 0.4% — and it's those percentages that show where the catalog is leaking.

See how it works

Price: after a free audit
Guarantee: 30 days after project sign-off
What are we doing?: we calculate the catalog, rebuild the site map, remove repetitions, add depth to the bypass

Price: after a free audit
Guarantee: 30; days after project sign-off
What are we doing?: we calculate the catalog, rebuild the site map, remove repetitions, add depth to the bypass
The term of the first cycle: 15–45; working days, then a monthly installment
The largest catalog in its own dimension: 54,302; products and 635 categories
The largest site map in size: 59,189; addresses: 18 subcards for 3,000 and another for 1,321
Speed by 54 thousand items: 413.8; ms to the first byte - volume is not a sentence
What we do not promise: number of pages in the index: this is the decision of the search engine
Before the estimate: free indexing slice, 3-5 business days

Process steps

Transparent stages with approval at every step

Total duration:15–45 days

A slice of indexing status
3-5 working days
We count the catalog using two methods, we analyze the map by submaps, we look at repeats, language branches, update dates and response time. Everything with the date of measurement - there will be something to compare with later.
Comparison with what the search engine sees
2-4 working days
Search Console data is next to our external count. The main thing becomes visible here: how many addresses you offer to the bot and how many of them actually reached the index.
Reconstruction of the site map
3-8 working days
Index file, reasonably sized submaps, last modified working dates, all language branches, map declaration in robots.txt. One entry per address, no exceptions.
Duplication cleaning and canonicalization
4-12 working days
Duplication patterns—different category paths, filter parameters, language prefixes—are covered by rules. We warn you in advance: the number of pages in the index will decrease for some time, and this is expected.
Depth reach and speed
3-10 working days
Relinking to deep nodes, cache of categories, card return under bypass. We measure on real addresses from the end of the directory, not on the main one, because the main one is almost always fast.
Monthly slice
monthly, from launch
Five numbers once a month: addresses in the map, unique addresses, pages in the index, time of appearance of a new product, response time. Catches a broken regen before it eats a block.

Technologies & integrations

What we build on and what it connects to

Stack

sitemap-index from submaps: format limit — 50,000 addresses per file, but cut earlier, 3,000–3,500
lastmod: only works as long as the dates actually change - a frozen date is worse than no date at all
bypassing the map with a script: actually counts the addresses; on tens of thousands of addresses, it is done slowly, so as not to put the site down
standard CMS search as the second method: gives a number where the search gives a full output, and is silent where it does not
Search Console: the only source of the number of indexed; data with download delay and limitation
Screaming Frog: shows the depth of clicks; does not see pages to which there are no links
structured data Product and Offer: change the appearance of the snippet, not items
nginx and category cache: keep yield under bypass but hide slow requests instead of treating them

Integrations

Google Search Console
Bing Webmaster Tools
Google Merchant Center
GA4
Cloudflare

What's included

Complete list of work and what you get as a result

Calculating the directory by two independent methods — sitemap traversal and regular CMS search — with a resolution of the discrepancy: the size of the directory ceases to be a matter of trust in the admin report
Sitemap as an index of submaps of several thousand addresses: the file is checked by eyes, quickly generated and updated in parts
Last Modified Dates That Really Change: A date set blindly by the generator invalidates the entire file
All language branches in the map — or a separate file for each language so that the second version is not left out of the list
Cleanup of repetitions: one address - one record, so that the detour goes to new pages, and not to the tenth repetition of a known
Canonical URLs for Filters and Sorts - options stop multiplying copies of the same category
Relinking based on depth: the path to the card from the end of the directory should be several clicks, not just a line in the site map
Directory throughput under traversal load - category cache and card response time measured on real deep addresses, not root
Structured data Product, Offer and breadcrumbs on cards: it affects the appearance of the snippet, but not the position
Monthly slice of five numbers: addresses in the map, unique addresses, pages in the index, time of appearance of a new product, response time

When this service isn't right

What's not included — so there are no surprises at delivery

Writing descriptions for thousands of cards is a separate job with a separate budget
Guarantee of the number of pages in the index: the decision is made by the search engine, not the contractor
Completion of the catalog and expansion of the assortment
Advertising and buying traffic for the period until the directory reaches the index
Rewriting the engine: if the CMS does not know how to render the correct map, this is a development task

Who it's for

Situations where this service delivers results

Scenario 1 of 4

The catalog is growing faster than the search engine bypasses it

New positions are added every week, and last year's cut lives on in the search. The reason is usually not in the budget: the bot goes in circles to the same addresses, because the site map gives them several times, and new pages lie without any internal link. First, we measure how much repetition eats up the bypass, and only then we talk about the rest.

We'll review your situation in a free audit

Work on indexing vs. "let's put a map generation module"

How this option differs from the alternative

	Site map box module	Our approach
Directory size	taken from the admin's report: as much as in the database, as much in the file	we calculate by two methods and show the difference between them
Repeat	the file includes all category paths, so the product is included several times	one entry per address, duplicate patterns are closed by rules
Language versions	often only the default language	all branches in the map or a separate file per language, language codes are reconciled
Checking the result	looking at the response code: 200 with an empty body is considered a working card	we read the body of the answer and count the records
What's next	the file lives until the generator breaks — and no one knows about it	five figures every month and analysis of changes

Free audit

A free slice of your directory's indexing status

In a catalog of tens of thousands of items, the most difficult question is simple: how many of them are generally available to the search engine. The answer can be obtained from the outside - and we do this before talking about money, because it is this figure that determines the amount of work.

What we measure

Directory size by two methodsWe count on the site map and cross-check with standard search, then show the discrepancy. In the equipment store, it was 0.4% — and it is this difference that indicates a hidden problem.
Site map structureIs it divided into sub-maps, how many addresses are in each, does the file not rest against the format limits.
RepeatHow many entries per unique address. In a large directory, this is the main consumer of the bypass.
Catalog return speedAnswer time on the card and in the category. A reference from the measurement: a catalog with 54,302 products gives the first byte in 413.8 ms — the size does not oblige the site to be slow.
Language branchesIs the catalog duplicated by languages, are all versions present in the map, and are the language codes declared correctly?
Dates of last modificationAre they actually updating or have they frozen since the last generation failure.

What you get

The size of the directory calculated by two independent methods, with the difference between them.
State of the site map: submaps, addresses, repeats, dates.
A list of what prevents the bot from going deep into the directory, in order of impact.
Conversation for 40 minutes on the document.

Timeline: 3-5 working days

Why is it free

Because this is measured work, and without numbers, both parties are trading blindly. We count your catalog ourselves, from the outside, and show exactly how we counted - the method is verified.

What's next

Next is an estimate by stages with calendar dates. We warn honestly: the site map does not guarantee indexing, and 55 thousand addresses in it do not mean 55 thousand pages in the index.

Short form: your contact and site URL

Did not find your case?

Describe how it works on your side — we will tell you whether “Indexing a large directory” fits and what it means in your situation. No brief and no call: one question, one answer.

Contract and paperwork

We work officially

Contract, act and invoice — for every project, not just large ones.

Trusted by6 brands

Terms fixed in writing8 points

Contract, act and 30-day warranty
Every project gets a written contract: scope, deadlines, amount, acceptance procedure. After delivery — act and invoice, then 30 calendar days of warranty.
Sole proprietor & bank transfer
The contractor is a registered sole proprietor. Payment by invoice with closing documents.
Rights & access — yours
Code, design and materials transfer to you after full payment. Domain, hosting, repository and analytics are registered to you.
Client portal instead of email chains
During the project you get access to a portal: contracts, invoices, acts and project status in one place.
Sites in nine countries
Our sites run in nine countries: besides Ukraine — Poland, Germany, Austria, England, Norway, Bulgaria, Moldova and Spain.
Verifiable numbers
Every case in the portfolio comes with a link to a live site and a technical measurement.
Audit first, then pricing
There is no price list on the site intentionally: the scope of the same work differs multiples between clients.
We say "no" when unsure
If the task isn't ours or the deadline is unrealistic — we tell you upfront.

Contract, act and 30-day warranty
Every project gets a written contract: scope, deadlines, amount, acceptance procedure. After delivery — act and invoice, then 30 calendar days of warranty.
Sole proprietor & bank transfer
The contractor is a registered sole proprietor. Payment by invoice with closing documents.
Rights & access — yours
Code, design and materials transfer to you after full payment. Domain, hosting, repository and analytics are registered to you.
Client portal instead of email chains
During the project you get access to a portal: contracts, invoices, acts and project status in one place.
Sites in nine countries
Our sites run in nine countries: besides Ukraine — Poland, Germany, Austria, England, Norway, Bulgaria, Moldova and Spain.
Verifiable numbers
Every case in the portfolio comes with a link to a live site and a technical measurement.
Audit first, then pricing
There is no price list on the site intentionally: the scope of the same work differs multiples between clients.
We say "no" when unsure
If the task isn't ours or the deadline is unrealistic — we tell you upfront.

We sign NDAs on request. After completing the work, we delete our access to your services ourselves.

What affects the price

Why two seemingly identical tasks are priced differently

The actual size of the directoryNot the one in the admin, but the one visible from the outside. The difference between 5,561 items and 54,302 is a difference in the number of map files, traversal time, and reconciliation volume, not a percentage of the same job.
How many repetitions per addressRepetitions are not done piecemeal: they fall into patterns, and one rule is written on the pattern. But first these patterns need to be found, and this is a traversal of the entire map, not a sample of ten addresses.
Card generator statusThe file is built in seconds - we work with the existing one. It gives an empty body or assembles in tens of seconds - first we fix the generator itself, and this is a separate line of the estimate.
The number of language branchesEach branch doubles the list of addresses and adds a check: whether all versions are in the map, whether the language codes are correctly declared. In a store with 54,302 products, it is the second language that gives 110,096 addresses in the file.
Recoil speed under bypassThe directory responsible for 413.8 ms, the bot bypasses as calmly as a small site. A catalog with an 840.5ms response must be accelerated first, otherwise the traversal will limit itself.
Access to Search ConsoleThere is access — we can see how many pages are actually in the index, and we work by the number. There is none - we work on what is visible from the outside, and part of the result remains unverified.

Cases

Tasks and results in numbers — all metrics measured by us

online store of tactical equipment, catalog of more than 50,000 items

Task: Confirm the real size of the directory and check how it is presented to the search engine.
Solution: Bypassing the index file of the site map with a selective check of submaps — the first, sixteenth, thirty-first, and last; cross-counting by full-time search of 100 items per page.
Result: 54,302 products and 635 categories. The map is assembled from 32 commodity files: the checked submaps contain exactly 3,500 addresses each, the last one — 104; a total of 110,096 addresses in two languages. Cross-validation by search yielded 54,501–54,600 positions, a discrepancy with the map of 0.4%. First response 413.8 ms, full download 511.1 ms, nine types of structured data. A defect was found: the language codes in the markup are set incorrectly - two branches exist, but they are declared as one language. Measured on 07/31/2026.

online store of garden equipment and spare parts, the largest site map in the selection

Task: Count the spare parts catalog that does not fit into one sitemap file.
Solution: We actually went through the submaps, page by page, checked the map announcement in robots.txt and took the response time on the most difficult pages.
Result: 55,321 commodity addresses — 18 subcards of 3,000 and 1,321 in the nineteenth — plus 3,868 categorical: at least 59,189 addresses. The map is explicitly declared in robots.txt, a markup of six types of structured data. The first response of 840.5 ms with 454.5 KB of markup is the first point of work. Separately, the stack age is visible: PHP 7.3.33, not supported since December 2021. Measured on 07/31/2026.

online store of professional cosmetics, catalog of over 5,000 items

Task: Find out why a catalog of this size is almost not represented in search.
Solution: Counting the directory by regular CMS search, checking the body of the /sitemap.xml response and the composition of the counters on the main page.
Result: 5,561 products in the live catalog — 55 search pages for 100 items and 61 for the last one. At the same time, /sitemap.xml returns code 200 with a body of zero bytes: the search engine does not receive a list of addresses at all. One language version with no language markup, no analytics counter on master, first response 801.2ms with 153.8KB of markup. Measured on 07/31/2026.

What we need from you

We can't start without this — best to prepare in advance

Site address and sitemap address - this is enough to make the first cut without any accesses.
Access to Search Console: no one, including us, can see the real number of indexed pages without it.
Access to CMS or your developer to change map generation, give updates and canonical addresses.
Understanding how often the catalog is updated and how many items are added per week: the frequency of regeneration depends on this.
The decision of what to do with product variants — separate addresses for size and color or parameters of one card. This is a business decision, not a technical one.
One person on your side with the right to approve changes in the address structure.

If something is missing — let us know, we'll help you gather it or do it as a separate task.

FAQ

Most frequently asked questions — with concrete answers

How do you calculate the size of the catalog - from my words?

No, by two independent methods. The first is a walkthrough of the site map with a count of addresses in each submap. The second is a standard CMS search with an empty query and the maximum number of items per page. Then we show the discrepancy. In the tactical equipment store, the map returned 54,302 items, the search returned 54,501–54,600 items: a discrepancy of 0.4%. This difference is interesting, because it shows how many products live in the catalog, but did not make it to the list for the bot.

Is a large catalog necessarily slow?

No. A catalog of 54,302 products renders the first byte in 413.8 ms and complete HTML in 511.1 ms. A catalog for 5,561 items in the same sample — 801.2 ms. Pulls down the template, module set, and stack age, not the number of rows in the database. The slowest directory measured has PHP 7.3.33 under the hood, a version that was deprecated in December 2021.

How many addresses to put in one map file?

The format limit is 50,000 addresses per file, but you should split it earlier. In the measured directories, submaps are made for 3,000 and 3,500 addresses, and this is a convenient size: the file is quickly assembled, it can be opened and counted by eye, and when a part of the directory is changed, only the necessary piece has to be regenerated. Our recommendation is to keep the submap within three to four thousand addresses, even when the format allows ten times more.

Will all products be included in the index?

No, and it would be untrue to promise such a thing. A sitemap is a request to be crawled, not a guarantee of indexation, Google writes about it directly. A card without description, without distinction from neighboring cards and without demand may remain outside the index during any technical work. Our part: to make every page accessible, non-duplicated, and quickly delivered. The decision about the index is left to the search engine.

Our sitemap opens, so everything is fine?

It doesn't mean It is not the response code that should be checked, but the body. In the store of professional cosmetics, /sitemap.xml returns a code of 200 and zero bytes: the browser shows an empty page, monitoring considers the site to be working, and the bot does not receive any address from the directory for 5,561 items. This is the cheapest breakdown of all and the most expensive in terms of consequences, because it goes unnoticed for months.

Does it make sense to index every position of a large directory?

Not always. When the addresses are entered at the level of size and color, the number of pages grows many times without an increase in demand: a hundred variants of one model is a hundred almost identical pages competing with each other. It is often more correct to keep the model in the index, and leave the options with the card parameters. The decision here is made by the business, because it is about the assortment; we only show what it costs in the bypass.

How often to check indexing status?

In the big catalog - every month. The status changes by itself: products are added, old ones disappear, after updating the CMS, the regeneration of the map breaks. Then the file remains in place, and the dates in it freeze, and outwardly everything looks normal. A five-figure slice once a month takes little time and catches this one before it eats a quarter.

After cleaning the duplicates, will the number of pages in the index drop?

Most likely yes, and this is the expected course. When one card is accessible in three ways, three weak pages lived in the index instead of one strong one. After cleaning, one thing remains: the number of "pages in the index" decreases, and the traversal stops being spent on repetitions. Therefore, we fix it to the works and show it next to the number of unique addresses - otherwise the normal course of events is read as a deterioration.

Demo

Your catalogue on a phone screen

This is how your store can look on a phone: a collection banner, popular products and a cart one tap away. Tap “Add to cart” and the counter in the header updates, just like in a real store.

Collection bannerThe first screen shows what is new, not a logo.
Product cardsPhoto, price and a button — no extra page to open.
One-tap cartThe counter in the header updates instantly, like in a real store.

Stores in the portfolio

This is a demo interface, not a client's store. Live projects are in the portfolio.

SHOP

New collectionAutumn / WinterExplore

PopularAll products ›

Essentials Hoodie1 290 ₴
Urban Backpack1 490 ₴
Classic Cap690 ₴
Basic Tee590 ₴
Street Sneakers2 390 ₴
Active Shorts890 ₴

Submit your site address and sitemap address.

In response, how many items are in the catalog according to two counting methods, how many addresses are in the map, how many of them are duplicates, and what prevents the bot from reaching the depth. If the catalog is small, we will say so in the first email and we will not sell extra.

From measured cases54,302 products and 635 categories

View cases

Reply within 2 hours
No commitment
We work under a contract

There is no price list on the site on purpose: the same work differs several times over between two clients, and a “from” figure explains nothing in that case. First a free audit — we count your pages, duplicates and speed — then we name the sum and the deadline and fix both in the contract.

Indexing of a large online store catalog

Transparent stages with approval at every step

A slice of indexing status

Comparison with what the search engine sees

Reconstruction of the site map

Duplication cleaning and canonicalization

Depth reach and speed

Monthly slice