Crawling large ecom sites using sitemap or API

I was thinking that maybe a good improvement for ecom and bigger sites might be a way to actually crawl all (product) pages. (for example via I api connection, or simpler maybe via a sitemap.

To get the best out of AI is to get enough existing content to train the model. I noticed for example that when I manually entered multiple URL's of out website that it was only able to find about 60 pages, even though we have about 1700 pages on our website and I added in all collection pages, some product pages etc.

Anyway, maybe its of help:)