AWS Builder Center: Learn, Build and Connect with builders in the AWS community

AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.

Google bots crawl rate

Hello, According to Google, we have maximized the crawl rate on our servers, which can cause excessive crawling that affects the performance of the websites. Is there a way to limit the crawl rate? Ideally without this causing Google to eliminate products from Adwords campaigns because it cannot access the product landing pages.

主題: Retail
標籤: 電子商務
語言: English

Marc

已提問 2 年前檢視次數 396 次

2 個答案

最新
最多得票
最多評論

這些答案是否有幫助？支持正確答案，以幫助社區從您的知識中受益。

Hi, find the official doc here on this very item. https://developers.google.com/search/docs/crawling-indexing/reduce-crawl-rate

專家

Giovanni Lauria

已回答 2 年前

You can use Robots.txt

here is a sample file for you

User-agent: *
Disallow: /private/
Disallow: /temp/
Disallow: /test/

User-agent: Googlebot
Allow: /
Disallow: /example-subdirectory/
Crawl-delay: 10

User-agent: Bingbot
Allow: /
Disallow: /example-subdirectory/
Crawl-delay: 10

# Sitemap link
Sitemap: https://www.yoursite.com/sitemap.xml

khnaz35

已回答 2 年前

Google bots crawl rate

新增您的答案

相關內容