AWS Builder Center: Learn, Build and Connect with builders in the AWS community

AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.

Google bots crawl rate

Hello, According to Google, we have maximized the crawl rate on our servers, which can cause excessive crawling that affects the performance of the websites. Is there a way to limit the crawl rate? Ideally without this causing Google to eliminate products from Adwords campaigns because it cannot access the product landing pages.

主题: 零售
标签: 电子商务
语言: English

Marc

已提问 2 年前398 查看次数

2 回答

最新
投票最多
评论最多

这些答案有用吗？为正确答案投票，以帮助社区从您的知识中受益。

Hi, find the official doc here on this very item. https://developers.google.com/search/docs/crawling-indexing/reduce-crawl-rate

专家

Giovanni Lauria

已回答 2 年前

You can use Robots.txt

here is a sample file for you

User-agent: *
Disallow: /private/
Disallow: /temp/
Disallow: /test/

User-agent: Googlebot
Allow: /
Disallow: /example-subdirectory/
Crawl-delay: 10

User-agent: Bingbot
Allow: /
Disallow: /example-subdirectory/
Crawl-delay: 10

# Sitemap link
Sitemap: https://www.yoursite.com/sitemap.xml

khnaz35

已回答 2 年前

Google bots crawl rate

添加回答

相关内容