When Google Merchant Center Website Crawl Is Not an Option

Google Merchant Center is a platform that allows businesses to showcase their products on Google. It provides a way for customers to find and buy products from retailers who sell online. One of the essential features of the Merchant Center is the ability to use automated feeds, which uses crawls of your website to read structured data and sitemap information to extract the most up-to-date information about relevant products from your website.

However, if you’re having trouble with the website crawl option on Merchant Center, it can be challenging to get your products to show up on Google. In this article, we’ll explore some common issues that can prevent the website crawl option from working correctly and how to resolve them.

The Problem: Website Crawl Option Not Available

As mentioned in an email from a Google technical support member, automated feeds rely on website crawls to extract information about products. However, in order for website crawls to work for automated feeds, you’ll need to add schema.org structured data markup to your website. Specifically, you’ll need to add this markup to all product landing pages.

If you’re missing any of the required attributes (title, price, availability, image_link, etc), the website crawl option may not be available. Additionally, if your structured data markup is not correctly implemented, you may not be able to use the website crawl option.

If you don’t see “website crawl” as the input method in your Merchant Center account, make sure:

Your structured data markup is correctly implemented for all required attributes.
You have removed or don’t currently have a feed in your account.
Your account is not suspended.
Your robots.txt settings are correct, and Googlebot is not blocked.
The Solution: Check Your Structured Data Markup
To make sure that Googlebot and Googlebot-image can crawl your site, add the following directives to your robots.txt file:

User-agent: Googlebot
Disallow:

User-agent: Googlebot-image
Disallow:

By implementing these directives, you can ensure that your site is accessible to both Googlebot and Googlebot-image, allowing for website crawls to be conducted and for automated feeds to work correctly.

Conclusion

In conclusion, automated feeds can be a valuable tool for businesses looking to increase their visibility on Google Merchant Center. If you’re having difficulty using the website crawl option for automated feeds, make sure to check your structured data markup and ensure that you meet all the requirements. Additionally, don’t forget to check your robots.txt file and add the necessary directives to allow Googlebot and Googlebot-image to crawl your site. By doing so, you can take advantage of this powerful feature and reach potential customers with ease.