Story Based Question
You’ve been monitoring your mobile site’s performance on Google Search Console, and you start to notice a significant drop in mobile traffic. After digging deeper, you discover that your mobile site has crawl errors that are preventing Googlebot from properly accessing some of your key pages. How can you resolve these crawl errors to ensure that Google can index your mobile site properly?
Exact Answer
To resolve crawl errors on mobile websites, you should identify the errors in Google Search Console, fix issues like blocked resources, incorrect redirects, and broken links, and then validate the fixes to ensure Googlebot can properly crawl your mobile site.
Explanation
Crawl errors on mobile websites can severely impact your search rankings because Googlebot won’t be able to properly index your pages if it encounters errors. Resolving these crawl issues involves using Google Search Console to identify problems, fixing the issues, and then validating the fixes to ensure that Googlebot can successfully crawl and index your mobile site. Here’s a step-by-step approach to solving crawl errors:
Identify Crawl Errors in Google Search Console
The first step is to look at the Coverage and Crawl Errors reports in Google Search Console. These reports provide a list of pages Googlebot couldn’t access, along with details about the type of error.Common types of crawl errors include:
- 404 errors (Page Not Found): This occurs when a page is deleted or moved without a proper redirect.
- Server errors (5xx): These errors happen when the server is down or not responding.
- Blocked resources (robots.txt): Resources like JavaScript, CSS, or images may be blocked from Googlebot’s crawl, preventing pages from being rendered properly.
- Go to Google Search Console > Coverage > Errors.
- Review the list of crawl errors and identify which pages or resources are affected.
Fix Blocked Resources (robots.txt)
One of the most common crawl errors happens when Googlebot is blocked from accessing essential files like JavaScript, CSS, or images. These files are necessary for rendering and displaying pages correctly.How to fix it:
- Check your robots.txt file to ensure you’re not blocking important resources.If JavaScript or CSS files are blocked, Googlebot may not be able to render the page properly on mobile devices.Update your robots.txt file to allow Googlebot to access these resources.
User-agent: Googlebot
Disallow: /js/
Disallow: /css/
You can remove or modify these lines to allow Googlebot access:
User-agent: Googlebot
Allow: /js/
Allow: /css/
Fix Redirect Issues
Redirects can sometimes cause crawl errors, especially if a page has been moved or the URL structure has changed. If there are redirects on mobile pages that don’t lead to the correct destination, it could cause issues with Googlebot indexing the page.
How to fix it:
- Check your site for 301 or 302 redirects that might be pointing to incorrect or non-existent pages.
- Ensure that all redirects are pointing to the correct, updated pages.
- Avoid multiple redirects, as they can slow down the crawling process and cause errors.
- Use the URL Inspection Tool in Google Search Console to inspect affected URLs and see if they are being redirected properly.
Fix 404 Errors (Page Not Found)
404 errors occur when Googlebot tries to access a page that doesn’t exist. These errors often happen when pages are deleted, moved, or not linked properly.
How to fix it:
- Check the Coverage report in Google Search Console for 404 errors.
- If a page has been moved, ensure a 301 redirect is set up from the old URL to the new one.
- If the page no longer exists, either create a new page or set up a proper redirect to a relevant page.
- For broken links, update internal links so they point to the correct, live pages.
Fix Server Errors (5xx)
Server errors, like 500 or 503 errors, indicate that Googlebot couldn’t access the page due to server issues. These are often temporary, but if they persist, they can harm your rankings.
How to fix it:
- Check with your hosting provider to make sure your server is running smoothly.
- Ensure that your server has enough resources to handle traffic, especially on mobile devices.
- Use Google Search Console’s URL Inspection Tool to check whether the server error has been resolved.
Test Mobile Version of Pages
Since Google uses mobile-first indexing, it’s crucial to make sure the mobile version of your site is accessible and properly indexed. Use the Mobile Usability report in Google Search Console to check if there are any issues with your mobile pages.
How to fix it:
- Fix any mobile usability issues, such as text that’s too small to read, buttons that are too close together, or content that’s not visible on mobile.
- Use the Mobile-Friendly Test to verify that Googlebot can easily access and render the mobile version of your pages.
Validate Fixes in Google Search Console
Once you’ve resolved the crawl errors, it’s important to let Google know that the issues have been fixed. You can do this by clicking on Validate Fix in Google Search Console.
How to validate:
- Go back to the Coverage report and select the fixed errors.
- Click Validate Fix to request Google to crawl and index the pages again.
- Monitor the Crawl Errors report over the next few days to ensure that Googlebot is able to access and index your pages without issues.
Example
Let’s say you run an e-commerce website, and recently you’ve noticed a significant dip in mobile traffic. After checking Google Search Console, you discover that multiple pages have been flagged for 404 errors and blocked resources. These issues are preventing Googlebot from accessing key product pages.
First, you look at your robots.txt file and find that it’s blocking access to essential JavaScript and CSS files. You update the file to allow Googlebot to crawl these resources.
Next, you find that a few product pages were deleted and not properly redirected, resulting in 404 errors. You set up 301 redirects from the old URLs to the new ones, ensuring that users and Googlebot will land on the correct product pages.
Lastly, you notice that a server error occurred when Googlebot tried to crawl one of your category pages. You contact your hosting provider to resolve the issue, and the server is up and running smoothly.
After fixing these errors, you go back to Google Search Console and click Validate Fix. Within a few days, you see that the crawl errors have been resolved, and your mobile traffic begins to improve as Googlebot can now properly index your site’s pages.
Resolving crawl errors on mobile websites is crucial for ensuring that your site is indexed properly by Google, especially with mobile-first indexing. By using Google Search Console to identify issues like blocked resources, redirects, 404 errors, and server errors, you can fix these problems and improve your mobile site’s visibility in search results.