Introduction
Blocked URLs and resources can harm an e-commerce website’s SEO, user experience, and overall functionality. When search engines cannot access important pages, images, JavaScript, or CSS files, it can lead to poor indexing, lower rankings, and a frustrating customer shopping experience.
At teclan, we take a proactive approach to identifying and resolving blocked URLs and resources, ensuring that our client’s websites remain fully accessible, optimised, and search engine-friendly. Below is our step-by-step process for detecting and fixing these issues.
1. Identifying Blocked URLs & Resource
teclan uses a combination of tools to detect blocked content and resources:
- Google Search Console → “Coverage” & “Blocked Resources” reports highlight pages and files that search engines cannot access.
- An SEO Spider → Identifies URLs returning errors or restricted by robots.txt.
- Google Mobile-Friendly Test → Ensures mobile resources are fully accessible.
- Google PageSpeed Insights → Flags blocked scripts, images, and other critical page elements.
2. Checking & fixing robots.txt Issues
Review & update the robots.txt File:
- Access robots.txt at https://www.yourwebsite.com/robots.txt.
- Identify blocked resources – Look for Disallow rules preventing access to important files.
Validate in Google Search Console:
- Use the robots.txt Tester to confirm that search engines can now access the correct resources.
- Submit changes to Google for a fresh crawl.
3. Resolving Meta Robots & Canonical Issues
- Remove incorrect “no index” tags
- We ensure only non-essential pages (like admin logins) use this tag.
- Correct Canonical Tags should accurately point to the correct page version to prevent indexing confusion.
4. Fixing Server & Security Blocking Issues
- 403 Forbidden Errors
- If Googlebot is blocked, check .htaccess and firewall settings.
- Firewall & CDN Restrictions
- Ensure Cloudflare, Sucuri, or security plugins do not accidentally block search engines.
- Whitelist Googlebot IPs to prevent unnecessary restrictions.
5. Updating Sitemap & Requesting Reindexing
- Ensure all important pages are included in the sitemap
- Generate an updated XML sitemap via Yoast SEO (WordPress), Shopify’s built-in tool, or a sitemap generator.
- Submit it through Google Search Console → Sitemaps.
- Use Google’s URL Inspection Tool
- Inspect affected pages in the Search Console.
- Click Request Indexing to speed up Google’s re-crawling process.
6. Ongoing Monitoring & Prevention
teclan implements a proactive monitoring strategy to prevent future URL and resource blocking:
- Regular SEO Audits – Run SEO Crawler scans for blocked resources.
- Pre-launch Testing – Check robots.txt & meta tags before launching new product pages.
- Google Search Console Monitoring – Resolve any newly detected blocked URLs.
- Mobile Usability Checks – Ensure that mobile-friendly versions load properly.
Conclusion
By systematically detecting and fixing blocked URLs & resources, teclan ensures that e-commerce websites remain fully crawlable, indexed, and accessible to users and search engines. This leads to better search rankings, increased traffic, and an improved shopping experience for customers.