What Lies Beneath: Website Crawl Analysis





Blog Date

May 27, 2024


UK, Manchester

Follow us on


Table of Contents

What Lies Beneath: Website Crawl Analysis

The Unseen Perils of Technical SEO

As an SEO professional, I’ve always believed that what lies beneath the surface can make or break your website’s performance. It’s like an optical illusion – what looks perfectly fine on the outside may be harbouring sinister gremlins that could send your rankings plummeting.

Take the case of a new client I recently started working with. Their search visibility had been steadily declining, and it was my job to get to the root of the problem. Armed with my trusty crawling tools, I dove headfirst into a thorough analysis of their website. And let me tell you, what I uncovered was enough to send a shiver down any SEO’s spine.

Uncovering the 302 Redirect Nightmare

The first red flag that caught my eye was the sheer volume of 302 redirects I was seeing in the crawl data. Now, a few redirects here and there are to be expected, but this was on a whole other level. Thousands upon thousands of pages were 302 redirecting straight to the homepage.

As any seasoned SEO knows, that’s a major no-no. 302 redirects are essentially telling Google that the page is temporarily moving, and the search engine may end up dropping those pages altogether. Not exactly the recipe for long-term organic success.

To get a clearer picture, I decided to triangulate the data by running crawls across multiple tools – Screaming Frog, DeepCrawl, and Sitebulb. The result? They all told the same story. Those pesky 302 redirects were very much real, and Googlebot was encountering them just like my crawlers were.

Digging into Google Search Console

With the crawl data in hand, I turned my attention to Google Search Console (GSC). The new index coverage reporting there is a goldmine for identifying technical issues, and it didn’t disappoint.

Sure enough, the Excluded section in GSC revealed the same 302 redirect problem I had uncovered. I exported the top 1000 redirects and started manually checking them, and lo and behold – many of them were resolving with 200 status codes when I accessed them directly, but the crawlers were still being redirected.

It was clear that there was a disconnect between how the website was behaving for regular users versus how it was being seen by Googlebot. A classic case of an optical illusion that could have disastrous consequences for SEO.

Bringing in the Dev Team

Armed with this information, I quickly put together a detailed report for the client, outlining the 302 redirect issue in all its gory details. They wasted no time in escalating the problem to their development team, who soon uncovered the root cause – a load balancing issue that was quietly redirecting the crawlers without anyone’s knowledge.

It just goes to show that what appears fine on the surface can be hiding all sorts of sinister technical problems underneath. And as we all know, those kinds of issues can wreak havoc on your search engine rankings if left unaddressed.

Lessons Learned: Staying Vigilant

This case study serves as a stark reminder of the importance of regular, thorough technical audits. By proactively crawling and analysing your website, you can uncover potential issues that might be invisible to the naked eye. Combine that with digging into the data in Google Search Console, and you’ve got a powerful one-two punch for keeping those pesky SEO gremlins at bay.

As they say, “what lies beneath” can be downright scary. But with the right tools and a keen eye for detail, you can stay one step ahead of the game. And who knows, you might even uncover a few optical illusions along the way – just be sure to slay them before they slay your rankings.

Now, if you’ll excuse me, I’ve got some more websites to crawl. You never know what kind of technical skeletons might be lurking in those digital closets. Happy hunting, fellow SEO adventurers!

The Power of Crawl Analysis

One of the most powerful weapons in an SEO’s arsenal is the humble website crawl. By systematically analysing the technical structure and implementation of a website, you can uncover a wealth of insights that can make or break your SEO efforts.

Think of it like exploring an uncharted cave – you never know what kind of hidden dangers and treasures might be waiting for you in the darkness. And just like caving, website crawling requires a keen eye, a methodical approach, and a willingness to venture into the unknown.

Crawling vs. Manual Audits: A Winning Combination

When it comes to technical SEO, I believe the most effective approach is to combine comprehensive crawls with thorough manual audits. On their own, each technique has its own strengths and weaknesses, but together they form a dynamic duo that can expose even the most well-hidden problems.

Manual audits allow you to dive deep into the nitty-gritty of a website’s code, uncovering nuanced issues that might slip through the cracks of an automated crawl. But they can also be time-consuming and prone to human error.

That’s where crawling comes in. By automating the process of scouring a website from top to bottom, you can quickly identify potential land mines that might have been missed during the manual inspection. Plus, with the ability to analyse millions of pages in a matter of minutes, crawling is an absolute godsend for larger websites.

Leveraging GSC for Powerful Insights

Of course, website crawling is just one piece of the puzzle. To truly get a comprehensive understanding of a site’s technical health, you need to combine that data with the wealth of information available in Google Search Console (GSC).

The index coverage reporting in GSC is a true treasure trove of insights. By diving into the Excluded category, you can uncover all sorts of juicy details about the issues Google is encountering – from redirect problems to indexing roadblocks and beyond.

And the best part? GSC is giving you a direct line into the mind of the search engine itself. So when you see something flagged as an issue, you can be confident that it’s a problem that’s worthy of your attention.

Uncovering Optical Illusions

One of the most insidious challenges in technical SEO is the dreaded “optical illusion” – where something appears to be working fine on the surface, but is actually causing all sorts of havoc under the hood.

Think about it like those mind-bending images that trick your eyes into seeing something that isn’t really there. Except in the world of SEO, these optical illusions can have very real consequences for your website’s performance.

As I’ve written about before, even the slightest typo or character out of place in your code can create these kinds of illusions. And when you’re dealing with a website of any significant size, it becomes all too easy for these problems to slip through the cracks.

That’s where the combination of crawling and GSC data really shines. By triangulating the information from multiple sources, you can cut through the smoke and mirrors and expose these optical illusions for what they really are. And once you’ve done that, you’re one step closer to slaying those pesky SEO gremlins once and for all.

Staying One Step Ahead

At the end of the day, successful SEO is all about staying vigilant and proactive. It’s about embracing the role of a digital explorer, venturing into the unknown and uncovering the hidden challenges that could be holding your website back.

And when it comes to technical SEO, that means regularly crawling your site, digging into the data in GSC, and keeping a keen eye out for those sneaky optical illusions. Because as the saying goes, “what lies beneath” can be the difference between ranking success and search engine oblivion.

So don’t be afraid to get your hands dirty. Embrace the power of crawling, harness the insights of GSC, and stay one step ahead of the game. Who knows, you might just uncover a few hidden gems along the way – and maybe even have a few laughs at the expense of those tricky optical illusions.

Happy crawling, my fellow SEO adventurers. The journey awaits!

Copyright 2023 © MCRSEO.ORG