SEO SOS: 5 Lessons Learned from a Brush with Disaster – A Noindex Incident
Feb 7, 2024 • 5 min read
Join me on a journey into the world of technical SEO, where I’m about to spill the beans on some real-life lessons.
But there’s a catch – I can’t spill the actual beans about the website, so I’m using a made-up one called puzzle.hub. It’s a real story with a fictional name to keep things hush-hush. In other words, the website’s true identity is under wraps, but the lessons we’re about to uncover are real.
Puzzle HUB is a digital haven that captivates puzzle enthusiasts with a blend of brainy challenges and entertainment.
While the website features an extensive repertoire of puzzles, not all of them are equal when it comes to their influence over the revenue stream. Only a small group of them bring in the big bucks, supporting my first principle of growth which says that 10% of the work will account for 90% of the results; in this case, a handful of pages generate the lion’s share of income. What’s more, Puzzle HUB isn’t just about puzzles; it’s a versatile platform hosting various types of content. Insightful guides and informative articles attract substantial traffic, playing a crucial role in driving visitors, but with modest financial impact compared to its Revenue Queens.
While traffic metrics may not spotlight these high-income pages, It’s a journey where SEO mastery meets the art of engagement to support business.
The Plot
As a diligent SEO, my daily routine starts with checking dashboards and rankings across various projects. One morning kicked off with a concerning revelation – the keywords crucial for an important project disappeared from SERP. Searching for those pages myself, I couldn’t find them at all. Search Console quickly revealed the culprit – an accidental noindex tag. The bug was quickly resolved thanks to a very responsive tech team. After overcoming a few challenges to get those pages back into the index and rankings, everything ended well, revealing valuable lessons about all aspects of being a digital VIP.
Get ready for a lesson that resonates with every webmaster and SEO enthusiast!
Lesson 1: Internet VIPs – Pages that Shine in Google’s Top Tier Index
In one of the first episodes of Search Off the Recod (the most enlightening SEO Podcast), Gary Illyes from Google demystified the tiered storage approach to index content.
- RAM (Random Access Memory): Fastest and most expensive
- SSD (Solid State Drive): Very fast but cost prohibitive
- HDD (Hard Disk Drive): Slowest and least costly
Shortly, when Google decides where to store pages, it considers how often people search for them. Guess where our puzzles landed? Right at the top tier(s) – the fastest and most exclusive storage club; they are internet VIPs. And, as in real life, celebrity status has a double-edged nature. It’s not just about being in the top-tier storage; it significantly impacts how often Google crawls, processes changes, and reindexes these pages. The downsides of this prominence became evident when an unintended bug was promptly executed.
This unexpected consequence introduces our next lesson about navigating the fast-paced world of digital celebrityhood.
Lesson 2: VIPs Require Real-time Attention and Tools
In the spotlights of Google’s top tiers, real-time attention is non-negotiable. These pages, so frequently demanded by users, require tools that operate in real time.
My heroe tools were Keyword.com (which allowed me to check the ranking in each step of the incident) and Search Console (which helped identify the bug and ask for reindexing).
Search Console alone or any other keyword tracking tool I use hadn’t issued any meaningful alert yet. The drop was still very recent; it only happened hours ago. The traffic impact was not so visible in the overall traffic and the variety of content.
In the fast-paced game of SEO, staying on top means having tools that match the speed of change.
Lesson 3: SEO Tempo – Understanding the Fast Track to Google Indexing and Removal
How long does it take to index pages or remove pages from Google’s index using noindex meta tags?
It’s all about how fast you can get Google to crawl them. This process, intricately tied to the tier status discussed in Lesson 1, highlights that for VIPs, indexing or removal happens faster than for regular pages.
As per Google’s official stance, Google needs to crawl your page to recognize <meta> tags and HTTP headers, impacting how swiftly it gets removed or indexed. The importance of the page dictates the frequency of Googlebot revisits, potentially taking months for less critical pages. To expedite the process, we can leverage the URL Inspection tool or sitemap submit to request a recrawl.
This lesson also highlights a challenge in performing quality assurance (QA) since the page status and Google’s recrawl timeline remain uncertain.
Lesson 4: Harnessing Search Console’s Secret Weapons – URL Inspection Tool & Sitemaps Report
This lesson sheds light on the hidden gems within the Search Console and their intricacies.
URL Inspection Tool
It is a tool we can use to get real-time insights into the index status of a URL, troubleshoot a missing page, or request indexing.
Troubleshooting with Live Test and its limitations
When troubleshooting, URL Inspection’s Live Test feature can be helpful but has limitations. It provides real-time insights into the URL’s status but may not reflect its indexed status accurately, it may overlook some details or meet cache challenges. In my experience, the live test misled me a little indicating that pages could be indexed after the fix but it also motivated me to better learn how it works. While the live test confirms Googlebot’s access to the page, other conditions could also influence indexing.
Request Indexing
While there’s a daily quota of around 10 URLs for reindexing, unlimited inspections are permitted.
It’s worth noting that the daily quota resets every 24 hours, contrary to the confusing documentation stating it as “daily.” This distinction is key to maximizing the tool’s effectiveness.
Sitemaps Report
We can use it to manage sitemaps. Submit sitemap serves as a backup plan when the daily quota is exceeded. By resubmitting the sitemap in the Search Console, we can ask Google to crawl the site or the pages in the sitemap again, trying to expedite the indexing process. Though not guaranteed, this workaround underscores the importance of being treated as a VIP in Google’s eyes. In my case, it helped me get my VIPs indexed before waiting 24 hours for the URL Inspection Tool quota to reset.
Lesson 5: Rankings Resilience
After the ordeal of sending back the pages into the index, guess what? They didn’t just make a comeback to the index; they landed right back to the top like they never left the party! Not a single rank was lost in the shuffle. It’s like they took a little vacation and returned even stronger proving once again the resilience of a sustainable SEO strategy.
Closing Thoughts
Navigating the SEO twists and turns isn’t easy, but with the right tools, strategies, team collaboration, and a dash of perseverance, success is within reach. Have your own SEO stories or tips to share? Let’s keep the conversation going!