Author Topic: Why are certain discussion threads here invisible to search engines?  (Read 443 times)

0 Members and 1 Guest are viewing this topic.

Offline openvista

  • Forecaster
  • *****
  • Posts: 419
    • marquetteweather.com
I see in the source code on certain threads (SHT31 humidity, various Atlas threads, etc) that the meta robots tag is set to "noindex" rendering it invisible to search engines.

I also see that this isn't universal and there are certainly other threads that are being indexed. So are administrators setting certain threads to "noindex"? If so, any reason for that?

I also checked the robots.txt file to see if there were specific instructions for an alternate path for robots to index and couldn't find any.

That would explain why I never get any results pages when I search on terms I know are present in some threads here.
Davis Vantage Pro2 FARS | https://marquetteweather.com | EW7933

Offline openvista

  • Forecaster
  • *****
  • Posts: 419
    • marquetteweather.com
Re: Why are certain discussion threads here invisible to search engines?
« Reply #1 on: October 23, 2018, 10:40:47 AM »
OK, I think I may have figured out what was happening. If you visit a link to a particular post (for example, if you click on a link in a notification that alerted you to the latest message in the thread), the HTML source will set that specific URL to noindex. Example URL: http://www.wxforum.net/index.php?topic=35327.msg361682#msg361682.

However, the base URL (http://www.wxforum.net/index.php?topic=35327) for the thread is not set to noindex (which means that it should be indexed).

I'm guessing this arrangement is meant to avoid the dreaded duplicate content penalty that Google can assess for having two different links to the same content.

Still not sure why I can't get returns for direct quotes in Google of forum content. But that is a separate issue.
Davis Vantage Pro2 FARS | https://marquetteweather.com | EW7933

Offline saratogaWX

  • Administrator
  • Forecaster
  • *****
  • Posts: 6874
  • Saratoga, CA, USA Weather - free PHP scripts
    • Saratoga-Weather.org
Re: Why are certain discussion threads here invisible to search engines?
« Reply #2 on: October 23, 2018, 01:09:49 PM »
AFAIK, it's not something we've done deliberately.  A topic listing seems to include the
Code: [Select]
<meta name="robots" content="noindex" /> in the header.  The main page, and the main page for a board do not include that header.

It must be a SMF built-in feature, since I've seen no admin option to set/unset that behavior.
Ken True/Saratoga, CA, USA main site: saratoga-weather.org
Davis VP1+ FARS, Boltek-PCI/NexStorm, microSferics ToA, Blitzortung RED, GRLevel3, WD, WL, VWS, Cumulus, Meteobridge/hub
Free weather PHP scripts/website templates - update notifications on Twitter saratogaWXPHP

Offline openvista

  • Forecaster
  • *****
  • Posts: 419
    • marquetteweather.com
Re: Why are certain discussion threads here invisible to search engines?
« Reply #3 on: October 23, 2018, 01:21:28 PM »
At this point, after further research, I'm confident this behavior is built into the forum software. Looks to be an SEO optimization.

Thanks for the response. 
Davis Vantage Pro2 FARS | https://marquetteweather.com | EW7933