WXforum.net

Administration => WXForum Bugs/Suggestions => Topic started by: openvista on October 23, 2018, 09:22:05 AM

Title: Why are certain discussion threads here invisible to search engines?
Post by: openvista on October 23, 2018, 09:22:05 AM

I see in the source code on certain threads (SHT31 humidity, various Atlas threads, etc) that the meta robots tag is set to "noindex" rendering it invisible to search engines.

I also see that this isn't universal and there are certainly other threads that are being indexed. So are administrators setting certain threads to "noindex"? If so, any reason for that?

I also checked the robots.txt file to see if there were specific instructions for an alternate path for robots to index and couldn't find any.

That would explain why I never get any results pages when I search on terms I know are present in some threads here.

Title: Re: Why are certain discussion threads here invisible to search engines?
Post by: openvista on October 23, 2018, 10:40:47 AM

OK, I think I may have figured out what was happening. If you visit a link to a particular post (for example, if you click on a link in a notification that alerted you to the latest message in the thread), the HTML source will set that specific URL to noindex. Example URL: http://www.wxforum.net/index.php?topic=35327.msg361682#msg361682.

However, the base URL (http://www.wxforum.net/index.php?topic=35327) for the thread is not set to noindex (which means that it should be indexed).

I'm guessing this arrangement is meant to avoid the dreaded duplicate content penalty that Google can assess for having two different links to the same content.

Still not sure why I can't get returns for direct quotes in Google of forum content. But that is a separate issue.

Title: Re: Why are certain discussion threads here invisible to search engines?
Post by: saratogaWX on October 23, 2018, 01:09:49 PM

AFAIK, it's not something we've done deliberately.  A topic listing seems to include the

Code: [Select]

<meta name="robots" content="noindex" /> in the header.  The main page, and the main page for a board do not include that header.

It must be a SMF built-in feature, since I've seen no admin option to set/unset that behavior.

Title: Re: Why are certain discussion threads here invisible to search engines?
Post by: openvista on October 23, 2018, 01:21:28 PM

At this point, after further research, I'm confident this behavior is built into the forum software. Looks to be an SEO optimization.

Thanks for the response.