Welcome to the Q&A Forum

Hello Moz World! I have been reading up on robots.txt files, and I understand the basics. I am looking for a deeper understanding on when to deploy particular tags, and when a page should be disallowed because it will affect SEO. I have been working with a software company who has a News & Events page which I don't think should be indexed. It changes every week, and is only relevant to potential customers who want to book a demo or attend an event, not so much search engines. My initial thinking was that I should use noindex/follow tag on that page. So, the pages would not be indexed, but all the links will be crawled. I decided to look at some of our competitors robots.txt files. Smartbear (https://smartbear.com/robots.txt), b2wsoftware (http://www.b2wsoftware.com/robots.txt) & labtech (http://www.labtechsoftware.com/robots.txt). I am still confused on what type of tags I should use, and how to gauge which set of tags is best for certain pages. I figured a static page is pretty much always good to index and follow, as long as it's public. And, I should always include a sitemap file. But, What about a dynamic page? What about pages that are out of date? Will this help with soft 404s? This is a long one, but I appreciate all of the expert insight. Thanks ahead of time for all of the awesome responses. Best Regards, Will H.

Had a bit of a "Gotcha" in Magento. We had Yoast Canonical Links extension which worked well , but then we installed Mageworx SEO Suite.. which broke Canonical Links. Unfortunately it started putting www.mysite.com/catalog/product/view/id/516/   as the Canonical Link -  and all URLs with /catalog/productview/*  is blocked in Robots.txt So unfortunately We told Google that the correct page is also a blocked page. they haven't been removed as far as I can see but traffic has certainly dropped. We have also , at the same time had some Site changes grouping some pages & having 301 redirects. Resubmitted site map & did a fetch as google. Any other ideas?  And Idea how long it will take to become unblocked?

Could anyone offer me some help regarding Moz Rank? Of all the metrics this seems to be one where we want to improve but just have not been able to. Does anyone have any advice or tips that we could look at implementing to get this thing to move at all?

Our website www.turbocupones.com dropped drastically in ranking today. I have no idea why this happened as we have not outsourced any SEO and sticked to google´s guidelines. Can you please check the web and give me Ideas why this happened? Our traffic dropped by 1000%. All our content is unique. I have years of experience in SEO and never had a problem like this. The rankings dropped drastically for everything. As said all SEO done is white hat. Thanks!

We have asked a few questions related to this process on Moz and wanted to give a breakdown of our journey as it will likely be helpful to others! A couple of months ago, we updated our robots.txt file with several pages that we did not want to be indexed.  At the time, we weren't checking WMT as regularly as we should have been and in a few weeks, we found that apparently one of the robots.txt files we were blocking was a dynamic file that led to the blocking of over 950,000 of our pages according to webmaster tools.  Which page was causing this is still a mystery, but we quickly removed all of the entries. From research, most people say that things normalize in a few weeks, so we waited.  A few weeks passed and things did not normalize.  We searched, we asked and the number of "blocked" pages in WMT which had increased at a rate of a few hundred thousand a week were decreasing at a rate of a thousand a week.  At this rate it would be a year or more before the pages were unblocked. This did not change.  Two months later and we were still at 840,000 pages blocked. We posted on the Google Webmaster Forum and one of the mods there said that it would just take a long time to normalize.  Very frustrating indeed considering how quickly the pages had been blocked. We found a few places on the interwebs that suggested that if you have an issue/mistake with robots.txt that you can submit a reconsideration request.  This seemed to be our only hope.  So, we put together a detailed reconsideration request asking for help with our blocked pages issue. A few days later, to our horror, we did not get a message offering help with our robots.txt problem.  Instead, we received a message saying that we had received a penalty for inbound links that violate Google's terms of use.  Major backfire.  We used an SEO company years ago that posted a hundred or so blog posts for us.  To our knowledge, the links didn't even exist anymore.  They did.... So, we signed up for an account with removeem.com.  We quickly found many of the links posted by the SEO firm as they were easily recognizable via the anchor text.  We began the process of using removem to contact the owners of the blogs.  To our surprise, we got a number of removals right away!  Others we had to contact another time and many did not respond at all.  Those we could not find an email for, we tried posting comments on the blog. Once we felt we had removed as many as possible, we added the rest to a disavow list and uploaded it using the disavow tool in WMT.  Then we waited... A few days later, we already had a response.  DENIED.  In our request, we specifically asked that if the request were to be denied that Google provide some example links.  When they denied our request, they sent us an email and including a sample link.  It was an interesting example.  We actually already had this blog in removem.  The issue in this case was, our version was a domain name, i.e. www.domainname.com and the version google had was a wordpress sub domain, i.e. www.subdomain.wordpress.com. So, we went back to the drawing board.  This time we signed up for majestic SEO and tied it in with removem.  That added a few more links.  We also had records from the old SEO company we were able to go through and locate a number of new links.  We repeated the previous process, contacting site owners and keeping track of our progress.  We also went through the "sample links" in WMT as best as we could (we have a lot of them) to try to pinpoint any other potentials. We removed what we could and again, disavowed the rest.  A few days later, we had a message in WMT.  DENIED AGAIN!  This time it was very discouraging as it just didn't seem there were any more links to remove.  The difference this time, was that there was NOT an email from Google.  Only a message in WMT.  So, while we didn't know if we would receive a response, we responded to the original email asking for more example links, so we could better understand what the issue was. Several days passed we received an email back saying that THE PENALTY HAD BEEN LIFTED!  This was of course very good news and it appeared that our email to Google was reviewed and received well. So, the final hurdle was the reason that we originally contacted Google.  Our robots.txt issue.  We did not receive any information from Google related to the robots.txt issue we originally filed the reconsideration request for.  We didn't know if it had just been ignored, or if there was something that might be done about it.  So, as a last ditch final effort, we responded to the email once again and requested help as we did the other times with the robots.txt issue. The weekend passed and on Monday we checked WMT again.  The number of blocked pages had dropped over the weekend from 840,000 to 440,000!  Success!  We are still waiting and hoping that number will continue downward back to zero. So, some thoughts: 1.  Was our site manually penalized from the beginning, yet without a message in WMT?  Or, when we filed the reconsideration request, did the reviewer take a closer look at our site, see the old paid links and add the penalty at that time?  If the latter is the case then... 2.  Did our reconsideration request backfire?  Or, was it ultimately for the best? 3.  When asking for reconsideration, make your requests known?  If you want example links, ask for them.  It never hurts to ask!  If you want to be connected with Google via email, ask to be! 4.  If you receive an email from Google, don't be afraid to respond to it.  I wouldn't over do this or spam them.  Keep it to the bare minimum and don't pester them, but if you have something pertinent to say that you have not already said, then don't be afraid to ask. Hopefully our journey might help others who have similar issues and feel free to ask any further questions. Thanks for reading! TheCraig

How can SEOmoz "most Common Errors*" under "Crawl Diagnostics" advice can be right for a good site organic? Site is well ranked top 3 on Google (difficult KW). If I go ahead and fix these errors, I might hurt my SEO , no? like: Too Many On-Page Links 302 (Temporary Redirect) Title Element Too Long (> 70 Characters) Missing Meta Description Tag

Please take a look at 7city.com This landing page contains links to: www.7city.co.uk www.7city.ae www.7city.com.sg and our US website which is also www.7city.com It is programmed so: If you are a first time user and type www.7city.com you go to the landing page above. If you then click on AMERICAS, it sets a cookie and directs you to  http://www.7city.com/home .  When you revisit www.7city.com in the future as the cookie is set you will be automatically sent to the AMERICAS website i.e http://www.7city.com/home. Our US websites is nor performing well on organic ranking compared to other regional website.  Is the above technique hindering our organic ranking in the US. Also, I have been led to believe that you get a higher ranking if the domain is specific to a country. Is this true? Does 7city.com receive higher ranking than if I created it as 7city.us for example? Many Thanks Mark