How to block Rogerbot From Crawling UTM URLs
-
I am trying to block roger from crawling some UTM urls we have created, but having no luck. My robots.txt file looks like:
User-agent: rogerbot Disallow: /?utm_source* This does not seem to be working. Any ideas? -
Shoot! There may be something else going on. Give us a shout at help@moz.com and we'll see if we can figure it out!
-
FYI - I tried this and it did not work. Rogerbot is still picking up URL's we don't need. It's making my crawl report a mess!
-
The only difference there is the * wildchar. The string with that character will limit the crawler from accessing any URL with that string of characters in it.
-
What is the difference between Disallow: /*?utm_ and Disallow: /?utm_ ?
-
Hi there! Tawny from the Customer Support team here!
You should be able to add a disallow directive for that parameter and any others to block our crawler from accessing them. It would look something like this:
User-agent: Rogerbot
Disallow: ?utmetc., until you have blocked all of the parameters that may be causing these duplicate content errors. It looks like the _source* might be what's giving our tools some trouble. It looks like Logan Ray has made an excellent suggestion - give that formatting a try and see if it helps!
You can also use the wild card user-agent * in order to block all crawlers from those pages, if you prefer. Here is a great resource about the robots.txt file that might be helpful: https://moz.com/learn/seo/robotstxt We always recommend checking your robots.txt file with a handy Robots Checker Tool once you make changes to avoid any nasty surprises.
-
Skyler,
You're close, give this a shot:
Disallow: /*?utm_
This will be inclusive of all UTM tags regardless of what comes before the tag or what element you have first.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved 503 Service Unavailable (temporary?) Rogerbot takes a break
A lot of my Moz duties seem to be setting hundreds of issues to ignore because my site was getting crawled while under maintenance. Why can't Rogerbot take a break after running into a few of these and then try again later? Is there an official code for Temporary Service Unavailability that can smart bots pause crawls so that they are not wasting compute, bandwidth, crawl budget and my time?
Product Support | | awilliams_kingston0 -
Why does Moz see short Russian & Chinese urls as too long
We are translating content into Russian and Chinese on our website, the number of errors are increasing mainly around URL too long, each time we create a page with a Chinese or Russian url. If you click on the link below for a Chinese content page: https://www.westbourneschool.com/zh-hans/%E5%AE%BF%E8%88%8D%E5%8F%8A%E5%AF%84%E5%AE%BF%E5%AE%B6%E5%BA%AD/%E5%AE%BF%E8%88%8D%E7%94%9F%E6%B4%BB You will notice the url displayed by the browser is actually not very long, is there a way for MOZ not to see it as it appears above? Below is a page in Russian https://www.westbourneschool.com/ru/%D0%A8%D0%BA%D0%BE%D0%BB%D0%B0%20%D0%9F%D1%80%D0%BE%D0%B6%D0%B8%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5 Any help will be much appreciated.
Product Support | | mariedetitomount0 -
Is is possible to revert reporting to a past crawl date?
Site Crawl report defaults to the last crawl. Is there a way to get data from a previous crawl for comparison?
Product Support | | JThibode1 -
Haven't received an update on site crawl issues more than a week
Hello, my account has be scheduled to have the next updated report on 1st March. However, up till now, the latest data I have for our site crawl issues is made on 21st Feb. May I know if there is any issue related to this? Any way that i can draw the data for this week?
Product Support | | Robylin10 -
Why is comment moderation blocking all my comments? There are no links in them.
For the past few days, every time I try posting a comment, no matter how long or short, how informative or simple, I've been getting the error message that the comment isn't being allowed. I only tried a few times, once on several different posts (not spamming, God forbid!), and I never include links in my comments... So what's wrong?! So frustrated, I am so not up to this right now. Why can't I share my insights and questions on Moz anymore? Gaaaaaaa!
Product Support | | whiteonlySEO1 -
No crawl data anymore
Using moz quite some time, but I don't have any crawl data anymore. What happened? (www.kbc.be)
Product Support | | KBC
http://analytics.moz.com/settings/campaign/517920.11285160 -
MOZ not accepting our recent changes it still showing us old Crawl Diagnostics report
Hi, 507 Temporary Redirect We made changes for 302 redirects which are listed in crawl diagnostics report. Now "Compare" and "Wishlist" links are already removed from our source code. All required changes are made but still your report listed Compare and wishlist links. We made changes on Friday (14/8/2015) and waiting for new updated report. Link: http://www.stopwobble.com/ Please let us know what is the exact issue. So that we can fix it.
Product Support | | torbett0 -
SEO Moz PRO app Isn't Crawling Anymore
Hi, We find the SEO Moz PRO app a great tool for us. What is the reason that it is not re-crawling the websites included in our campaigns anymore?
Product Support | | solution.advisor0