Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • Case Studies
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      Discover Brand Authority
      Moz Pro

      Discover Brand Authority

      Learn More
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      What is your Brand Authority?
      Moz

      What is your Brand Authority?

      Take the quiz
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • SEO Q&A

        Insights & discussions from an SEO community of 500,000+.

      June 3 & 4, 2024, Seattle
      MozCon

      June 3 & 4, 2024, Seattle

      Get tickets
    • Blog
    • Why Moz
      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • Case Studies

        Explore how Moz drives ROI with a proven track record of success.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature: Moz Pro

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Intermediate & Advanced SEO
    4. Robots.txt: Can you put a /* wildcard in the middle of a URL?

    Robots.txt: Can you put a /* wildcard in the middle of a URL?

    Intermediate & Advanced SEO
    2
    2
    886
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • IHSwebsite
      IHSwebsite last edited by

      We have noticed that Google is indexing the language/country directory versions of directories we have disallowed in our robots.txt.

      For example:

      Disallow: /images/ is blocked just fine

      However, once you add our /en/uk/ directory in front of it, there are dozens of pages indexed.

      The question is: Can I put a wildcard in the middle of the string, ex. /en/*/images/, or do I need to list out every single country for every language in the robots file. Anyone know of any workarounds?

      1 Reply Last reply Reply Quote 0
      • irvingw
        irvingw last edited by

        Yes, wildcards work, thank god.

        1 Reply Last reply Reply Quote 1
        • 1 / 1
        • First post
          Last post

        Got a burning SEO question?

        Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


        Start my free trial


        Browse Questions

        Explore more categories

        • Moz Tools

          Chat with the community about the Moz tools.

        • SEO Tactics

          Discuss the SEO process with fellow marketers

        • Community

          Discuss industry events, jobs, and news!

        • Digital Marketing

          Chat about tactics outside of SEO

        • Research & Trends

          Dive into research and trends in the search industry.

        • Support

          Connect on product support and feature requests.

        • See all categories

        Related Questions

        • fcromwell

          400+ deleted pictures, 404 on URLs, can they be deleted without penalty?

          A client had a website redesign and over 400 pictures (she's a photographer) were lost. The URL's (ending in .png) are hanging out there and according to the site scan, they have a matching current URL. Since the pictures are gone can the URL be deleted?

          Intermediate & Advanced SEO | | fcromwell
          0
        • BeckyKey

          SEO Strategy - Content/Outreach/Links

          Hi everyone I'm trying to prioritise my tasks for 2018 & wondered if anyone had any useful templates they use? In terms of SEO tasks, my priority was going to be content/outreach/links - Focusing on user guides/blogs onsite Then outreach articles/some PR that doesn't go against Google guidelines offsite. My struggle with the onsite content/blogs we produce is we have no real social media plan/manager so my content outreach always seems hampered by this. I've tried taking on some of the social stuff, but this ends up being too much for just me to do. I wondered if there were any other SEOs who face this issue and who have found some good solutions? I'm stuck in a bit of a rut and can't seem to effectively push forward with outreach/content writing. Thank you Becky

          Intermediate & Advanced SEO | | BeckyKey
          1
        • RickyShockley

          How Does Yelp Create URLs?

          Hi all, How does Yelp (or other sites) go about creating URLs for just about every service and city possible ending with the search? in the URL like this https://www.yelp.com/search?cflt=chiropractors&find_loc=West+Palm+Beach%2C+FL. They clearly aren't creating all of these pages, so how do you go about setting a meta title/optimization formula that allows these pages to exist AND to be crawled by search engines and indexed?

          Intermediate & Advanced SEO | | RickyShockley
          0
        • RosemaryB

          Client wants to remove mobile URLs from their sitemap to avoid indexing issues. However this will require SEVERAL billing hours. Is having both mobile/desktop URLs in a sitemap really that detrimental to search indexing?

          We had an enterprise client ask to remove mobile URLs from their sitemaps.  For their website both desktop & mobile URLs are combined into one sitemap.  Their website has a mobile template (not a responsive website) and is configured properly via Google's "separate URL" guidelines.  Our client is referencing a statement made from John Mueller that having both mobile & desktop sitemaps can be problematic for indexing.  Here is the article https://www.seroundtable.com/google-mobile-sitemaps-20137.html 
          We would be happy to remove the mobile URLs from their sitemap.  However this will unfortunately take several billing hours for our development team to implement and QA.   This will end up costing our client a great deal of money when the task is completed. Is it worth it to remove the mobile URLs from their main website to be in adherence to John Mueller's advice?  We don't believe these extra mobile URLs are harming their search indexing.   However we can't find any sources to explain otherwise. Any advice would be appreciated. Thx.

          Intermediate & Advanced SEO | | RosemaryB
          0
        • rootwaysinc

          Default Robots.txt in WordPress - Should i change it??

          I have a WordPress site as using theme Genesis  i am using default robots.txt. that has a line Allow: /wp-admin/admin-ajax.php, is it okay or any problem. Should i change it?

          Intermediate & Advanced SEO | | rootwaysinc
          0
        • partnerf

          Will Canonical tag on parameter URLs remove those URL's from Index, and preserve link juice?

          My website has 43,000 pages indexed by Google.  Almost all of these pages are URLs that have parameters in them, creating duplicate content.  I have external links pointing to those URLs that have parameters in them. If I add the canonical tag to these parameter URLs, will that remove those pages from the Google index, or do I need to do something more to remove those pages from the index? Ex:  www.website.com/boats/show/tuna-fishing/?TID=shkfsvdi_dc%ficol  (has link pointing here)
                 www.website.com/boats/show/tuna-fishing/   (canonical URL) Thanks for your help. Rob

          Intermediate & Advanced SEO | | partnerf
          0
        • VipinLouka78

          How can I block unwanted urls being indexed on google?

          Hi, I have to block unwanted urls (not that page) from being indexed on google. I have to block urls like example.com/entertainment not the exact page example.com/entertainment.aspx . Is there any other ways other than robot.txt? If i add this to robot.txt will that block my other url too? Or should I make a 301 redirection from example.com/entertainment to example.com/entertainment.aspx. Because some of the unwanted urls are linked from other sites. thanks in advance.

          Intermediate & Advanced SEO | | VipinLouka78
          0
        • AndreVanKets

          Is it safe to redirect multiple URLs to a single URL?

          Hi, I have an old Wordress website with about 300-400 original pages of content on it. All relating to my company's industry: travel in Africa. It's a legitimate site with travel stories, photos, advice etc. Nothing spammy about. No adverts on it. No affiliates. The site hasn't been updated for a couple of years and we no longer have a need for it. Many of the stories on it are quite out of date. The site has built up a modest Mozrank value over the last 5 years, and has a few hundreds organically achieved inbound links. Recently I set up a swanky new branded website on ExpressionEngine on a new domain. My intention is to: Shut down the old site Focus all attention on building up content on the new website Ask the people linking to the old site to my new site instead (I wonder how many will actually do so...) Where possible, setup a 301 redirect from pages on the old site to their closest match on the new site Setup a 301 redirect from the old site's home page to new site's homepage Sounds good, right? But there is one issue I need some advice on... The old site has about 100 pages that do not have a good match on the new site. These pages are outdated or inferior quality, so it doesn't really make sense to rewrite them and put them on the new site. I call these my "black sheep pages". So... for these "black sheep pages" should I (A) redirect the urls to the new site's homepage (B) redirect the urls the old site's home page (which in turn, redirects to the new site's homepage, or (C) not redirect the urls, and let them die a lonely 404 death? OPTION A: oldsite.com/page1.php -> newsite.com
          oldsite.com/page2.php -> newsite.com
          oldsite.com/page3.php -> newsite.com
          oldsite.com/page4.php -> newsite.com
          oldsite.com/page5.php -> newsite.com
          oldsite.com -> newsite.com OPTION B: oldsite.com/page1.php -> oldsite.com
          oldsite.com/page2.php -> oldsite.com
          oldsite.com/page3.php -> oldsite.com
          oldsite.com/page4.php -> oldsite.com
          oldsite.com/page5.php -> oldsite.com
          oldsite.com -> newsite.com OPTION 😄 oldsite.com/page1.php : do not redirect, let page 404 and disappear forever
          oldsite.com/page2.php : do not redirect, let page 404 and disappear forever
          oldsite.com/page3.php : do not redirect, let page 404 and disappear forever
          oldsite.com/page4.php : do not redirect, let page 404 and disappear forever
          oldsite.com/page5.php : do not redirect, let page 404 and disappear forever
          oldsite.com -> newsite.com My intuition tells me that Option A would pass the most "link juice" to my new site, but I am concerned that it could also be seen by Google as a spammy redirect technique. What would you do? Help 😐

          Intermediate & Advanced SEO | | AndreVanKets
          1
        Moz logo
        • Contact
        • Community
        • Free Trial
        • Terms & Privacy
        • Accessibility
        • Jobs
        • Help
        • News & Press
        • MozCon
        © 2021 - 2024 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.

        Looks like your connection to Moz was lost, please wait while we try to reconnect.