Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • Case Studies
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      Discover Brand Authority
      Moz Pro

      Discover Brand Authority

      Learn More
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      What is your Brand Authority?
      Moz

      What is your Brand Authority?

      Take the quiz
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • SEO Q&A

        Insights & discussions from an SEO community of 500,000+.

      June 3 & 4, 2024, Seattle
      MozCon

      June 3 & 4, 2024, Seattle

      Get tickets
    • Blog
    • Why Moz
      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • Case Studies

        Explore how Moz drives ROI with a proven track record of success.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature: Moz Pro

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Intermediate & Advanced SEO
    4. Google can't access/crawl my site!

    Google can't access/crawl my site!

    Intermediate & Advanced SEO
    4
    16
    4114
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • granitgash
      granitgash last edited by

      Hi

      I'm dealing with this problem for a few days. In fact i didn't realize it was this serious until today when i saw most of my site "de-indexed" and losing most of the rankings.

      [URL Errors: 1st photo]

      8/21/14 there were only 42 errors but in 8/22/14 this number went to 272 and it just keeps going up.

      The site i'm talking about is gazetaexpress.com (media news, custom cms) with lot's of pages.

      After i did some research i came to the conclusion that the problem is to the firewall, who might have blocked google bots from accessing the site. But the server administrator is saying that this isn't true and no google bots have been blocked.

      Also when i go to WMT, and try to Fetch as Google the site, this is what i get:

      [Fetch as Google: 2nd photo]

      From more than 60 tries, 2-3 times it showed Complete (and this only to homepage, never to articles).

      What can be the problem? Can i get Google to crawl properly my site and is there a chance that i will lose my previous rankings?

      Thanks a lot
      Granit

      FvhvDVR.png dKx3m1O.png

      1 Reply Last reply Reply Quote 0
      • Travis_Bailey
        Travis_Bailey @granitgash last edited by

        What did you do specifically to mitigate the problem? You can PM me, if you would like.

        1 Reply Last reply Reply Quote 0
        • Travis_Bailey
          Travis_Bailey @KeriMorgret last edited by

          This applies to the guy from Albania.

          Oh, this IS the guy from Albania. Never mind.

          1 Reply Last reply Reply Quote 0
          • KeriMorgret
            KeriMorgret @granitgash last edited by

            Great, thanks for letting us know what happened with this!

            Travis_Bailey 1 Reply Last reply Reply Quote 0
            • granitgash
              granitgash last edited by

              Hi all

              Just wanted to let you know that we fixed the problem. We disabled CloudFlare which we found out was blocking Google bots. More about this issue can be found at: https://support.cloudflare.com/hc/en-us/articles/200169806-I-m-getting-Google-Crawler-Errors-What-should-I-do-

              KeriMorgret Travis_Bailey 2 Replies Last reply Reply Quote 3
              • granitgash
                granitgash @Travis_Bailey last edited by

                Hi Travis, thank you for your time.

                Great for your friend, I also suggest to visit Kosovo someday, you will have great time here, for sure 🙂

                Back to the issue:

                Here is an interesting issue that is happening with the crawler.

                Our own cms uses htaccess for rewrite purposes. I created 2 new files that are independent from CMS and tried to fetch them with WMT, and it worked like a charm.

                These 2 independent files are:

                www.gazetaexpress.com/test_manaferra.php

                www.gazetaexpress.com/xhezidja.php

                Then, I created an ajax page with our CMS, which contains only plain text, tried to fetch it by WMT and strangely enough it didn't work. To make sure that the .htaccess file is not affecting this behavior, I deleted the htaccess and tried to fetch it, but it didn't worked.

                The ajax page is: www.gazetaexpress.com/page/xhezidja/?pageSEO=false

                The site works perfectly for humans which access it via the browser.

                I'm more than confused now!

                ac857dfbf02a316d92d378bc48f9c395.png

                1 Reply Last reply Reply Quote 0
                • Travis_Bailey
                  Travis_Bailey last edited by

                  A friend of mine just got back from Kosovo. It was the last stop on a tour of the Balkans. He had a pretty good time. Moving along...

                  I crawled about 12K URLs and hit almost 90 Internal Server Errors (500). It's probably not your core problem, but it's something to look at. Here are a few examples:

                  http://www.gazetaexpress.com/blihet/?search_category_id=1&searchFilter=1

                  http://www.gazetaexpress.com/shitet/?category_id=134&searchFilter=1

                  http://www.gazetaexpress.com/me-qera/?category_id=131&searchFilter=1

                  There was one actual page that threw a 500 at the time of crawl:

                  http://www.gazetaexpress.com/mistere/edhe-kesaj-i-thuhet-veze-22591/

                  The edhe kesaj page now resolves fine. (I'm not even going to pretend to understand or write Albanian.)

                  So there may be some issues with the server or hosting. If you haven't already, try this troubleshooter from Cloudflare.

                  granitgash 1 Reply Last reply Reply Quote 0
                  • Andy.Drinkwater
                    Andy.Drinkwater @granitgash last edited by

                    Ah OK - well keep us updated with what you find. Someone else will chip in with other info if they have some 🙂

                    -Andy

                    1 Reply Last reply Reply Quote 0
                    • granitgash
                      granitgash @Andy.Drinkwater last edited by

                      We are suspecting that CloudFlare might be causing these troubles. We are trying everything, in the meantime i'm looking here to see if anyone has any similar experience or an idea for solution.

                      As for warnings, the only warning we had was the one last week (8/23/14) saying that Google bot can't acces our site:

                      Over the last 24 hours, Googlebot encountered 316 errors while attempting to connect to your site. Your site's overall connection failure rate is 7.5%.

                      -Granit

                      Andy.Drinkwater 1 Reply Last reply Reply Quote 0
                      • Andy.Drinkwater
                        Andy.Drinkwater @granitgash last edited by

                        It doesn't look like a firewall, as I can crawl it with Screaming Frog. However, the server logs will be able to answer that one for you.

                        Without looking in depth, I'm not seeing anything that stands out to me - do you think that there have been changes to the server that could cause issues? What firewall is the server running? Also, if there were errors in crawling the site, you would see a warning about this.

                        -Andy

                        granitgash 1 Reply Last reply Reply Quote 1
                        • granitgash
                          granitgash @Andy.Drinkwater last edited by

                          In mid-march website changed it's CMS but i don't think that could be the reason because until this week everything was working perfectly. I don't think it could have been compromised too. I'm still suspecting it could be the firewall blocking bots from crawling the site, but the server administrator couldn't find any evidence of this.

                          Andy.Drinkwater 1 Reply Last reply Reply Quote 0
                          • Andy.Drinkwater
                            Andy.Drinkwater last edited by

                            Hi Granit,

                            Has any work been done to the site in the last 2-3 months? Have you had any warnings in webmaster tools at all? I did once see a strange problem where Google wasn't crawling a site correctly because it had been compromised, but after checking, there is nothing like this on yours.

                            -Andy

                            granitgash 1 Reply Last reply Reply Quote 1
                            • granitgash
                              granitgash @KeriMorgret last edited by

                              No prb. Thanks a lot for your time. Let just hope that someone in the community will help with a solution 🙂

                              1 Reply Last reply Reply Quote 0
                              • KeriMorgret
                                KeriMorgret @granitgash last edited by

                                Unfortunately, I don't have a quick answer for you. Looking forward to seeing what other community members have to say on this one!

                                granitgash 1 Reply Last reply Reply Quote 1
                                • granitgash
                                  granitgash @KeriMorgret last edited by

                                  I'm looking at the http version in GWT

                                  KeriMorgret 1 Reply Last reply Reply Quote 0
                                  • KeriMorgret
                                    KeriMorgret last edited by

                                    If I do a site:gazetaexpress.com in Google, I get some results that are http, and some results that are https. The https ones say there is an SSL connection error.

                                    Are you looking at the http or https version in GWT?

                                    granitgash 1 Reply Last reply Reply Quote 1
                                    • 1 / 1
                                    • First post
                                      Last post

                                    Got a burning SEO question?

                                    Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                                    Start my free trial


                                    Browse Questions

                                    Explore more categories

                                    • Moz Tools

                                      Chat with the community about the Moz tools.

                                    • SEO Tactics

                                      Discuss the SEO process with fellow marketers

                                    • Community

                                      Discuss industry events, jobs, and news!

                                    • Digital Marketing

                                      Chat about tactics outside of SEO

                                    • Research & Trends

                                      Dive into research and trends in the search industry.

                                    • Support

                                      Connect on product support and feature requests.

                                    • See all categories

                                    Related Questions

                                    • SEOhmygod

                                      Does content revealed by a 'show more' button get crawled by Google?

                                      I have a div on my website with around 500 words of unique content in, automatically when the page is first visited the div has a fixed height of 100px, showing a couple of hundred words and fading out to white, with a show more button, which when clicked, increases the height to show the full content. My question is, does Google crawl the content in that div when it renders the page? Or disregard it? Its all in the source code. Or worse, do they consider this cloaking or hidden content? It is only there to make the site more useable for customers, so i don't want to get penalised for it. Cheers

                                      Intermediate & Advanced SEO | | SEOhmygod
                                      0
                                    • plumvoice

                                      What can you do when Google can't decide which of two pages is the better search result

                                      On one of our primary keywords Google is swapping out (about every other week) returning our home page, which is more transactional, with a deeper more information based page. So if you look at the Analysis in Moz you get an almost double helix like graph of those pages repeatedly swapping places. So there seems to be a bit of cannibalizing happening that I don't know how to correct. I think part of the problem is the deeper page would ideally be "longer" tail searches that contain the one word keyword that is having this bouncing problem as a part of the longer phrase. What can be done to try prevent this from happening? Can internal links help? I tried adding a link on that term to the deeper page to our homepage, and in a knee jerk reaction was asked to pull that link before I think there was really any evidence to suggest that that one new link made a positive or negative effect. There are some crazy theories floating around at the moment, but I am curious what others think both about if adding a link from a informational to a transactional page could in fact have a negative effect, and what else could be done/tried to help clarify the difference between the two pages for the search engines.

                                      Intermediate & Advanced SEO | | plumvoice
                                      0
                                    • andreas.wpv

                                      Can Google index PDFs with flash?

                                      Does anyone know if Google can index PDF with Flash embedded? I would assume that the regular flash recommendations are still valid, even when embedded in another document. I would assume there is a list of the filetype and version which Google can index with the search appliance, but was not able to find any. Does anyone have a link or a list?

                                      Intermediate & Advanced SEO | | andreas.wpv
                                      0
                                    • VentaMarketing

                                      Why did this website disappear from Google's SERPs?

                                      For the first several months this website, WEBSITE, ranked well in Google for several local search terms like, "Columbia MO spinal decompression" and "Columbia, MO car accident therapy." Recently the website has completely disappeared from Google's SEPRs. It does not even exist when I copy and paste full paragraphs into Google's search bar. The website still ranks fine in Bing and Yahoo, but something happened that caused it to be removed from Google. Beside for optimizing the meta data, adding headers, alt tags, and all of the typical on-page SEO stuff, we did create a guest post for a relevant, local blog. Here is the post: Guest Post. The post's content is 100% unique. I realize the post has way to many internal/external links, which we definitely did not recommend, but can anyone find a reason why this website was removed from Google's SERPs? And possibly how we should go about getting it back into Google's SERPs? Thanks in advance for any help.

                                      Intermediate & Advanced SEO | | VentaMarketing
                                      0
                                    • kbbseo

                                      Will Google bots crawl tablet optimized pages of our site?

                                      We are in the process of creating a tablet experience for a portion of our site. We haven’t yet decided if we will use a one URL structure for pages that will have a tablet experience or if we will create separate URLs that can only be access by tablet users. Either way, will the tablet versions of these pages/URLs be crawled by Google bots?

                                      Intermediate & Advanced SEO | | kbbseo
                                      0
                                    • SamCUK

                                      How can this site rank post panda/penguin?

                                      I am doing link building for an adult dating comparison website. One of the main competitors though, having checked their backlink profile have anchor text that is not varied at all. In fact many, many links that are all the same. How can they possibly rank in the post panda/penguin era? In fact they're at number 2! The site is an adult site and it www.f hypen buddy.co.uk if anyone wants to runa backlink check on OSE. Any help greatly appreciated!

                                      Intermediate & Advanced SEO | | SamCUK
                                      0
                                    • jamestown

                                      Google suddenly indexing and displaying URLs that haven't existed for years?

                                      We recently noticed google is showing approx 23,000 indexed .jsp urls for our site. These are ancient pages that haven't existed in years and have long been 301 redirected to valid urls. I'm talking 6 years. Checking the serps the other day (and our current SEOMoz pro campaign), I see that a few of these urls are now replacing our correct ones in the serps for important, competitive phrases. What the heck is going on here? Is Google suddenly ignoring rewrite rules and redirects? Here's an example of the rewrite rules that we've used for 6+ years: RewriteRule ^(.*)/xref_interlux_antifoulingoutboards&keels.jsp$ $1/userportal/search_subCategory.do?categoryName=Bottom%20Paint&categoryId=35&refine=1&page=GRID [R=301] Now, this 'bottom paint' url has been incredibly stable in the serps for over a half decade. All of a sudden, a google search for 'bottom paint' (no quotes) brings up the jsp page at position 2-3. This is just one example of something very bizarre happening. Has anyone else had something similar happen lately? Thank You <colgroup><col width="64"></colgroup>
                                      | RewriteRule ^(.*)/xref_interlux_antifoulingoutboards&keels.jsp$ $1/userportal/search_subCategory.do?categoryName=Bottom%20Paint&categoryId=35&refine=1&page=GRID [R=301] |

                                      Intermediate & Advanced SEO | | jamestown
                                      0
                                    • DWJames

                                      Blog/Shop/Forum site structure - are we right to make these changes?

                                      We run a fairly large online community with a popular blog and Europe's largest online shop for drift-specific motor sport parts and our website has been around since 2004 I believe. Since it was launched, the blog (or previous CMS system) has been at the domain root, the forums have been located at /forum and the shop at /shop (or similar) but we have decided to move things around a bit and would like some comments as to whether we are doing the right thing or if you would make any addition or different changes to us. Currently the entire website gets around 3m page views per month from 500,000 visitors, but this is split roughly 75% to the forums, 10% to the shop and 15% to the blog (but remember the blog is at the root so anyone who visits our homepage "visits" the blog). We plan to move the shop to the domain root (since the shop provides the income for the business - surely it should be the 1st thing visitors see?), the blog from root to /blog and the forums will stay where they are at /forum. We have read Steven Macdonald's post here, and have taken notes to help minimize traffic loss and disruption to our army of users and hopefully avoid too many penalties from Google and plan to: 301 redirect old URLs to new ones where they have changed. Submit new site maps to search engines. Update old links where we have control (such as forums where we are paid traders etc.). Send out a newsletter to our subscribers. Update our forum members. Fix errors via WMT before and after the re-structure. Should we be taking this opportunity to actually set each of the three sections of the site to it's own sub domain? Our thoughts are that if we are disrupting things, it's surely best to have lots of disruption once rather than a little bit of disruption several times over a 3-6 month period? OSE shows us to have roughly 1500 inbound links to /shop, 2100 to /forum and 4800 to the root / - if we proceed with our plan and put 301 redirects in place this seems to be the best plan to retain the value of these links but if we were to switch to sub domains would the 301s lose most of the link values due to them being on "different" domains? Any help, advise or suggestions are very welcome but comments from experience are what we are seeking ideally! Thanks Jay

                                      Intermediate & Advanced SEO | | DWJames
                                      0
                                    Moz logo
                                    • Contact
                                    • Community
                                    • Free Trial
                                    • Terms & Privacy
                                    • Accessibility
                                    • Jobs
                                    • Help
                                    • News & Press
                                    • MozCon
                                    © 2021 - 2024 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.

                                    Looks like your connection to Moz was lost, please wait while we try to reconnect.