Skip to main content

Google is trying to crawl excluded and blocked pages, more than 16 million

My colleagues sent me a very interesting situation:

  1. Google started crawling a very large volume of blocked pages, already found 16 million)) Blocked URL in the robots.txt, meta noindex, and then added X-Robots-Tag noindex
  2. in the server has increased load, almost 1000 times
  3. there are no 503 errors, the server can withstand loading
  4. an XML sitemap, everything is clean
  5. the site is 1 year old
  6. clicks and impressions are growing in Google Search Console, but for now
  7. very strange, but google does not go so far on the pages... and here 27 slashes (/.../.../.../.../... etc) in the link +/- 25 clicks from the main page
  8. in the log file is really a lot of Googlebot - 500 thousand per day
  9. the backlinks are also clean, no spam

P.S. Screenshots in comments

What do you think?

submitted by /u/Andrew-Chornyy
[link] [comments]

from Search Engine Optimization: The Latest SEO News https://ift.tt/3f00ylh
Labels:

Comments

Post a Comment

Popular posts from this blog

Wordpress Tag Div Composer

Can somebody help me with scroll to class ( call to action ) button for long article, I wanna drive readers to specific place on the page… I keep adding the name and save then click on it but it’s not working? I cleared cache and still not working… Help please! submitted by /u/NadaGamalEldean [link] [comments] from Search Engine Optimization: The Latest SEO News https://ift.tt/3Bi5uf2
Post a Comment
Read more

HTTP-HTTPS redirects: domain level or each URL?

Googling for answers seems to give conflicting answers here. We have a domain that has been HTTP but some individual URLs have been HTTPS (like form pages). We are about to deploy HTTPS redirect on the server to get everything going to https://www.domain.com . Do we have to go to the effort of a big 301 redirect list of all page-level URLs without Google dropping our earned page rank? Thanks all! submitted by /u/runtmc2 [link] [comments] from Search Engine Optimization: The Latest SEO News https://ift.tt/31dJ82b
Post a Comment
Read more

One website, two properties (domains)

I wasn't sure how to frame that title so let me explain; Due to error on my part, I have ended up with, a www.xyz.com , version of my website, and one that's just xyz.com. Now I'm tracking my traffic differently on GSC and GA. I'm not sure how to feel about this. Is there anyone with a similar situation? Should I just go on like usual or are there things I need to start doing differently? Kindly advise. submitted by /u/Blessed_Dude_101010 [link] [comments] from Search Engine Optimization: The Latest SEO News https://ift.tt/87Z2MRA
Post a Comment
Read more