Skip to main content

Crawling bots which to allow/block in robots.txt??

I just built my 1st website, hosted on AWS Amplify w/ a Cloudflare domain. Cloudflare suggested robots txt file and there are so many bots i don't know which to block. I know i need to allow Google, Microsoft for my SEO and Google page rank, but which ones do i add to my robots.txt file to block (like the ones for ai training, using content without permission and giving false web visit analytics)?

Thanks for your help! 🙂

submitted by /u/spinspin_suga
[link] [comments]

from Search Engine Optimization: The Latest SEO News https://ift.tt/QzoUShw
Labels:

Comments

Post a Comment

Popular posts from this blog

Legit Reviews

The google review thing is insane I have gotten 2 legit verified reviews that i ask client to screenshot and observe goolge not post them, pretend like they never happened and when i sent them the image a policy message was sent . As I was browsing through the google forum a lot of ppl are getting hit with legit reviews being removed. While all this is going on i have observe a company go from 12 reviews to 65 reviews in a two month period with some stuff that dont make sense on some of them. Seems like a new business or profile is being limited while aged accounts can have a free for fall but who really knows. Still waiting for a resolution on a if any. submitted by /u/Ok-Bowl-6167 [link] [comments] from Search Engine Optimization: The Latest SEO News https://ift.tt/zkbfgDH
Post a Comment
Read more

Local seo vs. natiowide seo?

I've done SEO for local businesses but I recently got my first client that sells an item nation wide. ​ Any suggestions for doing nationwide SEO? ​ I am used to making geopages for local towns. I was going to do the same with some input from the client about what cities or towns he would like to show up in? submitted by /u/Letmeinterviewyou [link] [comments] from Search Engine Optimization: The Latest SEO News http://bit.ly/2JHy0k0
Post a Comment
Read more

are my YT keywords being ignored?

I have 3hr+ YT videos, so I chapter them out quite extensively. I've read in a couple places if I have too many keywords just grouped together they will all be ignored. Are my keywords being ignored...or are they even being crawled...through my chapters section? Also, is hashtagging in a chapter name good, bad, or not at all relevant? example chapters: Bookmarks: 0:00 Intro 4:25 Little Panda #littlepanda 13:04 Royale With Cheese Megaways #royalewithcheesemegaways 24:48 Big Bass Bonanza #bigbassbonanza 33:02 Book of Cats #bookofcats 44:11 Tractor Beam #tractorbeam submitted by /u/bauerwelson [link] [comments] from Search Engine Optimization: The Latest SEO News https://ift.tt/3Gx8IPZ
Post a Comment
Read more