Skip to main content

Crawling bots which to allow/block in robots.txt??

I just built my 1st website, hosted on AWS Amplify w/ a Cloudflare domain. Cloudflare suggested robots txt file and there are so many bots i don't know which to block. I know i need to allow Google, Microsoft for my SEO and Google page rank, but which ones do i add to my robots.txt file to block (like the ones for ai training, using content without permission and giving false web visit analytics)?

Thanks for your help! 🙂

submitted by /u/spinspin_suga
[link] [comments]

from Search Engine Optimization: The Latest SEO News https://ift.tt/QzoUShw
Labels:

Comments

Post a Comment

Popular posts from this blog

Local seo vs. natiowide seo?

I've done SEO for local businesses but I recently got my first client that sells an item nation wide. ​ Any suggestions for doing nationwide SEO? ​ I am used to making geopages for local towns. I was going to do the same with some input from the client about what cities or towns he would like to show up in? submitted by /u/Letmeinterviewyou [link] [comments] from Search Engine Optimization: The Latest SEO News http://bit.ly/2JHy0k0
Post a Comment
Read more

GMB territory inching with service company

Seeking advice on extending our service area, while our main GMB profile states SD County, we only rank in our neighborhood but we have the team to service much more jobs. The boss wants us to get addresses in these hot territories within the county and use call rail numbers to validate the GMB profile. Question: Does this still work? Will it catch up with us? OR is there a better way to rank in a territory for a service. We are 4.9 stars with 120+ reviews submitted by /u/SoCal619TradesPro [link] [comments] from Search Engine Optimization: The Latest SEO News https://ift.tt/9RWmh6i
Post a Comment
Read more