![block dotbot in .htaccess block dotbot in .htaccess](https://lionzeal.com/wp-content/uploads/2014/11/directory-selection-300x226.png)
Block dotbot in .htaccess full#
This happens far too often to be accounted for by a few random incorrect links.) Do they get their shopping list from someone else, like scraping a search engine's full listings? I would like to know how they find out about deep-interior URLs, like /directory/subdir/pagename.html when they have never seen /directory/subdir/ which is the only way to get there. If it was just DotBot I'd say maybe it's a dumb crawler.Oh, it’s definitely a dumb crawler.
Block dotbot in .htaccess update#
( Psst! Browser developers! Wouldn't it be clever to update bookmarks automatically when you see a protocol redirect?) I have a few specific pages where I do see humans getting redirected, but those are high-interest pages that would have been bookmarked before the site went HTTPS. Other than search engines, most links to the pages I'm especially interested in are from a curated directory, whose listings are correct. Especially since, if there were incorrect HTTP links out there, other search engines would be following them too. Is your site popular with forums or blogs and maybe they're linking to your pages with insecure URLs?Not that I know of, and definitely not to this wide extent. I am currently leaning toward the option of simply blocking all HTTP requests from DotBot. They are perfectly capable of using HTTPS (I've met a handful of law-abiding robots that aren’t) they just prefer HTTP, even if it means maintaining a ratio of about fifteen 301s to every one 200. What infuritates me to no end is that those redirects include pages that have never existed as HTTP, so if they claim to be following links they are lying in their teeth. Every few months there is a fresh flurry of DotBot requests, all redirected, on top of their usual sporadic visits. I keep a very close eye on redirects-currently mostly due to HTTP>HTTPS from a move made in late 2019. (I generally avoid the position of “If it doesn’t personally benefit me, I don’t want it.” After all, I don’t personally benefit from the street in front of your house, but that’s no reason not to maintain it.) Who benefits from DotBot? Asking seriously, not rhetorically.