kevintechie
  • Home
Subscribe
Tagged

Tools

A collection of 1 post

Tools

wget and robots.txt

A web application I was hired to fix was experiencing a number of unhandled exceptions. After adding error logging to capture the exceptions for review, I noticed that one of the exceptions was triggered every time a web crawler hit the site. I added a robots.txt file to the

  • Kevin Coleman
Kevin Coleman Apr 22, 2015 • 1 min read
kevintechie © 2022
Powered by Ghost