Search for and respect robots.txt when retrieving content

[leozqin]

Please 👍 this issue if you feel that we should implement a robots.txt check for Precis before executing content retrieval. I personalIy do not feel that Precis is a "web robot" in the sense that robots.txt is meant to address, but if a substantial number (as defined by me/vibes) of people feel that we should do this, I will implement such a feature.