Google posts draft to formalize Robots Exclusion Protocol Specification

Posted on July 1, 2019 by gftshappy

Google open sources its robots.txt parser as part of this announcement.

Please visit Search Engine Land for the full article.