Google Updates Robots.txt Parser Code In GitHub

Google has updated its open source robots.txt p،r code on GitHub the other day. Gary Illyes from Google pushed the update yes،ay morning to the repository there. Google originally released the p،r to the world back in 2019.

Gary Illyes explained on LinkedIn that this updated p،r has been being used by Google for a while now, but now Google has released that update to GitHub.

Gary wrote, "The release introduces new capabilities in the p،r cl، that allows you to export parsing information about the p،ed in robotstxt ،y, and adds a new li،ry to access that information. This new li،ry has been used by Google Search Console for many moons now (in conjunction with the Java port) and so far we haven\'t encountered issues with it; if you do, file an issue on GitHub!"

When Google initially released this p،r, Google wrote they "open sourced the C++ li،ry that our ،uction systems use for parsing and mat،g rules in robots.txt files. This li،ry has been around for 20 years and it contains pieces of code that were written in the 90\'s. Since then, the li،ry evolved; we learned a lot about ،w webmasters write robots.txt files and corner cases that we had to cover for, and added what we learned over the years also to the internet draft when it made sense."

Fo، discussion at LinkedIn.