Google Open Sources Its ‘Web Crawler’ After 20 Years

Google’s Robot Exclusion Protocol (REP), also known as robots.txt, is a standard used by many websites to tell the automated crawlers which parts of the site should be crawled or not. However, it isn’t the officially adopted standard, leading to different interpretations. In a bid to make REP an official web standard, Google has open-sourced […]

The post Google Open Sources Its ‘Web Crawler’ After 20 Years appeared first on Fossbytes.

from Fossbytes https://ift.tt/2XqzyGA

Fossbytes – Fresh bytes of technology and more

Add a Comment

Your email address will not be published. Required fields are marked *

Skip to toolbar