We analysed the robots.txt of 3,000+ new product launches. Here's who's most (not) wanted. Common Crawl takes the crown. More insights at stackscope.dev/insights #AI #WebCrawlers #RobotsTxt #BuildInPublic #IndieHackers

4 likes 0 replies