Looks like Google has started rolling out this Web Bot Auth thing which seems li... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

wilg 20 days ago | parent | context | favorite | on: Someone is running mass vulnerability scans, spoof...

Looks like Google has started rolling out this Web Bot Auth thing which seems like something that should gain adoption or become an open standard. https://developers.google.com/crawling/docs/crawlers-fetcher...

Seems like the crawler companies would be incentivized to not want to take responsibility for people spoofing their user agents.

gavinhking 20 days ago [–]

Yup, in fact most of them are already. That's one of the ways this data is verifying whether the visits are spoofed or not: https://knownagents.com/insights#spoofing-and-security

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact