School · Module 5 · Lesson 1 of 3
How robots.txt decides allow vs block
RFC 9309: the most specific matching group wins; longest rule wins; Allow wins a tie.
A named User-agent group replaces the * group for that bot. If you Disallow /admin under *, copy those lines into each named group you care about.
404 on /robots.txt means 'everything allowed'. 5xx means well-behaved crawlers should try later or treat the site as blocked.
ShipSafe's AI bot checker shows the exact deciding line for 23 AI crawlers.