Generate optimized robots.txt directives to guide Googlebot, Bingbot, and search engine crawlers with custom allow/disallow paths.
