robots.txt endpoint cannot be removed and how to prevent search engine indexingSummary Issue The /robots.txt endpoint can be reached on ServiceNow instances, including self-hosted (on-premises) instances where the file isn't meant to be used or exposed. Facts Customers want to know: - Whether the robots.txt endpoint can be removed or restricted.- How to keep ServiceNow pages out of search engine results, such as Google. Cause In ServiceNow, robots.txt is a plain-text resource that tells web crawlers and search engine bots, such as Googlebot, which pages or directories they may access and index. The robots.txt file can't be removed from the instance, including on self-hosted deployments. It isn't stored as a file on the underlying file system. Instead, the ServiceNow application server generates the response at the /robots.txt endpoint dynamically. By default, the robots.txt file doesn't expose sensitive information. Its default content is: User-agent: *Disallow: / Where: - User-agent: * applies the rule to all web crawlers.- Disallow: / tells crawlers not to access or index any content on the instance. The robots.txt file only gives instructions to crawlers. It is not an access-control mechanism. To keep pages out of search results or to limit access, Google recommends using the noindex tag or password protection instead of robots.txt. Resolution NOTES: - The below steps must be performed with a user with admin privileges.- Verify the Google Custom Search Integration plugin is installed(com.snc.integration.google_custom_search) 1. Keep the default robots.txt configuration. On self-hosted instances where robots.txt isn't used, the default content already tells all crawlers not to index the instance. No action is needed.2. Prevent specific pages from being indexed. Add the noindex tag and the relevant meta tags to the Service Portal page. For the steps, see the ServiceNow documentation Search engine optimization for portal pages.3. Restrict access. To limit access to pages, use password protection (require authentication) instead of relying on robots.txt.4. Customize robots.txt (optional). To manage or customize how robots.txt behaves, go to: All > Custom Search Integration > Robots.txt Definitions For additional details, see the Knowledge Base articles listed in Related Links. Related Links Block Search indexing with noindexIntroduction to robots.txtKB0787281 - Customise the Robots.txt File on a ServiceNow InstanceKB0684534 - How to allow customers to edit robots.txt file on customer instanceSearch engine optimization for portal pages