Best practices for Adobe Commerce robots.txt
This article provides best practices for using
robots.txt in Adobe Commerce. This includes configuration and security. The
robots.txt file is a text file that instructs web robots (typically search engine robots) how to crawl pages on their website.
Affected products and versions
- By not configuring the
robots.txt file, site performance can be negatively impacted, leading to a potential for site outages.
- Configure the
robots.txt file to avoid unnecessary Bots scanning and indexing the wrong content. For steps, refer to Search Engine Robots in our user guide.
Do not expose your Commerce Admin path in your
robots.txt file. Having the Admin path exposed is a vulnerability for site hacking and potential loss of data. Remove the Admin path from the
For steps to edit the
robots.txt file and remove all entries of the Admin path, refer to Marketing User Guide > SEO and Search > Search Engine Robots in our user guide.
If assistance is required or if there are questions or concerns, submit an Adobe Commerce Support ticket.
In our support knowledge base: