This article provides best practices for using
robots.txt in Adobe Commerce. This includes configuration and security. The
robots.txt file is a text file that instructs web robots (typically search engine robots) how to crawl pages on their website.
Affected products and versions
- Adobe Commerce on-premises, all supported versions
- Adobe Commerce on cloud infrastructure, all supported versions
- By not configuring the
robots.txtfile, site performance can be negatively impacted, leading to a potential for site outages.
- Configure the
robots.txtfile to avoid unnecessary Bots scanning and indexing the wrong content. For steps, refer to Search Engine Robots in our user guide.
Do not expose your Commerce Admin path in your
robots.txt file. Having the Admin path exposed is a vulnerability for site hacking and potential loss of data. Remove the Admin path from the
For steps to edit the
robots.txt file and remove all entries of the Admin path, refer to Marketing User Guide > SEO and Search > Search Engine Robots in our user guide.
If assistance is required or if there are questions or concerns, submit an Adobe Commerce Support ticket.
In our support knowledge base: