This article provides best practices for using
robots.txt in Magento. This includes configuration and security. The
robots.txt file is a text file that instructs web robots (typically search engine robots) how to crawl pages on their website.
Affected products and versions
- By not configuring the
robots.txtfile, site performance can be negatively impacted leading to a potential for site outages.
- Configure the
robots.txtfile to avoid unnecessary Bots scanning and indexing the wrong content. For steps, refer to Magento User Guide Search Engine Robots.
Do not expose your Magento Admin path in your
robots.txt file. Having the Admin path exposed is a vulnerability for site hacking and potential loss of data. Remove the Admin path from the
For steps to edit the
robots.txt file and remove all entries of the admin path, refer to Magento Marketing User Guide > SEO and Search > Search Engine Robots.
If assistance is required or if there are questions or concerns, submit a Magento Support ticket.
- KB How to block malicious traffic for Magento Commerce Cloud on Fastly level
- KB robots.txt gives a 404 error in Magento Commerce Cloud 2.3.x