The article provides a solution for when you have configured
robots.txt correctly, for example per Best practices for Magento robots.txt but the
robots.txt is not getting updated or is displaying the default settings.
Affected products and versions
- Magento Commerce (Cloud) 2.3.x, 2.4.x
Cannot change the default
Steps to reproduce:
- Access the Magento Admin panel.
- Add content to Content > Design > Configuration > Edit Custom instruction of
robots.txtfile such as the text "hello" and save the changes.
robots.txt has the saved text.
robots.txt file does not change.
Indexing by search engines is turned off.
Enable indexing by search engines. See Configure indexing by search engine in Magento Developer Documentation.