robots.txt not updated or displaying default settings
The article provides a solution for when you have configured
robots.txt correctly, for example per Best practices for Adobe Commerce robots.txt but the
robots.txt is not getting updated or is displaying the default settings.
Affected products and versions
- Adobe Commerce on cloud infrastructure 2.3.x, 2.4.x
Cannot change the default
Steps to reproduce:
- Access the Admin panel.
- Add content to Content > Design > Configuration > Edit Custom instruction of
robots.txt file such as the text "hello" and save the changes.
- Visit the
robots.txt has the saved text.
robots.txt file does not change.
Indexing by search engines is turned off.
Enable indexing by search engines. See Configure indexing by search engine in our developer documentation.