robots.txt not updated or displaying default settings

The article provides a solution for when you have configured robots.txt correctly, for example per Best practices for Adobe Commerce robots.txt but the robots.txt is not getting updated or is displaying the default settings.

Affected products and versions

• Adobe Commerce on cloud infrastructure 2.3.x, 2.4.x

Issue

Cannot change the default robots.txt setting.

Steps to reproduce:

1. Access the Admin panel.
2. Add content to Content > Design > Configuration > Edit Custom instruction of robots.txt file such as the text "hello" and save the changes.
3. Visit the robots.txt url.

Expected result: robots.txt has the saved text.

Actual result:

robots.txt file does not change.

Cause

Indexing by search engines is turned off.

Solution

Enable indexing by search engines. See Configure indexing by search engine in our developer documentation.

Related reading

• Add site map and search engine robots in our developer documentation.