This article provides a fix for when the
robots.txt file throws a 404 error in Magento Commerce Cloud 2.3.x.
Affected products and versions
Magento Commerce Cloud versions 2.3.x
robots.txt file is not working and throws a Nginx exception. The
robots.txt file is generated dynamically "on the fly." The
robots.txt file is not accessible by the
/robots.txt URL because Nginx has a rewrite rule which forcibly redirects all
/robots.txt requests to the
/media/robots.txt file that does not exist.
This typically happens when Nginx is not configured properly.
The solution is to disable the Nginx rule.
Submit a Magento Support ticket requesting removal of the Nginx redirect rule from
/robots.txt requests to