This article provides a fix for when the robots.txt file throws a 404 error in Magento Commerce Cloud 2.3.x.
Affected products and versions
Magento Commerce Cloud versions 2.3.x
Issue
The robots.txt file is not working and throws a Nginx exception. The robots.txt file is generated dynamically "on the fly." The robots.txt file is not accessible by the /robots.txt URL because Nginx has a rewrite rule which forcibly redirects all /robots.txt requests to the /media/robots.txt file that does not exist.
Cause
This typically happens when Nginx is not configured properly.
Solution
The solution is to disable the Nginx rule.
Submit a Magento Support ticket requesting removal of the Nginx redirect rule from /robots.txt requests to /media/robots.txt.