How to block malicious traffic for Magento Commerce Cloud on Fastly level

This article provides the steps you could take to block malicious traffic, when you suspect that your Magento Commerce Cloud store is experiencing a DDoS attack. 

Affected products and versions:

• Magento Commerce Cloud 2.3.x

In this article we assume that you already have the malicious IPs and/or their country and userAgents. Magento Commerce Cloud users would typically get this information from Magento Support. The following sections provide steps for blocking traffic based on this information. All the changes should be done in the Production environment.

Block traffic by IP

For the Magento Commerce Cloud store, the most effective way to block traffic by specific IP addresses and subnets is adding an ACL for Fastly in the Magento Admin. Following are the steps with links to more detailed instructions: 

1. In the Magento Admin, navigate to Stores > Configuration > Advanced > System > Full Page Cache > Fastly Configuration.
2. Create a new ACL with a list of IP addresses or subnets you're going to block.
3. Add it to the ACL list and block as described in the Blocking guide for the Fastly_Cdn Magento module. 

Block traffic by country 

For the Magento Commerce Cloud store, the most effective way to block traffic by country(s) is adding an ACL for Fastly in the Magento Admin.  

1. In the Magento Admin, navigate to Stores > Configuration > Advanced > System > Full Page Cache > Fastly Configuration.
2. Select the countries and configure blocking using ACL as described in the Blocking guide for the Fastly_Cdn Magento module. 

Block traffic by user agent

To establish blocking based on user agent, you need to add a custom VCL snippet to your Fastly configuration. To do this, take the following steps:

1. In the Magento Admin, navigate to Stores > Configuration > Advanced > System > Full Page Cache > Fastly Configuration > Custom VCL Snippets.
2. Create the new custom snippet as described in the Custom VCL snippets guide for the Fastly_Cdn module. You can use the following code sample as an example. This sample disallows traffic for the AhrefsBot and SemrushBot user agents.

name: block_bad_useragents
  type: recv
  priority: 5
  VCL:
  ```
  if ( req.http.User-Agent ~ "(AhrefsBot|SemrushBot)" ) {
      error 405 "Not allowed";

  }
  ```

Rate Limiting (experimental Fastly functionality)

There is an experimental Fastly functionality for Magento Commerce Cloud which allows you to specify the rate limit for particular paths and crawlers. Please reference the Fastly module documentation for details.

The functionality must be extensively tested on staging, before being used on production, because it might block legitimate traffic. 

Recommended: consider updating robot.txt

Updating your robots.txt file could help to keep certain search engines, crawlers, and robots from crawling certain pages. Examples of pages that should not be crawled are search result pages, checkout, customer information and so on. Keeping robots from crawling these pages could help to decrease the number of requests generated by those robots.

 There are two important considerations when using robots.txt:

• Robots can ignore your robots.txt. Especially malware robots, that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
• The robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

The basic information and default Magento robots.txt configuration can be found in the Search Engine Robots DevDocs article. 

For general information and recommendations about robots.txt, see:

• Create a robots.txt file by Google Support
• About /robots.txt by robotstxt.org

Work with your developer and/or SEO expert to determine what User Agents you want to allow, or those you want to disallow.