Managed alerts for Magento Commerce: Apdex critical alert

This article provides troubleshooting steps when you receive an Apdex critical alert for Magento Commerce in New Relic. The Apdex score measures users' satisfaction to the response time of web applications and services. Immediate action is required to remedy the issue. 

The alert will look something like the following, depending on the alert notification channel you selected.

Affected products and versions

• Magento Commerce Cloud Pro.
• Magento Commerce Cloud Starter.

Issue

You will receive a managed alert in New Relic if you have signed up to Managed alerts for Magento Commerce and one or more of the alert thresholds have been surpassed. These alerts were developed by Magento to give customers a standard set using insights from Support and Engineering.

Do!

• Abort any deployment scheduled until this alert is cleared.
• Put your site into maintenance mode immediately if your site is or becomes completely unresponsive. For steps refer to DevDocs Installation Guide> Enable or disable maintenance mode. Make sure to add your IP to the exempt IP address list to ensure that you are still able to access your site for troubleshooting. For steps, refer to DevDocs Maintain the list of exempt IP addresses.

Don't!

• Launch additional marketing campaigns which may bring additional pageviews to your site.
• Run indexers or additional crons which may cause additional stress on CPU or disk.
• Do any major administrative tasks (i.e., Magento Admin, data imports / exports).
• Clear your cache.

Doing the above when you have received a critical alert, before you have troubleshooted the cause of the alert, may lead to your site becoming non-responsive, if you are not already experiencing a site outage.

Solution

Follow these steps to identify and troubleshoot the cause.

Because this is a critical alert, it is highly recommended you complete Step 1 before you try to troubleshoot the issue (Step 2 onwards).

1. Check if a Magento support ticket exists. For steps, refer to KB Track your support tickets. Support may have received a New Relic threshold alert, created a ticket and started working on the issue. If no ticket exists, create one. The ticket should have the following information:
   • Contact Reason: select “New Relic CRITICAL alert received"
   • Description of the alert
   • New Relic Incident link. This is included in your Managed alerts for Magento Commerce.
2. To identify the source of the problem, use New Relic APM's Transaction page to identify transactions with performance issues:
   • Sort transactions by ascending Apdex scores. Apdex refers to user satisfaction to the response time of your web applications and services. A low Apdex score can indicate a bottleneck (a transaction with a higher response time). Usually it is the database, Redis, or PHP. For steps, refer to New Relic View transactions with highest Apdex dissatisfaction.
   • Sort transactions by highest throughput, the slowest average response time, most time-consuming, and other thresholds. For steps, refer to New Relic Find specific performance problems. If you are still struggling to identify the issue use New Relic APM's Infrastructure page. 
3. Use New Relic APM's Infrastructure page to identify resource intensive processes. For steps, refer to New Relic Infrastructure monitoring Hosts page > Processes tab:
   1. If services like Redis or MySQL are the top source of memory consumption, try the following:
      1. Check that you are on the latest version. Newer versions can sometimes fix memory leaks. If you are not on the latest version, consider upgrading. For steps, refer to Magento Commerce Cloud > Services > Change Services. 
      2. Check for MySQL issues like long running queries, Primary keys not defined, and duplicate indexes. For steps, refer to Most Common database Issues in Magento Commerce Cloud. 
      3. Check for PHP issues. Review running processes by running ps aufx in the CLI/Terminal. In the terminal output you will see cron jobs and processes that are currently being executed. Check the output for the processes' execution time. If there is a cron with a long execution time, the cron may be hanging. Refer to KB Slow performance, slow and long running crons and KB Cron job stuck in "running" status for troubleshooting steps. 
4. Once the source is identified, SSH into the environment to investigate further. For steps refer to DevDocs Magento Commerce Cloud: Technologies and requirements: SSH into your environment. 
5. If you are still struggling to identify the source, review recent trends to identify issues with recent code deployments or configuration changes (for example, new customer groups and large changes to the catalog). It is recommended that you review the past 7 days of activity for any correlations in code deployments or changes.
6. If you are unable to find a solution within a reasonable time, request an upsize or place site into maintenance mode if you have not already done so. For steps, refer to KB How to request temp resize and DevDocs Installation Guide > Enable or disable maintenance mode.
7. If the upsize returns the site to normal operations, consider requesting a permanent upsize (contact your CSM), or try to reproduce the problem in your Dedicated Staging by running a load test and optimize queries, or code that reduces pressure on services. Refer to DevDocs Magento Commerce Cloud  > Test Deployment > Load and stress testing.