Managed alerts on Magento Commerce: memory critical alert

This article provides troubleshooting steps when you receive a memory critical alert for Magento Commerce in New Relic. Immediate action is required to remedy the issue.

The alert will look something like the following, depending on the alert notification channel you selected.

Affected products and versions

All versions of Magento Commerce Cloud Pro.

Issue

You will receive a managed alert in New Relic if you have signed up to Managed alerts for Magento Commerce and one or more of the alert thresholds have been surpassed. These alerts were developed by Magento to give customers a standard set using insights from Support and Engineering.

Do!

• Abort any deployment scheduled until this alert is cleared
• Put your site into maintenance mode immediately if your site is or becomes completely unresponsive. For steps refer to DevDocs Installation Guide > Enable or disable maintenance mode. Make sure to add your IP to the exempt IP address list to ensure that you are still able to access your site for troubleshooting. For steps, refer to DevDocs Maintain the list of exempt IP addresses.

Don't!

• Launch additional marketing campaigns which may bring additional pageviews to your site.
• Run indexers or additional crons which may cause additional stress on CPU or disk.
• Do any major administrative tasks (i.e., Magento Admin, data imports/exports).
• Clear your cache.

Your site may become non-responsive, (if you are not already experiencing a site outage) if you do any of the "Don't" actions before you have investigated and solved the cause of the alert.

Solution

Follow these steps to identify and troubleshoot the cause.

Because this is a critical alert, it is highly recommended you complete Step 1 before you try to troubleshoot the issue (Step 2 onwards).

1. Check if a Magento support ticket exists. For steps, refer to KB Track your support tickets. Support may have already received a New Relic threshold alert, created a ticket and started working on the issue. If no ticket exists, create one. The ticket should have the following information:
   1. Contact Reason: select “New Relic CRITICAL alert received"
   2. Description of the alert
   3. New Relic Incident link. This is included in your Managed alerts for Magento Commerce.
2. Use New Relic APM's Infrastructure page to identify top memory intensive processes. For steps, refer to New Relic Infrastructure monitoring Hosts page > Processes tab:
   • If services like Redis, MySQL, or PHP are the top sources of memory consumption, try the following:
     1. Check that you are on the latest versions. Newer versions can sometimes fix memory leaks. If you are not on the latest version, consider upgrading. For steps, refer to Magento Commerce Cloud > Services > Change Services. 
     2. If the problem with the service is not version related, try the following:
        1. MySQL: Check for issues like long running queries, Primary keys not defined, and duplicate indexes. For steps, refer to Most Common database Issues in Magento Commerce Cloud. 
        2. Redis: If Redis is a top source of memory consumption, submit a support ticket.
        3. PHP: If PHP is a top source of memory consumption, review running processes by running ps aufx in the CLI/Terminal. In the terminal output you will see cron jobs and processes that are currently being executed. Check the output for the processes' execution time. If there is a cron with a long execution time, the cron may be hanging. Refer to KB Slow performance, slow and long running crons and KB Cron job stuck in "running" status for troubleshooting steps.
3. If you are still struggling to identify the source of the problem, use New Relic APM's Transaction page to identify transactions with performance issues:
   • Sort transactions by ascending Apdex scores. Apdex refers to user satisfaction to the response time of your web applications and services. A low Apdex score can indicate a bottleneck (a transaction with a higher response time). Usually it is the database, Redis, or PHP. For steps, refer to New Relic View transactions with highest Apdex dissatisfaction.
   • Sort transactions by highest throughput, the slowest average response time, most time-consuming, and other thresholds. For steps, refer to New Relic Find specific performance problems. If you are still struggling to identify the issue, use New Relic APM's Infrastructure page. 
4. If you cannot identify the cause of increased memory consumption, review recent trends to identify issues with recent code deployments or configuration changes (for example, new customer groups and large changes to the catalog). It is recommended that you review the past 7 days of activity for any correlations in code deployments or changes.
5. If the above methods do not help you find the cause and/or solution within a reasonable time, request an upsize or place site into maintenance mode if you have not already. For steps, refer to KB How to request temp resize and DevDocs Installation Guide > Enable or disable maintenance mode.
6. If the upsize returns the site to normal operations, consider requesting a permanent upsize (contact your CSM), or try to reproduce the problem in your Dedicated Staging by running a load test and optimize queries, or code that reduces pressure on services. Refer to DevDocs Magento Commerce Cloud  > Test Deployment > Load and stress testing.