Magento Index Management

Jul 1, 2020

6182 Aleksander Kutseika

Magento Index Management

Indexing is important for Magento 2 as for all systems that process large amounts of data in multiple tables. It facilitates platform’s performance. In this article, we are going to discuss the core aspects of Magento index management that you need to be aware of and we also demonstrate how you can create your own custom index.

What is indexing in Magento 2?
Default indexes in Magento 2
Terminology of indexation
Magento 2 index update
Magento index management via admin panel
Index commands for Magento 2
Index Status Tables in Database
Mechanism of indexation by schedule, Mview
Partial and complete indexation
Magento index management FAQs

What is indexing in Magento 2?

An index is a table of a certain format in a database. Magento index processing converts data to increase performance. It is necessary because the data which is required for an online store operation is stored in a complex structure of tables. For example, if you want to find out the current product price, it involves connecting and processing a lot of tables and their connections.

In their turn, indexes store partly processed and formatted data decreasing server load and increasing the store’s speed. However, indexing is required after any change of the data that this index is applied to.

Default indexes in Magento 2

There are several out of the box indexes in Magento 2 Open Source:

Name ID Class Description
Catalog Product Rule catalogrule_product Magento\CatalogRule\Model\Indexer\Product\ProductRuleIndexer
Catalog Rule Product catalogrule_rule Magento\CatalogRule\Model\Indexer\Rule\RuleProductIndexer
Catalog Search catalogseatch_fulltext Magento\CatalogSearch\Model\Indexer\Fulltext
Category Products catalog_category_product Magento\Catalog\Model\Indexer\Category\Product It creates the category – products connection
Customer Grid customer_grid Magento\Framework\Indexer\Action\Entity It restructures Customer Grid index
Design Config Grid design_config_grid Magento\Theme\Model\Indexer\Design\Config
Product Categories catalog_product_category Magento\Catalog\Model\Indexer\Product\Category It creates product – categories connection
Product EAV catalog_product_attribute Magento\Catalog\Model\Indexer\Product\Eav It restructures the EAV structure into a flat structure
Product Price catalog_product_price Magento\Catalog\Model\Indexer\Product\Price It pre-calculates product prices
Stock cataloginventory_stock Magento\CatalogInventory\Model\Indexer\Stock

Magento 2 Commerce Edition includes all the indexes from Open Source as well as some additional ones:

Name ID Class Description
Inventory inventory Magento\InventoryIndexer\Indexer\InventoryIndexer Multi-Source Inventory index
Product/Target Rule targetrule_product_rule Magento\TargetRule\Model\Indexer\TargetRule\Product\Rule It creates product – rule connection
Target Rule/Product targetrule_rule_product Magento\TargetRule\Model\Indexer\TargetRule\Rule\Product It creates rule – product connection
Sales Rule salesrule_rule Magento\AdvancedSalesRule\Model\Indexer\SalesRule Rule indexing

Besides the indexes listed above, you can also create a custom one. We are going to discuss it in detail below.

Magento 2 Indexation Terminology

  • Dictionary is the original regular data entered into the system.
  • Index is the representation of the original data for optimized reading and searching. It should be updated when there are any changes in the dictionary.
  • Indexer is an object that creates and updates an index.

Magento 2 index update

In order to operate up-to-date data, indexes should be updated from time to time. There are three ways to do it:

  1. Scheduled updates (launched by cron);
  2. Magento index update on save (It is launched when the data has been edited and saved);
  3. Manual update (via terminal).

By the type of processed data, Magento reindex can be launched as:

  • Full reindex processes all the indexed data again. It can always be launched manually via console.
  • Partial reindex processes again only the data that has been changed.

The logic of Magento indexer operation is described below:

Magento 2 indexation diagram

 

Magento index management via admin panel

In the admin panel, you can control index modes and observe the current information about the indexes (System> Index Management).

Magento 2 indexation

In the Mode column, we can see how index is updated. In the Status column, we can see whether reindex is required (Reindex Required) or not required (Ready). In the Updated column, you can see the date of the latest update.

On this page, we can change the reindex mode. In order to do it, you should choose the necessary mode in a checkbox and press Submit.

magento index management reindex magento 2

 

The default opportunities for managing and monitoring index operations are quite limited but there are multiple free modules for managing and reorganizing indexes from the Magento 2 admin panel.

Indexer 240px

Indexer for Magento 2

Optimize the indexing process in your Magento 2 backend.

Free download

What are the index commands for Magento 2?

We cannot launch reindex from the admin panel. In order to do it, we need access to the server via terminal. If you put in php bin/magento indexer, you will see the command description for indexing:

indexer:info                  Shows allowed Indexers.
indexer:reindex            Reindexes Data
indexer:reset                 Resets indexer status to invalid
indexer:set-mode         Sets index mode type
indexer:show-mode    Shows Index Mode
indexer:status              Shows status of Indexer

Executing this command, we can see how to work with the whole group of indexes as well as with specific ones adding their names as an argument.

indexer:info

This command will show you the list of all indexes as a code-name used in the system. We can use index code as an argument for all the commands listed below.

Here is the execution example:

Magento 2 index

indexer:show-mode

This command will show you the indexer modes.

Magento 2

indexer:set-mode {realtime|schedule} [indexer]

This command can be used to change reindex mode. realtime sets Update on Save; schedule sets reindex mode to Update by Schedule. Using the indexer argument, you can change the mode of a specific index. Using no argument, the mode will be set for all indexes.

indexer:status

This command demonstrates the indexer status. It is one more way to define whether reindex is required or data is up-to-date and reindex is not required.

magento 2 index management

indexer:reindex

It is the most often used command that launches reindexing. If there are no arguments, all the indexes will be reindexed. If you use index code as an [index] argument, it will launch reindex only for this one. It is very convenient as it allows launching reindex only for the indexes that require it.

indexer:reset

Sometimes an error may occur during reindexing and the process cannot finish. For example, after executing reindex command, we come across this:

Magento 2

In order to fix it, we can pause the index with the reset command In this situation, you should execute the php bin/magento indexer:reset stock command. It will unblock the reindex process.

Index Status Tables in Database

The following tables in a database are used for index status storage:

  • indexer_state
  • mview_state

Mechanism of indexation by schedule, Mview

When there are any changes in the object of a database that the Magento 2 system has indexes for, a new trigger appears in the table with the _cl postfix if there is such a table. If there is no such table, a new one is generated. The mview.xml files are responsible for defining the data which changes will be tracked in the _cl (change log) table.

For example, Magento/Catalog/etc/mview.xml


From this extract, you can see that the catalog_category_product_cl will be created. It will contain the information about the changes in catalog_category_entity and catalog_category_entity_int tables.

All the change log tables are created by a certain INDEXER_TABLE_NAME + ‘_cl’ rule. The tables contain the version_id columns as autoincrement and entity_id that contains the object identifier that needs to be reindexed. For every table listed in the table nodes, three types of MYSQL AFTER triggers are created: INSERT, UPDATE, DELETE.

For example, for the catalog_category_entity table, the following triggers will be created:


The Mview component checks version_id in the *_cl tables if it is different from the one stored in the mview_state table. If the numbers are different, reindex launches.

Partial and complete indexation

Here we are going to discuss partial and complete indexation in Magento 2.

Partial indexation

Partial indexation updates indexes only in the objects that were modified. Partial indexation occurs both with Update on Save and Update by Schedule index modes. It is an indexation type where the change log tables are used that we described above.

Complete indexation

Complete indexation is required by some extensions in Magento 2 as some indexes may become invalid. Complete indexation launches by cron schedule or manually.

Magento index management FAQs

How create custom index

In order to do it, we will create a custom module. First of all, the registration.php file:

Igor Dragun

Partner With Us

Let's discuss how to grow your business. Get a Free Quote.
Talk to Igor


And the etc/module.xml file:


So, now we define our index. We need to create a etc/indexer.xml file for it:


In this file, we define a new indexer with attributes:

id is a new unique index identifier. It is necessary particularly for launching reindex via terminal.
view_id is an identifier in the mview.xml file.
class possesses certain processing methods (executeFull, executeList, executeRow)

Simple indexer should contain the following elements:

title of an index that will be displayed in index grid
description of an index that will be displayed in index grid

Then, we create a etc/mview.xml file. It is necessary to detect changes in table objects and turn on indexers.


In this file, we define the identifier of our mview and class where the execute() method will launch if there are any data changes. In the subscriptions node, we define the tables that contain the table name and column which value will be transferred to the execute() method if the value is changed. So when the value of entity_id will be transferred to the Belvg\CronIndexer\Model\Indexer\Example::execute() method if the values are changed in the catalog_product_entity table.

In the indexer.xml and mview.xml files, we have defined the Belvg\CronIndexer\Model\Indexer\Example class where the methods for changing data processing should be. Now, we need to create Model/Indexer/Example.php


 

How to make Magento 2 index automatically at a set time?

So, we need to change the indexing frequency through cron. By default, indexation is performed every minute. You can see it for yourself by opening the file vendor / magento / module-indexer / etc / crontab.xml.


We are primarily interested in the <job name = “indexer_reindex_all_invalid”> tag. Five asterisks indicate that all invalid indexes are updated every minute. Asterisks  are given according to this table:

* * * * * command to be executed
| | | | |
| | | | +—– Day of week (0 – 7) (Sunday=0 or 7)
| | | +——- Month (1 – 12)
| | +——— Day of the month (1 – 31)
| +———– Hour (0 – 23)
+————- Minute (0 – 59)

You can also set the numerical values, with the asterisk simply meaning “every minute (hour, day, etc.)”. Therefore, if you want to update crown indexes only once a day at a certain time, you can write, for example, this: 0 4 * * *. This entry will indicate: 0 minutes – 4 hours – every day (4 am every day).

But we cannot make changes to the files that the vendor folder contains. For this you will need to create your own module. We can use the module created before Belvg_CronIndexer.

And now in the etc folder we will create a crontab.xml file with the following contents:


So this is how we redefined the time; now the reindexing of invalid indexes will be performed not every minute, but once a day.

This is all to the topic of index management in Magento 2. We hope this article was helpful to you. If you have any comments or questions, please leave them down below.

Igor Dragun

Partner With Us

Looking for a partner to grow your business? We are the right company to bring your webstore to success. Talk to Igor

3 Comments

  1. Really Helpful and very knowledgeable, Thanks. Please share your other blogs as well on M2.

  2. Thanks for this guide, this should be a default option in Magento. To many actions will cause a full reindex during the day.

  3. Really nice solution about Run cron job at particular time.
    I was looking for something like this how can we override core module crontab.xml file.
    This worked
    Thanks.

Post a new comment

BelVG Newsletter
Subscribe to our mailing list and get interesting stuff and updates to your email inbox.
Email *