Skip to main content
Skip to main content
Edit this page

ReplacingMergeTree

The engine differs from MergeTree in that it removes duplicate entries with the same sorting key value (ORDER BY table section, not PRIMARY KEY).

Data deduplication occurs only during a merge. Merging occurs in the background at an unknown time, so you can't plan for it. Some of the data may remain unprocessed. Although you can run an unscheduled merge using the OPTIMIZE query, do not count on using it, because the OPTIMIZE query will read and write a large amount of data.

Thus, ReplacingMergeTree is suitable for clearing out duplicate data in the background in order to save space, but it does not guarantee the absence of duplicates.

Note

A detailed guide on ReplacingMergeTree, including best practices and how to optimize performance, is available here.

Creating a Table​

For a description of request parameters, see statement description.

Note

Uniqueness of rows is determined by the ORDER BY table section, not PRIMARY KEY.

ReplacingMergeTree Parameters​

ver​

ver — column with the version number. Type UInt*, Date, DateTime or DateTime64. Optional parameter.

When merging, ReplacingMergeTree from all the rows with the same sorting key leaves only one:

  • The last in the selection, if ver not set. A selection is a set of rows in a set of parts participating in the merge. The most recently created part (the last insert) will be the last one in the selection. Thus, after deduplication, the very last row from the most recent insert will remain for each unique sorting key.
  • With the maximum version, if ver specified. If ver is the same for several rows, then it will use "if ver is not specified" rule for them, i.e. the most recent inserted row will remain.

Example:

is_deleted​

is_deleted — Name of a column used during a merge to determine whether the data in this row represents the state or is to be deleted; 1 is a "deleted" row, 0 is a "state" row.

Column data type — UInt8.

Note

is_deleted can only be enabled when ver is used.

No matter the operation on the data, the version should be increased. If two inserted rows have the same version number, the last inserted row is kept.

By default, ClickHouse will keep the last row for a key even if that row is a delete row. This is so that any future rows with lower versions can be safely inserted and the delete row will still be applied.

To permanently drop such delete rows, enable the table setting allow_experimental_replacing_merge_with_cleanup and either:

  1. Set the table settings enable_replacing_merge_with_cleanup_for_min_age_to_force_merge, min_age_to_force_merge_on_partition_only and min_age_to_force_merge_seconds. If all parts in a partition are older than min_age_to_force_merge_seconds, ClickHouse will merge them all into a single part and remove any delete rows.

  2. Manually run OPTIMIZE TABLE table [PARTITION partition | PARTITION ID 'partition_id'] FINAL CLEANUP.

Example:

Query clauses​

When creating a ReplacingMergeTree table the same clauses are required, as when creating a MergeTree table.

Deprecated Method for Creating a Table
Note

Do not use this method in new projects and, if possible, switch old projects to the method described above.

All of the parameters excepting ver have the same meaning as in MergeTree.

  • ver - column with the version. Optional parameter. For a description, see the text above.

Query time de-duplication & FINAL​

At merge time, the ReplacingMergeTree identifies duplicate rows, using the values of the ORDER BY columns (used to create the table) as a unique identifier, and retains only the highest version. This, however, offers eventual correctness only - it does not guarantee rows will be deduplicated, and you should not rely on it. Queries can, therefore, produce incorrect answers due to update and delete rows being considered in queries.

To obtain correct answers, users will need to complement background merges with query time deduplication and deletion removal. This can be achieved using the FINAL operator. For example, consider the following example:

Querying without FINAL produces an incorrect count (exact result will vary depending on merges):

Adding final produces a correct result:

For further details on FINAL, including how to optimize FINAL performance, we recommend reading our detailed guide on ReplacingMergeTree.