What's an anomaly?

In statistical terms, abnormal values are those that significantly deviate from other observations in the data concentration. The abnormalities may result from measurement errors, experiments errors or represent real but very few extremes. Identifying and deciding how to deal with abnormal values (delegate, amend or retain) is a central step in data cleansing.

Common test algorithm description

IQR (quadratic distance): No assumption of data distribution is required. It calculates the difference (i.e. IQR) between Q1 (first Quadrant) and Q3 (third Quadrant). Usually, any less than Q1 - 1.5*IQR Or greater than Q3 + 1.5*IQR is considered an anomaly.
Z-Score (Standard Score): Assume that data match the normal distribution. It measures the standard multiple of deviations from the average for each point. When Z-Score has an absolute value greater than 2 or 3, the tag is usually abnormal.

Best practices

When abnormalities are detected, they should not be deleted directly and arbitrarily. It is recommended that the entry errors be first checked for source data; If the data are true, the abnormal value needs to be assessed for the presence of important business information (e.g., unusual transaction identification in financial controls).