Skip to main content

​
What is Anomaly Detection?

Corelayer continuously monitors your data for anomalies using statistical baselines. When data deviates from expected patterns, Corelayer flags it as an anomaly with a severity level and provides context to help you debug the issue. There are two ways to set up anomaly detection:
  • Table Monitoring — Connect a database integration and monitor tables directly. Corelayer tracks row volume, column values, and schema changes automatically.
  • SDK Metrics — Instrument your code with the Corelayer SDK to track custom metrics from any data pipeline, ETL job, or application.

​
How It Works

​
1. Configure a Rule

Select a database table or codebase and configure what to monitor. Choose between volume rules, column rules, and schema detection.

​
2. Baseline Collection

Corelayer collects data to build a statistical baseline for each partition. During this phase, the rule shows a Collecting status. No anomalies are reported until the baseline is ready.

​
3. Anomaly Detection

Once the baseline is ready, Corelayer compares new data against expected ranges. Values outside the threshold (calculated using mean, standard deviation, and a k-sigma multiplier) are flagged as anomalies.

​
4. Review Findings

Anomalies appear on the detail page with severity levels (Critical, High, Medium, Low, Info). Each finding shows the observed value, expected value, and the threshold bounds.

​
Key Concepts

  • Partition — A logical grouping of data. For time-series data, this is a time bucket. For categorical data, this is a unique combination of partition column values.
  • Volume Rule — Monitors the row count per partition. Detects unexpected spikes or drops in data volume.
  • Column Rule — Monitors the values of specific numerical columns. Detects shifts in column distributions.
  • Schema Rule — Detects unexpected changes to table schema (added, removed, or modified columns).
  • Baseline — The statistical model (mean, standard deviation, data points) built from historical data. Used to calculate expected ranges.
  • Frequency — How often data is expected to arrive (hourly, daily, weekly, monthly). Helps Corelayer calibrate detection sensitivity.

​
Getting Started

Table Monitoring

Monitor database tables for volume, column, and schema anomalies.

SDK Metrics

Track custom metrics from your data pipelines using the Corelayer SDK.
Need help? Contact support for assistance with anomaly detection.