Anomaly Detection for Data Pipelines

​

What is Anomaly Detection?

Corelayer runs anomaly rules against your data. Each rule produces a stream of numeric observations, learns a statistical baseline for every partition in that stream, and raises a finding when a new observation falls outside the expected band. There are two ways to feed observations into a rule:

• Table rules — Connect a database integration and attach a rule to a table. You describe what to watch in plain English, Corelayer compiles it to a read-only SQL query, and runs that query on a schedule.
• API metrics — Instrument your own code with the Corelayer SDK and push observations from any pipeline, ETL job, or service.

Both paths share the same baseline engine, the same detectors, and the same findings UI.

​

How It Works

​

1. Create a rule

Go to Anomaly Rules in the sidebar and click Add. Pick a database table or a codebase, then configure the rule. See Table Monitoring and API Metrics for the specifics of each.

​

2. Set the cadence and detector

Every rule has an expected cadence — hourly, daily, weekly, or monthly — which sets how observations are bucketed and, for table rules, how often the query runs. Every rule also has a detector: the statistical method that scores each observation. For API metrics you choose it yourself, and Baseline is the default. For table rules Corelayer derives it from Analyze Setup.

​

3. Build the baseline

A rule needs history before it can judge anything. Until a partition has enough observations, it shows as Collecting and reports nothing.

• Table rules can train instantly using Save & Backfill, which runs the rule over historical rows.
• API metrics collect forward from the first observation your SDK sends.

Once a partition has enough data points, it flips to Ready.

​

4. Detection

On each run, Corelayer compares the new observation against that partition’s expected band. Anything outside the band is recorded as a finding, with a severity scaled to how far past the boundary it landed.

​

5. Review and triage

Findings appear on the rule’s detail page with a severity (Critical, High, Medium, Low, Info), the observed value, the expected value, and the band that was breached. Findings are also escalated for automated triage, which can open a Corelayer issue — the finding then links to that issue and shows whether it is still open. Breaches that only marginally clear the boundary are recorded but not escalated, so near-misses do not generate issues.

​

Detectors

Pick the detector that matches the shape of your metric. Each detector keeps its own rolling window, so switching detectors on an existing rule retrains it from stored history.

​

Key Concepts

• Observation — One numeric value for one partition at one point in time. A rule run produces a batch of them.
• Partition — A logical slice of the stream, identified by its partition keys. Every partition gets its own independent baseline. Partitioning by region and pipeline gives you a separate baseline for each region-pipeline pair.
• Baseline — The statistical state learned from a partition’s recent observations, used to compute the expected band.
• Detector — The method that turns a baseline plus a new observation into a pass or a finding.
• Minimum data points — How many observations a partition must accumulate before the detector is allowed to flag anything. Leave it blank to use the detector’s default. Lower values alert sooner but are noisier.
• Cadence — How often observations are expected to arrive: hourly, daily, weekly, or monthly. It sets how observations are bucketed and, for table rules, how often the query runs.
• Finding — A recorded anomaly, with severity, observed value, expected value, and the breached bounds.

​

Getting Started

Need help? Contact support for assistance with anomaly detection.