Protections — cooldown, backoff, limits

Protections: cooldown 360min, max executions/day 3, pending check, failure-backoff skip after 5 fails/24h, circuit breaker auto-pause (status paused) above 50% errors. Budget-cap fields are configurable; enforcement is coming.

Written By Salvatore Sinigaglia

Last updated About 1 hour ago

Protections: cooldown 360min, max executions/day 3, pending check, failure-backoff skip after 5 fails/24h, circuit breaker auto-pause (status paused) above 50% errors. Budget-cap fields are configurable; enforcement is coming.

Protections — cooldown, backoff, limits

Wevion enforces protections to prevent rules from causing harm: cooldown (per-entity time-between-acts), max executions per day, a pending-action check, a failure-backoff skip, and a circuit breaker that auto-pauses misbehaving rules (setting status back to paused). Budget-change-limit and budget-daily-cap fields are also configurable on a rule; their enforcement is coming (see the note below).

Who is this for

Anyone designing rules that change live spend. Protections are how you sleep at night.

Why protections matter

A rule without protections can:

  • Pause the same adset repeatedly as transient metric blips trigger conditions
  • Increase budget 10× in one day chasing a noisy ROAS spike
  • Cascade errors when the platform API is down, locking the rule in failure mode

Each protection guards a specific failure mode.

Protection 1: cooldown_minutes

After acting on an entity, the rule won't act on that entity again for cooldown_minutes.

  • Default: 360 (6 hours)
  • Minimum for destructive actions (pause, decrease_budget_pct, relaunch): 60 minutes
  • Per-entity scope: cooldown is per-rule-per-entity; the same rule can still act on OTHER entities during cooldown

Use case: prevents thrashing when a metric oscillates around the threshold. Adset's CPA crosses 30 → pause. 5 minutes later metric refreshes and shows 29.50 → without cooldown, would activate. With cooldown, action holds for 6 hours.

Protection 2: max_executions_per_day

Hard cap on total executions across all entities, per rule, per UTC day.

  • Default: 3
  • Hard cap: 50 (cannot raise above this)

Use case: a scale-winner rule capped at 3/day prevents runaway scaling. Once the rule reaches its daily cap it stops evaluating for the rest of the UTC day (this is a rule-level stop, not a per-entity skip reason).

budget_change_limit_pct and budget_daily_cap (configurable, enforcement coming)

The rule lets you configure two budget-cap fields — budget_change_limit_pct (a single-action % cap) and budget_daily_cap (a cumulative per-entity per-day $ cap). These values are saved on the rule.

Today these fields are configurable but not yet enforced at execution time — the evaluator and execution worker do not currently read them to block a budget action. Enforcement is on the roadmap. Until it ships, do not rely on these caps as a hard safety net: bound your scaling with conservative pct values on the budget action plus max_executions_per_day.

Protection: pending check (30 min)

Before executing, the worker checks: is there already a pending action for this rule + entity in the last 30 minutes? If yes, skip + record reason.

Prevents duplicate actions when:

  • Two consecutive cron ticks both enqueue the rule (rare edge case)
  • The previous action is still in flight (SQS retry, slow platform API)

Not user-configurable; always active.

Circuit breaker (auto-pause)

If a rule's error rate goes critically high, it auto-pauses by setting the rule's status back to paused (there is no separate error status).

Trigger conditions (both must hold):

  • Error rate above 50% over the last settled execution
  • At least 20 total errors in that execution

When tripped:

  • Rule status flips to paused
  • auto_paused_at timestamp set
  • auto_pause_reason field captures cause
  • cb_reset_after timestamp marks when auto-resume eligibility starts

To recover: review last_error, fix the upstream issue (e.g. token expired, platform API change), then manually toggle status back to active. Or wait until cb_reset_after for automatic resume eligibility.

Failure backoff (per-entity)

When an entity's action fails repeatedly, the evaluator stops acting on that entity for a while:

  • After 5 consecutive failures for an entity within a 24-hour window, the entity is skipped on subsequent evaluations with reason failure_backoff
  • This is per-entity, distinct from the circuit breaker (which pauses the whole rule)

Note: the SQS layer has its own delivery retries with a base backoff; the rule-level guard above is a skip-after-threshold, not an exponential per-attempt backoff.

Protection priority order

When multiple protections could apply, they're checked in this order:

  1. Rule status (not active → skip)
  2. Schedule (not due yet → skip)
  3. Daily cap (max_executions_per_day reached → rule stops for the day)
  4. Cooldown (within window → skip)
  5. Failure backoff (5+ consecutive failures for the entity in 24h → skip)
  6. Pending check (in-flight action → skip)
  7. Redundant action (entity already in the target state → skip)
  8. Action attempted

Each skip is recorded per entity with the specific reason (cooldown, failure_backoff, pending, no_tokens, redundant_action).

Visible in execution history

/rules/:id → execution timeline shows:

  • entities_skipped with a per-entity reason. Real reasons are: cooldown, failure_backoff, pending, no_tokens, redundant_action.
  • entities_errored with the last error message

Useful for tuning: if lots of entities skip for cooldown, the cooldown may be too conservative for how often the rule evaluates.

For new rules:

  • Cooldown: 360 min (6h) — default. Drop to 60 only for explicit reasons.
  • Max executions/day: 3 (default). Raise carefully.
  • Budget change limit / daily cap: you can configure these fields, but enforcement is not yet active (it's coming — see the section above) — do not rely on them as a hard safety net; use conservative pct values and max_executions_per_day instead.

Common mistakes

  • Cooldown too short on pause action: rule re-pauses + re-activates on metric noise. Default 360 min exists for this reason.
  • Relying on budget caps as a hard limit: use conservative pct and max_executions_per_day to bound scaling — the budget-cap fields are configurable but not yet enforced (see note above).
  • Resuming an auto-paused rule without fixing root cause: it immediately re-trips the circuit breaker (which sets status back to paused). Review last_error first.
  • Setting max_executions_per_day: 50: hard cap; rule can fire 50 times in a day. Almost always overkill — start at 3.

FAQ

What protections does Wevion apply to rules?

Wevion enforces cooldown_minutes (per-entity time between actions), max_executions_per_day, a 30-minute pending-action check, and a failure-backoff skip (5+ consecutive failures for an entity in 24h). A circuit breaker auto-pauses misbehaving rules by setting status back to paused. The rule also lets you configure budget_change_limit_pct and budget_daily_cap fields, but these are not yet enforced at execution time (enforcement is coming) — don't treat them as a hard cap yet.

What is the default cooldown for a Wevion rule?

The default cooldown_minutes is 360 (6 hours), with a 60-minute minimum for destructive actions like pause, decrease_budget_pct, and relaunch. Cooldown is per-rule-per-entity, so the same rule can still act on other entities during the window. It prevents thrashing when a metric oscillates around the threshold.

When does Wevion's circuit breaker auto-pause a rule?

The circuit breaker trips when both conditions hold: an error rate above 50% on the last settled execution and at least 20 total errors in that execution. The rule's status is set back to paused (there is no separate error status), and it records auto_paused_at, auto_pause_reason, and cb_reset_after. Recover by fixing the root cause, then toggle the status back to active.

How many times can a rule act per day?

max_executions_per_day defaults to 3 and cannot be raised above the hard cap of 50. It caps total executions across all entities per rule per UTC day. Once the cap is reached the rule stops evaluating for the rest of the day (a rule-level stop, not a per-entity skip reason). Start at 3 and raise carefully.

How do I see why a rule skipped an entity?

Wevion's execution history at /rules/:id shows entities_skipped broken down by reason. The real reasons are: cooldown, failure_backoff, pending, no_tokens, and redundant_action. entities_errored shows the last error message. If lots of entities skip for cooldown every cycle, the cooldown may be too conservative for the evaluation cadence.