Alerts and Self-Healing

Monitoring

Turns the telemetry each endpoint already reports into threshold and service checks that raise alerts through the same deduplicated, audited pipeline as everything else. It exists so a busy fleet is watched automatically — a sustained CPU/disk spike or a stopped service pages you instead of being noticed only when something breaks. For the tenant it means their machines are proactively monitored on their own thresholds, scoped to just the devices that matter, with no extra tooling to run.

Fleet utilisation: A live table of every endpoint’s CPU, memory and disk (as meters), load average, uptime, status and last-seen — straight from each heartbeat. The summary shows how many endpoints are reporting metrics, active breaches, and the fleet average CPU / memory / disk. Click a row to open that device; device.token.revoke shows an inline Remove.
Checks: Add check (needs rule-manage) opens a rule: a threshold check (e.g. CPU above 80% held for a number of seconds) or a "service stopped" check. A rule can be scoped to a device role, group and/or tags so a Windows-only or server-only check targets just those devices; leave the scope empty for tenant-wide. Only a sustained breach — not a momentary spike — raises an alert. Each rule row shows its condition, scope and for-seconds, and can be disabled or deleted.
Active breaches: Open incidents raised by these checks are listed at the top with their summary and age, so a current problem is visible before you scroll the rule list.
Watched services: A matrix of the last state each endpoint reported for every watched service. "—" means that endpoint has not reported it yet (the agent picks up the watch list on its next heartbeat; very old agents never will) — deliberately not shown as an error. To remediate a stopped service, open the device and use the Service action (start/stop/restart), which is allowlisted on the agent and second-operator approved.
Certificate expiry: TLS endpoints you name with "Watch endpoint" are handshaked every 6 hours and listed soonest-expiry first. Because an agent fails closed on TLS, an expired certificate takes endpoints offline — this warns you first, distinguishing "expired", "days left" and "unreachable" from "not checked yet". "Check now" runs one immediately; Delete stops watching it.

Alerts

Durable incidents raised by monitors, offline detection, sign-in monitoring and other rules — deduplicated, tenant-scoped, and routed through the notification pipeline — plus the rules and destinations that drive them. It exists so problems are captured once and chased to closure instead of scrolling past in a log, and so the right people are notified on the right channel. For the tenant it is their incident queue and their own alert rules and delivery targets.

Summary: Three counters head the view: Open incidents, Managed rules, and Delivery failures (incidents whose notification did not get through). A "🔔 Notification settings" button (top) opens your personal notification preferences.
Alert rules: Tenant-scoped rules, each a trigger event (e.g. device.offline, job.failed, file_transfer.failed, agent_token.revoke, agent_certificate.revoke, agent.version_change) with a cooldown that stops immediate reopening. "Add rule" creates one, Delete removes it (existing incidents are kept), and "Send test alert" raises a test sign-in incident so you can confirm the badge, dashboard and your channel.
Incidents: Each open incident stays deduplicated until acknowledged (resolving closes it); both are audited. Per row you can Acknowledge (stops re-notification), Retry delivery (re-runs the bounded webhook attempt) on a failed one, Isolate host (contain the machine the alert is about), and Hide/Unhide a noisy one. "Hide resolved" is on by default and remembered per browser; hidden incidents are never lost.
Notification destinations: Where alerts are pushed: phone push (ntfy), Slack/Teams, Discord, or a custom endpoint — each formatted for its channel with a minimum-severity floor. "Add destination" creates one; Test sends a sample, Delete stops delivery there. This is on top of the server’s configured signed webhook and any operators who turned on their own email alerts.

Self-Healing Rules

Log-driven auto-remediation. SentinelOps reads each online endpoint’s recent logs and, when a vetted rule matches a known problem, runs a safe, pre-approved fix through the ordinary job system and then verifies it worked — so recurring faults heal themselves instead of waiting for a human. Safety is the whole point: only vetted built-in rules auto-apply, and only genuinely safe, idempotent, reversible fixes; anything with outage potential ships monitor-only. The AI only suggests fixes for errors no rule covers — it never runs them. Every machine can opt out with a tag, a tenant on legal hold is frozen, and every single decision is recorded and auditable. This is a separate engine from App Self-Healing, which restores removed applications.

The safety model (read this first): Four layers protect you. (1) Only vetted BUILT-IN rules auto-apply, and the catalog is deliberately conservative — a fix that could cause an outage or touch data ships monitor-only until you deliberately enable it. (2) The AI never acts: it only proposes a rule for your approval. (3) Any machine tagged no-autoheal is skipped entirely (set it from the device drawer). (4) A tenant on legal hold has all auto-changes frozen for preservation. On top of that, every rule has a cooldown and a daily attempt cap so a flapping fault can never loop.
Global controls: Enabled is the master switch — off means nothing is detected, suggested, applied or emailed. Monitor-only mode makes the WHOLE engine detect-and-email but change nothing, which is the safe way to watch what would happen before letting fixes run. AI suggestions lets the assistant propose fixes for unknown recurring errors (they queue for approval, never run). Email reports sends a batched digest of activity and pending suggestions to your alert recipients (needs SMTP configured). These four are platform-wide, not per-organization, and only a platform-wide operator can change them.
Schedule & email (platform-wide): Under the engine switches, “Schedule & email” sets: how often machines are checked (default every 10 minutes, 5–1440) and how many machines per check (default 60 — raise it for large fleets); how often the digest is sent (default hourly) or sooner once N events queue (default 25); extra digest recipients, optionally REPLACING the alert recipients instead of adding to them; and which no-action observations get emailed — critical only (default), warning and above, everything, or never. Fixes applied, verified or failed are always emailed regardless. Leave a number blank for the default. Changes take effect on the next check with no restart.
Where each setting applies — organization, group, device: Pick an organization in the top bar, then choose the scope under “Per-rule overrides apply to”: the whole organization, one device group, or a single device. For that scope you can turn each rule off, make it observe-only, and set its cooldown and daily cap (blank = inherit — the greyed number shows what it inherits and from where). The most specific setting wins in both directions: device beats group beats organization beats the global rule, so a device can turn back ON a rule its group turned off, or use a shorter cooldown. Deleting a device, group or organization removes its overrides too.
Maintenance windows: Next to the scope picker, “Set window” restricts WHEN fixes may run for that organization, group or device — for example weekdays 22:00–06:00 in America/Los_Angeles (an end earlier than the start is an overnight window; the days are the days it starts). Outside the window a matched problem is recorded as “deferred” and fixed on the first check inside the window if it is still being logged; detection, observe-only and email carry on at any time. The most specific window wins, and “Fix any time” lets a device or group opt out of a broader window. Remove a window to follow the broader scope again.
Email per organization: With an organization selected, “Self-Healing email for …” sends that organization its own digest covering only its machines. A platform-wide operator can also take the organization out of the platform-wide digest (it then goes only to its own recipients) and give it its own severity threshold. Customer-scoped operators can manage their organization’s recipients but not those two settings.
Rules: Each rule watches a log regex on a chosen OS/role scope and, when it matches enough times in its window, runs its fix. Toggle Enabled and Monitor-only right in the list. Built-in rules carry a badge and are maintained by SentinelOps — a platform-wide operator can tune their toggles, severity, cooldown and daily cap (Tune), and they cannot be deleted (disable instead). Severity decides whether the rule’s no-action observations reach the email digest. Your own rules get a full editor and a delete. Use + Rule to add one: name it, give a regex pattern, scope it by OS/role, set the trigger count/window, choose the action (restart a named service, or run a script/command, or just suppress a benign alert), add a verify step, and set the guardrails. The action editor warns you that it runs on real machines — leave a new rule monitor-only until you have watched it behave.
AI suggestions: When AI suggestions is on, the assistant looks for a recurring error that no rule covers and proposes ONE safe fix, with the sample log lines and its rationale. It never runs anything. Approve turns the suggestion into an operator-owned rule that starts monitor-only — so it records what it WOULD do without acting — and you turn monitor-only off once the activity log convinces you it is safe. Dismiss discards it.
Activity log: Every decision, newest first: detected (matched, nothing done yet), monitor-only (would have acted but monitor-only is on), applied (fix dispatched as a job), verified (fix confirmed to have worked), failed (job or verification failed), suppressed (a benign/known-noise match, no alert, no action), deferred (a fix is waiting for the machine’s maintenance window), recurring (the same fix was needed on this machine at least the recurring threshold — default 3 — times in 24 hours: it works each time but the cause keeps coming back, so investigate). When a fix is verified, Self-Healing also closes the matching log alerts on that machine and, unless turned off under Schedule & email, emails a “Resolved by Self-Healing” notice right away. The trigger column shows the matched log line with secrets redacted.
Per-machine opt-out: On any device’s drawer there is an Auto-heal control. Opt a machine out and it gets the no-autoheal tag — the engine then skips it entirely (no detection, no fixes) while still healing the rest of the fleet. Turn it back on to remove the tag. This needs enrollment.manage (the same permission that sets device tags).

Self-Healing

SentinelOps watches the apps you care about on every managed endpoint and puts one back if it goes missing — so a removal (a bad uninstall, a failed patch, a CVE remediation that removed a broken app) does not leave a machine without it. It only ever restores an app the device actually had before, and the reinstall it runs is verify-or-rollback: if the fresh install fails it restores the previous copy, so a machine is never left with nothing.

Enable self-healing: When on, every managed endpoint’s installed-software inventory is checked on a schedule against the watched-apps list. If a watched app that was previously present disappears, a Self-healing alert is raised and (if Auto-restore is on) a reinstall is dispatched. Apps the device never had are never installed.
Auto-restore: Automatically reinstall a missing watched app when SentinelOps has a clean-reinstall recipe for it (Google Chrome today). The reinstall is verify-or-rollback: it downloads and size-checks the installer first, tries an in-place upgrade before any removal, backs up the existing install, and if a removal+reinstall ever fails it restores the backup — the endpoint is never left without the app. Watched apps with no recipe raise an alert to redeploy from the Software Library instead.
Watched apps: Comma-separated name patterns matched against installed-software names (e.g. “google chrome”). Default watchlist covers apps with an auto-restore recipe. Add your own to get missing-app alerts for them even without an automatic reinstall.
Recent self-heal activity: A log of what the guardian did: Missing (a watched app vanished), Auto-restore dispatched (a reinstall was queued), Recovered (the app came back), Blocked (could not auto-restore — e.g. installs are locked on the device, the device is offline, or no recipe exists). Blocked reasons tell you exactly what to fix.
When a device needs hands-on help: If an endpoint’s Windows Installer subsystem is itself damaged (msiexec runs but installs nothing), auto-restore is blocked and the reason is logged — that box needs a servicing-stack repair (DISM/SFC, msiexec re-register, or a reboot) before any install, SentinelOps or otherwise, can succeed.
Opt one machine out (stop the alerts for it): If a machine legitimately has a watched app come and go — for example Chrome that updates itself on a box whose installer can’t be repaired — the daily “managed app is missing” alert is just noise for that one machine. Open its device drawer and set App self-healing to “opt out” (this adds the no-selfheal tag). That machine is then skipped entirely by the guardian — no missing-app alerts and no auto-reinstalls — while the rest of the fleet keeps healing. Turn it back on from the same place any time. This is separate from Auto-heal (the log-driven rules engine, the no-autoheal tag) right above it in the drawer.

Next: Keep machines patched →