afnix/infra
0
1
Fork 6
Code Issues 47 Pull requests 11 Projects 1 Releases Packages Wiki Activity

Alert on flapping / restart-looping systemd units #370

New issue
Open
opened 2025-10-05 01:36:52 +00:00 by delroth · 0 comments
delroth commented 2025-10-05 01:36:52 +00:00
Owner
Copy link

We recently had a situation on lix-zulip01 where some of the zulip queue runners were restart-looping at a high rate. Since they were never actually in the failed state we never saw this in our alerting until further symptoms ended up causing problems. I don't know off-hand how we could monitor for this, but this is something we should really have imo.

We recently had a situation on `lix-zulip01` where some of the zulip queue runners were restart-looping at a high rate. Since they were never actually in the `failed` state we never saw this in our alerting until further symptoms ended up causing problems. I don't know off-hand how we could monitor for this, but this is something we should really have imo.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
floral-wiki
buildkite-lzbt
consume-machine-identity-secrets
workloads
rewire-vault
renovate/mainlock-file-maintenance
staging
goaccess
alerting-to-zulip
better-backup-monitoring
forgejo-runners-sprinkles
abm
afnix-social-logins
hyp01
benchmarks
afnix-oidc
voidlinux-tenant
x86_builder_v6_only
besadii-release-branches
tom-hubrecht/audit
repair-ofborg
i-wish-i-had-jj-and-gerrit
scheduled-builds
gerrit-hydra
single-node-ceph
storage-project
cache-lix
email-groups
format
No results found.
Labels
Clear labels   
Compat/Breaking

Breaking change that won't be backward compatible

  
Difficulty/Architectural

Requires changes in multiple pieces of the stack: the Nix interpreter, etc.

  
Difficulty/Easy

Infrastructure beginner friendly tasks

  
Difficulty/Hard

Requires prior knowledge, research and design

  
Help Wanted

Looking for a first task to pick? This one is looking for somefew to do it as well!

  
Kind/Bug

Something is not working

  
Kind/Documentation

Documentation changes

  
Kind/Enhancement

Improve existing functionality

  
Kind/Feature

New functionality

  
Kind/Testing

Issue or pull request related to testing

  
Priority/Critical

The priority is critical

  
Priority/High

The priority is high

  
Priority/Low

The priority is low

  
Priority/Medium

The priority is medium

  
Reviewed/Confirmed

Issue has been confirmed

  
Reviewed/Duplicate

This issue or pull request already exists

  
Reviewed/Invalid

Invalid issue

  
Reviewed/Won't Fix

This issue won't be fixed

  
Security

This is a security issue

  
Silenced Alert

This issue was created to track a silenced alert which needs remediation work.

  
Status/Abandoned

Somebody has started to work on this but abandoned work

  
Status/Blocked

Something is blocking this issue or pull request

  
Status/Need More Info

Feedback is required to reproduce issue or to continue work

  
Status/Postponed

We can do this later

  
Tracking Issue

Umbrella issue of many things at the same time

No labels
Compat/Breaking
Difficulty/Architectural
Difficulty/Easy
Difficulty/Hard
Help Wanted
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Testing
Priority/Critical
Priority/High
Priority/Low
Priority/Medium
Reviewed/Confirmed
Reviewed/Duplicate
Reviewed/Invalid
Reviewed/Won't Fix
Security
Silenced Alert
Status/Abandoned
Status/Blocked
Status/Need More Info
Status/Postponed
Tracking Issue
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
afnix/infra#370
No description provided.