afnix/infra
0
1
Fork 6
Code Issues 47 Pull requests 11 Projects 1 Releases Packages Wiki Activity

Spurious alerts on CI coredumps #325

New issue
Open
opened 2025-09-06 22:34:15 +00:00 by raito · 6 comments
raito commented 2025-09-06 22:34:15 +00:00
Owner
Copy link

Certain CI workloads will generate broken coredumps as part of their integration testing.

This appears in our alerting as spurious systemd units failing when such a coredump happen.

We should find a way to exclude an entire cgroup tree, e.g. the Nix build cgroup tree.

Certain CI workloads will generate broken coredumps as part of their integration testing. This appears in our alerting as spurious systemd units failing when such a coredump happen. We should find a way to exclude an entire cgroup tree, e.g. the Nix build cgroup tree.
delroth commented 2025-09-07 01:32:30 +00:00
Owner
Copy link

Is there maybe a way to just drop those coredumps on the floor instead of having them get processed by systemd-coredumpd? I'd prefer we isolate the CI workloads from their host as much as possible.

Is there maybe a way to just drop those coredumps on the floor instead of having them get processed by systemd-coredumpd? I'd prefer we isolate the CI workloads from their host as much as possible.
raito commented 2025-09-07 16:10:37 +00:00
Author
Owner
Copy link

I do not see an option in coredump.conf to achieve this.

I do not see an option in `coredump.conf` to achieve this.
delroth commented 2025-09-08 06:07:22 +00:00
Owner
Copy link

I was thinking e.g. new Lix option to RLIMIT_CORE=0.

I was thinking e.g. new Lix option to RLIMIT_CORE=0.
delroth commented 2025-09-08 06:09:01 +00:00
Owner
Copy link

Wait, actually looking into this RLIMIT_CORE=0 should be the default unless we explicitly --option enable-core-dumps true, what gives?

Wait, actually looking into this RLIMIT_CORE=0 should be the default unless we explicitly `--option enable-core-dumps true`, what gives?
raito commented 2025-09-08 13:03:42 +00:00
Author
Owner
Copy link

I think RLIMIT_CORE=0 has only effects if kernel.core_pattern is a filename and not a pipe, i.e. when the kernel writes the coredump, not when systemd processes it.

I think RLIMIT_CORE=0 has only effects if kernel.core_pattern is a filename and not a pipe, i.e. when the kernel writes the coredump, not when systemd processes it.
raito commented 2025-09-08 17:13:39 +00:00
Author
Owner
Copy link

I noticed [6736486.194464] coredump: 3234(liblixmain-test): RLIMIT_CORE is set to 1, aborting core in my EPYC logs BTW.

I noticed `[6736486.194464] coredump: 3234(liblixmain-test): RLIMIT_CORE is set to 1, aborting core` in my EPYC logs BTW.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
floral-wiki
buildkite-lzbt
consume-machine-identity-secrets
workloads
rewire-vault
renovate/mainlock-file-maintenance
staging
goaccess
alerting-to-zulip
better-backup-monitoring
forgejo-runners-sprinkles
abm
afnix-social-logins
hyp01
benchmarks
afnix-oidc
voidlinux-tenant
x86_builder_v6_only
besadii-release-branches
tom-hubrecht/audit
repair-ofborg
i-wish-i-had-jj-and-gerrit
scheduled-builds
gerrit-hydra
single-node-ceph
storage-project
cache-lix
email-groups
format
No results found.
Labels
Clear labels   
Compat/Breaking

Breaking change that won't be backward compatible

  
Difficulty/Architectural

Requires changes in multiple pieces of the stack: the Nix interpreter, etc.

  
Difficulty/Easy

Infrastructure beginner friendly tasks

  
Difficulty/Hard

Requires prior knowledge, research and design

  
Help Wanted

Looking for a first task to pick? This one is looking for somefew to do it as well!

  
Kind/Bug

Something is not working

  
Kind/Documentation

Documentation changes

  
Kind/Enhancement

Improve existing functionality

  
Kind/Feature

New functionality

  
Kind/Testing

Issue or pull request related to testing

  
Priority/Critical

The priority is critical

  
Priority/High

The priority is high

  
Priority/Low

The priority is low

  
Priority/Medium

The priority is medium

  
Reviewed/Confirmed

Issue has been confirmed

  
Reviewed/Duplicate

This issue or pull request already exists

  
Reviewed/Invalid

Invalid issue

  
Reviewed/Won't Fix

This issue won't be fixed

  
Security

This is a security issue

  
Silenced Alert

This issue was created to track a silenced alert which needs remediation work.

  
Status/Abandoned

Somebody has started to work on this but abandoned work

  
Status/Blocked

Something is blocking this issue or pull request

  
Status/Need More Info

Feedback is required to reproduce issue or to continue work

  
Status/Postponed

We can do this later

  
Tracking Issue

Umbrella issue of many things at the same time

No labels
Compat/Breaking
Difficulty/Architectural
Difficulty/Easy
Difficulty/Hard
Help Wanted
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Testing
Priority/Critical
Priority/High
Priority/Low
Priority/Medium
Reviewed/Confirmed
Reviewed/Duplicate
Reviewed/Invalid
Reviewed/Won't Fix
Security
Silenced Alert
Status/Abandoned
Status/Blocked
Status/Need More Info
Status/Postponed
Tracking Issue
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
afnix/infra#325
No description provided.