Teach besadii to react to gerrit comments (eg rerunning a build) #407

Open
opened 2026-04-14 12:03:19 +00:00 by blokyk · 4 comments

This was discussed in passing a few times on the lix matrix and other places, but it'd be nice to have besadii be able to relaunch a failed CI run, especially since it can be pretty flaky, and the only other way to force a rerun is to rebase or change the code in some insignificant way, which isn't great.

Would something like reacting to comments that contain @besadii rerun-build work?

Note that I know basically nothing about CI, so I have no idea about the limitations you might want to put on such an ability. It might be harder to implement if we need to check who's requesting a change, or if a certain quota has been reached, etc.

See the gerrit hook docs for the info we're passed for new comments.

This was discussed in passing a few times on the lix matrix and other places, but it'd be nice to have besadii be able to relaunch a failed CI run, especially since it can be pretty flaky, and the only other way to force a rerun is to rebase or change the code in some insignificant way, which isn't great. Would something like reacting to comments that _contain_ `@besadii rerun-build` work? Note that I know basically nothing about CI, so I have no idea about the limitations you might want to put on such an ability. It might be harder to implement if we need to check who's requesting a change, or if a certain quota has been reached, etc. See [the gerrit hook docs](https://gerrit-documentation.storage.googleapis.com/Documentation/2.11/config-hooks.html#_comment_added) for the info we're passed for new comments.
Owner

Dumping some state here: I've heard second hand that there were vague complaints about a "retry via review comments" workflow - they make the review thread more spammy, notify people for unactionable events, etc. (I fully realize that "vague second hand" makes this a bit of a discussion stopper - unfortunately a lot of Lix discussions happen behind closed doors and I'm not privy to them...).

Ideally we'd have another flow to retry builds, presumably a web interface behind SSO with a button. Possibly implemented as a Gerrit plugin so we can piggy back on auth and so on?

But I think that if nobody picks that up within 1-2 months we should just go with the review comment solution which is trivial-ish to implement (like, < 1 day of work) and evaluate whether the pros sufficiently balance the cons. I believe they will, but tbf the "better" solution also isn't a ton of work.

Dumping some state here: I've heard second hand that there were vague complaints about a "retry via review comments" workflow - they make the review thread more spammy, notify people for unactionable events, etc. (I fully realize that "vague second hand" makes this a bit of a discussion stopper - unfortunately a lot of Lix discussions happen behind closed doors and I'm not privy to them...). Ideally we'd have another flow to retry builds, presumably a web interface behind SSO with a button. Possibly implemented as a Gerrit plugin so we can piggy back on auth and so on? But I think that if nobody picks that up within 1-2 months we should just go with the review comment solution which is trivial-ish to implement (like, < 1 day of work) and evaluate whether the pros sufficiently balance the cons. I believe they will, but tbf the "better" solution also isn't a ton of work.
Author

looking at the gerrit plugin docs as well as existing plugins and the API docs, i think this is slightly above what i can do as a side-quest currently (mostly because of every "extra" task: testing it, giving it a way to be configured, packaging it, making it work with nix-gerrit, etc.), so if that's the direction this is going (which i do think is a better idea than comments), i can't help right now (so if someone wants to pick up, that'd be great ;)

looking at [the gerrit plugin docs](https://gerrit-review.googlesource.com/Documentation/dev-plugins.html) as well as [existing plugins](https://gerrit.googlesource.com/plugins/cookbook-plugin/+/refs/heads/master/src/main/java/com/googlesource/gerrit/plugins/cookbook/client) and [the API docs](https://javadoc.io/doc/com.google.gerrit/gerrit-plugin-api/latest/index.html), i think this is slightly above what i can do as a side-quest currently (mostly because of every "extra" task: testing it, giving it a way to be configured, packaging it, making it work with `nix-gerrit`, etc.), so if that's the direction this is going (which i _do_ think is a better idea than comments), i can't help right now (so if someone wants to pick up, that'd be great ;)
Author

update: i couldn't help looking more into it, and apparently there's already a whole bunch of infrastucture for this built into gerrit... which besadii already kinda uses, but only for displaying test results. we could probably extend it to also support sending jobs to buildkite from gerrit (without a separate listener like besadii), and have a rerun button, like the official instance:

a list of successful tasks with a "rerun" button at the end of each tile

now i just need to understand how you actually get that button to appear...

update: i couldn't help looking more into it, and apparently there's already [a whole bunch of infrastucture for this](https://gerrit-review.googlesource.com/Documentation/pg-plugin-checks-api.html) built into gerrit... which besadii [already kinda uses](https://git.afnix.fr/afnix/infra/src/commit/5e61cba283d148ed94e44c1588a4543893bb31d5/overlays/gerrit-plugins/buildkite-checks-afnix/buildkite-checks.js), but only for displaying test results. we could probably extend it to also support sending jobs to buildkite from gerrit (without a separate listener like besadii), and have a rerun button, like the official instance: ![a list of successful tasks with a "rerun" button at the end of each tile](/attachments/c8d40fcf-9e76-4078-b006-bf15956243f2) now i just need to understand _how_ you actually get that button to appear...
Author

to continue this saga: looking around a bit more (because another job of mine broke and i got frustrated), it seems like all we need to really add is an actions array to this constructor, an instance of which you can see in the wild in this random repo. according to the buildkite docs, we need to make a (write_builds authenticated) PUT call to /v2/organizations/afnix/lix/jobs/{job.id}/retry (but it has the caveat that it can only be called once, and retrying it anymore requires keeping state from each call to the next, but i guess we can add support for that later).

i don't think there's anything more complex than that to put in place here, but i just really don't have the infra knowledge to set it up properly or even just hack/test it, so it'll have to be done by someone else. hopefully i've been helpful enough to the person who'll implement this :)

to continue this saga: looking around a bit more (because another job of mine broke and i got frustrated), it seems like all we need to really add is an `actions` array to [this constructor](https://git.afnix.fr/afnix/infra/src/commit/5e61cba283d148ed94e44c1588a4543893bb31d5/overlays/gerrit-plugins/buildkite-checks-afnix/buildkite-checks.js#L127-L136), an instance of which you can see in the wild [in this random repo](https://github.com/jenkinsci/gerrit-checks-api-plugin/blob/e0ad91c77a92f1a478616a2ea42c8d1e09fe108f/README.md?plain=1#L67-L85). according to [the buildkite docs](https://buildkite.com/docs/apis/rest-api/jobs#retry-a-job), we need to make a (`write_builds` authenticated) PUT call to `/v2/organizations/afnix/lix/jobs/{job.id}/retry` (but it has the caveat that [it can only be called once](https://buildkite.com/docs/apis/rest-api/jobs#retry-a-job:~:text=You%20can%20only%20retry%20each%20job%2Eid%20once%2E%20To%20retry%20a%20%22second%20time%22%20use%20the%20new%20job%2Eid%20returned%20in%20the%20first%20retry%20query), and retrying it anymore requires keeping state from each call to the next, but i guess we can add support for that later). i don't think there's anything more complex than that to put in place here, but i just really don't have the infra knowledge to set it up properly or even just hack/test it, so it'll have to be done by someone else. hopefully i've been helpful enough to the person who'll implement this :)
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
afnix/infra#407
No description provided.