Unbalanced prioritization in the images buildbot? (main branch deprioritized too much)

[Petr Štetiar]

Thibaut <hacks at slashdirt.org> [2023-11-13 22:20:28]:

Hi,

> GitPoller accepts a single poll interval. What you’re suggesting would
> require separating each branch, i.e. returning to the previous situation.

IIUC then you can have multiple GitPoller instances, so wouldn't something
along this lines work?

	diff --git a/phase1/master.cfg b/phase1/master.cfg
	index 6f6c650cf54c..6cbd0c8d9609 100644
	--- a/phase1/master.cfg
	+++ b/phase1/master.cfg
	@@ -341,15 +341,16 @@ populateTargets()
	 # about source code changes.

	 c["change_source"] = []
	-c["change_source"].append(
	-    GitPoller(
	-        repo_url,
	-        workdir=work_dir + "/work.git",
	-        branches=branchNames,
	-        pollAtLaunch=True,
	-        pollinterval=300,
	+for branch in branchNames:
	+    c["change_source"].append(
	+        GitPoller(
	+            repo_url,
	+            branch=branch,
	+            workdir=work_dir + "/work.git",
	+            pollAtLaunch=True,
	+            pollinterval=pollinterval_for_branch(branch),
	+        )
	     )
	-)

> Also note that even with a 24h poll interval you could still starve the master builds.

Sure, but meanwhile it would still allow to build quite a bunch of the master
targets, so improving current situation.

> or we switch to a periodic build schedule

What about handling of security fixes in this once a week periodic build
proposal?

Going forward with this approach, we would still probably need some custom
scheduler which would be able to skim through the commit content and trigger
build when certain patterns are found, like a word 'security' or CVE number?

> This would have the added benefit of ensuring *consistency*
> across targets, i.e. ensuring that whichever commit is built is built on
> *all* targets.

I'm probably missing something, but what is this consistency good for?

I would actually prefer to select say top X targets (ideally have them in
testbeds for automatic runtime testing) and increase build priorities for
those in phase1/phase2 builds, making sure, that we provide builds to actual
users/testers first and build the rest afterwards.

Those targets would be clearly marked as such in the source tree, for example:

 FEATURES+=ci-tier1

 (or introduce some new variable?)

we would need to define and document this selection process of course as well :)

Cheers,

Petr