Discussion:
Disposable Jenkins & GitOps do not make a good match
domi
2018-11-30 13:51:04 UTC
Permalink
Hi,

Since we now have JCasC and the job-dsl plugin it is easier then ever to treat jenkins as a disposable resource and just recreate it from scratch whenever you feel the need for it.
We do have build everything around this idea and reinstall our whole build infrastructure every time we need to update jenkins or even just a plugin (we create a docker image with all required plugins preinstalled).

But there are two issues which hurt us quite a bit:

1. Pipeline jobs do not listen on any commit webhooks before the job was not executed at least once (in our case Bitbucket, but the same is true for Github)
2. A multi branch pipeline (organizationFolder, *-branch-source-plugin) job will trigger a build for every single branch just discovered. This is quite a problem in case you have build your processes around git (some call it GitOps). e.g. if every time your master branch gets build something special happens: build a new release version of a library or even install something to some environment.

These two issues together cause us quite some problem, because every time we setup a new environment the following happens:

The branch source plugin (in our case Bitbucket) scans the whole organisation for branches, will trigger builds and therefore: new releases of libraries and artefacts get build and some stuff gets installed because the builds for many of our ‘master’ branches are triggered too.
Then I somehow have to trigger a build for all other pipelines (not managed by the branch-source-plugin) which should listen on any git webhooks, because otherwise they will never be triggered


Does anyone have the same experience? How do you work around them?

I have some ideas on how to solve/ workaround it:
- problem 1: write a plugin which is able to abort the build in case it is the first run and the job is configured to listen on webhooks (e.g. Bitbucket or GitHub), then after installing the fresh environment, just loop through all jobs (with groovy) and trigger all which have a hook configured and let the first run fail on purpose.
- problem 2: extends https://github.com/jenkinsci/basic-branch-build-strategies-plugin to some how skip some branches at the initial organisation scanning, I already created an issue for this: https://issues.jenkins-ci.org/browse/JENKINS-54861

But to be very honest, most of this sound some how hackisch and I hope someone else has a better idea on how to do this stuff properly.

Many thanks
Domi
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/FA18D443-D4CA-4512-BAD1-255D6E3385C8%40fortysix.ch.
For more options, visit https://groups.google.com/d/optout.
James Strachan
2018-11-30 14:02:16 UTC
Permalink
we solved this in Jenkins X through the use of Prow for handling webhooks &
orchestrating the serverless jenkins pod and then using kubernetes custom
resources to maintain state (pipeline activities, environments, teams,
releases etc)
https://jenkins-x.io/news/serverless-jenkins/

we then use an immutable docker image of jenkins that starts up (thanks to
the custom war packager) and processes each pipeline execution in a
separate container.
Post by domi
Hi,
Since we now have JCasC and the job-dsl plugin it is easier then ever to
treat jenkins as a disposable resource and just recreate it from scratch
whenever you feel the need for it.
We do have build everything around this idea and reinstall our whole build
infrastructure every time we need to update jenkins or even just a plugin
(we create a docker image with all required plugins preinstalled).
1. Pipeline jobs do not listen on any commit webhooks before the job was
not executed at least once (in our case Bitbucket, but the same is true for
Github)
2. A multi branch pipeline (organizationFolder, *-branch-source-plugin)
job will trigger a build for every single branch just discovered. This is
quite a problem in case you have build your processes around git (some call
it GitOps). e.g. if every time your master branch gets build something
special happens: build a new release version of a library or even install
something to some environment.
These two issues together cause us quite some problem, because every time
The branch source plugin (in our case Bitbucket) scans the whole
organisation for branches, will trigger builds and therefore: new releases
of libraries and artefacts get build and some stuff gets installed because
the builds for many of our ‘master’ branches are triggered too.
Then I somehow have to trigger a build for all other pipelines (not
managed by the branch-source-plugin) which should listen on any git
webhooks, because otherwise they will never be triggered
Does anyone have the same experience? How do you work around them?
- problem 1: write a plugin which is able to abort the build in case it is
the first run and the job is configured to listen on webhooks (e.g.
Bitbucket or GitHub), then after installing the fresh environment, just
loop through all jobs (with groovy) and trigger all which have a hook
configured and let the first run fail on purpose.
- problem 2: extends
https://github.com/jenkinsci/basic-branch-build-strategies-plugin to some
how skip some branches at the initial organisation scanning, I already
https://issues.jenkins-ci.org/browse/JENKINS-54861
But to be very honest, most of this sound some how hackisch and I hope
someone else has a better idea on how to do this stuff properly.
Many thanks
Domi
--
You received this message because you are subscribed to the Google Groups
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-dev/FA18D443-D4CA-4512-BAD1-255D6E3385C8%40fortysix.ch
.
For more options, visit https://groups.google.com/d/optout.
--
James
-------
Twitter: @jstrachan
Blog: https://medium.com/@jstrachan/

Jenkins X: a CI/CD solution for modern cloud applications on Kubernetes
http://jenkins-x.io/
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CALvxejm05axySHoqVEGrvX%3DrCPj4a4EfK3Q39oqDLrMwjrT-hQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
domi
2018-12-03 08:11:10 UTC
Permalink
Thanks James, thats an interesting path, but it does look like we would have to build quite a lot around this to make it a good replacement of our current infrastructure. We also did have a look at jenkins-x, but to be honest, it is just way to much for us: first of all we don’t use k8s (we use cloud foundry) and I would say that running jenkins-x cost us more then our production environment, because of the number of pods (yes, we don’t have a micro service architecture)
/Domi
we solved this in Jenkins X through the use of Prow for handling webhooks & orchestrating the serverless jenkins pod and then using kubernetes custom resources to maintain state (pipeline activities, environments, teams, releases etc)
https://jenkins-x.io/news/serverless-jenkins/ <https://jenkins-x.io/news/serverless-jenkins/>
we then use an immutable docker image of jenkins that starts up (thanks to the custom war packager) and processes each pipeline execution in a separate container.
Hi,
Since we now have JCasC and the job-dsl plugin it is easier then ever to treat jenkins as a disposable resource and just recreate it from scratch whenever you feel the need for it.
We do have build everything around this idea and reinstall our whole build infrastructure every time we need to update jenkins or even just a plugin (we create a docker image with all required plugins preinstalled).
1. Pipeline jobs do not listen on any commit webhooks before the job was not executed at least once (in our case Bitbucket, but the same is true for Github)
2. A multi branch pipeline (organizationFolder, *-branch-source-plugin) job will trigger a build for every single branch just discovered. This is quite a problem in case you have build your processes around git (some call it GitOps). e.g. if every time your master branch gets build something special happens: build a new release version of a library or even install something to some environment.
The branch source plugin (in our case Bitbucket) scans the whole organisation for branches, will trigger builds and therefore: new releases of libraries and artefacts get build and some stuff gets installed because the builds for many of our ‘master’ branches are triggered too.
Then I somehow have to trigger a build for all other pipelines (not managed by the branch-source-plugin) which should listen on any git webhooks, because otherwise they will never be triggered
Does anyone have the same experience? How do you work around them?
- problem 1: write a plugin which is able to abort the build in case it is the first run and the job is configured to listen on webhooks (e.g. Bitbucket or GitHub), then after installing the fresh environment, just loop through all jobs (with groovy) and trigger all which have a hook configured and let the first run fail on purpose.
- problem 2: extends https://github.com/jenkinsci/basic-branch-build-strategies-plugin <https://github.com/jenkinsci/basic-branch-build-strategies-plugin> to some how skip some branches at the initial organisation scanning, I already created an issue for this: https://issues.jenkins-ci.org/browse/JENKINS-54861 <https://issues.jenkins-ci.org/browse/JENKINS-54861>
But to be very honest, most of this sound some how hackisch and I hope someone else has a better idea on how to do this stuff properly.
Many thanks
Domi
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/FA18D443-D4CA-4512-BAD1-255D6E3385C8%40fortysix.ch <https://groups.google.com/d/msgid/jenkinsci-dev/FA18D443-D4CA-4512-BAD1-255D6E3385C8%40fortysix.ch>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
--
James
-------
Jenkins X: a CI/CD solution for modern cloud applications on Kubernetes
http://jenkins-x.io/ <http://jenkins-x.io/>
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CALvxejm05axySHoqVEGrvX%3DrCPj4a4EfK3Q39oqDLrMwjrT-hQ%40mail.gmail.com <https://groups.google.com/d/msgid/jenkinsci-dev/CALvxejm05axySHoqVEGrvX%3DrCPj4a4EfK3Q39oqDLrMwjrT-hQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/9B0B05A3-E884-419F-B85E-471F22B66AC6%40fortysix.ch.
For more options, visit https://groups.google.com/d/optout.
Jesse Glick
2018-11-30 16:59:18 UTC
Permalink
The simplest approach, starting from what you are doing now, would be
to keep the build records and other state files from your Jenkins
installation, recreating only the configuration files from Git.
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1W_8-%3DU8GJSuw_oazA0PSiXGyYDhv8mSKzLV0bScStFg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
domi
2018-12-03 08:15:00 UTC
Permalink
Thanks Jesse, it probably is, but even though - do you see a chance to fix the issues I mentioned?
e.g. why do I need to run a pipeline once before it accepts webhooks from GitHub or Bitbucket? I do understand that I can configure additional SCM sources from within a pipeline, but why does it not listen on the one I have defined on the I have defined to load the "Pipeline from SCM”?
/Domi
Post by Jesse Glick
The simplest approach, starting from what you are doing now, would be
to keep the build records and other state files from your Jenkins
installation, recreating only the configuration files from Git.
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1W_8-%3DU8GJSuw_oazA0PSiXGyYDhv8mSKzLV0bScStFg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/D3849AFD-8449-48A8-88F5-172B2111C421%40fortysix.ch.
For more options, visit https://groups.google.com/d/optout.
Tomas Bjerre
2018-12-03 08:25:56 UTC
Permalink
Regarding configuration of webhooks with pipelines. This is why I always
configure that with Job DSL. I configure the Jenkins job and the Git
service (usually some rest-call) in the same Groovy (Job DSL) script.
Post by domi
Thanks Jesse, it probably is, but even though - do you see a chance to fix
the issues I mentioned?
e.g. why do I need to run a pipeline once before it accepts webhooks from
GitHub or Bitbucket? I do understand that I can configure additional SCM
sources from within a pipeline, but why does it not listen on the one I
have defined on the I have defined to load the "Pipeline from SCM”?
/Domi
Post by Jesse Glick
The simplest approach, starting from what you are doing now, would be
to keep the build records and other state files from your Jenkins
installation, recreating only the configuration files from Git.
--
You received this message because you are subscribed to the Google
Groups "Jenkins Developers" group.
Post by Jesse Glick
To unsubscribe from this group and stop receiving emails from it, send
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1W_8-%3DU8GJSuw_oazA0PSiXGyYDhv8mSKzLV0bScStFg%40mail.gmail.com.
Post by Jesse Glick
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/d21bf102-f345-4c6b-a234-4ab8ad3c6995%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...