The future of LTS certification

[Lucie Votypkova]

Hello,

I would like to start discussion about ATH selenium testing. After year of
attempts to help with maintenance and LTS testing, we are dealing with
problem that maintenance is becoming very time demanding and added value of
it is not sufficient. The tests were created for replacing manual testing
LTS with the most used plugins, showing that changes in Jenkins core does
not break widely used plugins and of course testing of Jenkins UI. In
current state is better then manual testing, but it is still time demanding
to keep the step with UI changes in plugins and core and in the same time
deal with browsers, driver, selenium issues and bugs in Jenkins (and its
plugins). Due to a lot of constant failures (which are not connected with
UI issues or integration of Jenkins with plugins in many cases), it can not
be used for watching impact of every Jenkins release on current plugins and
it does not save a lot of time in LTS testing (if we consider the time for
checking constant failures plus time for maintenance) . I collected the
main problems I had to deal with. And a few questions which we should think
about and suggestions what could help. Please feel free to share your
thoughts.


Problems

-

Maintenance is very time consuming.
-

It is revealed only for very small amount of bugs in core or plugins
(most failures are connected with changes in UI).
-

The most of revealed plugin bugs in last year can be detected by
functional testing.
-

Some failures which are real bugs in plugins have a very low priority
because it has a small impact on real user. But it sometime bloks whole
testing and building workaround would not fit the main purpose - use
jenkins as a real user do. This causes that there are real failures
(sometime in the beginning of the tests) which will not be fixed in near
future.
-

There are a lot of tests failures because of changes in plugin UI. When
it happens with widely used plugin which is a dependency of many other
popular plugins (credentials, for example), the ATH tests are useless until
this changes are fixed. For example:
https://issues.jenkins-ci.org/browse/JENKINS-49026
-

Vast majority of breakage caused by UI changes is mostly fixed by
maintainers of ATH because plugins/core contributors are not generally
aware of ATH existence, that they have broke some of the tests or even how
to fix that.
-

Reliability of ATH suite is now stuck in a cycle: ATH is not reliable
because there are always quite a lot of failures and it never gives zero
failures result. The most failures are caused by changes in plugin UI or
low priority bugs in plugins which mainteners are not keen to fix. After a
year attempts to fix ATH failures and achieve +- zero failures at least for
LTS and the last versions of plugins, I realized that the issues appears
faster than I am able to fix them. Now I am able to keep with them but not
to decrease their amount.
-

Due to many failures caused by UI changes, plugin bugs, browser upgrade,
marionette/geckodriver bugs are not very useful tool for showing successful
integration new core versions (and LTS) with the most widely used plugins.
Currently to get such a measurement, it is necessary to verify about 30-50
failures in ATH (if they are still caused only by plugin bugs, UI
changes
.). Which is quite time consuming in comparing with situation when
there is 5-10 failures.
-

Important integration aspects has no coverage in automated testing?
-

The testsuite takes ~20 hours to complete.



Questions:

-

How to deal with tests that are broken (failing without detecting bug)
for long time and there is noone to fix the test?
-

How to deal with known bugs which breaks ATH test and do not seem to get
fixed in near future?
-

How to make maintainers of plugins to feel that it is beneficial to
participate and help with maintenance of ATH, ideally having them actively
engaged in making ATH compatible with their plugin changes?



Suggestions:

-

Avoid ATH tests that duplicates JTH tests in core/plugin?
-

Drawback: Would require PCT to be part of LTS certification.
-

Use only subset of ATH for LTS certification?
-

Drawback: even less attention to the second class citizens.
-

Guideline of removing fragile tests.
-

How to detect fragility?
-

Guideline for including (or excluding) plugin tests into ATH based on
the size of the userbase?
-

Guideline for reporting broken plugin tests to plugin/test maintainer
including test removal if not fixed in time?
-

What would be the deadline?
-

Avoid trivial tests in favor of meaningful use-cases
-

Will cause collapsing/removal of many tests.
-

Guideline for maximal number of tests / maximal runtime per plugin?


-

Suggest perform ATH testing for pull requests of plugins which are
stable for long time in ATH by changing Jenkinsfile (only tests for given
plugin will be performed, not whole ATH)
-

It will definitely make ATH more visible make people more cooperate,
but!
-

Will it be acceptable from time perspective (ATH is running for
long time)?
-

Will it complicate the pull request? ATH tests does not fail only
because of bugs in given plugin, it does not have to be connected with
given pull request.
-

Can those tests be moved to plugin repositories altogether?

[Mark Waite]

My replies are inline. I'm a plugin maintainer. I am deeply attached to
automated tests, especially for the plugins I maintain.

I have not yet had a positive experience with automated tests that use a
web browser to perform those tests. I've been in software development for
a long time. I've seen several attempts, but no personal positive
experiences yet.

My replies are based on those biases.

Post by Lucie Votypkova
Hello,
I would like to start discussion about ATH selenium testing. After year of
attempts to help with maintenance and LTS testing, we are dealing with
problem that maintenance is becoming very time demanding and added value of
it is not sufficient. The tests were created for replacing manual testing
LTS with the most used plugins, showing that changes in Jenkins core does
not break widely used plugins and of course testing of Jenkins UI. In
current state is better then manual testing, but it is still time demanding
to keep the step with UI changes in plugins and core and in the same time
deal with browsers, driver, selenium issues and bugs in Jenkins (and its
plugins). Due to a lot of constant failures (which are not connected with
UI issues or integration of Jenkins with plugins in many cases), it can not
be used for watching impact of every Jenkins release on current plugins and
it does not save a lot of time in LTS testing (if we consider the time for
checking constant failures plus time for maintenance) . I collected the
main problems I had to deal with. And a few questions which we should think
about and suggestions what could help. Please feel free to share your
thoughts.
Problems
-
Maintenance is very time consuming.
-
It is revealed only for very small amount of bugs in core or plugins
(most failures are connected with changes in UI).
-
The most of revealed plugin bugs in last year can be detected by
functional testing.
-
Some failures which are real bugs in plugins have a very low priority
because it has a small impact on real user. But it sometime bloks whole
testing and building workaround would not fit the main purpose - use
jenkins as a real user do. This causes that there are real failures
(sometime in the beginning of the tests) which will not be fixed in near
future.
-
There are a lot of tests failures because of changes in plugin UI.
When it happens with widely used plugin which is a dependency of many other
popular plugins (credentials, for example), the ATH tests are useless until
https://issues.jenkins-ci.org/browse/JENKINS-49026
-
Vast majority of breakage caused by UI changes is mostly fixed by
maintainers of ATH because plugins/core contributors are not generally
aware of ATH existence, that they have broke some of the tests or even how
to fix that.
-
Reliability of ATH suite is now stuck in a cycle: ATH is not reliable
because there are always quite a lot of failures and it never gives zero
failures result. The most failures are caused by changes in plugin UI or
low priority bugs in plugins which mainteners are not keen to fix. After a
year attempts to fix ATH failures and achieve +- zero failures at least for
LTS and the last versions of plugins, I realized that the issues appears
faster than I am able to fix them. Now I am able to keep with them but not
to decrease their amount.
-
Due to many failures caused by UI changes, plugin bugs, browser
upgrade, marionette/geckodriver bugs are not very useful tool for showing
successful integration new core versions (and LTS) with the most widely
used plugins. Currently to get such a measurement, it is necessary to
verify about 30-50 failures in ATH (if they are still caused only by plugin
bugs, UI changes
.). Which is quite time consuming in comparing with
situation when there is 5-10 failures.
-
Important integration aspects has no coverage in automated testing?
-
The testsuite takes ~20 hours to complete.
-
How to deal with tests that are broken (failing without detecting bug)
for long time and there is noone to fix the test?
-
How to deal with known bugs which breaks ATH test and do not seem to
get fixed in near future?
-
How to make maintainers of plugins to feel that it is beneficial to
participate and help with maintenance of ATH, ideally having them actively
engaged in making ATH compatible with their plugin changes?
-
Avoid ATH tests that duplicates JTH tests in core/plugin?
-
Drawback: Would require PCT to be part of LTS certification.
-
Use only subset of ATH for LTS certification?
-
Drawback: even less attention to the second class citizens.
-
Guideline of removing fragile tests.
-
How to detect fragility?
-
Guideline for including (or excluding) plugin tests into ATH based on
the size of the userbase?
-
Guideline for reporting broken plugin tests to plugin/test maintainer
including test removal if not fixed in time?
-
What would be the deadline?
-
Avoid trivial tests in favor of meaningful use-cases
-
Will cause collapsing/removal of many tests.
-
Guideline for maximal number of tests / maximal runtime per plugin?
-
Suggest perform ATH testing for pull requests of plugins which are
stable for long time in ATH by changing Jenkinsfile (only tests for given
plugin will be performed, not whole ATH)
-
It will definitely make ATH more visible make people more
cooperate, but!
-
Will it be acceptable from time perspective (ATH is running for
long time)?

No, it won't be acceptable from a time perspective (at least it seems
unlikely to me). PCT tests were added to the git plugin. The cycle time
for pull request evaluation increased beyond acceptable limits (from 15
minutes to an hour or more if I remember correctly). That might have been
acceptable except for the other issues (described later in this message)

Post by Lucie Votypkova
-
Will it complicate the pull request? ATH tests does not fail
only because of bugs in given plugin, it does not have to be connected with
given pull request.

PCT tests were added to the git plugin. Random failures happened when
there were no relevant code changes. As a plugin maintainer, I had no
interest in investigating the failure of a long-running test which fails
for unrelated or transient reasons. I didn't see the benefit of
investigating those failures. They were not telling me something
especially useful about the problem they were reporting. Time researching
an unrelated or transient failure is time invested in something that rarely
helps the plugin I maintain and rarely helps the Jenkins community.

I've shifted my "end to end" testing for the git plugin to use one
multibranch pipeline job for each bug that I'm investigating (see
jenkins-bugs <https://github.com/MarkEWaite/jenkins-bugs>). I use the
Jenkins Pipeline code and the build tools of my choice to perform checks
that specific conditions of the specific bug are satisfied or not
satisfied. However, those end to end tests involve no web browser and tend
to remain tightly focused on verifying a very specific bug, not an entire
use case.

Use case verification still requires that I test interactively.

Post by Lucie Votypkova
-
Can those tests be moved to plugin repositories altogether?
--
You received this message because you are subscribed to the Google Groups
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-dev/CAMGCEQ1%2BQMESwByc_78p7r17b2cxxM9%3DqXz3zSBgE9ZNOMACGg%40mail.gmail.com
<https://groups.google.com/d/msgid/jenkinsci-dev/CAMGCEQ1%2BQMESwByc_78p7r17b2cxxM9%3DqXz3zSBgE9ZNOMACGg%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.