Re: [ELISA Development Process WG] Maintainers Expectations vs. Maintainers Reality: An Analysis of Organisational and Maintenance Structure of the Linux Kernel


Very interesting work, Lukas - thanks for bringing to our attention.
And there is certainly plenty of work to follow up.

Actually - I find the results to be quite complimentary.  Given the open-source mindset and heavy reliance on voluntary contributions, the findings are actually better than my expectations.
As noted by Pia (page 22):
​Our analysis shows that a high ratio of all recent patches was not integrated according to process, even though the majority was.  This does not implicitly mean that 20% of all integrated patches during that time window were directly harmful or integrated by maintainers who completely lack expertise.  Since the patches were found on public mailing lists before integration, it is safe to assume that the majority of these patches were exposed to discussion and reviewing.
What this means is that this research work relates to the quality of the release / CM process, not necessarily to the quality of the code itself (design, development, testing).
80% compliance for a voluntary community is quite amazing.  I wonder what comparable rates would be for commercial software development entities with mandatory processes - based on intuition only (no evidences yet) from my long-term experience, I would not guess the compliance rate to be as high as 80%.  But then again, an interesting point for comparison.

In practice, this means we only see here the opening shot.  More evidences to be collected and analyzed.  But indeed, a very interesting starting point.
Thanks, Lukas, for sharing.

From: development-process@... <development-process@...> on behalf of Lukas Bulwahn <lukas.bulwahn@...>
Sent: Monday, November 23, 2020 12:14 PM
To: devel@... <devel@...>; development-process@... <development-process@...>
Cc: Pia Eichinger <pia.eichinger@...>; Ralf Ramsauer <ralf.ramsauer@...>; Wolfgang Mauerer <wolfgang.mauerer@...>; Paul Albertella <paul.albertella@...>
Subject: [ELISA Development Process WG] Maintainers Expectations vs. Maintainers Reality: An Analysis of Organisational and Maintenance Structure of the Linux Kernel
Dear all,

Pia Eichinger, a student at OTH Regensburg, mentored by Ralf Ramsauer
and Wolfgang Mauerer, has written her bachelor thesis on Maintainers
Expectations vs. Maintainers Reality: An Analysis of Organisational
and Maintenance Structure of the Linux Kernel. Simply quoting her

"We showed that around 20% of all patches were theoretically wrongly
integrated when strictly analysing MAINTAINERS. The reality of
integration and maintenance structure is more complicated than that,
which we also explored. Furthermore, we identified 12 major subsystems
of the Linux kernel. This is very helpful for an overview of the
organisational structure, realistic grouping of subsystems and further
Linux kernel topology discussions."

I have placed a copy of her thesis in the Google Drive folder (under
Technical Community/Development Process Working
Group/Development-Process-Data-Mining) here:

Pia, Ralf, Wolfgang, congratulations on this work and thank you very
much for your investigation.

Just to give a quick insight how the various strings of work are related:

- The development process working group has set the goal to understand
the development management and change management of the Linux kernel.
Due to priorities, the group has not worked on the distributed
responsibility model for different parts of the overall code base and
responsibility for change management in much detail yet. I would
expect that the group eventually claims that "the person that is
responsible for the acceptance of a change (a patch) is determined by
the information provided in the MAINTAINERS file."

- Based on that hypothesis, Pia determined the evidence for this claim
and truth of that statement. The evidences suggest that this is 80%

- The remaining challenge is now to determine if the expectation of a
person assessing the development process and the evidences provided
meet. In other words, is "80% truth" for this claim good enough? If
this 80% conformance statement based on evidence is not sufficient, an
extended and refined claim, data preparation, collection, measurement
and interpretation is required. Hopefully, the Evidence Working Group
can eventually formulate such research questions, interpret results
appropriately and refine the analysis, e.g., create scripts to have
the documentation reflect the executed reality, determine the "right"
rules for interpretation and document those, and explain how to come
to actual risk assessments.

Some personal notes:

That 20% are not integrated according to the description in
MAINTAINERS might be surprising to outsiders and especially to
assessors that believe that the working model is fully described and
documented in a 100% structured and reality-reflected manner. The
challenge is really to understand if these 20% are A. due to other
implicit rules that have not been stated explicitly and outdated data
on organisational structures (which implies only a low risk on the
delivered product) or B. due to chaotic integration schemes, i.e., for
20% changes anyone can add anything (which implies a higher risk on
the delivered product).

If it is due to A, the Evidence Working Group can make those implicit
rules more explicit and update the data on organisation, and hence new
measurements will lead to lowering the criticality and risk of the
unknown on the challenge above.
If it is due to B, the change management induces a risk with a
specific known upper bound for specific areas, which needs to be
considered by the risk management of the downstream integrator. In the
worst case, ending with the decision that the responsibility of the
observed code integration is too chaotic to meet the business needs of
the downstream integrator. (In other words, not everything in the
kernel repository is gold; and if you rely on crappy code in the
repository, you will ship crap or if you are well informed, decide not
to ship crap...)

So... I guess the Evidence Working Group has created yet another work
result, even without kicking off... :) Paul, I certainly appreciate if
you would like to continue the investigation of these evidence
together with me (and maybe, Ralf and Wolfgang) and future prospective
bachelor and master students.


Join to automatically receive all group messages.