Malicious patches propagated all the way down to the stable trees


Roberto Bagnara
 

Hi there.

I guess we prefer waiting a little bit for the dust to settle,
but a discussion on the topic seems to be unavoidable.

Is it too soon to say the following, just to let the ball rolling?

1) Wouldn't the UMN people publish their paper, the thing may
have been discovered much later.

2) Patches can go all the way down to the stable trees also
because no sensible coding standard is enforced and static
analysis is not systematically used to detect patches introducing
defects and vulnerabilities.

3) Patches are accepted (also in the stable trees) with insufficient
peer review. (The absence of a sensible, systematically enforced
coding standard is a contributory factor to insufficient peer review.)

And, consequently:

4) In order to come up with something fit for use in safety application,
much has to be done for the redefinition of the development process.

Kind regards,

Roberto

Roberto Bagnara, Ph.D.

CEO/CTO, BUGSENG (http://bugseng.com)
Member, ISO/IEC JTC1/SC22/WG14 - C Standardization Working Group
Member, MISRA C Working Group
Mobile: +39 339 8593517


Oscar Slotosch
 

Hi Roberto,

> 4) In order to come up with something fit for use in safety application,

>   much has to be done for the redefinition of the development process.

 

I completely agree with you and we will now work more on the development process (together with BMW)

 

It's just surprising that my proposal for a talk about that topic was not accepted for the workshop.

So I am not sure if people are aware of the need for a process and the contributions we can make here.

 

Kind regards,

                Oscar

PS

Obviously the gaps that you mentioned have to be addressed by the process that we call "SW Completion Process" at Validas

and it looks in our notation as follows:

 

 

Kind regards,

 

    Roberto

 

Roberto Bagnara, Ph.D.

 

CEO/CTO, BUGSENG (http://bugseng.com)

Member, ISO/IEC JTC1/SC22/WG14 - C Standardization Working Group Member, MISRA C Working Group

Mobile: +39 339 8593517

 

 

 

 


Wenhui
 

Probably an addon auto-reply should be added in the email based patching system, an agreement has to be assigned by the first time commitor  to get the email registered as a valid email address. And only valid email addresses could read and reply through the system.

The agreement should state: 
(1) all communications messages are politeness and for good purpose 
(1) all patches are submitted for good faith 

Otherwise, the account might be banned, and also some legal issues might be raised.


Also, each commitor should submit a report of fuzzing tests and regression tests to be merged. And the process group will give out some guidance on what tools should be used , what tests should be done and for how long the tests should take at least.

On Fri, Apr 23, 2021, 3:26 AM Oscar Slotosch <slotosch@...> wrote:

Hi Roberto,

> 4) In order to come up with something fit for use in safety application,

>   much has to be done for the redefinition of the development process.

 

I completely agree with you and we will now work more on the development process (together with BMW)

 

It's just surprising that my proposal for a talk about that topic was not accepted for the workshop.

So I am not sure if people are aware of the need for a process and the contributions we can make here.

 

Kind regards,

                Oscar

PS

Obviously the gaps that you mentioned have to be addressed by the process that we call "SW Completion Process" at Validas

and it looks in our notation as follows:

 

 

Kind regards,

 

    Roberto

 

Roberto Bagnara, Ph.D.

 

CEO/CTO, BUGSENG (http://bugseng.com)

Member, ISO/IEC JTC1/SC22/WG14 - C Standardization Working Group Member, MISRA C Working Group

Mobile: +39 339 8593517

 

 

 

 


Sudip Mukherjee
 

Hi Roberto, All,

On 23/04/2021 8:15 am, Roberto Bagnara wrote:

Hi there.

I guess we prefer waiting a little bit for the dust to settle,
but a discussion on the topic seems to be unavoidable.

Is it too soon to say the following, just to let the ball rolling?

1) Wouldn't the UMN people publish their paper, the thing may
  have been discovered much later.
Just my personal opinion, I think its too soon to say anything, many of
the maintainer have replied to the revert list of patches saying they
have re-reviewed the patches and the patch is correct. We will know in
next two weeks how many of those patches are actually reverted.


2) Patches can go all the way down to the stable trees also
  because no sensible coding standard is enforced and static
  analysis is not systematically used to detect patches introducing
  defects and vulnerabilities.
All the patches are not accepted in stable, only patches which fixes
something.


3) Patches are accepted (also in the stable trees) with insufficient
  peer review.  (The absence of a sensible, systematically enforced
  coding standard is a contributory factor to insufficient peer review.)
Pavel (he reviews stable patches on behalf of CIP) brought it to Greg's
attention on one of the buggy patch that was sent for inclusion in
stable. I think that shows that review process for stable is working.
But we do need more volunteers for review.


And, consequently:

4) In order to come up with something fit for use in safety application,
  much has to be done for the redefinition of the development process.
This incident triggered a discussion and the maintainers are already
discussing and rethinking the acceptance policy for trivial patches. You
can check the discussion at
https://lore.kernel.org/ksummit/afc5664dc2b60f912dd97abfa818b3f7c4237b92.camel@HansenPartnership.com/



--
Regards
Sudip


Paoloni, Gabriele
 

Hi All

I fully agree with everything said by Sudip and I would like
to add that static code analysis is just one measure to reduce
the risk (valuable indeed); in general we are quite aware that
we have gaps in the current Linux Development process to
claim robustness and we have actually evaluated some of
these gaps for the test and verification phase for example.
As of today we are figuring out a way to fill these gaps
including architectural and design docs, safety analyses, SCA,
kernel tests, etc.
WRT the code review process though I must disagree as
any patch must be reviewed at least by one maintainer
and for the stable tree the maintainers turn to be at least
two.

Is the code review process of closed-source ASIL or SIL rated
OSs stricter?
How is that guaranteed in practice?

Thanks
Gab

-----Original Message-----
From: development-process@lists.elisa.tech <development-
process@lists.elisa.tech> On Behalf Of Sudip Mukherjee
Sent: Friday, April 23, 2021 10:39 AM
To: Roberto Bagnara <roberto.bagnara@bugseng.com>; development-
process@lists.elisa.tech
Subject: Re: [ELISA Development Process WG] Malicious patches propagated
all the way down to the stable trees

Hi Roberto, All,

On 23/04/2021 8:15 am, Roberto Bagnara wrote:

Hi there.

I guess we prefer waiting a little bit for the dust to settle,
but a discussion on the topic seems to be unavoidable.

Is it too soon to say the following, just to let the ball rolling?

1) Wouldn't the UMN people publish their paper, the thing may
  have been discovered much later.
Just my personal opinion, I think its too soon to say anything, many of
the maintainer have replied to the revert list of patches saying they
have re-reviewed the patches and the patch is correct. We will know in
next two weeks how many of those patches are actually reverted.


2) Patches can go all the way down to the stable trees also
  because no sensible coding standard is enforced and static
  analysis is not systematically used to detect patches introducing
  defects and vulnerabilities.
All the patches are not accepted in stable, only patches which fixes
something.


3) Patches are accepted (also in the stable trees) with insufficient
  peer review.  (The absence of a sensible, systematically enforced
  coding standard is a contributory factor to insufficient peer review.)
Pavel (he reviews stable patches on behalf of CIP) brought it to Greg's
attention on one of the buggy patch that was sent for inclusion in
stable. I think that shows that review process for stable is working.
But we do need more volunteers for review.


And, consequently:

4) In order to come up with something fit for use in safety application,
  much has to be done for the redefinition of the development process.
This incident triggered a discussion and the maintainers are already
discussing and rethinking the acceptance policy for trivial patches. You
can check the discussion at
https://lore.kernel.org/ksummit/afc5664dc2b60f912dd97abfa818b3f7c4237b
92.camel@HansenPartnership.com/



--
Regards
Sudip



---------------------------------------------------------------------
INTEL CORPORATION ITALIA S.p.A. con unico socio
Sede: Milanofiori Palazzo E 4
CAP 20094 Assago (MI)
Capitale Sociale Euro 104.000,00 interamente versato
Partita I.V.A. e Codice Fiscale 04236760155
Repertorio Economico Amministrativo n. 997124
Registro delle Imprese di Milano nr. 183983/5281/33
Soggetta ad attivita' di direzione e coordinamento di
INTEL CORPORATION, USA

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


Wenhui
 

Just a little adjustification, 
For the reviewers, the two reviewers should not have conflict of interest with the patch commitor. That is to say, the commitor and reviewers are not from the same organization, are not direct relatives, are not former advisors/mentors and students.


On Fri, Apr 23, 2021, 4:54 AM Paoloni, Gabriele <gabriele.paoloni@...> wrote:
Hi All

I fully agree with everything said by Sudip and I would like
to add that static code analysis is just one measure to reduce
the risk (valuable indeed); in general we are quite aware that
we have gaps in the current Linux Development process to
claim robustness and we have actually evaluated some of
these gaps for the test and verification phase for example.
As of today we are figuring out a way to fill these gaps
including architectural and design docs, safety analyses, SCA,
kernel tests, etc.
WRT the code review process though I must disagree as
any patch must be reviewed at least by one maintainer
and for the stable tree the maintainers turn to be at least
two.

Is the code review process of closed-source ASIL or SIL rated
OSs stricter?
How is that guaranteed in practice? 

Thanks
Gab

> -----Original Message-----
> From: development-process@... <development-
> process@...> On Behalf Of Sudip Mukherjee
> Sent: Friday, April 23, 2021 10:39 AM
> To: Roberto Bagnara <roberto.bagnara@...>; development-
> process@...
> Subject: Re: [ELISA Development Process WG] Malicious patches propagated
> all the way down to the stable trees
>
> Hi Roberto, All,
>
> On 23/04/2021 8:15 am, Roberto Bagnara wrote:
> >
> > Hi there.
> >
> > I guess we prefer waiting a little bit for the dust to settle,
> > but a discussion on the topic seems to be unavoidable.
> >
> > Is it too soon to say the following, just to let the ball rolling?
> >
> > 1) Wouldn't the UMN people publish their paper, the thing may
> >   have been discovered much later.
>
> Just my personal opinion, I think its too soon to say anything, many of
> the maintainer have replied to the revert list of patches saying they
> have re-reviewed the patches and the patch is correct. We will know in
> next two weeks how many of those patches are actually reverted.
>
> >
> > 2) Patches can go all the way down to the stable trees also
> >   because no sensible coding standard is enforced and static
> >   analysis is not systematically used to detect patches introducing
> >   defects and vulnerabilities.
>
> All the patches are not accepted in stable, only patches which fixes
> something.
>
> >
> > 3) Patches are accepted (also in the stable trees) with insufficient
> >   peer review.  (The absence of a sensible, systematically enforced
> >   coding standard is a contributory factor to insufficient peer review.)
>
> Pavel (he reviews stable patches on behalf of CIP) brought it to Greg's
> attention on one of the buggy patch that was sent for inclusion in
> stable. I think that shows that review process for stable is working.
> But we do need more volunteers for review.
>
> >
> > And, consequently:
> >
> > 4) In order to come up with something fit for use in safety application,
> >   much has to be done for the redefinition of the development process.
>
> This incident triggered a discussion and the maintainers are already
> discussing and rethinking the acceptance policy for trivial patches. You
> can check the discussion at
> https://lore.kernel.org/ksummit/afc5664dc2b60f912dd97abfa818b3f7c4237b
> 92.camel@.../
>
>
>
> --
> Regards
> Sudip
>
>
>
>

---------------------------------------------------------------------
INTEL CORPORATION ITALIA S.p.A. con unico socio
Sede: Milanofiori Palazzo E 4
CAP 20094 Assago (MI)
Capitale Sociale Euro 104.000,00 interamente versato
Partita I.V.A. e Codice Fiscale  04236760155
Repertorio Economico Amministrativo n. 997124
Registro delle Imprese di Milano nr. 183983/5281/33
Soggetta ad attivita' di direzione e coordinamento di
INTEL CORPORATION, USA

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.






Lukas Bulwahn
 

On Fri, Apr 23, 2021 at 9:15 AM Roberto Bagnara
<roberto.bagnara@bugseng.com> wrote:


Hi there.

I guess we prefer waiting a little bit for the dust to settle,
but a discussion on the topic seems to be unavoidable.

Is it too soon to say the following, just to let the ball rolling?

1) Wouldn't the UMN people publish their paper, the thing may
have been discovered much later.
Roberto, what is "the thing"?

My interpretation #1:

An open community accepts changes from developers without extensive
background checks, nor checking of their intent.

Agree to that.

Trivial changes by newcomers and non-experts often break more than
what they intend to repair. (IMHO, that is why static analysis done
wrong is harmful.)

Anyone that really assessed the development practice knows this for many years.
Just consider how many newbies and kernel non-experts contribute to
the kernel with clean-up activities, and follow up in the janitor
mailing list.

There are other interpretations for "the thing", but Roberto, please
clarify what you really stated.

2) Patches can go all the way down to the stable trees also
because no sensible coding standard is enforced and static
analysis is not systematically used to detect patches introducing
defects and vulnerabilities.
Agree, patches can go quickly into stable, sometimes too fast.
But no, your root-cause analysis (no sensible coding standard is
enforced and static analysis is not systematically used) is not the
reason that it went into stable.

And as the example here shows: it was eventually detected and the
process is now under discussion. So, the ISO 9001 continuous
improvement process on the development process is taking effect.
Also, a single instance hardly proves a point in one or the other direction.

You need to look at all fixes integrated into stable and all
regressions. This was done by Henry Rosten and Marijo Simunovic in
2019 and 2020 in the ELISA Project, see
https://github.com/elisa-tech/workgroups/. However, nobody else
contributed to that investigation.

3) Patches are accepted (also in the stable trees) with insufficient
peer review. (The absence of a sensible, systematically enforced
coding standard is a contributory factor to insufficient peer review.)
The first hypothesis might be true, but how do you measure sufficient
peer review? Again, Basak's thesis did some first steps in this
direction; feel free to continue this work.

Clearly, a sensible, systematically enforced coding standard cannot
stop a malicious contributor to contribute bugs to an open development
community. The malicious contributor would follow the coding standard,
but still would intend to introduce a bug.

And, consequently:

4) In order to come up with something fit for use in safety application,
much has to be done for the redefinition of the development process.
Agree. You need to have a good detailed understanding of the actual
development process and practice in the large kernel community (>1,500
constant developers) to make a reasonable risk estimation due to
procedural development issues. You need to learn how you measure this
risk and how you can effectively impact the kernel development process
of those kernel developers. (Hint: it does not work by an outsider
without contributions requesting a process change and asking someone
else within the community to enforce it in this open community. So,
you first need to contribute to the project and experiment locally
with a process change within a small group of willing maintainers,
show the benefit and feasibility and advertise slowly its adoption.
That is what many contributors could successfully push forward
throughout the past decade.)


Lukas


Lukas Bulwahn
 

On Fri, Apr 23, 2021 at 10:39 AM Sudip Mukherjee
<sudip.mukherjee@codethink.co.uk> wrote:

Hi Roberto, All,

On 23/04/2021 8:15 am, Roberto Bagnara wrote:

Hi there.

I guess we prefer waiting a little bit for the dust to settle,
but a discussion on the topic seems to be unavoidable.

Is it too soon to say the following, just to let the ball rolling?

1) Wouldn't the UMN people publish their paper, the thing may
have been discovered much later.
Just my personal opinion, I think its too soon to say anything, many of
the maintainer have replied to the revert list of patches saying they
have re-reviewed the patches and the patch is correct. We will know in
next two weeks how many of those patches are actually reverted.


2) Patches can go all the way down to the stable trees also
because no sensible coding standard is enforced and static
analysis is not systematically used to detect patches introducing
defects and vulnerabilities.
All the patches are not accepted in stable, only patches which fixes
something.


3) Patches are accepted (also in the stable trees) with insufficient
peer review. (The absence of a sensible, systematically enforced
coding standard is a contributory factor to insufficient peer review.)
Pavel (he reviews stable patches on behalf of CIP) brought it to Greg's
attention on one of the buggy patch that was sent for inclusion in
stable. I think that shows that review process for stable is working.
But we do need more volunteers for review.


And, consequently:

4) In order to come up with something fit for use in safety application,
much has to be done for the redefinition of the development process.
This incident triggered a discussion and the maintainers are already
discussing and rethinking the acceptance policy for trivial patches. You
can check the discussion at
https://lore.kernel.org/ksummit/afc5664dc2b60f912dd97abfa818b3f7c4237b92.camel@HansenPartnership.com/
I fully agree with what Sudip said. I think my email I wrote slower in
parallel points out very similar insights.

If you really want to make a proper judgement, consider proper ways to
prove or disprove a hypothesis. The various thesis from Sebastian
Duda, Pia Eichinger and Basak Erdamar (with the support from Wolfgang
Mauerer and Ralf Ramsauer) are theses that look into these process
issues and try to evaluate hypotheses based on analysis of available
data from the executed development.

Lukas


Lukas Bulwahn
 

On Fri, Apr 23, 2021 at 11:04 AM Wenhui via lists.elisa.tech
<wenhui=gwmail.gwu.edu@lists.elisa.tech> wrote:

Just a little adjustification,
For the reviewers, the two reviewers should not have conflict of interest with the patch commitor. That is to say, the commitor and reviewers are not from the same organization, are not direct relatives, are not former advisors/mentors and students.
Requiring that as a hard criteria would block the whole Intel graphics
drivers development (just as a random example), as they are mainly
developed and reviewed by Intel developers.

Feel free to investigate how many commits only have committers and
reviewers from the same organization. (We have done that in the past.)
The data is available.

Lukas


Sudip Mukherjee
 

Almost all the maintainers and reviewers are volunteers, and many of the
trivial patches will only get maintainer review when its being added to
the tree. I dont think its possible to make two reviews or even a single
review compulsory unless you are proposing that companies/corporates
employ people who will review patches as part of their dayjob.

And its also very common that to have maintainers and contributors from
the same organization.


--
Regards
Sudip

On 23/04/2021 10:04 am, Wenhui Zhang wrote:
Just a little adjustification, 
For the reviewers, the two reviewers should not have conflict of
interest with the patch commitor. That is to say, the commitor and
reviewers are not from the same organization, are not direct relatives,
are not former advisors/mentors and students.

On Fri, Apr 23, 2021, 4:54 AM Paoloni, Gabriele
<gabriele.paoloni@intel.com <mailto:gabriele.paoloni@intel.com>> wrote:

Hi All

I fully agree with everything said by Sudip and I would like
to add that static code analysis is just one measure to reduce
the risk (valuable indeed); in general we are quite aware that
we have gaps in the current Linux Development process to
claim robustness and we have actually evaluated some of
these gaps for the test and verification phase for example.
As of today we are figuring out a way to fill these gaps
including architectural and design docs, safety analyses, SCA,
kernel tests, etc.
WRT the code review process though I must disagree as
any patch must be reviewed at least by one maintainer
and for the stable tree the maintainers turn to be at least
two.

Is the code review process of closed-source ASIL or SIL rated
OSs stricter?
How is that guaranteed in practice? 

Thanks
Gab

> -----Original Message-----
> From: development-process@lists.elisa.tech <development-
> process@lists.elisa.tech> On Behalf Of Sudip Mukherjee
> Sent: Friday, April 23, 2021 10:39 AM
> To: Roberto Bagnara <roberto.bagnara@bugseng.com
<mailto:roberto.bagnara@bugseng.com>>; development-
> process@lists.elisa.tech
> Subject: Re: [ELISA Development Process WG] Malicious patches
propagated
> all the way down to the stable trees
>
> Hi Roberto, All,
>
> On 23/04/2021 8:15 am, Roberto Bagnara wrote:
> >
> > Hi there.
> >
> > I guess we prefer waiting a little bit for the dust to settle,
> > but a discussion on the topic seems to be unavoidable.
> >
> > Is it too soon to say the following, just to let the ball rolling?
> >
> > 1) Wouldn't the UMN people publish their paper, the thing may
> >   have been discovered much later.
>
> Just my personal opinion, I think its too soon to say anything,
many of
> the maintainer have replied to the revert list of patches saying they
> have re-reviewed the patches and the patch is correct. We will know in
> next two weeks how many of those patches are actually reverted.
>
> >
> > 2) Patches can go all the way down to the stable trees also
> >   because no sensible coding standard is enforced and static
> >   analysis is not systematically used to detect patches introducing
> >   defects and vulnerabilities.
>
> All the patches are not accepted in stable, only patches which fixes
> something.
>
> >
> > 3) Patches are accepted (also in the stable trees) with insufficient
> >   peer review.  (The absence of a sensible, systematically enforced
> >   coding standard is a contributory factor to insufficient peer
review.)
>
> Pavel (he reviews stable patches on behalf of CIP) brought it to
Greg's
> attention on one of the buggy patch that was sent for inclusion in
> stable. I think that shows that review process for stable is working.
> But we do need more volunteers for review.
>
> >
> > And, consequently:
> >
> > 4) In order to come up with something fit for use in safety
application,
> >   much has to be done for the redefinition of the development
process.
>
> This incident triggered a discussion and the maintainers are already
> discussing and rethinking the acceptance policy for trivial
patches. You
> can check the discussion at
>
https://lore.kernel.org/ksummit/afc5664dc2b60f912dd97abfa818b3f7c4237b
<https://lore.kernel.org/ksummit/afc5664dc2b60f912dd97abfa818b3f7c4237b>
> 92.camel@HansenPartnership.com/
>
>
>
> --
> Regards
> Sudip
>
>
>
>

---------------------------------------------------------------------
INTEL CORPORATION ITALIA S.p.A. con unico socio
Sede: Milanofiori Palazzo E 4
CAP 20094 Assago (MI)
Capitale Sociale Euro 104.000,00 interamente versato
Partita I.V.A. e Codice Fiscale  04236760155
Repertorio Economico Amministrativo n. 997124
Registro delle Imprese di Milano nr. 183983/5281/33
Soggetta ad attivita' di direzione e coordinamento di
INTEL CORPORATION, USA

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.





Wenhui
 

Yup, you are right, we could not just go ahead and copy the standard for conflict of interest of academic conferences. 

Do you think a consent form stating the committers are on their good faith is enough? 


On Fri, Apr 23, 2021, 5:18 AM Lukas Bulwahn <lukas.bulwahn@...> wrote:
On Fri, Apr 23, 2021 at 11:04 AM Wenhui via lists.elisa.tech
<wenhui=gwmail.gwu.edu@...> wrote:
>
> Just a little adjustification,
> For the reviewers, the two reviewers should not have conflict of interest with the patch commitor. That is to say, the commitor and reviewers are not from the same organization, are not direct relatives, are not former advisors/mentors and students.
>

Requiring that as a hard criteria would block the whole Intel graphics
drivers development (just as a random example), as they are mainly
developed and reviewed by Intel developers.

Feel free to investigate how many commits only have committers and
reviewers from the same organization. (We have done that in the past.)
The data is available.

Lukas


Lukas Bulwahn
 

On Fri, Apr 23, 2021 at 11:24 AM Wenhui Zhang <wenhui@gwmail.gwu.edu> wrote:

Yup, you are right, we could not just go ahead and copy the standard for conflict of interest of academic conferences.

Do you think a consent form stating the committers are on their good faith is enough?
Well, but somebody not acting on their good faith would also just
consent with bad faith (but as a lie!). So, how does this question to
each contributor increase trust?

Lukas


Paul Albertella
 

Hi,

On 23/04/2021 09:38, Sudip Mukherjee wrote:
This incident triggered a discussion and the maintainers are already
discussing and rethinking the acceptance policy for trivial patches. You
can check the discussion at
https://lore.kernel.org/ksummit/afc5664dc2b60f912dd97abfa818b3f7c4237b92.camel@HansenPartnership.com/
Could this have implications for the work being done by the Tools Improvements group? On the one hand, it may be harder to get 'trivial' patches accepted, but there's also an opportunity to advance the principle of providing evidence to support a change. As the poster suggests, an acceptance policy that requires "no trivial bug fix without a pointer to the actual bug report or report from a trusted static checker" might be an improvement.

As Lukas notes, fostering positive change in a large open source developer community requires engagement with that community and an effort to understand their priorities and constraints. However, an incident like this may serve to accelerate the pace of change.

Regards,

Paul


Wenhui
 

Got it, it seems like the situation and process of  driver subgroup is a little different than the environment of virtual filesystem subgroup and Linux security module subgroup, where the review and merge process is extremely slow, and hardly any patches could be committed if you are not a member of the group already (  people trust your code natively). 


On Fri, Apr 23, 2021, 5:21 AM Sudip Mukherjee <sudip.mukherjee@...> wrote:
Almost all the maintainers and reviewers are volunteers, and many of the
trivial patches will only get maintainer review when its being added to
the tree. I dont think its possible to make two reviews or even a single
review compulsory unless you are proposing that companies/corporates
employ people who will review patches as part of their dayjob.

And its also very common that to have maintainers and contributors from
the same organization.


--
Regards
Sudip



On 23/04/2021 10:04 am, Wenhui Zhang wrote:
> Just a little adjustification, 
> For the reviewers, the two reviewers should not have conflict of
> interest with the patch commitor. That is to say, the commitor and
> reviewers are not from the same organization, are not direct relatives,
> are not former advisors/mentors and students.
>
> On Fri, Apr 23, 2021, 4:54 AM Paoloni, Gabriele
> <gabriele.paoloni@... <mailto:gabriele.paoloni@...>> wrote:
>
>     Hi All
>
>     I fully agree with everything said by Sudip and I would like
>     to add that static code analysis is just one measure to reduce
>     the risk (valuable indeed); in general we are quite aware that
>     we have gaps in the current Linux Development process to
>     claim robustness and we have actually evaluated some of
>     these gaps for the test and verification phase for example.
>     As of today we are figuring out a way to fill these gaps
>     including architectural and design docs, safety analyses, SCA,
>     kernel tests, etc.
>     WRT the code review process though I must disagree as
>     any patch must be reviewed at least by one maintainer
>     and for the stable tree the maintainers turn to be at least
>     two.
>
>     Is the code review process of closed-source ASIL or SIL rated
>     OSs stricter?
>     How is that guaranteed in practice? 
>
>     Thanks
>     Gab
>
>     > -----Original Message-----
>     > From: development-process@... <development-
>     > process@...> On Behalf Of Sudip Mukherjee
>     > Sent: Friday, April 23, 2021 10:39 AM
>     > To: Roberto Bagnara <roberto.bagnara@...
>     <mailto:roberto.bagnara@...>>; development-
>     > process@...
>     > Subject: Re: [ELISA Development Process WG] Malicious patches
>     propagated
>     > all the way down to the stable trees
>     >
>     > Hi Roberto, All,
>     >
>     > On 23/04/2021 8:15 am, Roberto Bagnara wrote:
>     > >
>     > > Hi there.
>     > >
>     > > I guess we prefer waiting a little bit for the dust to settle,
>     > > but a discussion on the topic seems to be unavoidable.
>     > >
>     > > Is it too soon to say the following, just to let the ball rolling?
>     > >
>     > > 1) Wouldn't the UMN people publish their paper, the thing may
>     > >   have been discovered much later.
>     >
>     > Just my personal opinion, I think its too soon to say anything,
>     many of
>     > the maintainer have replied to the revert list of patches saying they
>     > have re-reviewed the patches and the patch is correct. We will know in
>     > next two weeks how many of those patches are actually reverted.
>     >
>     > >
>     > > 2) Patches can go all the way down to the stable trees also
>     > >   because no sensible coding standard is enforced and static
>     > >   analysis is not systematically used to detect patches introducing
>     > >   defects and vulnerabilities.
>     >
>     > All the patches are not accepted in stable, only patches which fixes
>     > something.
>     >
>     > >
>     > > 3) Patches are accepted (also in the stable trees) with insufficient
>     > >   peer review.  (The absence of a sensible, systematically enforced
>     > >   coding standard is a contributory factor to insufficient peer
>     review.)
>     >
>     > Pavel (he reviews stable patches on behalf of CIP) brought it to
>     Greg's
>     > attention on one of the buggy patch that was sent for inclusion in
>     > stable. I think that shows that review process for stable is working.
>     > But we do need more volunteers for review.
>     >
>     > >
>     > > And, consequently:
>     > >
>     > > 4) In order to come up with something fit for use in safety
>     application,
>     > >   much has to be done for the redefinition of the development
>     process.
>     >
>     > This incident triggered a discussion and the maintainers are already
>     > discussing and rethinking the acceptance policy for trivial
>     patches. You
>     > can check the discussion at
>     >
>     https://lore.kernel.org/ksummit/afc5664dc2b60f912dd97abfa818b3f7c4237b
>     <https://lore.kernel.org/ksummit/afc5664dc2b60f912dd97abfa818b3f7c4237b>
>     > 92.camel@.../
>     >
>     >
>     >
>     > --
>     > Regards
>     > Sudip
>     >
>     >
>     >
>     >
>
>     ---------------------------------------------------------------------
>     INTEL CORPORATION ITALIA S.p.A. con unico socio
>     Sede: Milanofiori Palazzo E 4
>     CAP 20094 Assago (MI)
>     Capitale Sociale Euro 104.000,00 interamente versato
>     Partita I.V.A. e Codice Fiscale  04236760155
>     Repertorio Economico Amministrativo n. 997124
>     Registro delle Imprese di Milano nr. 183983/5281/33
>     Soggetta ad attivita' di direzione e coordinamento di
>     INTEL CORPORATION, USA
>
>     This e-mail and any attachments may contain confidential material for
>     the sole use of the intended recipient(s). Any review or distribution
>     by others is strictly prohibited. If you are not the intended
>     recipient, please contact the sender and delete all copies.
>
>
>     
>
>


Wenhui
 

Sellers and customers got banned by Amazon, facebook page and twitter etc. for malicious acts. I guess we could ask the first time committers sign this consent form, and if they are detected as malicious, their account could be banned for a period of time? 


On Fri, Apr 23, 2021, 5:29 AM Lukas Bulwahn <lukas.bulwahn@...> wrote:
On Fri, Apr 23, 2021 at 11:24 AM Wenhui Zhang <wenhui@...> wrote:
>
> Yup, you are right, we could not just go ahead and copy the standard for conflict of interest of academic conferences.
>
> Do you think a consent form stating the committers are on their good faith is enough?
>

Well, but somebody not acting on their good faith would also just
consent with bad faith (but as a lie!). So, how does this question to
each contributor increase trust?

Lukas


Sudip Mukherjee
 

On 23/04/2021 10:36 am, Wenhui via lists.elisa.tech wrote:
Sellers and customers got banned by Amazon, facebook page and twitter
etc. for malicious acts. I guess we could ask the first time committers
sign this consent form, and if they are detected as malicious, their
account could be banned for a period of time?
I think everyone from @umn.edu has been banned for now. :)



--
Regards
Sudip


John MacGregor
 

Hi Roberto, all

It may be too soon to discuss this event, but a number of issues have been placed on the back burner, which could possibly be warmed up again. If you interpret this as grandstanding, sorry.

First, let's put the incident in context. For us, this is not a security issue. It means that patches of arbitrary quality can be incorporated in the stable tree, including patches of dubious quality and / or impact on safety. This means that the integration process is faulty.

I talked to Greg Kroah-Hartmann about this a couple of years ago and he said that while in Paris, he was working with INRIA on a study of the accuracy of the stable update process (i.e. his decision process). As I remember (and I have a terrible memory for details), he said he was only right about 50% of the time, especially w.r.t type 1 and type 2 errors (accepting a patch that wasn't necessary, rejecting a patch that was necessary). For the sake of sticking to the truth, if anyone can provide this report, please speak up.

While I may use other sources for the following points, I have been thinking, and saying, these things for... years, mostly.

1) As Lukas Bulwahn claimed in the September Workshop[1. approximately minute 45] Kernel development is not a project or a product, but a consolidated change management process. In other words, it process is not a development process.

2) Kernel development does not have a quality management system

4) Linux was not developed for application in safety-critical systems.

3) The kernel is not "industrial-grade" software
- As Jason Smith pointed out in a (very) recent ELISA blog post, Linux is SOUP (Software of Unknown Provenance... for me Pedigree). The first instance of this term that I found was in a consultant's report to the British Government in 2001. That is, this is neither unknown nor deniable.
- Linux is miles away from being "QM" in the ISO 26262 sense
- ==> Whatever subset of the 27 million lines of non-safety critical Linux source code that comes into a safety critical system is not QM, and has not been developed according to the demands of safety standards. Using it in a safety-critical system is probably questionable.

4) The Linux Kernel Project makes software available in the sense that it is a somewhat coherent collection of source code. Executable Linux software comes from somewhere else or you make it yourself.

5) The criteria whether Linux, or open source software, is suitable for use in safety-critical systems is the residual risk of failure in the software as integrated into the system.
- failures can be eliminated by a conformant development process
- failures can be tolerated with appropriate compensation mechanisms

6) For me, the Linux development process has already stopped, and we're in an operational phase. At least, the overwhelming proportion of the code used will be used unmodified. (Modification and upstreaming the changes seems like a very tenuous approach to me). To an extent, this means that system integrators are reusing Linux, not using it.

7) I agree with your comments on the "development process" as far as they go.
- peer review
- static testing
- coding standards
yup.

8) As a continuation of the discussion we had in the sync telco this week, the Linux process must be understood in the context of commonly accepted (i.e. professional) systems, and software, engineering practice. And the role of Linux must be understood in the context of this practice applied to the development of safety-critical embedded systems. Whether the Linux process has been successful in other domains is irrelevant.

9) Maybe the development process should be redefined, but first of all the strict Linux Kernel process is not a development process in the first place. Second, I think the Linux community is satisfied with, even proud of, its process as it is. I'm not sure they want to have an "industrial grade" one.

10) As we have said from the beginning of the project that there is a tension between what the standards require, what safety-critical system developers would consider to be acceptably safe and what the open source community is willing to do in this direction. Each of these areas will have to be redefined, and the practice adapted accordingly, in order to enable Linux in safety-critical applications.

Well, that's it off the top of my head. It probably stirs up more dust, but the dust was laying around anyway.

Cheers

John

[1] https://drive.google.com/drive/folders/12DkfO3bRaPje75lG9XtoLBgtn383DezF

[2] https://www.hse.gov.uk/research/crr_pdf/2001/crr01336.pdf

On 23/04/2021 09:15, Roberto Bagnara wrote:
Hi there.
I guess we prefer waiting a little bit for the dust to settle,
but a discussion on the topic seems to be unavoidable.
Is it too soon to say the following, just to let the ball rolling?
1) Wouldn't the UMN people publish their paper, the thing may
  have been discovered much later.
2) Patches can go all the way down to the stable trees also
  because no sensible coding standard is enforced and static
  analysis is not systematically used to detect patches introducing
  defects and vulnerabilities.
3) Patches are accepted (also in the stable trees) with insufficient
  peer review.  (The absence of a sensible, systematically enforced
  coding standard is a contributory factor to insufficient peer review.)
And, consequently:
4) In order to come up with something fit for use in safety application,
  much has to be done for the redefinition of the development process.
Kind regards,
  Roberto
Roberto Bagnara, Ph.D.
CEO/CTO, BUGSENG (http://bugseng.com)
Member, ISO/IEC JTC1/SC22/WG14 - C Standardization Working Group
Member, MISRA C Working Group
Mobile: +39 339 8593517


Sudip Mukherjee
 

On 23/04/2021 9:38 am, Sudip Mukherjee wrote:
Hi Roberto, All,

On 23/04/2021 8:15 am, Roberto Bagnara wrote:

Hi there.

I guess we prefer waiting a little bit for the dust to settle,
but a discussion on the topic seems to be unavoidable.

Is it too soon to say the following, just to let the ball rolling?

1) Wouldn't the UMN people publish their paper, the thing may
  have been discovered much later.
Just my personal opinion, I think its too soon to say anything, many of
the maintainer have replied to the revert list of patches saying they
have re-reviewed the patches and the patch is correct. We will know in
next two weeks how many of those patches are actually reverted.
An announcement from "The LF Technical Advisory Board".

https://lore.kernel.org/lkml/202104221451.292A6ED4@keescook/



--
Regards
Sudip


Wenhui
 

IMHO, Linux could be runtime industry level acceptable to safety-critical systems.
While since source code of Linux is too large, it might be hard to ensure such a standard. 

However, the compiler could easily kick out the out of tree code, and trim the Linux for a specific safety-critical system, which is called " dead code analysis". 

Most of the bugs are on the out of tree code, I think. Then making Linux working for one safety-critical system is a tangible goal.

Then one question is, which part of the source code are commonly shared among all runtimes, and which part are specific to the particular application ( i.e. libc, arch and drivers)  

On Fri, Apr 23, 2021, 5:44 AM John MacGregor <open.john.macgregor@...> wrote:
Hi Roberto, all

It may be too soon to discuss this event, but a number of issues have
been placed on the back burner, which could possibly be warmed up again.
  If you interpret this as grandstanding, sorry.

First, let's put the incident in context.  For us, this is not a
security issue.  It means that patches of arbitrary quality can be
incorporated in the stable tree, including patches of dubious quality
and / or impact on safety.  This means that the integration process is
faulty.

I talked to Greg Kroah-Hartmann about this a couple of years ago and he
said that while in Paris, he was working with INRIA on a study of the
accuracy of the stable update process (i.e. his decision process).  As I
remember (and I have a terrible memory for details), he said he was only
right about 50% of the time, especially w.r.t type 1 and type 2 errors
(accepting a patch that wasn't necessary, rejecting a patch that was
necessary).  For the sake of sticking to the truth, if anyone can
provide this report, please speak up.

While I may use other sources for the following points, I have been
thinking, and saying, these things for... years, mostly.

1) As Lukas Bulwahn claimed in the September Workshop[1. approximately
minute 45]  Kernel development is not a project or a product, but a
consolidated change management process.  In other words, it process is
not a development process.

2) Kernel development does not have a quality management system

4) Linux was not developed for application in safety-critical systems.

3) The kernel is not "industrial-grade" software
   - As Jason Smith pointed out in a (very) recent ELISA blog post,
Linux is SOUP (Software of Unknown Provenance... for me Pedigree).  The
first instance of this term that I found was in a consultant's report to
the British Government in 2001. That is, this is neither unknown nor
deniable.
   - Linux is miles away from being "QM" in the ISO 26262 sense
   - ==> Whatever subset of the 27 million lines of non-safety critical
Linux source code that comes into a safety critical system is not QM,
and has not been developed according to the demands of safety standards.
  Using it in a safety-critical system is probably questionable.

4) The Linux Kernel Project makes software available in the sense that
it is a somewhat coherent collection of source code.  Executable Linux
software comes from somewhere else or you make it yourself.

5) The criteria whether Linux, or open source software, is suitable for
use in safety-critical systems is the residual risk of failure in the
software as integrated into the system.
   - failures can be eliminated by a conformant development process
   - failures can be tolerated with appropriate compensation mechanisms

6) For me, the Linux development process has already stopped, and we're
in an operational phase.  At least, the overwhelming proportion of the
code used will be used unmodified. (Modification and upstreaming the
changes seems like a very tenuous approach to me). To an extent, this
means that system integrators are reusing Linux, not using it.

7)  I agree with your comments on the "development process" as far as
they go.
   - peer review
   - static testing
   - coding standards
yup.

8) As a continuation of the discussion we had in the sync telco this
week, the Linux process must be understood in the context of commonly
accepted (i.e. professional) systems, and software, engineering
practice.  And the role of Linux must be understood in the context of
this practice applied to the development of safety-critical embedded
systems.  Whether the Linux process has been successful in other domains
is irrelevant.

9) Maybe the development process should be redefined, but first of all
the strict Linux Kernel process is not a development process in the
first place.  Second, I think the Linux community is satisfied with,
even proud of, its process as it is.  I'm not sure they want to have an
"industrial grade" one.

10) As we have said from the beginning of the project that there is a
tension between what the standards require, what safety-critical system
developers would consider to be acceptably safe and what the open source
community is willing to do in this direction.  Each of these areas will
have to be redefined, and the practice adapted accordingly, in order to
enable Linux in safety-critical applications.

Well, that's it off the top of my head.  It probably stirs up more dust,
but the dust was laying around anyway.

Cheers

John

[1]
https://drive.google.com/drive/folders/12DkfO3bRaPje75lG9XtoLBgtn383DezF

[2] https://www.hse.gov.uk/research/crr_pdf/2001/crr01336.pdf

On 23/04/2021 09:15, Roberto Bagnara wrote:
>
> Hi there.
>
> I guess we prefer waiting a little bit for the dust to settle,
> but a discussion on the topic seems to be unavoidable.
>
> Is it too soon to say the following, just to let the ball rolling?
>
> 1) Wouldn't the UMN people publish their paper, the thing may
>    have been discovered much later.
>
> 2) Patches can go all the way down to the stable trees also
>    because no sensible coding standard is enforced and static
>    analysis is not systematically used to detect patches introducing
>    defects and vulnerabilities.
>
> 3) Patches are accepted (also in the stable trees) with insufficient
>    peer review.  (The absence of a sensible, systematically enforced
>    coding standard is a contributory factor to insufficient peer review.)
>
> And, consequently:
>
> 4) In order to come up with something fit for use in safety application,
>    much has to be done for the redefinition of the development process.
>
> Kind regards,
>
>    Roberto
>
> Roberto Bagnara, Ph.D.
>
> CEO/CTO, BUGSENG (http://bugseng.com)
> Member, ISO/IEC JTC1/SC22/WG14 - C Standardization Working Group
> Member, MISRA C Working Group
> Mobile: +39 339 8593517
>
>
>
>
>






John MacGregor
 

Hi Wenhui

The point is that, in order to use Linux, manufacturers of safety-critical systems must ensure that they won't be sued for negligence and that they comply with regulatory standards.

In order to do that, they usually must demonstrate conformance to the relevant safety standards.

It doesn't appear that without the work the ELISA is doing, at least, they will be able to do that.

Whether Linux otherwise could be used in safety-critical systems is a bit beside the point.

Cheers

John

On 23/04/2021 12:00, Wenhui Zhang wrote:
IMHO, Linux could be runtime industry level acceptable to safety-critical systems.
While since source code of Linux is too large, it might be hard to ensure such a standard.
However, the compiler could easily kick out the out of tree code, and trim the Linux for a specific safety-critical system, which is called " dead code analysis".
Most of the bugs are on the out of tree code, I think. Then making Linux working for one safety-critical system is a tangible goal.
Then one question is, which part of the source code are commonly shared among all runtimes, and which part are specific to the particular application ( i.e. libc, arch and drivers)
On Fri, Apr 23, 2021, 5:44 AM John MacGregor <open.john.macgregor@gmail.com <mailto:open.john.macgregor@gmail.com>> wrote:
Hi Roberto, all
It may be too soon to discuss this event, but a number of issues have
been placed on the back burner, which could possibly be warmed up
again.
  If you interpret this as grandstanding, sorry.
First, let's put the incident in context.  For us, this is not a
security issue.  It means that patches of arbitrary quality can be
incorporated in the stable tree, including patches of dubious quality
and / or impact on safety.  This means that the integration process is
faulty.
I talked to Greg Kroah-Hartmann about this a couple of years ago and he
said that while in Paris, he was working with INRIA on a study of the
accuracy of the stable update process (i.e. his decision process). As I
remember (and I have a terrible memory for details), he said he was
only
right about 50% of the time, especially w.r.t type 1 and type 2 errors
(accepting a patch that wasn't necessary, rejecting a patch that was
necessary).  For the sake of sticking to the truth, if anyone can
provide this report, please speak up.
While I may use other sources for the following points, I have been
thinking, and saying, these things for... years, mostly.
1) As Lukas Bulwahn claimed in the September Workshop[1. approximately
minute 45]  Kernel development is not a project or a product, but a
consolidated change management process.  In other words, it process is
not a development process.
2) Kernel development does not have a quality management system
4) Linux was not developed for application in safety-critical systems.
3) The kernel is not "industrial-grade" software
   - As Jason Smith pointed out in a (very) recent ELISA blog post,
Linux is SOUP (Software of Unknown Provenance... for me Pedigree).  The
first instance of this term that I found was in a consultant's
report to
the British Government in 2001. That is, this is neither unknown nor
deniable.
   - Linux is miles away from being "QM" in the ISO 26262 sense
   - ==> Whatever subset of the 27 million lines of non-safety
critical
Linux source code that comes into a safety critical system is not QM,
and has not been developed according to the demands of safety
standards.
  Using it in a safety-critical system is probably questionable.
4) The Linux Kernel Project makes software available in the sense that
it is a somewhat coherent collection of source code.  Executable Linux
software comes from somewhere else or you make it yourself.
5) The criteria whether Linux, or open source software, is suitable for
use in safety-critical systems is the residual risk of failure in the
software as integrated into the system.
   - failures can be eliminated by a conformant development process
   - failures can be tolerated with appropriate compensation mechanisms
6) For me, the Linux development process has already stopped, and we're
in an operational phase.  At least, the overwhelming proportion of the
code used will be used unmodified. (Modification and upstreaming the
changes seems like a very tenuous approach to me). To an extent, this
means that system integrators are reusing Linux, not using it.
7)  I agree with your comments on the "development process" as far as
they go.
   - peer review
   - static testing
   - coding standards
yup.
8) As a continuation of the discussion we had in the sync telco this
week, the Linux process must be understood in the context of commonly
accepted (i.e. professional) systems, and software, engineering
practice.  And the role of Linux must be understood in the context of
this practice applied to the development of safety-critical embedded
systems.  Whether the Linux process has been successful in other
domains
is irrelevant.
9) Maybe the development process should be redefined, but first of all
the strict Linux Kernel process is not a development process in the
first place.  Second, I think the Linux community is satisfied with,
even proud of, its process as it is.  I'm not sure they want to have an
"industrial grade" one.
10) As we have said from the beginning of the project that there is a
tension between what the standards require, what safety-critical system
developers would consider to be acceptably safe and what the open
source
community is willing to do in this direction.  Each of these areas will
have to be redefined, and the practice adapted accordingly, in order to
enable Linux in safety-critical applications.
Well, that's it off the top of my head.  It probably stirs up more
dust,
but the dust was laying around anyway.
Cheers
John
[1]
https://drive.google.com/drive/folders/12DkfO3bRaPje75lG9XtoLBgtn383DezF
<https://drive.google.com/drive/folders/12DkfO3bRaPje75lG9XtoLBgtn383DezF>
[2] https://www.hse.gov.uk/research/crr_pdf/2001/crr01336.pdf
<https://www.hse.gov.uk/research/crr_pdf/2001/crr01336.pdf>
On 23/04/2021 09:15, Roberto Bagnara wrote:
>
> Hi there.
>
> I guess we prefer waiting a little bit for the dust to settle,
> but a discussion on the topic seems to be unavoidable.
>
> Is it too soon to say the following, just to let the ball rolling?
>
> 1) Wouldn't the UMN people publish their paper, the thing may
>    have been discovered much later.
>
> 2) Patches can go all the way down to the stable trees also
>    because no sensible coding standard is enforced and static
>    analysis is not systematically used to detect patches introducing
>    defects and vulnerabilities.
>
> 3) Patches are accepted (also in the stable trees) with insufficient
>    peer review.  (The absence of a sensible, systematically enforced
>    coding standard is a contributory factor to insufficient peer
review.)
>
> And, consequently:
>
> 4) In order to come up with something fit for use in safety
application,
>    much has to be done for the redefinition of the development
process.
>
> Kind regards,
>
>    Roberto
>
> Roberto Bagnara, Ph.D.
>
> CEO/CTO, BUGSENG (http://bugseng.com <http://bugseng.com>)
> Member, ISO/IEC JTC1/SC22/WG14 - C Standardization Working Group
> Member, MISRA C Working Group
> Mobile: +39 339 8593517
>
>
>
>
>