On Tue, Sep 14, 2021 at 3:05 PM Paul Albertella
<paul.albertella@...> wrote:

Hi,

The Development Process working group has been discussing proposals for
the evolution of the group. The WG has historically attempted to cover a
number of goals in a single context, which has occasionally created
confusion; to address this, we propose to split into two separate
working groups, each with a clearer focus.

Elana Copperman and I have proposed mission statements and planned
activities for these two new groups, which have been discussed in the
Dev Process WG weekly calls.

Having reached some consensus about a way forward in that forum, I'd now
like to share my proposal (see below) with the wider ELISA community for
comment, before seeking to formalise the group at the TSC. I hope that
Elana will share details of her proposal in due course.

Regards,

Paul

Paul, thanks for pushing this forward and not giving up.

On the scope, I fully agree with your proposal for a new working
group; you can probably find some emails a few years back pushing for
a working group going in that direction (the proposal for an "Evidence
WG" goes far back in history). So, I am all in... except for the
name...

So, that drives me directly to the bikeshedding on naming...

---------------------------------------------------------------------

Proposed working group name
---------------------------

Safety Engineering Process

I dislike the name for the following reasons:

With my personal interpretation of the Working Group's description, I
understand:

For me, safety engineering is "system engineering to modify an early
version of a system description", "safety analysis in the scope of a
specific system" and "structuring the created insights suitable for an
safety certification assessor". None of the three aspects are really
the core of WG. In detail:

While the results the WG creates eventually contributes to a safety
engineering process (at the integrator's site and desk), the large
part (let's say 90%) of safety engineering and the definition and
establishment of the safety engineering process really happens outside
of this working group.
E.g., I think the Automotive and Medical WG are much more involved in
safety engineering and need to determine the safety engineering
process they employ, e.g., how they write down their safety analysis,
how they execute STPA (as one safety analysis method the Medical WG
currently employs). So, they are much more a "Safety Engineering
Process" working group than what I read out of your WG description.

Also, your description states "specifying how a given claim and its
supporting evidence may be used as part of a certification process
should be considered out of scope", which really is the third
essential part of a safety engineering process for me.

My preference on naming, hence, would be on:

- Evidence WG
- Claims on Open-Source Software WG
- Open-Source Engineering Process WG

The name is important, just for newcomers and outsiders not to set the
wrong expectations.
The detailed working group description states your goal clearly; it is
just that name is not a good fit IMHO.

Proposed WG chair
-----------------

Paul Albertella

Proposed meeting schedule
-------------------------

Weekly on Thursday, in the same slot as the current Dev process call

Proposed mission statement
--------------------------

This working group aims to examine safety-related claims that we might
like to make about Linux as part of a system, and to explore how we can
gather and present evidence to support such claims as part of a safety
engineering process.

We are interested in two broad classes of claims and evidence:

a) Those relating to development practices for Linux as a whole; for
example, the peer review process for patches, the testing performed on a
kernel stable branch when preparing a release, or the testing performed
by a system integrator for a product that incorporates Linux

b) Those relating to specific properties or behaviour of Linux; for
example, features that we can enable or disable in a kernel config, the
(inferred) design of a subsystem, the characteristics of a driver, or
tests that can verify aspects of a given feature

For (b), we will focus on engineering process aspects (construction,
verification, integration, use and maintenance) relating to the feature
or property, rather than technical details of its design/implementation.

In both cases we may examine practices or processes that are undertaken
by the Linux kernel community, as well as communities or organisations
that consume Linux (e.g. AGL, LTP), or we may identify practices or
processes from other sources that might be used by organisations who
want to use Linux to build systems with safety requirements.

Just a detail:

I do not know if LTP (it stands for Linux Test Project, right?) is a
community or organisation; I would really just think LTP stands for
the test suite itself, i.e., the source code in the repository. The
community for LTP is not that large and there is unfortunately hardly
a strong organisation behind that. A few years back, when I last
checked the few main developers were not employed for that purpose and
did it in their spare time (nowadays, I think Suse and RedHat are
employing them... but still not a large community).

I think "the Debian community" might serve as a better example here.
They do have practices, and do organize themselves (e.g. Debian
conferences, etc.).

Well, just a nit...

Planned activities
------------------

Our objectives are to:

* Define claims that we would like to make about Linux and/or the
processes used to develop it or integrate it as part of a product
* Identify evidence that we might use to support these claims
* Identify strategies or tools that we can use to gather, generate or
reformulate evidence (perhaps in collaboration with the Tools WG)

We are of course always happy to collaborate and see what we (or our
high-performance server) can do for you.

* Examine the evidence gathered and document our findings
* Share and peer-review our findings in an elisa.tech Github repo

Possible claims might be proposed by another WG, perhaps in relation to
a specific use case, technique or safety argument, or we may select our
own. Claims that we examine may not all be safety claims, per se; some
might relate to software quality, for example, or functionality that
does not have an explicit safety role, since both of these may be needed
to support safety arguments.

I think this observation on the nature of claims is already quite a
valuable one.

We will use contributors’ knowledge of specific safety standards to
inform our work where appropriate - for example to determine what
criteria to apply for a given class of evidence - but specifying how a
given claim and its supporting evidence may be used as part of a
certification process should be considered out of scope.

Where we do not have consensus on results or conclusions, we will
document the different perspectives and/or open questions. We will
manage the material produced in an elisa.tech Github repo and
collectively review it in GitHub PRs, inviting review or input from the
rest of the ELISA community where appropriate.

I think that is a very important aspect for a working group in the
ELISA Project and should be the best practice for all WGs here. Some
have been more successful than others in that regard.

As part of this work, we also plan to write a summary of the results and
conclusions of the work done by the Development Process WG, which will
also be made available in an elisa.tech Github repo. This is intended to
provide background for new participants and other WGs, as well as a
baseline / conceptual framework for the work of the new group.

I think that is quite important; it might also be a separate group
(the continuation of the development process WG; those that have
invested and learned something in that group) but certainly should
happen. I would be very interested in the overall summary, results and
conclusions. I got lost on the way on what the consensus and lessons
learned really were in the course of all the discussions.

Collaboration
-------------

The group expects to collaborate with the existing working groups and
with the new group proposed by Elana, both to identify or discuss claims
and evidence for analysis, and to address wider questions relating to
safety engineering processes. Where we work with another group on given
topics, we may jointly present the results at an Elisa workshop.

Thanks again, Paul, for your investment and moving this forward. I
hope to see some previous work results from the students I mentored
being picked up in this new group.

Lukas