On Thu, Apr 18, 2019 at 09:14:41AM +0000, Lukas Bulwahn wrote:

Hi all,

From the previous discussion, I created a new Task Proposal. It is rather a Preparation Task for further activities.


Goal: Define Assumed System and Requirements to provide a starting point for further activities.



Rationale: Many further activities depend on the definition of an assumed system and safety requirements, including hardware selection, kernel version selection, kernel configuration selection.


Needed activities



1. Define a generic assumed software/system architecture and requirements

Lukas - this is right back to the fellacy we start at 5 years ago
that will not work - if we do it that way - why would the elements chosen
based on crude assumptions miracously match the actual needs ?
notably when chosen at the functional level not at the safety level ?

I believe that the only suitable starting point is "Intent" and derive the
needs in a technology agnostic manner - identifying all potential safety
issues and security issues along the way - and THEN we can start selecting
element cadidates.

A side node - NONE of them willl be generic - not in the first run - maybe
after we built 5-6 ssystems some elements will emerge as generic - but not from the start.

2. Define which hardware shall be assumed, for which a hardware safety manual is available

That list is known to be short - so how about turning this around a bit and
deriving the interaction points between HW and SW - with other words record
where we are making any HW assumptions so that we can re-open that issue
as soon as a real HW exists with all its limitatoins and possibly
unconsidered capabilities. MAtching against sasumptions is futil at this point
nobody knows what the HW will look like and notably what constraints
the HW manual might contain - if only we could convince one HW vendor to
stop trying to build SEooC and cooordinate there HW assurance process with
the SW assurance process - the synergies (in context of a specific Use-Case/Intent)
are, I believe, significant

3. Define a suitable kernel configuration for this setup with proper requirements traceability to allow others to understand the selected configuration.

3 would need to iterate over all element candidates rather than try to
define a kernel configuratoin only - if you do this for one configuration - what is
the probability of getting it right ? msot likely closs to 0
but we can develop the potentials with few constraints - e.g
1) what could seccomp provide - und constraint XYZ
2) what could CONFIG_ABC provide - under constraint ..

This will not be context agnostic building blocks but about as close
to that ideal as we can get - the evaluation is then in context of a specific
defined Use-Case (and in no way generic)

thx!
hofrat

Work Product

Documentation of assumed system and software and safety requirements in a suitable requirement engineering tool/format.
Extract of relevant sections related to the operating system from the hardware safety manual
Requirements Trace from requirements to Kernel Config Fragments.
Kernel configuration

Milestones tbd.

Best regards,

Lukas

Dear Nicholas,

19.04.2019, 12:00, "Nicholas Mc Guire" <der.herr@...>:

On Thu, Apr 18, 2019 at 09:14:41AM +0000, Lukas Bulwahn wrote:

 Hi all,

 From the previous discussion, I created a new Task Proposal. It is rather a Preparation Task for further activities.

 Goal: Define Assumed System and Requirements to provide a starting point for further activities.

 Rationale: Many further activities depend on the definition of an assumed system and safety requirements, including hardware selection, kernel version selection, kernel configuration selection.

 Needed activities

 1. Define a generic assumed software/system architecture and requirements

Lukas - this is right back to the fellacy we start at 5 years ago
that will not work - if we do it that way - why would the elements chosen
based on crude assumptions miracously match the actual needs ?
notably when chosen at the functional level not at the safety level ?

Who prevents you to discuss actual needs with ELISA stockholders? I guess that they can formulate high-level requirements, e.g. one may need support for multi-core CPUs (so, there are corresponding requirements, configurations and components), another one may need networking and cryptography but they may be run within an isolated container, etc.

I see the goal of ELISA not to demonstrate safety for an abstract base (especially, because of this is not likely possible at a very good level in a reasonable time) but to demonstrate safety for existing OS and library functionality that is highly required. Obviously, it is better to ensure safety for the latter cooperatively rather than to develop it again independently.

Unfortunately, it may turn out that ensuring required levels of safety for chosen functionality can be too expensive, but it is better to know in advance.

I believe that the only suitable starting point is "Intent" and derive the
needs in a technology agnostic manner - identifying all potential safety
issues and security issues along the way - and THEN we can start selecting
element cadidates.

A side node - NONE of them willl be generic - not in the first run - maybe
after we built 5-6 ssystems some elements will emerge as generic - but not from the start.

 2. Define which hardware shall be assumed, for which a hardware safety manual is available

That list is known to be short - so how about turning this around a bit and
deriving the interaction points between HW and SW - with other words record
where we are making any HW assumptions so that we can re-open that issue
as soon as a real HW exists with all its limitatoins and possibly
unconsidered capabilities. MAtching against sasumptions is futil at this point
nobody knows what the HW will look like and notably what constraints
the HW manual might contain - if only we could convince one HW vendor to
stop trying to build SEooC and cooordinate there HW assurance process with
the SW assurance process - the synergies (in context of a specific Use-Case/Intent)
are, I believe, significant

I am wondering are these problems new in context of safety?

Indeed, I guess that you should estimate potential risks related to hardware, e.g. if one would like to use NVMe SSDs you should estimate issues with an increasing throughput and develop additional test cases. Perhaps, it is easy to say but very hard to support.

 3. Define a suitable kernel configuration for this setup with proper requirements traceability to allow others to understand the selected configuration.

3 would need to iterate over all element candidates rather than try to
define a kernel configuratoin only - if you do this for one configuration - what is
the probability of getting it right ? msot likely closs to 0
but we can develop the potentials with few constraints - e.g
1) what could seccomp provide - und constraint XYZ
2) what could CONFIG_ABC provide - under constraint ..

This will not be context agnostic building blocks but about as close
to that ideal as we can get - the evaluation is then in context of a specific
defined Use-Case (and in no way generic)

thx!
hofrat

 Work Product

 Documentation of assumed system and software and safety requirements in a suitable requirement engineering tool/format.
 Extract of relevant sections related to the operating system from the hardware safety manual
 Requirements Trace from requirements to Kernel Config Fragments.
 Kernel configuration

 Milestones tbd.

 Best regards,

 Lukas

Hi Nicholas,

(sorry for the clumsy inline comments; I tried my best with the broken tools I get provided.)

On Thu, Apr 18, 2019 at 09:14:41AM +0000, Lukas Bulwahn wrote:

1. Define a generic assumed software/system architecture and requirements

Lukas - this is right back to the fellacy we start at 5 years ago
that will not work - if we do it that way - why would the elements chosen
based on crude assumptions miracously match the actual needs ?
notably when chosen at the functional level not at the safety level ?

I believe that the only suitable starting point is "Intent" and derive the
needs in a technology agnostic manner - identifying all potential safety
issues and security issues along the way - and THEN we can start selecting
element cadidates.


A side node - NONE of them willl be generic - not in the first run - maybe
after we built 5-6 ssystems some elements will emerge as generic - but not from the start.

Nicholas, I think you picked up on the word "generic" than I actually intended to say. But yes, I agree the word "generic" must be used with care and I should drop it and replace it with "incomplete" or "non-derived" and "collaboratively agreed with common understanding".

So, I will try to be more specific about the intentions of this activity (as I see it) and level of needed granularity:

1. Further activities that only make sense with any assumed system and kernel configuration:
- Safety impact analysis: crucial to determine if a bug is a safety goal violation or not; we need at least some assumed requirements to exercise this activity.
- Root cause analysis: assuming that certain subsystems are excluded helps to identify the actual relevant root causes.
- Static analysis assessment and tracking: you need to know roughly the assumed source code base to be relevant.

For example, we might generally assume the system to be fail-safe. So, a kernel oops is not a safety violation. For example, we do not intend to include a floppy disk driver; so, syzkaller findings in that subsystem are less relevant than in the scheduler.


2. Social aspect: Come to a common agreement and understanding on core aspects. Without further investigation of social studies, I believe that within a group that exchanged and agreed on core aspects, the willingness to collaborate is higher. This also includes defending the work towards management and supporting each other in the various activities. We need to achieve a collaborative working mode, not a mode of unrelated parallel and conflicting and competing activities.

I want this discussion to start to then move the activities forward WITH the support of ALL stakeholders.

2. Define which hardware shall be assumed, for which a hardware safety manual is available

That list is known to be short - so how about turning this around a bit and
deriving the interaction points between HW and SW - with other words record
where we are making any HW assumptions so that we can re-open that issue
as soon as a real HW exists with all its limitatoins and possibly
unconsidered capabilities. MAtching against sasumptions is futil at this point
nobody knows what the HW will look like and notably what constraints
the HW manual might contain - if only we could convince one HW vendor to
stop trying to build SEooC and cooordinate there HW assurance process with
the SW assurance process - the synergies (in context of a specific Use-Case/Intent)
are, I believe, significant

Agree. The goal here is to start this discussion with the hardware vendors and get them involved.

3. Define a suitable kernel configuration for this setup with proper requirements traceability to allow others to understand the selected configuration.

3 would need to iterate over all element candidates rather than try to
define a kernel configuratoin only - if you do this for one configuration - what is
the probability of getting it right ? msot likely closs to 0
but we can develop the potentials with few constraints - e.g
1) what could seccomp provide - und constraint XYZ
2) what could CONFIG_ABC provide - under constraint ..

This will not be context agnostic building blocks but about as close
to that ideal as we can get - the evaluation is then in context of a specific
defined Use-Case (and in no way generic)

In SIL2LinuxMP, we had this x86_minimal_config.
This was by its process creation mostly "assumed" and in fact, we never even wrote down the high-level properties and the general assumptions of how we came to the configuration. It is probably still all in your head.
Then, in various activities, we used this configuration for various analysis and measurements.

I am considering something similar here, again, just better documented a bit better and commonly and collaboratively agreed for further activities.

Again, the technical task is one thing. The more important task is the social activity: Everyone should agree on some same kernel configuration setup and hence a common basis for the collaborative effort. It might be painful, but that is life: collaboration with others and reality always include comprises, but then, once you committed to that configuration, everyone knows what they are considering and working on.

That what you call "potentials with constraints", I would consider a subtree of the requirements tree from "feature" requirements to "kernel configuration fragments".

I also expect that a proper requirements tree would then allow to carefully split out one subtree of this kernel configuration with some careful expert operation, of course. The re-use of that requirements tree is limited. I am more interested in reusing some of the analysis results or at least, claiming that certain statistics are not completely skewed for another similar configuration.

Best regards,


Lukas

Dear Lukas,

I am wondering why you do not consider old good testing that is strictly necessary for ensuring good levels of safety (if I understand properly)? I suggest that this is a challenge that nobody could solve yet. Developing detailed requirements and corresponding tests may be the most expensive work in this project, thus, it influences everything very much.

Unfortunately, I can not reason accurately about various issues of Linux kernel testing, but there should be appropriate experts in this list.

Best regards,
Evgeny

23.04.2019, 11:37, "Lukas Bulwahn" <lukas.bulwahn@...>:

Agree. I should have also mentioned testing. I am not sure if the testing really has much influence on other activities, though.

It is clear however that we need to assume a system context and kernel configuration to identify the relevant parts and properties that shall be under test.

Kernel CI, Kernel selftest and Kernel unit testing should be mentioned here.

Lukas

-----Ursprüngliche Nachricht-----
Von: devel@... [mailto:devel@...] Im Auftrag von Evgeny Novikov
Gesendet: Dienstag, 23. April 2019 21:00
An: devel@...
Betreff: Re: [devel] Task Proposal: Define Assumed System and Requirements

Dear Lukas,

I am wondering why you do not consider old good testing that is strictly necessary for ensuring good levels of safety (if I understand properly)? I suggest that this is a challenge that nobody could solve yet. Developing detailed requirements and corresponding tests may be the most expensive work in this project, thus, it influences everything very much.

Unfortunately, I can not reason accurately about various issues of Linux kernel testing, but there should be appropriate experts in this list.

Best regards,
Evgeny

23.04.2019, 11:37, "Lukas Bulwahn" <lukas.bulwahn@...>:

On Tue, 23 Apr 2019, Evgeny Novikov wrote:

Dear Lukas,

I am wondering why you do not consider old good testing that is strictly necessary for ensuring good levels of safety (if I understand properly)? I suggest that this is a challenge that nobody could solve yet. Developing detailed requirements and corresponding tests may be the most expensive work in this project, thus, it influences everything very much.

What sense does testing make on a moving target which has an endless
state space?

when you say influences everything very much, what do you mean?

Wouldn't it be a much more elegant and interesting challenge to avoid
this cost intensive work (testing)?

\BR
Markus

Unfortunately, I can not reason accurately about various issues of Linux kernel testing, but there should be appropriate experts in this list.

Best regards,
Evgeny

23.04.2019, 11:37, "Lukas Bulwahn" <lukas.bulwahn@...>:

Hi Nicholas,

(sorry for the clumsy inline comments; I tried my best with the broken tools I get provided.)

On Thu, Apr 18, 2019 at 09:14:41AM +0000, Lukas Bulwahn wrote:

 1. Define a generic assumed software/system architecture and requirements

 Lukas - this is right back to the fellacy we start at 5 years ago
 that will not work - if we do it that way - why would the elements chosen
 based on crude assumptions miracously match the actual needs ?
 notably when chosen at the functional level not at the safety level ?

 I believe that the only suitable starting point is "Intent" and derive the
 needs in a technology agnostic manner - identifying all potential safety
 issues and security issues along the way - and THEN we can start selecting
 element cadidates.

 A side node - NONE of them willl be generic - not in the first run - maybe
 after we built 5-6 ssystems some elements will emerge as generic - but not from the start.

Nicholas, I think you picked up on the word "generic" than I actually intended to say. But yes, I agree the word "generic" must be used with care and I should drop it and replace it with "incomplete" or "non-derived" and "collaboratively agreed with common understanding".

So, I will try to be more specific about the intentions of this activity (as I see it) and level of needed granularity:

1. Further activities that only make sense with any assumed system and kernel configuration:
  - Safety impact analysis: crucial to determine if a bug is a safety goal violation or not; we need at least some assumed requirements to exercise this activity.
  - Root cause analysis: assuming that certain subsystems are excluded helps to identify the actual relevant root causes.
  - Static analysis assessment and tracking: you need to know roughly the assumed source code base to be relevant.

For example, we might generally assume the system to be fail-safe. So, a kernel oops is not a safety violation. For example, we do not intend to include a floppy disk driver; so, syzkaller findings in that subsystem are less relevant than in the scheduler.

2. Social aspect: Come to a common agreement and understanding on core aspects. Without further investigation of social studies, I believe that within a group that exchanged and agreed on core aspects, the willingness to collaborate is higher. This also includes defending the work towards management and supporting each other in the various activities. We need to achieve a collaborative working mode, not a mode of unrelated parallel and conflicting and competing activities.

I want this discussion to start to then move the activities forward WITH the support of ALL stakeholders.

 2. Define which hardware shall be assumed, for which a hardware safety manual is available

 That list is known to be short - so how about turning this around a bit and
 deriving the interaction points between HW and SW - with other words record
 where we are making any HW assumptions so that we can re-open that issue
 as soon as a real HW exists with all its limitatoins and possibly
 unconsidered capabilities. MAtching against sasumptions is futil at this point
 nobody knows what the HW will look like and notably what constraints
 the HW manual might contain - if only we could convince one HW vendor to
 stop trying to build SEooC and cooordinate there HW assurance process with
 the SW assurance process - the synergies (in context of a specific Use-Case/Intent)
 are, I believe, significant

Agree. The goal here is to start this discussion with the hardware vendors and get them involved.

 3. Define a suitable kernel configuration for this setup with proper requirements traceability to allow others to understand the selected configuration.

 3 would need to iterate over all element candidates rather than try to
 define a kernel configuratoin only - if you do this for one configuration - what is
 the probability of getting it right ? msot likely closs to 0
 but we can develop the potentials with few constraints - e.g
 1) what could seccomp provide - und constraint XYZ
 2) what could CONFIG_ABC provide - under constraint ..

 This will not be context agnostic building blocks but about as close
 to that ideal as we can get - the evaluation is then in context of a specific
 defined Use-Case (and in no way generic)

In SIL2LinuxMP, we had this x86_minimal_config.
This was by its process creation mostly "assumed" and in fact, we never even wrote down the high-level properties and the general assumptions of how we came to the configuration. It is probably still all in your head.
Then, in various activities, we used this configuration for various analysis and measurements.

I am considering something similar here, again, just better documented a bit better and commonly and collaboratively agreed for further activities.

Again, the technical task is one thing. The more important task is the social activity: Everyone should agree on some same kernel configuration setup and hence a common basis for the collaborative effort. It might be painful, but that is life: collaboration with others and reality always include comprises, but then, once you committed to that configuration, everyone knows what they are considering and working on.

That what you call "potentials with constraints", I would consider a subtree of the requirements tree from "feature" requirements to "kernel configuration fragments".

I also expect that a proper requirements tree would then allow to carefully split out one subtree of this kernel configuration with some careful expert operation, of course. The re-use of that requirements tree is limited. I am more interested in reusing some of the analysis results or at least, claiming that certain statistics are not completely skewed for another similar configuration.

Best regards,

Lukas

On Tue, 23 Apr 2019, Lukas Bulwahn wrote:

Hi Nicholas,

(sorry for the clumsy inline comments; I tried my best with the broken tools I get provided.)

On Thu, Apr 18, 2019 at 09:14:41AM +0000, Lukas Bulwahn wrote:

1. Define a generic assumed software/system architecture and requirements

Lukas - this is right back to the fellacy we start at 5 years ago
that will not work - if we do it that way - why would the elements chosen
based on crude assumptions miracously match the actual needs ?
notably when chosen at the functional level not at the safety level ?

I believe that the only suitable starting point is "Intent" and derive the
needs in a technology agnostic manner - identifying all potential safety
issues and security issues along the way - and THEN we can start selecting
element cadidates.


A side node - NONE of them willl be generic - not in the first run - maybe
after we built 5-6 ssystems some elements will emerge as generic - but not from the start.

Nicholas, I think you picked up on the word "generic" than I actually intended to say. But yes, I agree the word "generic" must be used with care and I should drop it and replace it with "incomplete" or "non-derived" and "collaboratively agreed with common understanding".

So, I will try to be more specific about the intentions of this activity (as I see it) and level of needed granularity:

1. Further activities that only make sense with any assumed system and kernel configuration:
- Safety impact analysis: crucial to determine if a bug is a safety goal violation or not; we need at least some assumed requirements to exercise this activity.
- Root cause analysis: assuming that certain subsystems are excluded helps to identify the actual relevant root causes.
- Static analysis assessment and tracking: you need to know roughly the assumed source code base to be relevant.

For example, we might generally assume the system to be fail-safe. So, a kernel oops is not a safety violation. For example, we do not intend to include a floppy disk driver; so, syzkaller findings in that subsystem are less relevant than in the scheduler.


2. Social aspect: Come to a common agreement and understanding on core aspects. Without further investigation of social studies, I believe that within a group that exchanged and agreed on core aspects, the willingness to collaborate is higher. This also includes defending the work towards management and supporting each other in the various activities. We need to achieve a collaborative working mode, not a mode of unrelated parallel and conflicting and competing activities.

I want this discussion to start to then move the activities forward WITH the support of ALL stakeholders.

2. Define which hardware shall be assumed, for which a hardware safety manual is available

That list is known to be short - so how about turning this around a bit and
deriving the interaction points between HW and SW - with other words record
where we are making any HW assumptions so that we can re-open that issue
as soon as a real HW exists with all its limitatoins and possibly
unconsidered capabilities. MAtching against sasumptions is futil at this point
nobody knows what the HW will look like and notably what constraints
the HW manual might contain - if only we could convince one HW vendor to
stop trying to build SEooC and cooordinate there HW assurance process with
the SW assurance process - the synergies (in context of a specific Use-Case/Intent)
are, I believe, significant

Agree. The goal here is to start this discussion with the hardware vendors and get them involved.

3. Define a suitable kernel configuration for this setup with proper requirements traceability to allow others to understand the selected configuration.

3 would need to iterate over all element candidates rather than try to
define a kernel configuratoin only - if you do this for one configuration - what is
the probability of getting it right ? msot likely closs to 0
but we can develop the potentials with few constraints - e.g
1) what could seccomp provide - und constraint XYZ
2) what could CONFIG_ABC provide - under constraint ..

This will not be context agnostic building blocks but about as close
to that ideal as we can get - the evaluation is then in context of a specific
defined Use-Case (and in no way generic)

In SIL2LinuxMP, we had this x86_minimal_config.
This was by its process creation mostly "assumed" and in fact, we never even wrote down the high-level properties and the general assumptions of how we came to the configuration. It is probably still all in your head.
Then, in various activities, we used this configuration for various analysis and measurements.

I am considering something similar here, again, just better documented a bit better and commonly and collaboratively agreed for further activities.

x86_minimal_config was an (OT) internal working configuration created
because SIL2LinuxMP partner did deliver a configuration. We needed some
starting point which was according the developed architecture to make our
studies comparable (eg. PIT). Thats the reason we refered to "a minimal
i7 configuration".

Again, the technical task is one thing. The more important task is the social activity: Everyone should agree on some same kernel configuration setup and hence a common basis for the collaborative effort. It might be painful, but that is life: collaboration with others and reality always include comprises, but then, once you committed to that configuration, everyone knows what they are considering and working on.

That what you call "potentials with constraints", I would consider a subtree of the requirements tree from "feature" requirements to "kernel configuration fragments".

I also expect that a proper requirements tree would then allow to carefully split out one subtree of this kernel configuration with some careful expert operation, of course. The re-use of that requirements tree is limited. I am more interested in reusing some of the analysis results or at least, claiming that certain statistics are not completely skewed for another similar configuration.

Best regards,


Lukas

Hi Markus,

Wouldn't it be a much more elegant and interesting challenge to avoid this cost intensive work (testing)?

No, testing is required for Safety Standards and we can run tests automated (one we have created them).
They are a good indicator for changes and also for developers, to see if they have "damaged" something
by their changes.

Kind regards,
Oscar

-----Ursprüngliche Nachricht-----
Von: devel@... [mailto:devel@...] Im Auftrag von markus
Gesendet: Mittwoch, 24. April 2019 19:10
An: devel@...
Betreff: Re: [devel] Task Proposal: Define Assumed System and Requirements

On Tue, 23 Apr 2019, Evgeny Novikov wrote:

Dear Lukas,

I am wondering why you do not consider old good testing that is strictly necessary for ensuring good levels of safety (if I understand properly)? I suggest that this is a challenge that nobody could solve yet. Developing detailed requirements and corresponding tests may be the most expensive work in this project, thus, it influences everything very much.

What sense does testing make on a moving target which has an endless state space?

when you say influences everything very much, what do you mean?

\BR
Markus

Unfortunately, I can not reason accurately about various issues of Linux kernel testing, but there should be appropriate experts in this list.

Best regards,
Evgeny

23.04.2019, 11:37, "Lukas Bulwahn" <lukas.bulwahn@...>:

Hi Nicholas,

(sorry for the clumsy inline comments; I tried my best with the
broken tools I get provided.)

On Thu, Apr 18, 2019 at 09:14:41AM +0000, Lukas Bulwahn wrote:

 1. Define a generic assumed software/system architecture and
requirements

 Lukas - this is right back to the fellacy we start at 5 years ago
 that will not work - if we do it that way - why would the elements
chosen
 based on crude assumptions miracously match the actual needs ?
 notably when chosen at the functional level not at the safety level ?

 I believe that the only suitable starting point is "Intent" and
derive the
 needs in a technology agnostic manner - identifying all potential
safety
 issues and security issues along the way - and THEN we can start
selecting
 element cadidates.

 A side node - NONE of them willl be generic - not in the first run
- maybe
 after we built 5-6 ssystems some elements will emerge as generic - but not from the start.

Nicholas, I think you picked up on the word "generic" than I actually intended to say. But yes, I agree the word "generic" must be used with care and I should drop it and replace it with "incomplete" or "non-derived" and "collaboratively agreed with common understanding".

So, I will try to be more specific about the intentions of this activity (as I see it) and level of needed granularity:

1. Further activities that only make sense with any assumed system and kernel configuration:
  - Safety impact analysis: crucial to determine if a bug is a safety goal violation or not; we need at least some assumed requirements to exercise this activity.
  - Root cause analysis: assuming that certain subsystems are excluded helps to identify the actual relevant root causes.
  - Static analysis assessment and tracking: you need to know roughly the assumed source code base to be relevant.

For example, we might generally assume the system to be fail-safe. So, a kernel oops is not a safety violation. For example, we do not intend to include a floppy disk driver; so, syzkaller findings in that subsystem are less relevant than in the scheduler.

2. Social aspect: Come to a common agreement and understanding on core aspects. Without further investigation of social studies, I believe that within a group that exchanged and agreed on core aspects, the willingness to collaborate is higher. This also includes defending the work towards management and supporting each other in the various activities. We need to achieve a collaborative working mode, not a mode of unrelated parallel and conflicting and competing activities.

I want this discussion to start to then move the activities forward WITH the support of ALL stakeholders.

 2. Define which hardware shall be assumed, for which a hardware
safety manual is available

 That list is known to be short - so how about turning this around
a bit and
 deriving the interaction points between HW and SW - with other
words record
 where we are making any HW assumptions so that we can re-open that
issue
 as soon as a real HW exists with all its limitatoins and possibly
 unconsidered capabilities. MAtching against sasumptions is futil
at this point
 nobody knows what the HW will look like and notably what
constraints
 the HW manual might contain - if only we could convince one HW
vendor to
 stop trying to build SEooC and cooordinate there HW assurance
process with
 the SW assurance process - the synergies (in context of a specific
Use-Case/Intent)
 are, I believe, significant

Agree. The goal here is to start this discussion with the hardware vendors and get them involved.

 3. Define a suitable kernel configuration for this setup with proper requirements traceability to allow others to understand the selected configuration.

 3 would need to iterate over all element candidates rather than
try to
 define a kernel configuratoin only - if you do this for one
configuration - what is
 the probability of getting it right ? msot likely closs to 0
 but we can develop the potentials with few constraints - e.g
 1) what could seccomp provide - und constraint XYZ
 2) what could CONFIG_ABC provide - under constraint ..

 This will not be context agnostic building blocks but about as
close
 to that ideal as we can get - the evaluation is then in context of
a specific
 defined Use-Case (and in no way generic)

In SIL2LinuxMP, we had this x86_minimal_config.
This was by its process creation mostly "assumed" and in fact, we never even wrote down the high-level properties and the general assumptions of how we came to the configuration. It is probably still all in your head.
Then, in various activities, we used this configuration for various analysis and measurements.

I am considering something similar here, again, just better documented a bit better and commonly and collaboratively agreed for further activities.

Again, the technical task is one thing. The more important task is the social activity: Everyone should agree on some same kernel configuration setup and hence a common basis for the collaborative effort. It might be painful, but that is life: collaboration with others and reality always include comprises, but then, once you committed to that configuration, everyone knows what they are considering and working on.

That what you call "potentials with constraints", I would consider a subtree of the requirements tree from "feature" requirements to "kernel configuration fragments".

I also expect that a proper requirements tree would then allow to carefully split out one subtree of this kernel configuration with some careful expert operation, of course. The re-use of that requirements tree is limited. I am more interested in reusing some of the analysis results or at least, claiming that certain statistics are not completely skewed for another similar configuration.

Best regards,

Lukas

Dear Markus,

24.04.2019, 20:10, "markus" <mkreidl@...>:

On Tue, 23 Apr 2019, Evgeny Novikov wrote:

 Dear Lukas,

 I am wondering why you do not consider old good testing that is strictly necessary for ensuring good levels of safety (if I understand properly)? I suggest that this is a challenge that nobody could solve yet. Developing detailed requirements and corresponding tests may be the most expensive work in this project, thus, it influences everything very much.

What sense does testing make on a moving target which has an endless
state space?

Isn't this the case in avionics and automotive software? If this is the case there, then why they consider testing mandatory still?

I think that in general the Linux kernel is much more difficult than software that is currently used in safety-critical applications. But actually you can tailor it very much, so, it is still be very useful, but this will simplify assessment of quality. Then you will be able to enable more and more features adding necessary things for ensuring safety, e.g. new requirements and test cases, as Oscar suggested.

when you say influences everything very much, what do you mean?

More features you enable, much more efforts will be necessary to demonstrate a necessary level of quality (if we are speaking about safety-critical applications, not about just running existing static analyzers).

Wouldn't it be a much more elegant and interesting challenge to avoid
this cost intensive work (testing)?

It would be great if you will be able to replace testing with something equivalent in quality but cheaper. I do not think that you are able to do this. As far as I know even most advanced deductive verification does not replace testing.

\BR
Markus

 Unfortunately, I can not reason accurately about various issues of Linux kernel testing, but there should be appropriate experts in this list.

 Best regards,
 Evgeny

 23.04.2019, 11:37, "Lukas Bulwahn" <lukas.bulwahn@...>:
 > Hi Nicholas,
 >
 > (sorry for the clumsy inline comments; I tried my best with the broken tools I get provided.)
 >
 > On Thu, Apr 18, 2019 at 09:14:41AM +0000, Lukas Bulwahn wrote:
 >>>  1. Define a generic assumed software/system architecture and requirements
 >>  Lukas - this is right back to the fellacy we start at 5 years ago
 >>  that will not work - if we do it that way - why would the elements chosen
 >>  based on crude assumptions miracously match the actual needs ?
 >>  notably when chosen at the functional level not at the safety level ?
 >>
 >>  I believe that the only suitable starting point is "Intent" and derive the
 >>  needs in a technology agnostic manner - identifying all potential safety
 >>  issues and security issues along the way - and THEN we can start selecting
 >>  element cadidates.
 >>
 >>  A side node - NONE of them willl be generic - not in the first run - maybe
 >>  after we built 5-6 ssystems some elements will emerge as generic - but not from the start.
 >
 > Nicholas, I think you picked up on the word "generic" than I actually intended to say. But yes, I agree the word "generic" must be used with care and I should drop it and replace it with "incomplete" or "non-derived" and "collaboratively agreed with common understanding".
 >
 > So, I will try to be more specific about the intentions of this activity (as I see it) and level of needed granularity:
 >
 > 1. Further activities that only make sense with any assumed system and kernel configuration:
 >   - Safety impact analysis: crucial to determine if a bug is a safety goal violation or not; we need at least some assumed requirements to exercise this activity.
 >   - Root cause analysis: assuming that certain subsystems are excluded helps to identify the actual relevant root causes.
 >   - Static analysis assessment and tracking: you need to know roughly the assumed source code base to be relevant.
 >
 > For example, we might generally assume the system to be fail-safe. So, a kernel oops is not a safety violation. For example, we do not intend to include a floppy disk driver; so, syzkaller findings in that subsystem are less relevant than in the scheduler.
 >
 > 2. Social aspect: Come to a common agreement and understanding on core aspects. Without further investigation of social studies, I believe that within a group that exchanged and agreed on core aspects, the willingness to collaborate is higher. This also includes defending the work towards management and supporting each other in the various activities. We need to achieve a collaborative working mode, not a mode of unrelated parallel and conflicting and competing activities.
 >
 > I want this discussion to start to then move the activities forward WITH the support of ALL stakeholders.
 >
 >>>  2. Define which hardware shall be assumed, for which a hardware safety manual is available
 >>
 >>  That list is known to be short - so how about turning this around a bit and
 >>  deriving the interaction points between HW and SW - with other words record
 >>  where we are making any HW assumptions so that we can re-open that issue
 >>  as soon as a real HW exists with all its limitatoins and possibly
 >>  unconsidered capabilities. MAtching against sasumptions is futil at this point
 >>  nobody knows what the HW will look like and notably what constraints
 >>  the HW manual might contain - if only we could convince one HW vendor to
 >>  stop trying to build SEooC and cooordinate there HW assurance process with
 >>  the SW assurance process - the synergies (in context of a specific Use-Case/Intent)
 >>  are, I believe, significant
 >
 > Agree. The goal here is to start this discussion with the hardware vendors and get them involved.
 >
 >>>  3. Define a suitable kernel configuration for this setup with proper requirements traceability to allow others to understand the selected configuration.
 >>
 >>  3 would need to iterate over all element candidates rather than try to
 >>  define a kernel configuratoin only - if you do this for one configuration - what is
 >>  the probability of getting it right ? msot likely closs to 0
 >>  but we can develop the potentials with few constraints - e.g
 >>  1) what could seccomp provide - und constraint XYZ
 >>  2) what could CONFIG_ABC provide - under constraint ..
 >>
 >>  This will not be context agnostic building blocks but about as close
 >>  to that ideal as we can get - the evaluation is then in context of a specific
 >>  defined Use-Case (and in no way generic)
 >
 > In SIL2LinuxMP, we had this x86_minimal_config.
 > This was by its process creation mostly "assumed" and in fact, we never even wrote down the high-level properties and the general assumptions of how we came to the configuration. It is probably still all in your head.
 > Then, in various activities, we used this configuration for various analysis and measurements.
 >
 > I am considering something similar here, again, just better documented a bit better and commonly and collaboratively agreed for further activities.
 >
 > Again, the technical task is one thing. The more important task is the social activity: Everyone should agree on some same kernel configuration setup and hence a common basis for the collaborative effort. It might be painful, but that is life: collaboration with others and reality always include comprises, but then, once you committed to that configuration, everyone knows what they are considering and working on.
 >
 > That what you call "potentials with constraints", I would consider a subtree of the requirements tree from "feature" requirements to "kernel configuration fragments".
 >
 > I also expect that a proper requirements tree would then allow to carefully split out one subtree of this kernel configuration with some careful expert operation, of course. The re-use of that requirements tree is limited. I am more interested in reusing some of the analysis results or at least, claiming that certain statistics are not completely skewed for another similar configuration.
 >
 > Best regards,
 >
 > Lukas
 >
 >

In SIL2LinuxMP, we had this x86_minimal_config.
This was by its process creation mostly "assumed" and in fact, we never even wrote down the high-level properties and the general assumptions of how we came to the configuration. It is probably still all in your head.
Then, in various activities, we used this configuration for various analysis and measurements.

I am considering something similar here, again, just better documented a bit better and commonly and collaboratively agreed for further activities.

x86_minimal_config was an (OT) internal working configuration created
because SIL2LinuxMP partner did deliver a configuration. We needed some
starting point which was according the developed architecture to make our
studies comparable (eg. PIT). Thats the reason we refered to "a minimal
i7 configuration".

"A starting point which was according the developed architecture to make our studies comparable" and understandable and valuable to the stakeholders.
This is exactly what the collaborative group needs to determine, so that further studies and potentially motivated kernel improvements can be focused and prioritized accordingly.

ELISA is a new project with different and new stakeholders (than in SIL2LinuxMP), so all stakeholders should get a fair chance to engage in this discussion and propose a suitable kernel configuration to consider. Then, we determine a suitable agreement that will allow feasible work on relevant parts, given that kernel configuration.


Best regards,

Lukas

Dear Lukas,

I am wondering why you do not consider old good testing that is strictly necessary for ensuring good levels of safety (if I understand properly)? I suggest that this is a challenge that nobody could solve yet. Developing detailed requirements and corresponding tests may be the most expensive work in this project, thus, it influences everything very much.

What sense does testing make on a moving target which has an endless
state space?

Testing is currently the best available approach to check if complex functional requirements are implemented and as Oskar said it is rather easy to handle moving targets by re-running tests.

Wouldn't it be a much more elegant and interesting challenge to avoid
this cost intensive work (testing)?

Yes. It would, but suitable alternatives in software engineering have not yet evolved to a state that are serve a real alternatives to testing. So, testing is in the next years always a complement to various other activities in software engineering.

Dear Lukas,

Of course, you need to investigate currently available test suites and infrastructures for executing tests. But unlike, say, static analysis that is quite flexible and does not require considerable efforts according to standard requirements, I suggest that you will have to develop very many things related with tests since:

1. Tests are mandatory - certification bodies will not excuse you for missing requirements and test cases. Moreover, your customers will look at requirements and testing results first of all rather than on verification results from static analyzers.
2. Current tests are not intended for demonstrating safety. Thus, you have to develop all necessary requirements and test cases as well as to relate these artifacts with different configurations (profiles) of the Linux kernel in an adequate way. Although, concrete details are rather specific, there should be tools that simplify the whole workflow since the task is not new. And you should select an appropriate tool. Perhaps, some improvements in this tool will be necessary because of specifics of the OS kernel and the platform. Regarding concrete details you need to find appropriate experts and practitioners who have a large experience in testing of the Linux kernel and who will be able to estimate necessary efforts. Altogether I see that this task is the most outstanding and most labor-intensive in the whole project.

That's why I have been keeping in mind that testing influences many activities in your project actually. You will not be able to develop necessary artifacts for rather generic version and configuration of the Linux kernel in a reasonable time.

Best regards,
Evgeny

24.04.2019, 09:46, "Lukas Bulwahn" <lukas.bulwahn@...>:

Dear Evgeny, dear Markus, dear Oskar,

I fear this thread might develop in the direction of the value of testing and its relation to other activities.

If we continue this discussion, please change the subject line in further responses if you continue in this direction.

Thanks and best regards,

Lukas

Hello,
my availabity on 15:00 is
6th May,
7th May
(8th May)
10th May

Kind regards,
Oscar
PS
And no comments about testability ;-)

-----Ursprüngliche Nachricht-----
Von: Oscar Slotosch
Gesendet: Mittwoch, 24. April 2019 21:56
An: devel@...
Betreff: AW: [devel] Task Proposal: Define Assumed System and Requirements

Hi Markus,

Wouldn't it be a much more elegant and interesting challenge to avoid this cost intensive work (testing)?

No, testing is required for Safety Standards and we can run tests automated (one we have created them).
They are a good indicator for changes and also for developers, to see if they have "damaged" something by their changes.

Kind regards,
Oscar

-----Ursprüngliche Nachricht-----
Von: devel@... [mailto:devel@...] Im Auftrag von markus
Gesendet: Mittwoch, 24. April 2019 19:10
An: devel@...
Betreff: Re: [devel] Task Proposal: Define Assumed System and Requirements

On Tue, 23 Apr 2019, Evgeny Novikov wrote:

Dear Lukas,

I am wondering why you do not consider old good testing that is strictly necessary for ensuring good levels of safety (if I understand properly)? I suggest that this is a challenge that nobody could solve yet. Developing detailed requirements and corresponding tests may be the most expensive work in this project, thus, it influences everything very much.

What sense does testing make on a moving target which has an endless state space?

when you say influences everything very much, what do you mean?

\BR
Markus

Unfortunately, I can not reason accurately about various issues of Linux kernel testing, but there should be appropriate experts in this list.

Best regards,
Evgeny

23.04.2019, 11:37, "Lukas Bulwahn" <lukas.bulwahn@...>:

Hi Nicholas,

(sorry for the clumsy inline comments; I tried my best with the
broken tools I get provided.)

On Thu, Apr 18, 2019 at 09:14:41AM +0000, Lukas Bulwahn wrote:

 1. Define a generic assumed software/system architecture and
requirements

 Lukas - this is right back to the fellacy we start at 5 years ago
 that will not work - if we do it that way - why would the elements
chosen
 based on crude assumptions miracously match the actual needs ?
 notably when chosen at the functional level not at the safety level ?

 I believe that the only suitable starting point is "Intent" and
derive the
 needs in a technology agnostic manner - identifying all potential
safety
 issues and security issues along the way - and THEN we can start
selecting
 element cadidates.

 A side node - NONE of them willl be generic - not in the first run
- maybe
 after we built 5-6 ssystems some elements will emerge as generic - but not from the start.

Nicholas, I think you picked up on the word "generic" than I actually intended to say. But yes, I agree the word "generic" must be used with care and I should drop it and replace it with "incomplete" or "non-derived" and "collaboratively agreed with common understanding".

So, I will try to be more specific about the intentions of this activity (as I see it) and level of needed granularity:

1. Further activities that only make sense with any assumed system and kernel configuration:
  - Safety impact analysis: crucial to determine if a bug is a safety goal violation or not; we need at least some assumed requirements to exercise this activity.
  - Root cause analysis: assuming that certain subsystems are excluded helps to identify the actual relevant root causes.
  - Static analysis assessment and tracking: you need to know roughly the assumed source code base to be relevant.

For example, we might generally assume the system to be fail-safe. So, a kernel oops is not a safety violation. For example, we do not intend to include a floppy disk driver; so, syzkaller findings in that subsystem are less relevant than in the scheduler.

2. Social aspect: Come to a common agreement and understanding on core aspects. Without further investigation of social studies, I believe that within a group that exchanged and agreed on core aspects, the willingness to collaborate is higher. This also includes defending the work towards management and supporting each other in the various activities. We need to achieve a collaborative working mode, not a mode of unrelated parallel and conflicting and competing activities.

I want this discussion to start to then move the activities forward WITH the support of ALL stakeholders.

 2. Define which hardware shall be assumed, for which a hardware
safety manual is available

 That list is known to be short - so how about turning this around
a bit and
 deriving the interaction points between HW and SW - with other
words record
 where we are making any HW assumptions so that we can re-open that
issue
 as soon as a real HW exists with all its limitatoins and possibly
 unconsidered capabilities. MAtching against sasumptions is futil
at this point
 nobody knows what the HW will look like and notably what
constraints
 the HW manual might contain - if only we could convince one HW
vendor to
 stop trying to build SEooC and cooordinate there HW assurance
process with
 the SW assurance process - the synergies (in context of a specific
Use-Case/Intent)
 are, I believe, significant

Agree. The goal here is to start this discussion with the hardware vendors and get them involved.

 3. Define a suitable kernel configuration for this setup with proper requirements traceability to allow others to understand the selected configuration.

 3 would need to iterate over all element candidates rather than
try to
 define a kernel configuratoin only - if you do this for one
configuration - what is
 the probability of getting it right ? msot likely closs to 0
 but we can develop the potentials with few constraints - e.g
 1) what could seccomp provide - und constraint XYZ
 2) what could CONFIG_ABC provide - under constraint ..

 This will not be context agnostic building blocks but about as
close
 to that ideal as we can get - the evaluation is then in context of
a specific
 defined Use-Case (and in no way generic)

In SIL2LinuxMP, we had this x86_minimal_config.
This was by its process creation mostly "assumed" and in fact, we never even wrote down the high-level properties and the general assumptions of how we came to the configuration. It is probably still all in your head.
Then, in various activities, we used this configuration for various analysis and measurements.

I am considering something similar here, again, just better documented a bit better and commonly and collaboratively agreed for further activities.

Again, the technical task is one thing. The more important task is the social activity: Everyone should agree on some same kernel configuration setup and hence a common basis for the collaborative effort. It might be painful, but that is life: collaboration with others and reality always include comprises, but then, once you committed to that configuration, everyone knows what they are considering and working on.

That what you call "potentials with constraints", I would consider a subtree of the requirements tree from "feature" requirements to "kernel configuration fragments".

I also expect that a proper requirements tree would then allow to carefully split out one subtree of this kernel configuration with some careful expert operation, of course. The re-use of that requirements tree is limited. I am more interested in reusing some of the analysis results or at least, claiming that certain statistics are not completely skewed for another similar configuration.

Best regards,

Lukas

On Thu, 25 Apr 2019, Oscar Slotosch wrote:

Hello,
my availabity on 15:00 is
6th May,
7th May
(8th May)
10th May

I'm not available in that week. I will look forward to the next meeting.

julia

Kind regards,
Oscar
PS
And no comments about testability ;-)

-----Ursprüngliche Nachricht-----
Von: Oscar Slotosch
Gesendet: Mittwoch, 24. April 2019 21:56
An: devel@...
Betreff: AW: [devel] Task Proposal: Define Assumed System and Requirements

Hi Markus,

Wouldn't it be a much more elegant and interesting challenge to avoid this cost intensive work (testing)?

No, testing is required for Safety Standards and we can run tests automated (one we have created them).
They are a good indicator for changes and also for developers, to see if they have "damaged" something by their changes.

Kind regards,
Oscar

-----Ursprüngliche Nachricht-----
Von: devel@... [mailto:devel@...] Im Auftrag von markus
Gesendet: Mittwoch, 24. April 2019 19:10
An: devel@...
Betreff: Re: [devel] Task Proposal: Define Assumed System and Requirements

On Tue, 23 Apr 2019, Evgeny Novikov wrote:

Dear Lukas,

I am wondering why you do not consider old good testing that is strictly necessary for ensuring good levels of safety (if I understand properly)? I suggest that this is a challenge that nobody could solve yet. Developing detailed requirements and corresponding tests may be the most expensive work in this project, thus, it influences everything very much.

What sense does testing make on a moving target which has an endless state space?

when you say influences everything very much, what do you mean?

\BR
Markus

Unfortunately, I can not reason accurately about various issues of Linux kernel testing, but there should be appropriate experts in this list.

Best regards,
Evgeny

23.04.2019, 11:37, "Lukas Bulwahn" <lukas.bulwahn@...>:

Hi Nicholas,

(sorry for the clumsy inline comments; I tried my best with the
broken tools I get provided.)

On Thu, Apr 18, 2019 at 09:14:41AM +0000, Lukas Bulwahn wrote:

 1. Define a generic assumed software/system architecture and
requirements

 Lukas - this is right back to the fellacy we start at 5 years ago
 that will not work - if we do it that way - why would the elements
chosen
 based on crude assumptions miracously match the actual needs ?
 notably when chosen at the functional level not at the safety level ?

 I believe that the only suitable starting point is "Intent" and
derive the
 needs in a technology agnostic manner - identifying all potential
safety
 issues and security issues along the way - and THEN we can start
selecting
 element cadidates.

 A side node - NONE of them willl be generic - not in the first run
- maybe
 after we built 5-6 ssystems some elements will emerge as generic - but not from the start.

Nicholas, I think you picked up on the word "generic" than I actually intended to say. But yes, I agree the word "generic" must be used with care and I should drop it and replace it with "incomplete" or "non-derived" and "collaboratively agreed with common understanding".

So, I will try to be more specific about the intentions of this activity (as I see it) and level of needed granularity:

1. Further activities that only make sense with any assumed system and kernel configuration:
  - Safety impact analysis: crucial to determine if a bug is a safety goal violation or not; we need at least some assumed requirements to exercise this activity.
  - Root cause analysis: assuming that certain subsystems are excluded helps to identify the actual relevant root causes.
  - Static analysis assessment and tracking: you need to know roughly the assumed source code base to be relevant.

For example, we might generally assume the system to be fail-safe. So, a kernel oops is not a safety violation. For example, we do not intend to include a floppy disk driver; so, syzkaller findings in that subsystem are less relevant than in the scheduler.

2. Social aspect: Come to a common agreement and understanding on core aspects. Without further investigation of social studies, I believe that within a group that exchanged and agreed on core aspects, the willingness to collaborate is higher. This also includes defending the work towards management and supporting each other in the various activities. We need to achieve a collaborative working mode, not a mode of unrelated parallel and conflicting and competing activities.

I want this discussion to start to then move the activities forward WITH the support of ALL stakeholders.

 2. Define which hardware shall be assumed, for which a hardware
safety manual is available

 That list is known to be short - so how about turning this around
a bit and
 deriving the interaction points between HW and SW - with other
words record
 where we are making any HW assumptions so that we can re-open that
issue
 as soon as a real HW exists with all its limitatoins and possibly
 unconsidered capabilities. MAtching against sasumptions is futil
at this point
 nobody knows what the HW will look like and notably what
constraints
 the HW manual might contain - if only we could convince one HW
vendor to
 stop trying to build SEooC and cooordinate there HW assurance
process with
 the SW assurance process - the synergies (in context of a specific
Use-Case/Intent)
 are, I believe, significant

Agree. The goal here is to start this discussion with the hardware vendors and get them involved.

 3. Define a suitable kernel configuration for this setup with proper requirements traceability to allow others to understand the selected configuration.

 3 would need to iterate over all element candidates rather than
try to
 define a kernel configuratoin only - if you do this for one
configuration - what is
 the probability of getting it right ? msot likely closs to 0
 but we can develop the potentials with few constraints - e.g
 1) what could seccomp provide - und constraint XYZ
 2) what could CONFIG_ABC provide - under constraint ..

 This will not be context agnostic building blocks but about as
close
 to that ideal as we can get - the evaluation is then in context of
a specific
 defined Use-Case (and in no way generic)

In SIL2LinuxMP, we had this x86_minimal_config.
This was by its process creation mostly "assumed" and in fact, we never even wrote down the high-level properties and the general assumptions of how we came to the configuration. It is probably still all in your head.
Then, in various activities, we used this configuration for various analysis and measurements.

I am considering something similar here, again, just better documented a bit better and commonly and collaboratively agreed for further activities.

Again, the technical task is one thing. The more important task is the social activity: Everyone should agree on some same kernel configuration setup and hence a common basis for the collaborative effort. It might be painful, but that is life: collaboration with others and reality always include comprises, but then, once you committed to that configuration, everyone knows what they are considering and working on.

That what you call "potentials with constraints", I would consider a subtree of the requirements tree from "feature" requirements to "kernel configuration fragments".

I also expect that a proper requirements tree would then allow to carefully split out one subtree of this kernel configuration with some careful expert operation, of course. The re-use of that requirements tree is limited. I am more interested in reusing some of the analysis results or at least, claiming that certain statistics are not completely skewed for another similar configuration.

Best regards,

Lukas