Task proposal: Public safety compliance argumentation for Linux
|
Hi all,
From the discussion with Nicholas, I created a high-level summary of one first central task that I see around Enabling Linux in Safety Applications. As we do not have a public safety compliance route for Linux available and I see this as the "first" task in a waterfall model development, I consider this important and sketched this roughly below. This does not mean that other topics cannot be investigated independently; it is just that the results of those activities eventually will link into this central compliance route document. I tried to formulate the goal, the activity, the resulting work product, my personal effort estimation and rough milestones for the overall task. Public safety compliance argumentation for Linux Goal: Public safety compliance argumentation for Linux available and approved by five major companies in safety-related system development and three certification agencies Needed activities - Agree on common safety standard and general compliance route - Assess differences of: - IEC 61508 compliance route - ISO 26262 Part 6 compliance route - Result: Short white paper (~6 pages) on the suitability of the different standards to a Linux kernel verification from multiple authors from different companies and approved by multiple certification agencies - Provide fitting into ISO 26262 Ed 2 safety argumentation - Result: One section in the white paper - Define clear scope of compliance argumentation - Result: Definition of an assumed development interface agreement of the compliance route document - Agree on common interpretation - Result: A shared interpretation and compliance route document - Define possible methods to meet interpretation - Result: Mostly unified set of methods to achieve compliance - Note in case of disagreements, multiple methods can be listed as alternatives Work Product - Documentation on: - suitability of the different standards to a Linux kernel verification - fitting into an ISO 26262 Ed 2 safety argumentation - an assumed DIA - interpretation of clauses and identified/recommended activity - Assessment report from certification authorities My Personal Effort Estimation 14x 2-day expert workshops + 5 month offline effort per author: ~100 days work, i.e., 7 days offline work per workshop between workshops for each author = 6 month per author Expecting 5 to 7 main authors from the different companies. If companies have already prepared such documents internally, this could lower the effort for the author of that company. 3 - 4 months effort of a technical writer to assist in improving style, wording and presentation Milestones to measure progress and state of work M1: Experts identified, obtained commitment from authors and if needed, authors are contracted. Initial technical kick-off workshop planned. M2: First draft of paper with agreed structure. Agreement on general compliance route among authors; argumentation written and acknowledged by all authors. M3: argumentation documented and agreed for 50% of clauses M4: argumentation documented and agreed for all clauses except clauses with unresolved major disagreement M5: Resolution of last clauses done. Only stylistic and presentation issues outstanding. M6: Final document available and distributed to suitable journal. Any comments to this task proposal? Lukas |
|
|
|
Oscar Slotosch
Hi Lucas and Nicolas, and all
Very interesting discussions and good ideas. From my point of view speaking of "new compliance route" sounds a bit like writing a new standard, which I think we should avoid, even being suspect to this should be avoided in order to convince certification authorities and safety experts easier. From my point of view we can "tailor" the standards if necessary. However, at least the ISO states (2-6.4.5.1.b) that there should be an argument for it's safety. I agree with Proposal of Lucas, but would prefer to be more standard compliant in a way that we Have safety plan & safety case by - select the requirements (standard) to comply with - define the process, focussing on the verification of the selected requirements - argue compliance And generate: compliance report and check-lists for the concrete product. We using the freely available PMT tool from the research project SPEDIT from http://www.validas.de/en/tools/ That we have been using during certification of our qualification processes and that we plan to make open source. By using a more standard approach we definitely have less issues in convincing assessors ;-). Kind regards, Oscar PS Klever, cucinelle,.. are a cool tool, but we should must not forget that standards require to have requirements, systematic tests, code coverage, which I was missing a bit in the discussion. What about Gcov? Gdb?,... I would consider klever, etc. as great tools to check for guidelines (which we have to satisfy anyhow and we are free to select them;-). -----Ursprüngliche Nachricht----- Von: devel@... [mailto:devel@...] Im Auftrag von Lukas Bulwahn Gesendet: Donnerstag, 11. April 2019 16:17 An: devel@... Betreff: [devel] Task proposal: Public safety compliance argumentation for Linux Hi all, From the discussion with Nicholas, I created a high-level summary of one first central task that I see around Enabling Linux in Safety Applications. As we do not have a public safety compliance route for Linux available and I see this as the "first" task in a waterfall model development, I consider this important and sketched this roughly below. This does not mean that other topics cannot be investigated independently; it is just that the results of those activities eventually will link into this central compliance route document. I tried to formulate the goal, the activity, the resulting work product, my personal effort estimation and rough milestones for the overall task. Public safety compliance argumentation for Linux Goal: Public safety compliance argumentation for Linux available and approved by five major companies in safety-related system development and three certification agencies Needed activities - Agree on common safety standard and general compliance route - Assess differences of: - IEC 61508 compliance route - ISO 26262 Part 6 compliance route - Result: Short white paper (~6 pages) on the suitability of the different standards to a Linux kernel verification from multiple authors from different companies and approved by multiple certification agencies - Provide fitting into ISO 26262 Ed 2 safety argumentation - Result: One section in the white paper - Define clear scope of compliance argumentation - Result: Definition of an assumed development interface agreement of the compliance route document - Agree on common interpretation - Result: A shared interpretation and compliance route document - Define possible methods to meet interpretation - Result: Mostly unified set of methods to achieve compliance - Note in case of disagreements, multiple methods can be listed as alternatives Work Product - Documentation on: - suitability of the different standards to a Linux kernel verification - fitting into an ISO 26262 Ed 2 safety argumentation - an assumed DIA - interpretation of clauses and identified/recommended activity - Assessment report from certification authorities My Personal Effort Estimation 14x 2-day expert workshops + 5 month offline effort per author: ~100 days work, i.e., 7 days offline work per workshop between workshops for each author = 6 month per author Expecting 5 to 7 main authors from the different companies. If companies have already prepared such documents internally, this could lower the effort for the author of that company. 3 - 4 months effort of a technical writer to assist in improving style, wording and presentation Milestones to measure progress and state of work M1: Experts identified, obtained commitment from authors and if needed, authors are contracted. Initial technical kick-off workshop planned. M2: First draft of paper with agreed structure. Agreement on general compliance route among authors; argumentation written and acknowledged by all authors. M3: argumentation documented and agreed for 50% of clauses M4: argumentation documented and agreed for all clauses except clauses with unresolved major disagreement M5: Resolution of last clauses done. Only stylistic and presentation issues outstanding. M6: Final document available and distributed to suitable journal. Any comments to this task proposal? Lukas |
|
|
|
Nicholas Mc Guire
On Thu, Apr 11, 2019 at 05:41:57PM +0000, Oscar Slotosch wrote:
Hi Lucas and Nicolas, and allNew compliance routes just means interpretation of the standard - there is a lot of room for interpretation - e.g. 61508-3 7.4.2.13 describes Route 3_S (assessment of non-compliant develoment" in one Clause - there is a lot to define and clarify before one can take action. And writing a new standard for pre-existing software or open-source would not be that bad a thing to do ! From my point of view we can "tailor" the standards if necessary. However, at least the ISO states (2-6.4.5.1.b) that there should be an argument for it's safety.All standard state that - tailoring is formalized in many ways for Route 3S in SIL2LinuxMP - of course its not just putting together a wish-list - its a strict derivative of 61508 just like ISO 26262 is a derivative of 61508 (mostly Ed1 unfortunately for human operated cars using low-comlexity HW/SW - no clue why people are even trying to use it for complex systems....) I agree with Proposal of Lucas, but would prefer to be more standard compliant in a way that weNo difference with that aproach - the standard is 61508 Ed2 as the only standard that actually can accomodate highly-complex pre-existing SW (formally you can plug it into ISO 26262-8 Clause 12 if you like) - just that the term safety case is a highly unspecific term that everyone has a different idea about - that is in line with the intent here. - define the process, focussing on the verification of the selected requirementsWe do have to start a bit earlier in the process - getting the requirements right if you are building on pre-existing elements is a diferent story - you can not just write down the requirements for the Linux kernel and then verify them - thats going to be impossible (and if you did manage to do it it would be out-of-date before you completed it) - argue complianceSure - but what exactly is compliance for non-compliantly developed SW ? (and HW for that matter) - THAT has to be defined in accordance with 61508 and agreed with the CA - then we can assess and provide an argument. And generate: compliance report and check-lists for the concrete product.check-lists ? serious - I though check-list safety is finally done with... We using the freely available PMT tool from the research project SPEDIT from http://www.validas.de/en/tools/Make and re-use is too very different cans of worms ! Nobody here is suggesting a non-standard approach - defining your compliance route is the one of the early tasks in any safety project - but even ISO2626 leves you some flexibility - select methods (line-coverage or instruction coverage - or MCDC (if you want to waste time and money with little benefit)) So you never an simply "apply" a standard - and even less in complex systems because the standards (except for 61508) were written for low-complexity (Type-A) elements only. thx! hofrat |
|
|
|
Oscar Slotosch
Hello Nicholas, Thank you for your comments. Basically there are two things that I would recommend to avoid in a car * moving targets (so we should come to a stable, frozen and save version of linux to add to the car) that can be shown to satisfy safety requirements. * unconventional argumentations like "ISO 26262 is not applicable" or "MCDC being waste of time & money"
A safety case is very clearly defined by containing everything that you have planned to do (in your process/safety plan) to meet the standard requirements. It's so simple that we have created a model for it such that we can even automatically check consistency and completeness of it. Of course safety cases difference, because they depend on the processes and safety plans that are different. But the core idea of a safety plan is just the evidence of the compliance as planned in the safety plan.
With this approach is was very easy to convince any certification authority so far, and I think that's what we are aiming for.
Kind regards, Oscar PS we are re-using our approach as well by extending it to new standards (EN 50657) and new processes "Pre-Qualified Tools" and new scopes ISO 8-12, SEOOC. We can do so also for arguing safety of linux as SEOOC once we agree on a process to make it safe that has a chance to be compliant;-)
Here is a slide that I us to explain our approach:
-----Ursprüngliche Nachricht----- Von: devel@... [mailto:devel@...] Im Auftrag von Nicholas Mc Guire Gesendet: Freitag, 12. April 2019 01:30 An: devel@... Betreff: Re: [devel] Task proposal: Public safety compliance argumentation for Linux
On Thu, Apr 11, 2019 at 05:41:57PM +0000, Oscar Slotosch wrote:
> Hi Lucas and Nicolas, and all > Very interesting discussions and good ideas. > > From my point of view speaking of "new compliance route" sounds a bit > like writing a new standard, which I think we should avoid, even being suspect to this should be avoided in order to convince certification authorities and safety experts easier. >
New compliance routes just means interpretation of the standard - there is a lot of room for interpretation - e.g. 61508-3 7.4.2.13 describes Route 3_S (assessment of non-compliant develoment" in one Clause - there is a lot to define and clarify before one can take action.
And writing a new standard for pre-existing software or open-source would not be that bad a thing to do !
> From my point of view we can "tailor" the standards if necessary. However, at least the ISO states (2-6.4.5.1.b) that there should be an argument for it's safety. >
All standard state that - tailoring is formalized in many ways for Route 3S in SIL2LinuxMP - of course its not just putting together a wish-list - its a strict derivative of 61508 just like ISO 26262 is a derivative of 61508 (mostly Ed1 unfortunately for human operated cars using low-comlexity HW/SW - no clue why people are even trying to use it for complex systems....)
> I agree with Proposal of Lucas, but would prefer to be more standard compliant in a way that we > Have safety plan & safety case by > - select the requirements (standard) to comply with
No difference with that aproach - the standard is 61508 Ed2 as the only standard that actually can accomodate highly-complex pre-existing SW (formally you can plug it into ISO 26262-8 Clause 12 if you like) - just that the term safety case is a highly unspecific term that everyone has a different idea about - that is in line with the intent here.
> - define the process, focussing on the verification of the selected requirements
We do have to start a bit earlier in the process - getting the requirements right if you are building on pre-existing elements is a diferent story - you can not just write down the requirements for the Linux kernel and then verify them - thats going to be impossible (and if you did manage to do it it would be out-of-date before you completed it)
> - argue compliance
Sure - but what exactly is compliance for non-compliantly developed SW ? (and HW for that matter) - THAT has to be defined in accordance with 61508 and agreed with the CA - then we can assess and provide an argument.
> And generate: compliance report and check-lists for the concrete product.
check-lists ? serious - I though check-list safety is finally done with...
> We using the freely available PMT tool from the research project SPEDIT from http://www.validas.de/en/tools/ > That we have been using during certification of our qualification processes and that we plan to make open source.
Make and re-use is too very different cans of worms !
> > By using a more standard approach we definitely have less issues in convincing assessors ;-).
Nobody here is suggesting a non-standard approach - defining your compliance route is the one of the early tasks in any safety project - but even ISO2626 leves you some flexibility - select methods (line-coverage or instruction coverage - or MCDC (if you want to waste time and money with little benefit))
So you never an simply "apply" a standard - and even less in complex systems because the standards (except for 61508) were written for low-complexity (Type-A) elements only.
thx! hofrat
|
|
|
|
Evgeny Novikov <novikov@...>
Dear Oscar and others,
toggle quoted message
Show quoted text
I was not involved directly into any certification processes, so, it is interesting to hear opinions from experts. Without no doubts for such complex software as the Linux kernel it is impossible to identify and to eliminate all possible safety issues in any reasonable time (say, within 2-3 years). Even with rather restricted versions without numerous extensions the code size will be hundreds or thousands of KLOC (and it is very complicated code!) since one seems to want many features (advanced multiple-processor scheduling, memory managements, IPC, networking, file systems, cryptography, etc.). I suggest that nobody will have even simple line coverage that is more than 50-75% with standard tests. Moreover, nobody even will formulate all requirements since their number is extremely huge, especially, when considering many specific devices. Taking this into account, is it possible to get some meaningful certificates ever? By meaningful certificates I understand ones that will not say, that a development process is okay, but ones that will provide some confirmations that one can use certified software in safety systems (that is the final goal, is not it?). Let's assume that, say, ELISA will be able to get such the certificates one day in near future. But there still will be safety related issues in certified software. And one will be able to become aware of these issues from one or another source, e.g. from users, verification tool developers, hackers and so on. What's next? The software should be fixed since people health is more important than papers. And what about existing certificates? Of course, I assume that ELISA can obtain very permissive certificates that will not ensure any safety, but will this help to reach safety related industries? To some extent I agree with Nicholas who suggested to continuously improve and monitor the Linux kernel quality. But there are many related questions as well. -- Evgeny Novikov Linux Verification Center, ISP RAS http://linuxtesting.org 12.04.2019, 08:59, "Oscar Slotosch" <slotosch@...>: Hello Nicholas, |
|
|
|
Hi Oscar,
just one comment here: Nicholas might not be able to parse your picture.
If you meet him in person, you will know why.
Lukas
Von: devel@... [mailto:devel@...]
Im Auftrag von Oscar Slotosch
Hello Nicholas, Thank you for your comments. Basically there are two things that I would recommend to avoid in a car * moving targets (so we should come to a stable, frozen and save version of linux to add to the car) that can be shown to satisfy safety requirements. * unconventional argumentations like "ISO 26262 is not applicable" or "MCDC being waste of time & money"
A safety case is very clearly defined by containing everything that you have planned to do (in your process/safety plan) to meet the standard requirements. It's so simple that we have created a model for it such that we can even automatically check consistency and completeness of it. Of course safety cases difference, because they depend on the processes and safety plans that are different. But the core idea of a safety plan is just the evidence of the compliance as planned in the safety plan.
With this approach is was very easy to convince any certification authority so far, and I think that's what we are aiming for.
Kind regards, Oscar PS we are re-using our approach as well by extending it to new standards (EN 50657) and new processes "Pre-Qualified Tools" and new scopes ISO 8-12, SEOOC. We can do so also for arguing safety of linux as SEOOC once we agree on a process to make it safe that has a chance to be compliant;-)
Here is a slide that I us to explain our approach:
-----Ursprüngliche Nachricht----- Von: devel@... [mailto:devel@...] Im Auftrag von Nicholas Mc Guire Gesendet: Freitag, 12. April 2019 01:30 An: devel@... Betreff: Re: [devel] Task proposal: Public safety compliance argumentation for Linux
On Thu, Apr 11, 2019 at 05:41:57PM +0000, Oscar Slotosch wrote:
> Hi Lucas and Nicolas, and all > Very interesting discussions and good ideas. > > From my point of view speaking of "new compliance route" sounds a bit > like writing a new standard, which I think we should avoid, even being suspect to this should be avoided in order to convince certification authorities and safety experts easier. >
New compliance routes just means interpretation of the standard - there is a lot of room for interpretation - e.g. 61508-3 7.4.2.13 describes Route 3_S (assessment of non-compliant develoment" in one Clause - there is a lot to define and clarify before one can take action.
And writing a new standard for pre-existing software or open-source would not be that bad a thing to do !
> From my point of view we can "tailor" the standards if necessary. However, at least the ISO states (2-6.4.5.1.b) that there should be an argument for it's safety. >
All standard state that - tailoring is formalized in many ways for Route 3S in SIL2LinuxMP - of course its not just putting together a wish-list - its a strict derivative of 61508 just like ISO 26262 is a derivative of 61508 (mostly Ed1 unfortunately for human operated cars using low-comlexity HW/SW - no clue why people are even trying to use it for complex systems....)
> I agree with Proposal of Lucas, but would prefer to be more standard compliant in a way that we > Have safety plan & safety case by > - select the requirements (standard) to comply with
No difference with that aproach - the standard is 61508 Ed2 as the only standard that actually can accomodate highly-complex pre-existing SW (formally you can plug it into ISO 26262-8 Clause 12 if you like) - just that the term safety case is a highly unspecific term that everyone has a different idea about - that is in line with the intent here.
> - define the process, focussing on the verification of the selected requirements
We do have to start a bit earlier in the process - getting the requirements right if you are building on pre-existing elements is a diferent story - you can not just write down the requirements for the Linux kernel and then verify them - thats going to be impossible (and if you did manage to do it it would be out-of-date before you completed it)
> - argue compliance
Sure - but what exactly is compliance for non-compliantly developed SW ? (and HW for that matter) - THAT has to be defined in accordance with 61508 and agreed with the CA - then we can assess and provide an argument.
> And generate: compliance report and check-lists for the concrete product.
check-lists ? serious - I though check-list safety is finally done with...
> We using the freely available PMT tool from the research project SPEDIT from http://www.validas.de/en/tools/ > That we have been using during certification of our qualification processes and that we plan to make open source.
Make and re-use is too very different cans of worms !
> > By using a more standard approach we definitely have less issues in convincing assessors ;-).
Nobody here is suggesting a non-standard approach - defining your compliance route is the one of the early tasks in any safety project - but even ISO2626 leves you some flexibility - select methods (line-coverage or instruction coverage - or MCDC (if you want to waste time and money with little benefit))
So you never an simply "apply" a standard - and even less in complex systems because the standards (except for 61508) were written for low-complexity (Type-A) elements only.
thx! hofrat
|
|
|
|
Oscar Slotosch
Hi, I agree that it is not easy to parse and even worse to understand ;-)
I promise to make a podcast-episode explaining it in our podcast that we are launching. Will let you know when it is available.
Kind regards, Oscar
Von: devel@... [mailto:devel@...]
Im Auftrag von Lukas Bulwahn
Hi Oscar,
just one comment here: Nicholas might not be able to parse your picture.
If you meet him in person, you will know why.
Lukas
Von:
devel@... [mailto:devel@...]
Im Auftrag von Oscar Slotosch
Hello Nicholas, Thank you for your comments. Basically there are two things that I would recommend to avoid in a car * moving targets (so we should come to a stable, frozen and save version of linux to add to the car) that can be shown to satisfy safety requirements. * unconventional argumentations like "ISO 26262 is not applicable" or "MCDC being waste of time & money"
A safety case is very clearly defined by containing everything that you have planned to do (in your process/safety plan) to meet the standard requirements. It's so simple that we have created a model for it such that we can even automatically check consistency and completeness of it. Of course safety cases difference, because they depend on the processes and safety plans that are different. But the core idea of a safety plan is just the evidence of the compliance as planned in the safety plan.
With this approach is was very easy to convince any certification authority so far, and I think that's what we are aiming for.
Kind regards, Oscar PS we are re-using our approach as well by extending it to new standards (EN 50657) and new processes "Pre-Qualified Tools" and new scopes ISO 8-12, SEOOC. We can do so also for arguing safety of linux as SEOOC once we agree on a process to make it safe that has a chance to be compliant;-)
Here is a slide that I us to explain our approach:
-----Ursprüngliche Nachricht----- Von: devel@... [mailto:devel@...] Im Auftrag von Nicholas Mc Guire Gesendet: Freitag, 12. April 2019 01:30 An: devel@... Betreff: Re: [devel] Task proposal: Public safety compliance argumentation for Linux
On Thu, Apr 11, 2019 at 05:41:57PM +0000, Oscar Slotosch wrote:
> Hi Lucas and Nicolas, and all > Very interesting discussions and good ideas. > > From my point of view speaking of "new compliance route" sounds a bit > like writing a new standard, which I think we should avoid, even being suspect to this should be avoided in order to convince certification authorities and safety experts easier. >
New compliance routes just means interpretation of the standard - there is a lot of room for interpretation - e.g. 61508-3 7.4.2.13 describes Route 3_S (assessment of non-compliant develoment" in one Clause - there is a lot to define and clarify before one can take action.
And writing a new standard for pre-existing software or open-source would not be that bad a thing to do !
> From my point of view we can "tailor" the standards if necessary. However, at least the ISO states (2-6.4.5.1.b) that there should be an argument for it's safety. >
All standard state that - tailoring is formalized in many ways for Route 3S in SIL2LinuxMP - of course its not just putting together a wish-list - its a strict derivative of 61508 just like ISO 26262 is a derivative of 61508 (mostly Ed1 unfortunately for human operated cars using low-comlexity HW/SW - no clue why people are even trying to use it for complex systems....)
> I agree with Proposal of Lucas, but would prefer to be more standard compliant in a way that we > Have safety plan & safety case by > - select the requirements (standard) to comply with
No difference with that aproach - the standard is 61508 Ed2 as the only standard that actually can accomodate highly-complex pre-existing SW (formally you can plug it into ISO 26262-8 Clause 12 if you like) - just that the term safety case is a highly unspecific term that everyone has a different idea about - that is in line with the intent here.
> - define the process, focussing on the verification of the selected requirements
We do have to start a bit earlier in the process - getting the requirements right if you are building on pre-existing elements is a diferent story - you can not just write down the requirements for the Linux kernel and then verify them - thats going to be impossible (and if you did manage to do it it would be out-of-date before you completed it)
> - argue compliance
Sure - but what exactly is compliance for non-compliantly developed SW ? (and HW for that matter) - THAT has to be defined in accordance with 61508 and agreed with the CA - then we can assess and provide an argument.
> And generate: compliance report and check-lists for the concrete product.
check-lists ? serious - I though check-list safety is finally done with...
> We using the freely available PMT tool from the research project SPEDIT from http://www.validas.de/en/tools/ > That we have been using during certification of our qualification processes and that we plan to make open source.
Make and re-use is too very different cans of worms !
> > By using a more standard approach we definitely have less issues in convincing assessors ;-).
Nobody here is suggesting a non-standard approach - defining your compliance route is the one of the early tasks in any safety project - but even ISO2626 leves you some flexibility - select methods (line-coverage or instruction coverage - or MCDC (if you want to waste time and money with little benefit))
So you never an simply "apply" a standard - and even less in complex systems because the standards (except for 61508) were written for low-complexity (Type-A) elements only.
thx! hofrat
|
|
|
|
Hi Oscar,
Given that the final safety plan and safety case shall be used for multiple products in multiple companies, we need to split the safety plan and safety case in a staged way: 1. You have a basic argumentation how you interpret the objectives, what you think is the appropriate measure to show compliance towards the objective of a standard in the context of the Linux kernel/Linux kernel development. This basic argumentation should be "re-usable" independent of the specific version you select. E.g., we would not expect that a measure is sufficient for 4.4 but we believe that 4.9 it is not sufficient or vice versa. This is (at least in my understanding) the definition of the compliance route. You could also call it a functional safety plan draft or some kind of functional safety concept. 2. You have a specific product in mind: you have specific requirements in mind, you have selected a specific Linux kernel version, you have a specific configuration. Now, you apply this compliance route and you get a safety case. That does help to understand our terminology? I hope it is also clear why we want to split this in these two stages. Best regards, Lukas -----Ursprüngliche Nachricht----- Von: devel@... [mailto:devel@...] Im Auftrag von Oscar Slotosch Gesendet: Donnerstag, 11. April 2019 19:42 An: devel@... Betreff: Re: [devel] Task proposal: Public safety compliance argumentation for Linux Hi Lucas and Nicolas, and all Very interesting discussions and good ideas. From my point of view speaking of "new compliance route" sounds a bit like writing a new standard, which I think we should avoid, even being suspect to this should be avoided in order to convince certification authorities and safety experts easier. From my point of view we can "tailor" the standards if necessary. However, at least the ISO states (2-6.4.5.1.b) that there should be an argument for it's safety. I agree with Proposal of Lucas, but would prefer to be more standard compliant in a way that we Have safety plan & safety case by - select the requirements (standard) to comply with - define the process, focussing on the verification of the selected requirements - argue compliance And generate: compliance report and check-lists for the concrete product. We using the freely available PMT tool from the research project SPEDIT from http://www.validas.de/en/tools/ That we have been using during certification of our qualification processes and that we plan to make open source. By using a more standard approach we definitely have less issues in convincing assessors ;-). Kind regards, Oscar PS Klever, cucinelle,.. are a cool tool, but we should must not forget that standards require to have requirements, systematic tests, code coverage, which I was missing a bit in the discussion. What about Gcov? Gdb?,... I would consider klever, etc. as great tools to check for guidelines (which we have to satisfy anyhow and we are free to select them;-). -----Ursprüngliche Nachricht----- Von: devel@... [mailto:devel@...] Im Auftrag von Lukas Bulwahn Gesendet: Donnerstag, 11. April 2019 16:17 An: devel@... Betreff: [devel] Task proposal: Public safety compliance argumentation for Linux Hi all, From the discussion with Nicholas, I created a high-level summary of one first central task that I see around Enabling Linux in Safety Applications. As we do not have a public safety compliance route for Linux available and I see this as the "first" task in a waterfall model development, I consider this important and sketched this roughly below. This does not mean that other topics cannot be investigated independently; it is just that the results of those activities eventually will link into this central compliance route document. I tried to formulate the goal, the activity, the resulting work product, my personal effort estimation and rough milestones for the overall task. Public safety compliance argumentation for Linux Goal: Public safety compliance argumentation for Linux available and approved by five major companies in safety-related system development and three certification agencies Needed activities - Agree on common safety standard and general compliance route - Assess differences of: - IEC 61508 compliance route - ISO 26262 Part 6 compliance route - Result: Short white paper (~6 pages) on the suitability of the different standards to a Linux kernel verification from multiple authors from different companies and approved by multiple certification agencies - Provide fitting into ISO 26262 Ed 2 safety argumentation - Result: One section in the white paper - Define clear scope of compliance argumentation - Result: Definition of an assumed development interface agreement of the compliance route document - Agree on common interpretation - Result: A shared interpretation and compliance route document - Define possible methods to meet interpretation - Result: Mostly unified set of methods to achieve compliance - Note in case of disagreements, multiple methods can be listed as alternatives Work Product - Documentation on: - suitability of the different standards to a Linux kernel verification - fitting into an ISO 26262 Ed 2 safety argumentation - an assumed DIA - interpretation of clauses and identified/recommended activity - Assessment report from certification authorities My Personal Effort Estimation 14x 2-day expert workshops + 5 month offline effort per author: ~100 days work, i.e., 7 days offline work per workshop between workshops for each author = 6 month per author Expecting 5 to 7 main authors from the different companies. If companies have already prepared such documents internally, this could lower the effort for the author of that company. 3 - 4 months effort of a technical writer to assist in improving style, wording and presentation Milestones to measure progress and state of work M1: Experts identified, obtained commitment from authors and if needed, authors are contracted. Initial technical kick-off workshop planned. M2: First draft of paper with agreed structure. Agreement on general compliance route among authors; argumentation written and acknowledged by all authors. M3: argumentation documented and agreed for 50% of clauses M4: argumentation documented and agreed for all clauses except clauses with unresolved major disagreement M5: Resolution of last clauses done. Only stylistic and presentation issues outstanding. M6: Final document available and distributed to suitable journal. Any comments to this task proposal? Lukas |
|
|
|
Oscar Slotosch
Hi Lukas,
I agree completely, just I don't call it route ;-) The safety plans are generic, reusable and parameterized, as well as the corresponding checklists, But refer to a process (which can also be generic), main thing is that the checklist ensure all safety requirements to be satisfied. For example a parameter can be "MODULE", denoting the software modules in the product. However every single project has to define it's own parameters, e.g. names of software modules and has to apply the checklist to their products. A simplified checklist for a MODULE could be * Has MODULE been requirements defined? * coding guidelines in MODULE satisfied? * sufficient test methods used for MODULE? * MODULE successfully tested? * MODULE test coverage reached? The collection of all (successfully) answered checklists for all parameters is the content of safety case. So probably you would call the safety plan (which includes the compliance argumentation and the process) "route" Kind regards, Oscar -----Ursprüngliche Nachricht----- Von: devel@... [mailto:devel@...] Im Auftrag von Lukas Bulwahn Gesendet: Freitag, 12. April 2019 12:02 An: devel@... Betreff: Re: [devel] Task proposal: Public safety compliance argumentation for Linux Hi Oscar, Given that the final safety plan and safety case shall be used for multiple products in multiple companies, we need to split the safety plan and safety case in a staged way: 1. You have a basic argumentation how you interpret the objectives, what you think is the appropriate measure to show compliance towards the objective of a standard in the context of the Linux kernel/Linux kernel development. This basic argumentation should be "re-usable" independent of the specific version you select. E.g., we would not expect that a measure is sufficient for 4.4 but we believe that 4.9 it is not sufficient or vice versa. This is (at least in my understanding) the definition of the compliance route. You could also call it a functional safety plan draft or some kind of functional safety concept. 2. You have a specific product in mind: you have specific requirements in mind, you have selected a specific Linux kernel version, you have a specific configuration. Now, you apply this compliance route and you get a safety case. That does help to understand our terminology? I hope it is also clear why we want to split this in these two stages. Best regards, Lukas -----Ursprüngliche Nachricht----- Von: devel@... [mailto:devel@...] Im Auftrag von Oscar Slotosch Gesendet: Donnerstag, 11. April 2019 19:42 An: devel@... Betreff: Re: [devel] Task proposal: Public safety compliance argumentation for Linux Hi Lucas and Nicolas, and all Very interesting discussions and good ideas. From my point of view speaking of "new compliance route" sounds a bit like writing a new standard, which I think we should avoid, even being suspect to this should be avoided in order to convince certification authorities and safety experts easier. From my point of view we can "tailor" the standards if necessary. However, at least the ISO states (2-6.4.5.1.b) that there should be an argument for it's safety. I agree with Proposal of Lucas, but would prefer to be more standard compliant in a way that we Have safety plan & safety case by - select the requirements (standard) to comply with - define the process, focussing on the verification of the selected requirements - argue compliance And generate: compliance report and check-lists for the concrete product. We using the freely available PMT tool from the research project SPEDIT from http://www.validas.de/en/tools/ That we have been using during certification of our qualification processes and that we plan to make open source. By using a more standard approach we definitely have less issues in convincing assessors ;-). Kind regards, Oscar PS Klever, cucinelle,.. are a cool tool, but we should must not forget that standards require to have requirements, systematic tests, code coverage, which I was missing a bit in the discussion. What about Gcov? Gdb?,... I would consider klever, etc. as great tools to check for guidelines (which we have to satisfy anyhow and we are free to select them;-). -----Ursprüngliche Nachricht----- Von: devel@... [mailto:devel@...] Im Auftrag von Lukas Bulwahn Gesendet: Donnerstag, 11. April 2019 16:17 An: devel@... Betreff: [devel] Task proposal: Public safety compliance argumentation for Linux Hi all, From the discussion with Nicholas, I created a high-level summary of one first central task that I see around Enabling Linux in Safety Applications. As we do not have a public safety compliance route for Linux available and I see this as the "first" task in a waterfall model development, I consider this important and sketched this roughly below. This does not mean that other topics cannot be investigated independently; it is just that the results of those activities eventually will link into this central compliance route document. I tried to formulate the goal, the activity, the resulting work product, my personal effort estimation and rough milestones for the overall task. Public safety compliance argumentation for Linux Goal: Public safety compliance argumentation for Linux available and approved by five major companies in safety-related system development and three certification agencies Needed activities - Agree on common safety standard and general compliance route - Assess differences of: - IEC 61508 compliance route - ISO 26262 Part 6 compliance route - Result: Short white paper (~6 pages) on the suitability of the different standards to a Linux kernel verification from multiple authors from different companies and approved by multiple certification agencies - Provide fitting into ISO 26262 Ed 2 safety argumentation - Result: One section in the white paper - Define clear scope of compliance argumentation - Result: Definition of an assumed development interface agreement of the compliance route document - Agree on common interpretation - Result: A shared interpretation and compliance route document - Define possible methods to meet interpretation - Result: Mostly unified set of methods to achieve compliance - Note in case of disagreements, multiple methods can be listed as alternatives Work Product - Documentation on: - suitability of the different standards to a Linux kernel verification - fitting into an ISO 26262 Ed 2 safety argumentation - an assumed DIA - interpretation of clauses and identified/recommended activity - Assessment report from certification authorities My Personal Effort Estimation 14x 2-day expert workshops + 5 month offline effort per author: ~100 days work, i.e., 7 days offline work per workshop between workshops for each author = 6 month per author Expecting 5 to 7 main authors from the different companies. If companies have already prepared such documents internally, this could lower the effort for the author of that company. 3 - 4 months effort of a technical writer to assist in improving style, wording and presentation Milestones to measure progress and state of work M1: Experts identified, obtained commitment from authors and if needed, authors are contracted. Initial technical kick-off workshop planned. M2: First draft of paper with agreed structure. Agreement on general compliance route among authors; argumentation written and acknowledged by all authors. M3: argumentation documented and agreed for 50% of clauses M4: argumentation documented and agreed for all clauses except clauses with unresolved major disagreement M5: Resolution of last clauses done. Only stylistic and presentation issues outstanding. M6: Final document available and distributed to suitable journal. Any comments to this task proposal? Lukas |
|
|
|
Hi Oscar,
Basically there are two things that I would recommend to avoid in a car Do you agree that: - bug fixes in the implementation shall be applied to increase availability of the system, e.g., a bug is identified and has been assessed to potentially crash the system? - changes due to lessons learned in the design shall be applied once the new design has proven itself to be superior (of higher quality etc.)? - bug fixes that close security holes in the system must be applied within a controlled process and defined time limit? If you agree to at least one of those above, a (although comparatively slowly) moving target is unavoidable. One incident of each of those three cases will most likely appear in an connected device with a product life-time that goes beyond a few months. I think it is important to judge the contribution of each activity to eliminate hazards. In practice, this the main point what safety experts do. Best regards, Lukas |
|
|
|
Oscar, Nicholas, I think we are talking about the same thing, we certainly have different wording, we might differ in the level of how formal this thing must be recorded, and we might differ in our belief on the level of competence and reflection it requires to consume and use this thing (i.e., just calling it "checklists" suggest very little reflection and competence.).
Lukas -----Ursprüngliche Nachricht----- Von: devel@... [mailto:devel@...] Im Auftrag von Oscar Slotosch Gesendet: Freitag, 12. April 2019 12:25 An: devel@... Betreff: Re: [devel] Task proposal: Public safety compliance argumentation for Linux Hi Lukas, I agree completely, just I don't call it route ;-) The safety plans are generic, reusable and parameterized, as well as the corresponding checklists, But refer to a process (which can also be generic), main thing is that the checklist ensure all safety requirements to be satisfied. For example a parameter can be "MODULE", denoting the software modules in the product. However every single project has to define it's own parameters, e.g. names of software modules and has to apply the checklist to their products. A simplified checklist for a MODULE could be * Has MODULE been requirements defined? * coding guidelines in MODULE satisfied? * sufficient test methods used for MODULE? * MODULE successfully tested? * MODULE test coverage reached? The collection of all (successfully) answered checklists for all parameters is the content of safety case. So probably you would call the safety plan (which includes the compliance argumentation and the process) "route" Kind regards, Oscar -----Ursprüngliche Nachricht----- Von: devel@... [mailto:devel@...] Im Auftrag von Lukas Bulwahn Gesendet: Freitag, 12. April 2019 12:02 An: devel@... Betreff: Re: [devel] Task proposal: Public safety compliance argumentation for Linux Hi Oscar, Given that the final safety plan and safety case shall be used for multiple products in multiple companies, we need to split the safety plan and safety case in a staged way: 1. You have a basic argumentation how you interpret the objectives, what you think is the appropriate measure to show compliance towards the objective of a standard in the context of the Linux kernel/Linux kernel development. This basic argumentation should be "re-usable" independent of the specific version you select. E.g., we would not expect that a measure is sufficient for 4.4 but we believe that 4.9 it is not sufficient or vice versa. This is (at least in my understanding) the definition of the compliance route. You could also call it a functional safety plan draft or some kind of functional safety concept. 2. You have a specific product in mind: you have specific requirements in mind, you have selected a specific Linux kernel version, you have a specific configuration. Now, you apply this compliance route and you get a safety case. That does help to understand our terminology? I hope it is also clear why we want to split this in these two stages. Best regards, Lukas -----Ursprüngliche Nachricht----- Von: devel@... [mailto:devel@...] Im Auftrag von Oscar Slotosch Gesendet: Donnerstag, 11. April 2019 19:42 An: devel@... Betreff: Re: [devel] Task proposal: Public safety compliance argumentation for Linux Hi Lucas and Nicolas, and all Very interesting discussions and good ideas. From my point of view speaking of "new compliance route" sounds a bit like writing a new standard, which I think we should avoid, even being suspect to this should be avoided in order to convince certification authorities and safety experts easier. From my point of view we can "tailor" the standards if necessary. However, at least the ISO states (2-6.4.5.1.b) that there should be an argument for it's safety. I agree with Proposal of Lucas, but would prefer to be more standard compliant in a way that we Have safety plan & safety case by - select the requirements (standard) to comply with - define the process, focussing on the verification of the selected requirements - argue compliance And generate: compliance report and check-lists for the concrete product. We using the freely available PMT tool from the research project SPEDIT from http://www.validas.de/en/tools/ That we have been using during certification of our qualification processes and that we plan to make open source. By using a more standard approach we definitely have less issues in convincing assessors ;-). Kind regards, Oscar PS Klever, cucinelle,.. are a cool tool, but we should must not forget that standards require to have requirements, systematic tests, code coverage, which I was missing a bit in the discussion. What about Gcov? Gdb?,... I would consider klever, etc. as great tools to check for guidelines (which we have to satisfy anyhow and we are free to select them;-). -----Ursprüngliche Nachricht----- Von: devel@... [mailto:devel@...] Im Auftrag von Lukas Bulwahn Gesendet: Donnerstag, 11. April 2019 16:17 An: devel@... Betreff: [devel] Task proposal: Public safety compliance argumentation for Linux Hi all, From the discussion with Nicholas, I created a high-level summary of one first central task that I see around Enabling Linux in Safety Applications. As we do not have a public safety compliance route for Linux available and I see this as the "first" task in a waterfall model development, I consider this important and sketched this roughly below. This does not mean that other topics cannot be investigated independently; it is just that the results of those activities eventually will link into this central compliance route document. I tried to formulate the goal, the activity, the resulting work product, my personal effort estimation and rough milestones for the overall task. Public safety compliance argumentation for Linux Goal: Public safety compliance argumentation for Linux available and approved by five major companies in safety-related system development and three certification agencies Needed activities - Agree on common safety standard and general compliance route - Assess differences of: - IEC 61508 compliance route - ISO 26262 Part 6 compliance route - Result: Short white paper (~6 pages) on the suitability of the different standards to a Linux kernel verification from multiple authors from different companies and approved by multiple certification agencies - Provide fitting into ISO 26262 Ed 2 safety argumentation - Result: One section in the white paper - Define clear scope of compliance argumentation - Result: Definition of an assumed development interface agreement of the compliance route document - Agree on common interpretation - Result: A shared interpretation and compliance route document - Define possible methods to meet interpretation - Result: Mostly unified set of methods to achieve compliance - Note in case of disagreements, multiple methods can be listed as alternatives Work Product - Documentation on: - suitability of the different standards to a Linux kernel verification - fitting into an ISO 26262 Ed 2 safety argumentation - an assumed DIA - interpretation of clauses and identified/recommended activity - Assessment report from certification authorities My Personal Effort Estimation 14x 2-day expert workshops + 5 month offline effort per author: ~100 days work, i.e., 7 days offline work per workshop between workshops for each author = 6 month per author Expecting 5 to 7 main authors from the different companies. If companies have already prepared such documents internally, this could lower the effort for the author of that company. 3 - 4 months effort of a technical writer to assist in improving style, wording and presentation Milestones to measure progress and state of work M1: Experts identified, obtained commitment from authors and if needed, authors are contracted. Initial technical kick-off workshop planned. M2: First draft of paper with agreed structure. Agreement on general compliance route among authors; argumentation written and acknowledged by all authors. M3: argumentation documented and agreed for 50% of clauses M4: argumentation documented and agreed for all clauses except clauses with unresolved major disagreement M5: Resolution of last clauses done. Only stylistic and presentation issues outstanding. M6: Final document available and distributed to suitable journal. Any comments to this task proposal? Lukas |
|
|
|
Hi all,
I hope the discussion has clarified a bit the intended scope of this work. Who would be interested in contributing to the COLLABORATIVE writing/creation of this artefact? I would hope that the supporting companies can provide the necessary means for the experts to do this work. Best regards, Lukas -----Ursprüngliche Nachricht----- Von: devel@... [mailto:devel@...] Im Auftrag von Lukas Bulwahn Gesendet: Donnerstag, 11. April 2019 16:17 An: devel@... Betreff: [devel] Task proposal: Public safety compliance argumentation for Linux Hi all, From the discussion with Nicholas, I created a high-level summary of one first central task that I see around Enabling Linux in Safety Applications. As we do not have a public safety compliance route for Linux available and I see this as the "first" task in a waterfall model development, I consider this important and sketched this roughly below. This does not mean that other topics cannot be investigated independently; it is just that the results of those activities eventually will link into this central compliance route document. I tried to formulate the goal, the activity, the resulting work product, my personal effort estimation and rough milestones for the overall task. Public safety compliance argumentation for Linux Goal: Public safety compliance argumentation for Linux available and approved by five major companies in safety-related system development and three certification agencies Needed activities - Agree on common safety standard and general compliance route - Assess differences of: - IEC 61508 compliance route - ISO 26262 Part 6 compliance route - Result: Short white paper (~6 pages) on the suitability of the different standards to a Linux kernel verification from multiple authors from different companies and approved by multiple certification agencies - Provide fitting into ISO 26262 Ed 2 safety argumentation - Result: One section in the white paper - Define clear scope of compliance argumentation - Result: Definition of an assumed development interface agreement of the compliance route document - Agree on common interpretation - Result: A shared interpretation and compliance route document - Define possible methods to meet interpretation - Result: Mostly unified set of methods to achieve compliance - Note in case of disagreements, multiple methods can be listed as alternatives Work Product - Documentation on: - suitability of the different standards to a Linux kernel verification - fitting into an ISO 26262 Ed 2 safety argumentation - an assumed DIA - interpretation of clauses and identified/recommended activity - Assessment report from certification authorities My Personal Effort Estimation 14x 2-day expert workshops + 5 month offline effort per author: ~100 days work, i.e., 7 days offline work per workshop between workshops for each author = 6 month per author Expecting 5 to 7 main authors from the different companies. If companies have already prepared such documents internally, this could lower the effort for the author of that company. 3 - 4 months effort of a technical writer to assist in improving style, wording and presentation Milestones to measure progress and state of work M1: Experts identified, obtained commitment from authors and if needed, authors are contracted. Initial technical kick-off workshop planned. M2: First draft of paper with agreed structure. Agreement on general compliance route among authors; argumentation written and acknowledged by all authors. M3: argumentation documented and agreed for 50% of clauses M4: argumentation documented and agreed for all clauses except clauses with unresolved major disagreement M5: Resolution of last clauses done. Only stylistic and presentation issues outstanding. M6: Final document available and distributed to suitable journal. Any comments to this task proposal? Lukas |
|
|
|
Nicholas Mc Guire
On Fri, Apr 12, 2019 at 05:58:58AM +0000, Oscar Slotosch wrote:
Hello Nicholas,That would be nice - but it is not realistic - you will need updates in the field - even upgrades - we are talking about a larget set of elements that have significant numbers of findings over time (notably CVEs) and also feature increments that will be most likely demanded by customers. Try giving someone a 2.6.9 kernel on a laptop and see how long they accept that ! * unconventional argumentations like "ISO 26262 is not applicable" or "MCDC being waste of time & money"ISO 26262 is normatively not applicable to complex systems - this is not unconventional - just read IEC 61508 and the relation to the derived standards - all of them point back to 61508 for "complex" systems (IEC 61511, EN 50128, IEC 62061 and ISO 26262 was developed as consolidated SotA of MCUs running OSEK/AUTOSAR (classic that is) type systems). Go into a safety converence and ask the participants for a definition of safety case - it is unfortunately not a well defined term - which is why I prefere not to use it - but fine. The standard does not have simple requirements to offer - and simply checkboxing things does not entail safety - there are a number of clasues that are stating dependency on size,complexity, novelty of design and novelty of technology - those must be interpreted to make any sense - interpreted IN context. And satisfying all requirements of any of the standard does not assure that you built a safe system - and that is the goal. Aside from ISO 26262 not having adequate provisions for the use of pre-existning (non-safe) elements. I would disagree - safety planing is a set of measures, processes and competency that has the goal to build safe systems not compliant systems. Not me - I'm aiming for a system that will not kill too many people I frankly do not care to much about compliance to the wrong standard (ISO 26262) thx! hofrat |
|
|
