Linux and ioctls


Elana C
 

Hi,

To add to yesterday's discussion – see  https://www.kernel.org/doc/html/latest/process/botching-up-ioctls.html

This is basic common sense advice for any usage of ioctls.  There is a nice summary of well know pitfalls to avoid (some of which we could check using static analysis tools).

And the caveat that any usage of ioctls should be checked against preferred Linux options such as sysfs or debugfs.

 

Hardware dependent features such as the ioctl syscall, devmem, etc. should be avoided altogether whenever possible in safety critical applications, for reasons of safety as well as security.  And the only really necessary usage should be in driver code, not safety critical management code.

Regards

Elana


Robert Krutsch
 

Hi Elena,

The API of the watchdog is ioctl based in this case. Even if we would have a user-space API on top, as all complex drivers have we would still end up at the same ioctl as the interface between user space and kernel-space.

What can be done is to say: "our watchdog is to be considered as a module out-of-tree and we assume someone else provides us with a watchdog module that fulfils the FuSa needs". From our req. we can deduce requirements for the watchdog module and stop there.

This is a simplification of scope and would take out some parts of the analysis for this App but it would not solve the issues of someone that solves the problem for real. They would need to analyze the watchdog module and deal with the ioctl or rewrite this module altogether as a user-space driver. If that sounds familiar , it is basically what you have to do when you port to QNX... 

BR,
Robert

On Wed, Mar 10, 2021 at 9:29 AM Elana C <elana.copperman@...> wrote:

Hi,

To add to yesterday's discussion – see  https://www.kernel.org/doc/html/latest/process/botching-up-ioctls.html

This is basic common sense advice for any usage of ioctls.  There is a nice summary of well know pitfalls to avoid (some of which we could check using static analysis tools).

And the caveat that any usage of ioctls should be checked against preferred Linux options such as sysfs or debugfs.

 

Hardware dependent features such as the ioctl syscall, devmem, etc. should be avoided altogether whenever possible in safety critical applications, for reasons of safety as well as security.  And the only really necessary usage should be in driver code, not safety critical management code.

Regards

Elana


Elana C
 

Thanks, Robert, for your comment.

This is all fine and true.  And as noted, at the end of the day, the drivers end up calling the ioctl system interface.  But then my question ends up being, for qualification of the watchdog subsystem – will we need to qualify each driver and its implementation to support each specific hardware interface and each specific system call which accesses that hardware interface?

 

I am only concerned that this an endless saga.  We can do this (as you say) "for real", and spend weeks digging in to each low level system call called by each driver.

Moreover, if you check the header files for the watchdog core and dev utilities, you can readily see that the watchdog (as "simple" as it seems …) touches on many other complex subsystems – such as the file system and the memory management subsystem. 

Where does "for real" end?

And more generically – why is this definition of "for real" in the use case of Linux (with its ~30 million lines of code …) so much more stringent than the "normal" definition of "for real" in safety qualification of non-open source commercial software?  In the latter case, safety requirements are defined, implemented and validated on a much higher level.

Regards

Elana

 

From: safety-architecture@... <safety-architecture@...> On Behalf Of Robert Krutsch
Sent: Wednesday, March 10, 2021 12:49 PM
To: safety-architecture@...
Subject: Re: [ELISA Safety Architecture WG] Linux and ioctls

 

Hi Elena,

 

The API of the watchdog is ioctl based in this case. Even if we would have a user-space API on top, as all complex drivers have we would still end up at the same ioctl as the interface between user space and kernel-space.

 

What can be done is to say: "our watchdog is to be considered as a module out-of-tree and we assume someone else provides us with a watchdog module that fulfils the FuSa needs". From our req. we can deduce requirements for the watchdog module and stop there.

 

This is a simplification of scope and would take out some parts of the analysis for this App but it would not solve the issues of someone that solves the problem for real. They would need to analyze the watchdog module and deal with the ioctl or rewrite this module altogether as a user-space driver. If that sounds familiar , it is basically what you have to do when you port to QNX... 

 

BR,

Robert

 

On Wed, Mar 10, 2021 at 9:29 AM Elana C <elana.copperman@...> wrote:

Hi,

To add to yesterday's discussion – see  https://www.kernel.org/doc/html/latest/process/botching-up-ioctls.html

This is basic common sense advice for any usage of ioctls.  There is a nice summary of well know pitfalls to avoid (some of which we could check using static analysis tools).

And the caveat that any usage of ioctls should be checked against preferred Linux options such as sysfs or debugfs.

 

Hardware dependent features such as the ioctl syscall, devmem, etc. should be avoided altogether whenever possible in safety critical applications, for reasons of safety as well as security.  And the only really necessary usage should be in driver code, not safety critical management code.

Regards

Elana


Paoloni, Gabriele <gabriele.paoloni@...>
 

Hi Elana, all

 

WRT the watchdog subsystem I believe that the analysis should be focused on HW independent functions. I.e.:

# The WatchDog Timer Driver Core.

obj-$(CONFIG_WATCHDOG_CORE)    += watchdog.o

 

watchdog-objs  += watchdog_core.o watchdog_dev.o

 

watchdog-$(CONFIG_WATCHDOG_PRETIMEOUT_GOV)    += watchdog_pretimeout.o

 

obj-$(CONFIG_WATCHDOG_PRETIMEOUT_GOV_NOOP)    += pretimeout_noop.o

obj-$(CONFIG_WATCHDOG_PRETIMEOUT_GOV_PANIC)   += pretimeout_panic.o

from

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/watchdog/Makefile?h=linux-5.4.y

Note that at this stage I still don’t know if the settings CONFIG_WATCHDOG_PRETIMEOUT_GOV *  are applicable (to be figured out…)

 

Everything below this point in the Makefile is HW specific and I don’t think should be considered in the analysis (or at least it is lower priority)

Thanks

Gab

 

 

 

From: safety-architecture@... <safety-architecture@...> On Behalf Of Elana C
Sent: Wednesday, March 10, 2021 12:01 PM
To: Robert Krutsch <krutsch@...>; safety-architecture@...
Subject: Re: [ELISA Safety Architecture WG] Linux and ioctls

 

Thanks, Robert, for your comment.

This is all fine and true.  And as noted, at the end of the day, the drivers end up calling the ioctl system interface.  But then my question ends up being, for qualification of the watchdog subsystem – will we need to qualify each driver and its implementation to support each specific hardware interface and each specific system call which accesses that hardware interface?

 

I am only concerned that this an endless saga.  We can do this (as you say) "for real", and spend weeks digging in to each low level system call called by each driver.

Moreover, if you check the header files for the watchdog core and dev utilities, you can readily see that the watchdog (as "simple" as it seems …) touches on many other complex subsystems – such as the file system and the memory management subsystem. 

Where does "for real" end?

And more generically – why is this definition of "for real" in the use case of Linux (with its ~30 million lines of code …) so much more stringent than the "normal" definition of "for real" in safety qualification of non-open source commercial software?  In the latter case, safety requirements are defined, implemented and validated on a much higher level.

Regards

Elana

 

From: safety-architecture@... <safety-architecture@...> On Behalf Of Robert Krutsch
Sent: Wednesday, March 10, 2021 12:49 PM
To: safety-architecture@...
Subject: Re: [ELISA Safety Architecture WG] Linux and ioctls

 

Hi Elena,

 

The API of the watchdog is ioctl based in this case. Even if we would have a user-space API on top, as all complex drivers have we would still end up at the same ioctl as the interface between user space and kernel-space.

 

What can be done is to say: "our watchdog is to be considered as a module out-of-tree and we assume someone else provides us with a watchdog module that fulfils the FuSa needs". From our req. we can deduce requirements for the watchdog module and stop there.

 

This is a simplification of scope and would take out some parts of the analysis for this App but it would not solve the issues of someone that solves the problem for real. They would need to analyze the watchdog module and deal with the ioctl or rewrite this module altogether as a user-space driver. If that sounds familiar , it is basically what you have to do when you port to QNX... 

 

BR,

Robert

 

On Wed, Mar 10, 2021 at 9:29 AM Elana C <elana.copperman@...> wrote:

Hi,

To add to yesterday's discussion – see  https://www.kernel.org/doc/html/latest/process/botching-up-ioctls.html

This is basic common sense advice for any usage of ioctls.  There is a nice summary of well know pitfalls to avoid (some of which we could check using static analysis tools).

And the caveat that any usage of ioctls should be checked against preferred Linux options such as sysfs or debugfs.

 

Hardware dependent features such as the ioctl syscall, devmem, etc. should be avoided altogether whenever possible in safety critical applications, for reasons of safety as well as security.  And the only really necessary usage should be in driver code, not safety critical management code.

Regards

Elana

---------------------------------------------------------------------
INTEL CORPORATION ITALIA S.p.A. con unico socio
Sede: Milanofiori Palazzo E 4
CAP 20094 Assago (MI)
Capitale Sociale Euro 104.000,00 interamente versato
Partita I.V.A. e Codice Fiscale  04236760155
Repertorio Economico Amministrativo n. 997124
Registro delle Imprese di Milano nr. 183983/5281/33
Soggetta ad attivita' di direzione e coordinamento di
INTEL CORPORATION, USA

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.


Christopher Temple
 

Elena, Robert, this is a good discussion and getting to the details of the challenge – thanks!

 

Your statements are in line with my thinking around effort explosion.

 

To prevent this it makes sense to keep the safety argumentation away from the deep plumbing wherever possible in practical systems.

As presented in Lyon eGas, for example, resorts to an external watchdog.

 

We had also discussed the benefits of an external watchdog in the context of the tell-tale use case.

At the time there were voices advocating for doing it with a watchdog via Linux to understand where the challenges are as part of the investigation.

 

(Since the watchdog subsystem is part of the safety mechanism it would need more than qualification.)

 

As part of the investigation this is helpful insight that the project is obtaining.

 

I’d be interested in Lukas’s opinion.

 

Best regards

Chris

 

 

From: safety-architecture@... <safety-architecture@...> On Behalf Of Elana C via lists.elisa.tech
Sent: Mittwoch, 10. März 2021 12:01
To: Robert Krutsch <krutsch@...>; safety-architecture@...
Subject: Re: [ELISA Safety Architecture WG] Linux and ioctls

 

Thanks, Robert, for your comment.

This is all fine and true.  And as noted, at the end of the day, the drivers end up calling the ioctl system interface.  But then my question ends up being, for qualification of the watchdog subsystem – will we need to qualify each driver and its implementation to support each specific hardware interface and each specific system call which accesses that hardware interface?

 

I am only concerned that this an endless saga.  We can do this (as you say) "for real", and spend weeks digging in to each low level system call called by each driver.

Moreover, if you check the header files for the watchdog core and dev utilities, you can readily see that the watchdog (as "simple" as it seems …) touches on many other complex subsystems – such as the file system and the memory management subsystem. 

Where does "for real" end?

And more generically – why is this definition of "for real" in the use case of Linux (with its ~30 million lines of code …) so much more stringent than the "normal" definition of "for real" in safety qualification of non-open source commercial software?  In the latter case, safety requirements are defined, implemented and validated on a much higher level.

Regards

Elana

 

From: safety-architecture@... <safety-architecture@...> On Behalf Of Robert Krutsch
Sent: Wednesday, March 10, 2021 12:49 PM
To: safety-architecture@...
Subject: Re: [ELISA Safety Architecture WG] Linux and ioctls

 

Hi Elena,

 

The API of the watchdog is ioctl based in this case. Even if we would have a user-space API on top, as all complex drivers have we would still end up at the same ioctl as the interface between user space and kernel-space.

 

What can be done is to say: "our watchdog is to be considered as a module out-of-tree and we assume someone else provides us with a watchdog module that fulfils the FuSa needs". From our req. we can deduce requirements for the watchdog module and stop there.

 

This is a simplification of scope and would take out some parts of the analysis for this App but it would not solve the issues of someone that solves the problem for real. They would need to analyze the watchdog module and deal with the ioctl or rewrite this module altogether as a user-space driver. If that sounds familiar , it is basically what you have to do when you port to QNX... 

 

BR,

Robert

 

On Wed, Mar 10, 2021 at 9:29 AM Elana C <elana.copperman@...> wrote:

Hi,

To add to yesterday's discussion – see  https://www.kernel.org/doc/html/latest/process/botching-up-ioctls.html

This is basic common sense advice for any usage of ioctls.  There is a nice summary of well know pitfalls to avoid (some of which we could check using static analysis tools).

And the caveat that any usage of ioctls should be checked against preferred Linux options such as sysfs or debugfs.

 

Hardware dependent features such as the ioctl syscall, devmem, etc. should be avoided altogether whenever possible in safety critical applications, for reasons of safety as well as security.  And the only really necessary usage should be in driver code, not safety critical management code.

Regards

Elana

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.


Elana C
 

Fully agreed.  At this point – awareness is fundamentally important, but I doubt we can yet make any final decisions, where we draw the lines.

 

Thanks, Gab, for sharing the makefile – this helps to highlight the different system layers.  But the makefile only buries the original questions in a high level makefile. 

  • watchdog_core and watchdog_dev cover the watchdog infrastructure. 
  • Pretimeout is a trigger which is enabled before the actual reset time, to support multistage watchdog timers which allow corrective actions to be taken before the actual reset takes place.

Each of these end up calling other subsystems, including ioctls 😊.

And that's before we even touch on support for watchdogs in multitasking environments.  Or specific kernel configurations which impact watchdog behavior.

So we are still left with our original question – how "deep" is "real" for safety qualification?  At some point, we will need to define a model of "trust" for low-level infrastructure, based on intensive testing.

 

Anyhow, let's leave this email thread for additional opinions and feedback.

For the Development Process WG, I have been "feeding" github with some drafts, and we can continue to discuss the details in tomorrow's meeting.

https://github.com/elisa-tech/wg-development-process  (you should ignore the svg documents for now, I am playing around with using Inkspace as an open source drawing tool – but that is a side issue.)

Regards

Elana

 

 

From: Christopher Temple <Christopher.Temple@...>
Sent: Wednesday, March 10, 2021 2:05 PM
To: Elana Copperman <Elana.Copperman@...>; Robert Krutsch <krutsch@...>; safety-architecture@...
Subject: RE: [ELISA Safety Architecture WG] Linux and ioctls

 

Elena, Robert, this is a good discussion and getting to the details of the challenge – thanks!

 

Your statements are in line with my thinking around effort explosion.

 

To prevent this it makes sense to keep the safety argumentation away from the deep plumbing wherever possible in practical systems.

As presented in Lyon eGas, for example, resorts to an external watchdog.

 

We had also discussed the benefits of an external watchdog in the context of the tell-tale use case.

At the time there were voices advocating for doing it with a watchdog via Linux to understand where the challenges are as part of the investigation.

 

(Since the watchdog subsystem is part of the safety mechanism it would need more than qualification.)

 

As part of the investigation this is helpful insight that the project is obtaining.

 

I’d be interested in Lukas’s opinion.

 

Best regards

Chris

 

 

From: safety-architecture@... <safety-architecture@...> On Behalf Of Elana C via lists.elisa.tech
Sent: Mittwoch, 10. März 2021 12:01
To: Robert Krutsch <krutsch@...>; safety-architecture@...
Subject: Re: [ELISA Safety Architecture WG] Linux and ioctls

 

Thanks, Robert, for your comment.

This is all fine and true.  And as noted, at the end of the day, the drivers end up calling the ioctl system interface.  But then my question ends up being, for qualification of the watchdog subsystem – will we need to qualify each driver and its implementation to support each specific hardware interface and each specific system call which accesses that hardware interface?

 

I am only concerned that this an endless saga.  We can do this (as you say) "for real", and spend weeks digging in to each low level system call called by each driver.

Moreover, if you check the header files for the watchdog core and dev utilities, you can readily see that the watchdog (as "simple" as it seems …) touches on many other complex subsystems – such as the file system and the memory management subsystem. 

Where does "for real" end?

And more generically – why is this definition of "for real" in the use case of Linux (with its ~30 million lines of code …) so much more stringent than the "normal" definition of "for real" in safety qualification of non-open source commercial software?  In the latter case, safety requirements are defined, implemented and validated on a much higher level.

Regards

Elana

 

From: safety-architecture@... <safety-architecture@...> On Behalf Of Robert Krutsch
Sent: Wednesday, March 10, 2021 12:49 PM
To: safety-architecture@...
Subject: Re: [ELISA Safety Architecture WG] Linux and ioctls

 

Hi Elena,

 

The API of the watchdog is ioctl based in this case. Even if we would have a user-space API on top, as all complex drivers have we would still end up at the same ioctl as the interface between user space and kernel-space.

 

What can be done is to say: "our watchdog is to be considered as a module out-of-tree and we assume someone else provides us with a watchdog module that fulfils the FuSa needs". From our req. we can deduce requirements for the watchdog module and stop there.

 

This is a simplification of scope and would take out some parts of the analysis for this App but it would not solve the issues of someone that solves the problem for real. They would need to analyze the watchdog module and deal with the ioctl or rewrite this module altogether as a user-space driver. If that sounds familiar , it is basically what you have to do when you port to QNX... 

 

BR,

Robert

 

On Wed, Mar 10, 2021 at 9:29 AM Elana C <elana.copperman@...> wrote:

Hi,

To add to yesterday's discussion – see  https://www.kernel.org/doc/html/latest/process/botching-up-ioctls.html

This is basic common sense advice for any usage of ioctls.  There is a nice summary of well know pitfalls to avoid (some of which we could check using static analysis tools).

And the caveat that any usage of ioctls should be checked against preferred Linux options such as sysfs or debugfs.

 

Hardware dependent features such as the ioctl syscall, devmem, etc. should be avoided altogether whenever possible in safety critical applications, for reasons of safety as well as security.  And the only really necessary usage should be in driver code, not safety critical management code.

Regards

Elana

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.


Lukas Bulwahn
 

On Wed, Mar 10, 2021 at 1:05 PM Christopher Temple
<Christopher.Temple@...> wrote:


I’d be interested in Lukas’s opinion.
Yes, this is a good and interesting first step.

I need to catch up on the actual application's safety goal, which
faults were identified, the exact intended functional behavior of the
kernel watchdog, and which faults can effectively be mitigated and
which not. So:
step 0: describe the functional behaviors (the application, and the
kernel watchdog)
step 1 (risk/fault collection): Identify the faults of both functional
behaviors (e.g., brainstorm and cluster them)
step 2 (dependent faults): Look at the common faults of both; it is
clear that the common faults are largely overlapping given a largely
common context; at least from a purely enumerative consideration of
faults, without considering the actual probability each fault has.
This is quite tricky, e.g., just for illustration, compare and assess
the probability of a compiler introducing a non-terminating loop vs.
the probability of some developer doing that. Obviously that is
relevant, as you then could ask "does it matter that two different
compilers were responsible for compilation of application and kernel
watchdog or does it matter that two different developers that were
responsible for implementation of application and kernel watchdog?";
that leads to step 3...
step 3 (complexity vs. mitigation decision): Determine if the added
complexity of the kernel watchdog functionality is actually worth the
remaining mitigation of faults (after assessment of individual risks
and common dependent risks), or if other alternatives should be
considered.

I will try to dig into that, and then comment in more detail. If
someone from the group could quickly point me to those documents that
exist for step 0 (or even step 1), I will give it a read.


Lukas