In the automotive industry, we can see a clear trend towards a growing complexity of embedded software combined with more and more software being used in safety critical applications. In this context the ISO 26262 standard plays an important role, providing recommendations for the development process of embedded systems. In addition to the process, ISO 26262 also talks about the tools that are used to develop and test the system as well as the qualification of these tools. But a tool qualification is actually not needed for all tools. This article will describe how to find out if a tool needs to be qualified and how to perform a tool qualification. I will also give some examples in the context of Model-based software development and testing.
Does a tool need to be qualified?
The answer to this question depends on the use case and the project specific workflow. It is possible that a particular tool needs to be qualified in one project, while a tool qualification is not needed in another project. The main question is: „Could the tool be responsible for an error in the final product (=the embedded software/system)?“ This includes “the possibility that the malfunctioning software tool can ..fail to detect errors in a safety-related item….“ (ISO 26262-8, 11.2). If the answer is YES, then we need to have a high confidence in the tool and thus a tool qualification is required. So the need to qualify a tool or not actually depends on the confidence we need to have in this particular tool. ISO 26262 defines 3 levels of tool confidence. For the lowest level (TCL1), no confidence is needed so a tool qualification is not necessary.
The actual path towards this answer is basically relying on a use case analysis:
1. TI: Tool Impact
As a first step, we’ll have to determine the tool impact. If a tool can „introduce or fail to detect“ errors, the tool actually will have an impact on the final product quality. This is then called TI2. If a tool does not play any role regarding the final product quality, there is no tool impact (TI1).
Going into some examples, let’s first consider a code generator or a compiler. As these tools actually generate something which goes into production, a malfunction of this kind of tool can of course have an impact, leading to a TI2 classification. Looking at a test tool like BTC EmbeddedTester, we will also get a TI2 classification, as a tool malfunction might lead to a situation where we fail to detect an error. As a TI1 example, let’s consider a tool like Simulink Check looking for the fulfillment of modeling guidelines on Simulink level. While many of these design rules (e.g. colors of blocks) are quite useful in the development process, a tool malfunction would not directly introduce an error into the product (in this case meaning the behavior of model and production code). If a tool is classified as TI2, we need to look at the second step (see below), if we get TI1 we immediately assign the lowest tool confidence level TCL 1.
2. TD: Tool Error Detection
The second step in the process is the evaluation of the tool error detection level. For tools that can have an impact on the safety of the final product (TI2), we just need to ask one simple question: Assuming the tool has a malfunction, what is the likelihood of detecting it later in the process. If there is a high likelihood (TD1), we don’t need to have a high confidence in the tool behaving correctly. If we would not detect a tool malfunction in subsequent process steps (TD2, TD3), we need to have a high confidence in the correct tool behavior.
Let’s again look at some examples and start with the code generator and/or compiler I mentioned above. If in my process I’m not testing the source/object code, I have a low likelihood of detecting tool malfunctions. This means a tool error detection level TD3. But due to the huge amount of configuration options, a tool qualification for code generators/compilers is either almost impossible or it would mean a dramatic limitation regarding available configuration options (and still be extremely time consuming). If we want to avoid a tool qualification for these tools, we just have to design our workflow in a way that potential tool errors are detected by other measures. In this context it’s important to mention that these measures need to be part of the process and can’t be done upfront with a test suite. For a code generator and/or compiler the solution to get to TD1 is to actually run tests on source code and object code (e.g. with a highly automatable Back-to-Back Test which is anyway recommended by ISO 26262). Of course, we still want to have a reliable and high-quality code generator/compiler to ensure an efficient development workflow, but from an ISO 26262 point-of-view tool confidence is not important if we test the tool output. Assuming we do test the source code and object code, we also need to look at the tool error detection level for the test tool. Test tools typically perform test execution and reporting in a rather automatic way, and we usually don’t have any redundant process steps to verify the results. Therefore test tools typically get a tool error detection level of TD3.
ISO 26262: 2018 provides the following examples regarding the analysis of the tool error detection:
EXAMPLE 1 TD1 is selected for a code generator when the generated source code is verified in accordance with ISO 26262. TD3 is selected for a code generator when the generated source code is not verified.
EXAMPLE 3 TD2 is selected for a tool that statically verifies the absence in source code of potential for division by zero if testing is also applied for this purpose. TD3 is selected if absence of division by zero is verified by the tool alone.
3. TCL: Tool Confidence level
Based on the process described above, we have mainly two possible results:
TCL1: This is the lowest tool confidence level. The tool does not play an important role regarding the quality of our final product. Therefore, we do not need to have a high confidence in the correct tool behavior from an ISO 26262 view. A tool qualification is not needed.
TCL2/3: This corresponds to a medium/high tool confidence level. The tool plays an important role regarding the quality of our final product, so we need to have a certain level of confidence and therefore need to perform a tool qualification to demonstrate the reliability of that tool.
TCL2 and TCL3 are pretty similar, as they both demand tool qualification. There are just some minor differences regarding the recommend qualification methods depending on the ASIL level.
How to perform a tool qualification?
Let’s assume we ended up with a TCL 2 or TCL 3 classification for a particular tool. This means, we need to look at section 11.4.6 of ISO 26262-8 which lists different potential methods for performing a tool qualification. If a certain method is recommended or not depends on the ASIL level of the particular project. A tool qualification can only be performed by the user in context of a particular project, but of course tool providers can perform a lot of work upfront to make this task as easy as possible. The proposed methods are:
- Increased Confidence from Use: To apply this method, I’d need to have a previous project which was using the same tool versions and where I can demonstrate that the tool didn’t show any malfunction. But in new projects, the toolchain and the tool versions do slightly change most of the time, meaning that this method is often not applicable.
- Evaluation of the Development Process: This method would require a detailed analysis of the tool development process. Obviously, a user would not take the time to visit his tool providers and perform a time consuming assessment of the tool development process. But of course tool providers can invest into a generic pre-qualification with an authority like the German TÜV to provide the results of this measure upfront to the users.
- Validation of the Software Tool: This basically means, that we need to develop a test suite which covers all use cases of the software tool. According to [ISO 26262-8, 11.4.9] this can be done either by the user or by the tool vendor. If the tool vendor is able to invest in this, it can of course dramatically reduce or eliminate efforts on the user side.
- Development in accordance with a safety standard: Tools involved in embedded software development are mostly PC based and typically not developed according to a safety standard.
Tool qualification of BTC tools
To make this as easy as possible for our users, BTC has worked with German certification authority TÜV Süd to get a pre-qualification certificate that we provide free of charge to our customers. This certificate addresses the methods “b.” and “c.” mentioned above. For method “b.”, TÜV Süd has performed an in-depth analysis of our internal development process by looking at things like requirements management, change management, traceability as well as our processes and documentation inside of our quality assurance team (which by the way is and should be independent from the development department). For method c. we developed a test suite with over 600 different software units (e.g. Simulink models), more than 50% of them provided by customers to have realistic examples of real automotive embedded development projects. Both methods have been performed for all use cases of our toolchain incl. Requirements-based Testing, Back-to-Back Testing, code coverage analysis, automatic test generation, robustness analysis, formal specification and formal verification. Assuming a customer stays within the supported environment (meaning he should use one of the officially supported versions of Matlab, TargetLink, Windows etc.), there are no additional measures needed on his side. It is in particular not needed to run a validation suite on the customer side (which always costs a lot of time and money).
Conclusion
Within a safety critical project, tool qualification is only needed for some tools. To determine if a tool qualification is needed or not, ISO 26262 describes a simple two-step process to find out how much confidence we need to have in a particular tool. If no confidence is needed, we get the lowest tool confidence level (TCL 1) and a tool qualification is not needed. Test tools (like the BTC tool chain) usually require a high level of confidence and therefore a tool qualification. While this can be an expensive and time-consuming process, a tool pre-qualification from authorities like TÜV can almost eliminate the qualification efforts on the user side.
