What is the position of standards regarding formal methods in my industry segment?
(e.g. are they enforcing, highly recommending, etc.)
- Theme: External Factor Pushing for Formal Method Adoption (ExFac)
- Role: HM
Overview of Reference Standards for Safety Critical Systems
The landscape of standard includes both generic and domain specific standards. IEC61508 is the key generic safety standard and several domain specific standards specialise it, as shown in the following figure:
- IEC 61511 (published in 2003) addresses the industrial processes
- IEC 61513 (published in 2001) addresses the nuclear industry
- IEC 62061 (published in 2005) addresses the machine safety
- IEC 50126/50128/50129 where (respectively published/updated in 1999/2001/2003) target the railway sector. IEC 50128  targets the software.
- ISO 26262 (published in 2011) addresses the automotive sector
In the aeronautic and space sector, there are standards not directly linked to IEC61508.
In the software related to medical devices there are also specific standards, like IEC 62304 (2006).
Other generic standard may also have a more underlying role, for example IEC-12207 which specifies the software development life-cycle.
The above already shows a wide variety of standards. In addition to sector specific constraints, there are also a number of other reasons for such a variety of standards including historical reasons (uncoordinated work, national level) and market reasons (market protection) . Moreover those standard are also evolving hopefully to a better integration, so they are revised (the publication year is generally appended to their numerical identifier such as in EN 61508:2002) or more specific scheme (like the "B" in DO-178B).
Beyond this variety, standards exhibit common aspects:
- definition of safety assurance levels: all standards have a risk oriented approach of their safety functions and classify them in a number of dependability classes, typically by specifying PFD (Probability of Failure on Demand) and RRF (Risk Reduction Factor) figures. This classification varies across the standards for example is IEC61508, the classification ranges from SIL1 that is the least dependable to SIL4, that is the most dependable. The DO-178B has a letter-base classification ranging from level "E" that is the least dependable, to level "A" that is most dependable.
- prescription level: standards can be prescriptive (obligation to demonstrate compliance) or just give recommendations. However in practice deviating from recommendations can be difficult as it may requires non trivial work to motivate it and convince the certification body used to the well-established recommended practices. This plays a role w.r.t the introduction of formal methods depending if they are simply state, recommended, or highly recommended.
- scope: it can address the software, the hardware or the complex system as a whole. It can also focus on specific part of the development process (like development life-cycle, quality management, safety assessment, system certification...). In the following table, inspired from , we give a classification of standards according to scope and domain.
|Domain||System Certification||Development Process||Safety Assessment|
Overview of standards position wrt Formal Methods
Based on the SIL level, IEC 61508 classifies methodologies, techniques and activities as “not recommended”, “recommended”, “highly recommended”, etc. Among the “highly recommended” techniques for SIL4 are inspection and reviewing, use of an independent test team and the use of formal methods.
To assure the required safety, reliability and correctness, a single "highly recommended" techniques is not enough: it can only be reached using a carefully chosen combination of appropriate techniques. To reach this in a project requires the definition of a dedicated system engineering process as described in .
The EN50128 guidelines, issued by the European Committee for Electrotechnical Standardization (CENELEC), address the development of ”Software for Railway Control and Protection Systems”, and constitute the main reference for railway signalling equipment manufacturers in Europe, with their use spreading to the other continents and to other sectors of the railway (and other safety-related) industry.
In EN-50128, Formal Methods/Proofs are explicitly identified as relevant technique/measure for software requirements specification, software architecture, software design, implementation, verification and testing and data preparation techniques. More precisely they are "recommended" for SIL levels 1 and 2 and "Highly Recommended" for SIL levels 3 and 4. Particular example of Formal Methods cited are: CCS, CSP, HOL, LOTOS, OBJ, Temporal Logic, VDM, Z and B. In the on-going 2011 revision of the standard, additional constraints are put on tools, especially code/data generation tools with respect to the need of a specification and evidence that the implementations complies with the specification.
Despite this and success stories like METEOR, formal methods have not spread in the whole railway signalling industries, where much software is still written and tested in traditional ways (with testing effort usually summing up to more that 50% of the development effort). This lack of adoption is due to the investments needed to build up a formal methods culture, and to the high costs of commercial support tools. Moreover, equipment can conform to CENELEC without applying formal methods. 
However as EN-50128 requires that the bugs detection and fixing activities imply reviews of early phases. This causes high costs and stretched time. Consequently companies are becoming interested to apply formal methods in the specification and design phases as it is the only solution to shift back the effort to the design team.
Another barrier is that, although mentioned in the standard, the certification bodies might not be familiar with them as most of the system they certify follow a test-based approach. The first time a certification body is submitted a system whose justification relies on such formal argument, they might require additional information to be convinced. Typically, one should foresee some specific training-level information in the certification process. Consider the FAQ TSP-QAP-1 for insight on this point. Once acquired, new system can easily follow the same path. Siemens has gone through this process with the B-method, and it is now accepted by the certification bodies, so that the next projects have become easier to certify. In the specific market of metro lines, the B-method has even become the standard required by the market .
DO-178B was published a while ago, in 1992. It does not recommend or propose a specific development process or methodology. The certification approach is to demonstrate compliance of the process and produced artefacts with a set of goals related to:
- software planning process
- development process
- artefact verification activities: ranging from requirements to design to code.
- integration testing
- verification of process
- configuration management
As mentioned above, DO178-B ranks the software category in 5 dependability classes: from A (most dependable: catastrophic effect) to E (least dependable:no effect).
The scope of the application of formal methods is the artefact verification activities. In DO-18B, it can be achieved though a combination of three kind of activities:
- reviews: relying on common software engineering practices: checklists, looking for specific kinds of defects/flaws (ambiguities, inconsistencies, incompleteness...)
- analysis: typically coverage analysis, data and control flow analysis
- testing activities: requirements based and with the need to demonstrate traceability
Related to Formal Methods, DO-178B categorise them as "alternative methods" because, at the time the document was produced (1992), they were evaluated to have an inadequate maturity level. However they can be used "as long as they can be demonstrated to address the goals of the standard, and their usage is adequately planned and described (...)" 
The wikipedia:DO-178C revision of the standard was published in January 2012. It is replacing DO-178B as the primary document by which the certification authorities such as FAA, EASA and Transport Canada will approve all commercial software-based aerospace systems. DO-178C is explicitely addressing formal methods to complement dynamic testing, they can be used selectively or as primary source of evidence.
The sector specific standard is wikipedia:ISO 26262. It is based on the IEC50128 and was release in November 2011.
ISO 26262 recommends formal methods but semi-formal methods are highly recommend. In such ISO standards, only one development method can be "highly recommended" (above all others). On the other hand, many methods can be "recommended". At the beginning of the debate on ISO 26262, it was suggested that formal methods should be highly recommended. After a intensive lobbying from large automotive companies, which are convinced that formal method will incur a significant cost overhead, semi-formal method have taken the "highly recommended" status while formal method are just recommended.
Conclusion: general position of standards w.r.t Formal Methods and possible strategies
Some standards recommend or even highly recommend formal methods but very few standard describe how to manage a formal development.It is therefore difficult to prove that the development process comply with the standards. Certification authorities have to be convinced about this issue.
As formal methods are less widespread and certification authorities are less familiar with them as compared to the classical development methods, there is more work to be done in comparison with other methods, especially the first time, and despite the fact that better argument could be provided. However the investment might be worth the effort as shown by the Siemens case for B.
- ↑ European Committee for Electrotechnical Standardization. EN 50128, Railway Applications Communications, Signaling and Processing Systems Software for Railway Control and Protection Systems, 2001.
- ↑ 2.0 2.1 RTCA/DO-178B "Software Considerations in Airborne Systems and Equipment Certification", 1992
- ↑ http://www.ecss.nl
- ↑ 4.0 4.1 M. Bozzano, A. Willafiorita, Design and Safety Assessment of Critical Systems, Auerbach Publishers Inc, 2010.
- ↑ P. Kars. Formal Methods in the Design of s Storm Surge Barrier Control System. In Lectures on Embedded Systems, European Educational Forum, School on Embedded Systems, Grzegorz Rozenberg and Frits W. Vaandrager (Eds.). Springer-Verlag, London, UK, 353-367, 1996.
- ↑ S. Bacherini, A. Fantechi, M. Tempestini, N. Zingoni, “A Story about Formal Methods Adoption by a Railway Signaling Manufacturer”, in Proc. FM 2006, Hamilton, Canada, August 2006, Lecture Notes in Computer Science, 4085, 1996.
- ↑ DEPLOY Deliverable D7, Measurement Methodology Guide, http://www.deploy-project.eu/pdf/d7-revised-final.pdf