V. Ilyin.

Head of Quality Service at TopSBI

"If you do anything

wrong - no need
expect the right result."

Folk Chinese wisdom

Comprehensive solution for quality assurance tasks software tools involves the development and implementation of a particular quality management system (Quality Management System - QMS). In world practice, it is the system based on the requirements of international standards of the ISO 9000 series that has become most widespread, because it determines exactly the most General requirements, and to the PS, including, and thus, in general, already predetermines the initial maturity of the processes that is necessary to comply with many industry models and standards in the IT field .

But to the question whether the introduction of a quality system and successful certification guarantees the release of a quality product, it is necessary to answer honestly - "no".

Emphasizing that ISO 9000 is "a great idea," the Gartner Group recommends that ISO 9001 certification be viewed only as a starting point on the road to quality (1).

It lays, as it were, the "skeleton" of the quality system, and filling this system with "muscles" (professional content based on already special, industry standards and methodologies, such as CMM) can provide a quality level that meets the growing market requirements.

In connection with the foregoing, both from a methodological and practical point of view, many experts in the field of quality management consider it appropriate to build a strategy for the development of IT companies as follows:

First, develop and implement a QMS according to the ISO 9001:2000 model. (After all, most of the companies that are now at the 4th and 5th levels of SW-CMM first went through bringing their processes in line with the ISO model. As practice shows, this is the best option in terms of managing the development of the QMS and reducing risks).

And only then begin to develop and implement the key processes of the SW-CMM model and, if necessary, the CMMI model.

In order to understand how this is really correct, let's compare these models.

1. Review of applicants.

Let's spend short review the most popular standards that can be used by an IT company to optimize their business processes.

ISO9001. The most popular, and especially in Europe, is ISO 9001(2)

At the same time, methodically, in full accordance with the discipline of building complex systems, the ISO 9001 standard provides, on the one hand, for building an organizational system "top down": from the goals of the enterprise and its policy to organizational structure and the formation of business processes, and on the other hand, the iterative development of the organizational system through measurement and improvement mechanisms.

Simplified "quality", according to the series ISO standards 9000, this is a situation in which consumers receive from the manufacturer products that meet their direct requirements and implicit expectations. Therefore, quality management, in accordance with ISO 9000, involves the use of the so-called. "process approach", when the most optimal chain of "transformation-processes" is modeled and implemented, ensuring that the needs of consumers are perceived by the manufacturer and embodied in any product without distortion.

Many software development organizations successfully use this well-known series of ISO 9000 standards. The new version of the standards of this series was released in 2000 and already contains such concepts as the process approach, analysis and measurement, process improvement, borrowed from the CMM model and earlier. absent in previous versions of ISO 9000. True, it should be noted that the standards of this series are universal - they are not focused on any particular industry, do not take into account the specifics of the IT sphere and, in this sense, of course, in terms of the degree of specification, they are noticeably inferior to CMM. In addition, ISO 9000 does not imply any gradations (levels) of compliance and, thus, makes it difficult to determine the "true" capabilities of an organization and, accordingly, the ways of their further development.

CMM(Capability Maturity Model) was developed by the Software Engineering Institute at Carnegie Mellon University (USA) and describes the maturity model of software development processes in enterprises (3). Within the framework of this model, for each company, a certain level (one of five possible) can be compared, indicating the achieved quality of the software development process. Since these standards were developed primarily to streamline the process of selecting contractors for the US Department of Defense, they focus on software project management processes, while the technical aspects of development are less covered.

There are 316 Key Practices in SW-CMM v.1.1 (Capability Maturity Model for Software). Key Practices are what should be implemented in the enterprise and what the process evaluation team will pay attention to. They are combined into areas - Key Practices Areas (KPA) - these are already sets of interrelated processes that, when performed together, lead to the achievement of a certain set of goals.

CMMI(Capability Maturity Model Integration) - further development CMM models. In CMMI-SE/SW Version 1.02 (CMMI for Systems Engineering/Software Engineering), which is perhaps the most acceptable for developers of software systems, the number of Key Practices reaches 417.

The increase in Key Practices is related to the very purpose of developing CMMI - the model should help to avoid problems associated with the use of various industry-specific CMM models.

(Since 1991, CMMs have been developed for various applications, the most significant being:

Software development process maturity model (Capability Maturity Model for Software - SW-CMM)
- process maturity model for system reengineering (Electronic Industries Alliance Interim Standard - EIA/IS 731)
- the maturity model of integrated product development processes Integrated Product Development Capability Maturity Model - IPD-CMM)

Based on these models, CMMI was built. It has absorbed the best of these models, eliminating the ambiguity in the interpretation of some concepts due to the presence of many models - therefore, the number of key practices has increased significantly).

It is clear that this turned out to be a significantly more "heavy" model - see Fig. Rice. one, which, moreover, has not yet been sufficiently tested in practice (it was released only in 2002). In this regard, in my opinion, when implementing the model, great risks are possible, associated both with unjustified losses in the speed of software development, and with a simultaneous unambiguous increase in labor costs for the operation (and support) of the implemented KPA - see. Fig 1. As a practitioner who has already built a QMS in three different IT companies, it seems to me that in the CMMI model, the balance of what is necessary and sufficient is clearly violated - the IT company's staff (and this, as a rule, is mostly "code artists") simply "will not accept" such a number of controlled regulations (there is a very big risk to build a "Potemkin village")!

Rice. 1 Comparison of KPA composition in CMM and CMMI models.

In addition, Assessment for CMMI will be significantly more expensive, since authorized SEI Lead Assessor" there will still be very few ovs, and these services will be much more expensive than when assessing for compliance with the CMM model.

Moreover, many foreign experts in the field of quality management (whom I fully agree with at the moment) are rather skeptical about CMMI in the context of its usefulness for implementation in small and medium-sized organizations (it is precisely such organizations that are typical for Russia). ). There is even an opinion that after some time SEI will either have to release an adapted SW-CMM v.2, or take some similar steps. Those. if the market does not accept the model, and such prerequisites already exist at the time of this writing, then the SEI will have to adapt to the requirements of the market.

In connection with the foregoing, it seems appropriate to analyze the already mentioned balance of what is necessary and sufficient in all these basic QMS models.

Let's draw it in the following coordinates (see Fig. Rice. 2) :

the degree of regulation of development processes - let's denote this concept - RP,

the probability of achieving the planned results - denote this concept - PQ.

On Fig. 2 shows an expert assessment of the balance of the degree of regulation and the likelihood of achieving the planned results, carried out by the author based on the results of the practice of implementing the requirements of these models in the development and implementation of PS (software tools).

In mathematical terms, the value of the derivative: F(Q) = dPQ \ dRQ(increase in efficiency in achieving quality dPQ with an increase in the cost of working hours to support the fulfillment of requirements dRQ), decreases, respectively, in the following sequence : ISO 9000, CMM, CMMI.

Therefore Fig. 2 clearly and simply explains:

the popularity of the ISO 9000 model,

the correctness of the methodology: first ISO, and only then, if necessary, CMM,

some skepticism about the effectiveness of the CMMI model.

Rice. 2 Analysis of the balance between the degree of regulation and the likelihood of achieving the planned results (according to the author's expert assessment)

Let us now consider another guide that is widely used in IT companies and will be mentioned below when analyzing the issues of QMS implementation practice.

it PMBoK(Guide to the Project Management Body of Knowledge) is a project of the Project Management Institute, which has absorbed the accumulated knowledge in the field of project management. The latest version of the document was published in 2000 and at the same time received the status of the standard of the American Standards Institute ANSI (although the ANSI and IEEE standards are formally considered American, most of them are de facto international in nature). An important feature of PMBoK is that it considers project management in a general sense, without being tied to specific subject areas such as IT, and therefore cannot be applied on its own - below we will consider what effect this has when used in conjunction with ISO 9000.

Let us now consider how the requirements of the already popular ISO 9001:2000 standard correlate with the general properties of the increasingly popular ISO 9001:2000 standard. popular model SMM {3}- cm. Rice. 3.

Rice. 3. Correspondence between the general properties of the CMM and the elements of ISO 9001:2000

Each level of HMM, as mentioned above, is characterized by a set of areas of key processes - KPA (Key Process Areas) - cm. Fig.3. Achievement of all goals within KPA for a certain level, the CMM determines the compliance of the organization with this level. If at least one goal of at least one KPA for the CMM level is not achieved, then the organization cannot meet this CMM level. KPA can be divided into three categories: managers , organizational and providing (cm. Rice. four).

The CMM does not define all processes relevant to software development; it highlights only those that are necessary to achieve the level of SMM, and they are included in KPA. Each KPA is broken down into 5 Common Features: Commitment to perform (Comment to perform); Ability to Perform; Activities Performed; Measurement and Analysis (Measurement and Analysis); Verifying Implementation

General property " Actions performed" describes the actions to be taken to achieve the goals KPA, the remaining four general properties describe the formal factors that make the process part of the organizational culture. Full implementation of all key techniques (key practice) of all common properties ensures the achievement of goals KPA. Key ways of working describe what the workflow (or process element, or part of the infrastructure) should become, but do not determine how to achieve it (specific technologies or techniques), although general recommendations are given for some techniques. For different conditions, the same result can be achieved different ways. It's rather general principles work than specific actions.

Sequential execution of common properties actually implements a business process improvement cycle (Business-process Improvement - BPI-cm. Rice. 5.), i.e. continuous improvement of business processes (BP).

Rice. 5. The cycle of continuous improvement of business processes according to the CMM model and ISO 9000:2000.

The desire to obtain a certificate of conformity in the shortest possible time forces consulting companies and specialists involved in quality management to use the flexibility and framework requirements of all the above-mentioned high-level models for their own "mercenary" purposes.
As a result of this forcing of events, an organization, for example, which has received an ISO 9000:2000 certificate, has only the minimum required set of processes for compliance with ISO 9001, and not all the processes that the company needs to function effectively - see below. Rice. 2. In addition, the level of detail of the processes may not be sufficient for a clear understanding of what is happening inside the processes and who is responsible for what tasks within the process.
AT best case only a few test projects went through the new procedures, and after some time it becomes clear that they need to be corrected and supplemented. Often, immediately after QMS certification, processes are forgotten until the next observational audit, while forgetting about the spent financial resources and the enthusiasm of employees.
Indeed, when you act as an independent auditor, it is very difficult to prove that the accepted level of process detail is clearly not sufficient for the effective functioning of the company's QMS. But it is extremely difficult to prove the opposite in the time allocated for an ISO 9000 audit (this can be very successfully used when opposing an auditor). Practice shows that it is impossible to quickly build effective processes even of the 3rd level of maturity (as well as processes based on ISO 9000).
In order to achieve this, it is not enough just to describe the processes taking into account the requirements of the chosen model. The main difficulty lies in the fact that redesign the production culture within the organization .

And it is impossible to do this by a strong-willed decision of the leadership. That is why the approach that is defined in the CMM is simply more viable and realistic than in the ISO 9000 models - see. Rice. 5.

Let us now consider how in practice it is possible to build a QMS compatible with both models.

Expert review the degree of coverage of key CMM processes by the requirements of ISO 9000:2000, in accordance with the assessment of the authors of the CMM themselves (4), is shown in Fig.6.

The assessment itself was carried out according to two coordinates:

the degree of availability (in%) of the compliance of development processes (SWP) with the level of maturity within the CMM - " sufficiency";

the degree of possibility (in%) of such sufficiency, which gives ISO 9000:2000 - " possibility".

As seen from Rice. 6, ISO 9000:2000 requirements create a real opportunity to achieve even the highest (CMM Level 5) SWP maturity level.

However, in the sense of already ensuring the maturity of the SWP at least the third (CMM Level 3) level, the QMS according to the ISO 9000:2000 model needs to be slightly improved - namely, to develop and implement two more organizational procedures ( Organization process definition and focus) and general control procedure ( integrated software management ), the contents of which are not difficult for any IT company.

But it is possible and necessary to go further (CMM Level 4) - for example, in parentheses is the assessment of the author of this article (in the same coordinates - availability and capability), which corresponds to the QMS according to the ISO 9000:2000 model, in which the QMS process landscape is supplemented by processes project management in accordance with already the requirements of another standard mentioned above P.M. Bok- this will help you significantly increase the maturity of even such SWP, how:

Monitoring the progress of projects (Software project tracking and oversight);

• Project planning (Software project planning);

General software management (Integrated software management);

Process management through quantitative assessments (Quantitative process management).

Rice. 6. Expert assessment of the degree of coverage of key CMM processes by the requirements of ISO 9000:2000

As seen from Fig.6., the CMM model, according to its principles, is very close to the QMS built according to the ISO 9001:2000 standard and supplemented by project management processes in accordance with PM BoK..

In order not to do unnecessary work with simultaneous ISO 9000 certification and subsequent CMM assessment, I recommend that when defining your production processes, include (or maybe limit yourself to them - after all, this is for an IT company production processes!) are all required in the CMM KPA model. Thus, the company simultaneously:

fulfills the requirements ISO 9001:2000 on the implementation of the process approach;

documents all necessary CMM processes ( KPA);

fulfills a number of important requirements ISO 9001:2000 how:

metrics based process management (Quantitative process management);

Supplier management based on subcontract management ( Software subcontract management );

analysis of consumer requirements based on requirements management ( Requirements management );

control by human resourses based Staff training programs (Training program );

communication management based on creation of formal models of organizational processes ( Organization process definition );

launches an improvement mechanism (Plan-Dо-Check-Action) all described processes (SWP) through the consistent implementation of all five Common Features-cm. Rice. 5.

Thus, if we use KPA CMM as a BP and use the requirements for PS development project management procedures P.M.BoK, then the QMS constructed in this way can be fully evaluated on CMM Level 4 - see. Rice. 7.

Rice. 7. The scheme for achieving CMM Level 4 when using the QMS model according to ISO 9000 and the PM BoK 2000 guide.

In conclusion, for reasons of clarity (in the author's stylization), I present a scheme for the functioning of the QMS of an IT company with the consistent implementation of ISO 9000 and CMM models - see Fig. Rice. eight.

Rice. 8. Scheme of the QMS functioning with the consistent implementation of ISO 9000 and CMM models (author's stylization)

It is important to understand here that both the CMM and ISO 9001:2000 are in themselves just tools for continuous improvement.

Thus, certification according to ISO 9001:2000 and confirmation of the certificate should contribute to the growth of the quality of the company's processes, where the criterion for assessing the growth of the quality of processes is the company's exit to a new level. bpi, that is, their evaluation is already according to the model exactly CMM {3}.

Literature

"Software quality assessment", V. Lipaev, Sinteg, 2001

ISO 9001:2000. Quality Management System. Requirements.

Paulk M.C., Curtis B., Chrissis M.B., Weber C.V. Capability Maturity Model for Software (SW-CMM), version 1.1. // CMU/SEI-93-TR-024, February 1993.

05.04.2007 Vyacheslav Pankratov

There is a lot of talk these days about software quality and information systems, studies are being conducted that demonstrate the dependence of the quality and efficiency of automated business processes. The quality of software from an abstract and intangible concept is transformed into a comprehensive metric for evaluating a software solution, a project for its implementation, the creation process and the level of use of information systems as a whole. What determines the quality of programs and how can you influence it?

Today, much is said about the quality of software and information systems, research is being conducted that demonstrates the dependence of the quality and efficiency of automated business processes. The quality of software from an abstract and intangible concept is transformed into a comprehensive metric for evaluating a software solution, a project for its implementation, the creation process and the level of use of information systems as a whole. What determines the quality of programs and how can you influence it?

Obviously, the quality of software directly depends on the quality of its production process. Managing the production process and monitoring the performance indicators of all its technological stages, can affect the quality of the product. Speaking about the characteristics of programs, we can single out quantitative metrics that are easy to understand and analyze, related to the quality of the program code (cyclomatic code complexity - the complexity of the module structure, for example, the number of independent routes in it); the number of lines of code assigned to the artifacts of the design repository, etc.; tests (covering code branches and modules with test scripts, the ratio of the number of bugs found before and after the release of the product, the dynamics of bug detection, etc.); coverage of requirements for compliance with recommendations for the application interface and operating platforms. However, when moving to the process level of quality assurance of the developed programs, certain difficulties arise in understanding the quality of this process. Indeed, how, for example, to evaluate and measure the effectiveness of one or another development method, if there are practically no development projects for two identical software systems, and even more so, there are no two development teams that are identical in experience and skills? It is not possible to judge by the final result: in addition to the process conditions for software production (the methodology used, the structure of the project team, methods of communication with the customer), the conditions of the project (terms, cost and volume of resources) often vary greatly. A more detailed consideration of the software testing process - a technological component of the production process - reveals the problem of choosing test efficiency metrics.

In addition to a direct assessment of the current level of efficiency of a particular process, there is a more interesting task - increasing the level of efficiency or, as they say, maturity level processes. When basic problems planning and carrying out testing work in integration with the software development process are solved, the problem arises of finding optimal organizational and procedural schemes for performing work. Having answered the questions “who” and “when”, one has to look for answers to the questions “how, in what way”, “how to measure results”, “how to control”, “how to work more efficiently”, and also “how to manage and develop the process based on based on data and experience.

In everyday practice - when working with specific tools - IT professionals and project managers develop a strong opinion about the technological nature of all problems associated with the quality of software and the levels of maturity of the software development process that are achieved. As a result, vendor-promoted belief in the power of quality improvement software in practice becomes the breeding ground for unreasonable decisions to implement methodologies or automate development processes, starting with the implementation of specific software solutions. However, modern tools for automating development processes are such flexible and customizable platforms that they require preliminary deep study of the process component, followed by implementation and adaptation to specific production conditions.

In 1980, the Software Engineering Institute at Carnegie Mellon University developed software development process maturity model(Capability Maturity Model for Software, CMM), which was subsequently developed into CMMI (Capability Maturity Model Integration), a de facto certificate of development maturity level recognized in the software industry. By analogy with the world of software development and existing models of the maturity of their processes, we will consider models of the maturity of the testing process.

Test maturity model

Software Testing Maturity Model is a systematic approach to the development of the testing process, which offers a system of elements efficient processes and ways to achieve specific process goals. Based on the maturity model, two main tasks of process development can be solved: to understand and fix the current testing process and to identify areas for improvement. Practice shows that process changes are possible only on the basis of a clear understanding by management of the need to make such changes - any structural and procedural changes are impossible without the political will of management. In addition to obtaining management support and the necessary resources, making changes to the testing process requires careful planning, like any other project activity.

Test consultant Terry Weatheril was one of the first to compare existing test maturity models in 2001 and identified six models:

Testability Maturity Model (TMM);

Software Testing Maturity Model (TMMSW);

Test Process Improvement (TPI);

Test Organization Maturity (TOM);

Testing Assessment Program (TAM);

Proposed Evaluation and Test SW-CMM Key Process Areas (SW-CMM KPA).

Then fundamental changes were made to some models; Thus, the TOM model (it was created and developed by Gerrard Consulting) offers an updated set of metrics that describe the testing process itself (the duration of test iterations, the ratio of test scenarios and functional requirements, etc.), which are collected and analyzed at the stage of describing the existing process.

Among the six software testing maturity models, practitioners and consultants identify two: TMMSW, developed at the Illinois Institute of Technology, and TPI, proposed by Sogeti. Both models have become popular primarily due to their simplicity, as well as the proposed practices. internal audits, which can be produced by specialists of a company that has embarked on the path of process improvements. Consider the structure of software testing maturity models using the TMM model as an example.

The TMMSW model, chosen by Thomas Staab, one of the leading consultants in the field of software testing, is the most interesting to apply, because, along with the ease of understanding and using practices, it allows organizations to conduct assessments (assessment) on their own and gradually approach their development goals, controlling intermediate results. In our work, we also opted for this model, given the unpopularity of the practice of attracting third-party competence among domestic IT companies (companies try to save on investment projects, which in essence are consulting projects, as well as projects that bring indirect benefits, which include process changes), and we invite you to get acquainted with our experience, which demonstrates the readiness of companies to independently conduct internal changes by our own experts. The iteration of the approach is a key point for many companies when choosing one or another maturity model, as it allows you to control the timing of the project for the implementation of process changes.

The TMMSW model was developed by a group of specialists led by Ilene Barnstein in 1996 as an extension and addition to the SW-CMM model. Its advantages include the correspondence between the maturity levels of software testing processes and the maturity levels of development processes in the SW-CMM model. Also, the TMMSW model can be used in integration with CMMI, ISO-9001 recommendations and ISO / IEC Std 12207, which allow you to get formal certification and, with constant monitoring in the form of audits and recertifications, remain at a given quality level. On the one hand, this feature of TMMSW allows the implementation of process changes in the software testing department in the form of a dedicated project on a smaller scale than the implementation of CMMI throughout the organization (which saves money and provides transparency); on the other hand, this approach eliminates the cost of adapting and pairing several models. Speaking of a kind of relationship with CMMI, I would like to emphasize that this model is quite widespread in the IT world and has earned a high degree of trust in itself, which makes it much easier for management to motivate the cost of a process change project.

The TMMSW model offers easier planning, implementation and monitoring of the improvement process. Among the shortcomings of the model, one can note the limited literature: the book Practical Software Testing: A Process-oriented Approach, published by Springer Professional Computing, is perhaps the only significant work on this topic.

The TMMSW model, like the CMM, has five levels of maturity.

Level 1 - chaotic. The software testing process is chaotic in nature, which distinguishes most start-up companies. The testing process is not defined as a dedicated activity and is not separate from the code debugging process. Testing is done after the code is generated and the system is built or assembled. The purpose of testing is to show that the application works. This level is characterized by untrained personnel, lack of resources and tools. The software is released without the formal consent of the testers. Level goals are not defined.

Level 2 - development phase. Software testing is separated from coding and is highlighted as the next phase. the main objective testing - to show that the application meets the requirements. There are basic approaches and testing practices. Level objectives: define development and testing tasks, create appropriate procedures, initiate the test planning process, fix and describe basic testing procedures and techniques.

Level 3 - integrated. The testing process is integrated into the software development life cycle. Test objectives are based on requirements. There is an organization of testing, and testing itself is allocated in professional activity. Level objectives: to single out testing into a separate group and define a technical training program, integrate the testing process into the development life cycle, and also control the testing process itself.

Level 4 - control and measurement. Testing is a measurable and controlled process. Processes are critical inspections(review) of project artifacts (test plans and scenarios, error messages, final version status reports, etc.) refer to test activities. The product is tested for compliance with such quality metrics as reliability, usability, maintainability. Test scripts are recorded, stored in the test management system, and can be reused along with test datasets. Detected defects are not only fixed, but also analyzed according to formal criteria: criticality, “weight” of the defect, importance, lifetime, etc. Level Objectives: Implementation of critical review and audit programs at the organization/unit level on a par with a measured testing program. The quality of software products is assessed.

Level 5 - process optimization, error prevention and quality control. Testing is a defined and controlled process. The cost of testing along with performance indicators can be determined. Testing as a process lends itself to changes that clearly positively affect it. Error prevention and quality control practices are implemented and used. Automated testing is used as the main approach in testing. Designing tests, analyzing the results obtained, processing error descriptions, as well as metrics related to testing, is carried out using the appropriate tools. Widespread approach reuse process practices. Level objectives: optimization of the testing process, error prevention and quality control.

All of the listed maturity levels, except for the first, include development goals, which, in turn, contain subgoals, that is, they allow you to operate not only with high-level tasks of software development process quality management, but also formulate operational tasks for all performers in the project. Control and analysis of task performance is achieved by covering the so-called ATR-matrices (Activities - Tasks - Responsibilities) - project-level artifacts that project participants can work with without prior preparation or long-term training. ATR-matrices define the activities and tasks that must be performed at each specific stage of the model implementation, and serve both as a “road map” and a kind of “checklist” for the change implementation process. The presence of design artifacts offered by the model itself is often an essential criterion for the success of a project to adapt and implement the model. Each activity in the ATR matrix is ​​assigned a control task, which is performed by managers, developers/testers, or clients/users. We especially note that the control over the change project is not assigned to a selected group of people or a specific person in the project, but is a general project function, in which project participants of all levels are interested.

Up to the fifth level of maturity of software testing processes, no special tools or integrated solutions are required, in contrast to the maturity of development processes, where tools for collecting and analyzing requirements and version control, for example, are necessary condition reaching a certain level of maturity of the development process as a whole. At the fifth level of the TMMSW model, certain tools can be used statistical analysis code that allows solving one of the target tasks of this maturity level - preventing errors by identifying code sections containing logical errors in the absence of resource release operators or searching for unused variables or memory arrays. However, the use of a special tool is not mandatory even at this level; for example, the approach when there are two testers per coder, one of which - a higher-level developer - is busy with the tasks of critical code reviews, also allows us to solve the problem of error prevention.

The use of specific methodologies or following any of the chosen methodologies (RUP, MSF or Scrum) also does not guarantee the achievement of product quality or project success, since the software development methodology only works for a specific type of project. Similarly, for the software testing process, no methodology without adaptation to the conditions of a particular project gives a guaranteed result. The process maturity model is precisely the practice of achieving certain levels of process quality that is interesting for application, which is a structure of goals and approaches for achieving them, allowing you to use "inside" an almost arbitrary development methodology (with proper adaptation) and a set of tools. The model explains “what and how” to do, but not “what and in what order”.

Practice

By putting into practice the recommendations of the maturity models of the testing process, the company can not only see progress in the format of the collected metrics, but also really feel the qualitative changes in the work mode itself. There are several signs of process changes that are felt both by management and team members, as well as by customers and consumers of the product being developed.

Time-controlled process of releasing versions with a given level of quality. It does not say about the ideal quality of the manufactured product or the complete absence of defects - we are talking confidence in the state of the released version, both on the part of the project team and the testing team.

Regularity and predictable repeatability of all stages of the project. In the conditions of initial levels of maturity of software testing processes, testing activities follow as the final stage of work and often “suffer” due to limited time and lack of resources, which directly affects the stability of released versions and their quality. With the transition to higher levels of the testing process maturity model, testing activities are increasingly integrated into the development and release of product versions, which positively affects the allocation of the necessary resources and time to complete the work.

Changing this quality indicator characterizing the process of developing and releasing software, as the number of defects found after the release of the product version in experimental or even production operation. This indicator is very significant for the managerial staff and services. technical support, which can confirm the positive dynamics of improving the quality of the software by conducting a quantitative and qualitative analysis of registered requests from customers or users. The positive aspects, according to our estimates, are associated with practical improvements at the stage of planning and testing control, since often missed defects are caused precisely by the lack of time for planning and monitoring the testing work carried out.

Literature

Terry Weatherill, In the Testing Maturity Model Maze. Journal of Software Testing Professionals, 2001.

Thomas Staab, Using SW-TMM to Improve the Testing Process. Wind Ridge International.

Vyacheslav Pankratov ([email protected] ) - CEO QAExpert company (Kyiv, Ukraine).

Capability Maturity Model (CMM) offers

various levels. For this, 3 levels of elements are defined: organization maturity levels, key process areas and key practices. Most often, under the CMM model, they mean a model of maturity levels. The CMM is now considered obsolete and is being replaced by the CMMI model (see below).

o Maturity levels. The CMM describes the different levels of process maturity in organizations by defining 5 levels of organizations.

─ Level 1, initial. Organizations that develop software, but do not have a conscious development process, do not plan and evaluate their capabilities.

─ Level 2, repeatable. These organizations keep track of resource inputs and project progress, and establish project management rules based on experience.

─ Level 3, defined. In such organizations, there is an accepted, fully documented, real-world and accessible process for developing and maintaining software for the staff. It should include both managerial and technical sub-processes, as well as training employees to work with it.

─ Level 4, manageable. In these organizations, in addition to the established and described process, measurable indicators of the quality of products and the effectiveness of processes are used, which make it possible to accurately predict the amount of resources (time, money, personnel) required to develop a product with a certain quality.

─ Level 5, optimizing. In such organizations, in addition to the processes and methods for their evaluation, there are methods for identifying weaknesses, procedures for finding and evaluating new development methods and techniques, training staff to work with them and including them in general process organizations in case of increasing production efficiency.

o Key process areas. According to the CMM, the maturity levels of an organization can be determined by the use of well-defined techniques and procedures related to various key process areas. Each such area is a set of related activities that are aimed at achieving goals that are essential for the overall assessment of the effectiveness of the technological process. There are 18 regions in total. The set of areas that an organization must support expands as it moves to higher levels of maturity.

There are no requirements for the first level.

Maturity 2 organizations should support requirements management, project planning, project oversight, contractor management, software quality assurance, configuration management.

─ Organizations of the third level should, in addition to the activities of the second level, support the conduct of examinations, coordination of activities individual groups, software product development, integrated development and maintenance management, staff training, development and support of the organization's technological process, monitoring of compliance with the organization's technological process.

─ To the activities of organizations of the fourth level are added: software quality management and process management based on measurable indicators.

Maturity 5 organizations must additionally support process change management, technology change management, and defect prevention.

o Key practices. Key areas of the process are described using sets of key practices. Key practices are classified into several types: commitments (commitments to perform), capabilities (abilities to perform), activities (activities performed), measurements (measurements and analysis) and checks (verifying implementations). For example, requirements management is associated with the following practices.

─ Commitment. Projects must follow the organization's defined requirements management policy.

─ Possibilities. Each project should identify a person responsible for analyzing system requirements and linking them to hardware, software and other components of the system. Requirements must be documented. Adequate resources and budget should be allocated to requirements management. Staff should be trained in requirements management.

─ Activities. Before being included in the project, the requirements are analyzed for completeness, adequacy, consistency, etc. The selected requirements are used as the basis for planning and performing other work. Changes in requirements are analyzed and included in the project.

o Measurement. The status of the requirements and the status of their management activities are periodically determined.

─ Checks. Requirements management activities are periodically reviewed by senior managers. Requirements management activities are reviewed periodically and based on significant events by the project manager. The quality assurance team analyzes and audits the requirements management activities and reports on the results of this analysis.

Table 4 summarizes information on the number of practices various kinds assigned to different key process areas.

Table 4. Number of key practices in different process areas according to CMM version 1.1.

Models life cycle

All the standards discussed in one way or another try to describe how the any software development process. At the same time, they are forced to introduce too general software life cycle models that are difficult to use when organizing a specific project.

Within specific life cycle models, which prescribe the rules for organizing software development within a given industry or organization, more specific development processes are defined. They differ from the standards, first of all, in greater detail and a clear description of the links between certain types activities, defining data flows (documents and artifacts) during the life cycle. There are quite a few such models, because in fact every time an organization defines its own development process, some software life cycle model is developed as the basis of this process. In this lecture, we will consider only a few models. Unfortunately, it is very difficult to choose criteria by which one could give at least some useful classification of known life cycle models.

The most widely known and used for a long time remained the so-called cascading or waterfall life cycle model, which is believed to have been first clearly articulated in the work and subsequently embodied in the standards of the US Department of Defense in the seventies and eighties of the XX century. This model assumes the sequential execution of various activities, from requirements development to maintenance, with a clear definition of the boundaries between stages, in which the set of documents created in the previous stage is transferred as input to the next. Thus, each type of activity is performed at some one phase of the life cycle. The "classic" cascade model assumes only forward movement according to this scheme: everything necessary for carrying out the next activity must be prepared in the course of previous work. Development of system requirements Development of software requirements Operation

Testing

Coding

Design

However, if you carefully read the article, it turns out that it does not prescribe following this particular order of work, but rather represents a model iterative process- in her sequential form this model seems to have been fixed in the minds of officials from ministries and managers of companies working with these ministries under contracts. At real work under a one-way model, there are usually problems in detecting flaws and errors made in the early stages. But it is even more difficult to deal with changes in the environment in which software is developed (this can be changes in requirements, a change in contractors, changes in the policies of the developing or operating organization, changes in industry standards, the emergence of competing products, etc.).

You can only work according to this model if you can anticipate the possible ups and downs of the project in advance and carefully collect and integrate information in the early stages so that later you can use their results without regard to possible changes. Development of system requirements Development of software requirements

Exploitation

Testing

Coding

Design

Among developers and researchers who have dealt with the development of complex software, almost from the very beginning of the software production industry (see, for example,)

Rice. 6.1. Exemplary flight simulator architecture

Rice. Figure 6.1 shows a sketch of the architecture of such a flight simulator. Each of these components solves its own tasks that are necessary for the operation of the entire system. Together, they solve all the problems of the system as a whole. The arrows show the data and control flows between the components. The dotted arrows represent the data streams being sent for logging.

The architecture determines most of the quality characteristics of the software as a whole. The architecture also serves as the primary means of communication between developers, as well as between developers and all other stakeholders in the software.

The choice of architecture determines how the requirements for high level abstraction. It is the architecture that almost completely determines such software characteristics as reliability, portability and maintainability. It also significantly affects the usability and efficiency of the software, which, however, are highly dependent on the implementation of individual components. The influence of architecture on functionality is much less - usually a given functionality can be implemented using completely different architectures.

Therefore, the choice between one architecture or another is determined to a greater extent by non-functional requirements and the necessary properties of the software in terms of maintainability and portability. At the same time, in order to build a good architecture, it is necessary to take into account possible contradictions between the requirements for various characteristics and be able to choose compromise solutions, giving acceptable values ​​for all indicators.

So, to improve efficiency, in the general case, it is more profitable to use monolithic architectures in which a small number of components are allocated (in the limit, a single component). This saves both memory, since each component usually has its own data, and here the number of components is minimal, and operating time, since the ability to optimize the work of data processing algorithms is also available only within one component.

On the other hand, to improve maintainability, on the contrary, it is better to break the system into a large number of separate small components, so that each of them solves a small but well-defined part of the overall task. At the same time, if there are changes in the requirements or the project, they can usually be reduced to a change in the formulation of one, less often two or three such subtasks and, accordingly, only the components responsible for solving these subtasks can be changed.

On the third hand, to improve reliability, it is better to use either a small set of simple components or duplication of functions, i.e. make several components responsible for solving one subtask. Note, however, that errors in software are most often non-random in nature. They are repeatable, unlike hardware, where errors are often associated with random changes in the characteristics of the environment and can be overcome by simply duplicating components without changing their internal implementation. Therefore, with such reliability assurance, it is necessary to use quite different ways of solving the same problem in different components.

Another example of conflicting requirements are usability and security features. The stronger the system is protected, the more checks, identification procedures, etc. users need to go through. Accordingly, the less convenient for them to work with such a system. When developing real systems, some reasonable compromise has to be found in order to make the system sufficiently secure and able to put a tangible barrier to unauthorized access to its data and, at the same time, not scare away users with the complexity of working with it. List of standards governing the description of the architecture, which is the main component project documentation on the software, it looks like this:

· IEEE 1016-1998 Recommended Practice for Software Design Descriptions(recommended methods for describing design decisions for software).

· IEEE 1471-2000 Recommended Practice for Architectural Description of Software-Intensive Systems(recommended methods for describing the architecture of software systems).

This is, first of all, the very concept of architecture as a set of fundamental principles for organizing a system, embodied in a set of its components, their connections with each other and between them and the environment of the system, as well as principles for designing and developing the system.

This definition, unlike the one given at the beginning of this lecture, focuses not on the set of structures at the heart of the architecture, but on the principles of its construction. The IEEE 1471 standard also defines an architectural description as a coherent set of documents that describes an architecture from the point of view of a particular group of stakeholders using a set of models. An architecture can have multiple views that reflect the interests of different stakeholder groups.

The standard recommends that for each view, the views and interests reflected in it, the roles of persons who are interested in such a view of the system, the reasons for the need for such a review of the system, inconsistencies between elements of one view or between different views, as well as various service information about sources of information , dates of creation of documents, etc.

The IEEE 1471 standard notes the need to use the system architecture to solve problems such as the following.

Introduction

The most important part of modern complex systems are software products - the intellectual component. Software products are now used to solve management problems in almost all areas of human activity: in the economy, social, military and other areas. Security High Quality domestic software products during their mass development and delivery for various applications in the country and on the world market has become a strategic task.

Currently, there are two almost independent areas of standardization in software engineering and software product quality assurance, which can be conditionally called ISO (International Standards Organization) standards profiles and SEI (US Software Engineering Institute) maturity models. The first ones are quite fully represented in [ , ], and the second ones - in [ , ]. The main content of the article is devoted to the maturity models.

To ensure competitiveness in the world of complex software products and the possibility of their successful export, they must be developed and certified in accordance with the requirements profiles of international standards on the base ISO 9000:2000 or maturity models - CMMI:2003(Capability Maturity Model Integration - Integrated software engineering maturity assessment model). These two directions are methodologically very close and partially intersect through mutual references.

The improvement of technical and economic indicators and the quality of software products, as well as the prevention of errors and defects, is ensured by the use of modern technologies software engineering and systems computer-aided design. These are high-performance, resource-saving technologies for creating software complexes of high quality, reliability and safety, aimed at reducing total costs resources for the design, implementation and maintenance of software tools (PS). To do this, first of all, it is necessary to apply methods and tools for analysis and design, which provide concretization and the most accurate representation of the goals, purpose and functions from the beginning of the life cycle (LC) of the software and prevent the spread of possible system defects to subsequent stages of development. Such software engineering technologies make it possible to eliminate or significantly reduce the level of system, algorithmic and software errors in software products transferred for operation. In addition, they are effective in modifying and maintaining the software, as well as in changes external environment.

To certify the quality, reliability and safety of the use of complex, critical systems, the software products used in them are subjected to certification certified, problem-oriented test centers or laboratories. Such tests should be carried out when programs manage complex, critical processes or process information so important that defects in them or insufficient quality can cause significant damage. Certification tests should establish the compliance of software complexes with the requirements of the documentation and allow them to operate within the limits of changes in the parameters of the external environment studied during the tests. These types of tests are characterized by the greatest severity and depth of checks and should be carried out by specialists independent of developers and customers (users).

The basis for certification should be detailed and effective programs and methods for testing software packages for compliance with standardized customer requirements, specially designed test problems and generators for their formation, as well as high qualification and authority of testers. Application at enterprises-developers of software products, certified quality systems for ensuring the life cycle of PS based on requirements ISO 9000:2000 or CMMI:2003, guarantees high, sustainable quality management of processes and products of their life cycle, and also allows in many cases to facilitate the certification of the final software product. Therefore, clients of complex software projects tend to choose implementing contractors who have certificates certifying their application of quality assurance systems based on adapted international standards profiles or maturity models.

Gaps in teaching software engineering methods leave a wide field for the arbitrariness of specialists in assessing the quality of their work, as well as for the appearance of numerous defects and errors in software projects. The growing complexity and responsibility of modern tasks solved by programs, as well as the possible damage from the insufficient quality of their results, has significantly increased the relevance of mastering methods for a complete, standardized description of the requirements for quality characteristics and methods for measuring their real, achieved values ​​at various stages of the software life cycle. The need for specialists to know the concepts, definitions and methods for evaluating the characteristics of the quality of software products has sharply increased.

The rapid increase and complication of software complexes leads to the creation of large programming teams with a professional division of labor, in which it is necessary to regulate the coordinated activities of groups of specialists on a single project. Developers' promises in contracts to deliver high-quality software within agreed timeframes are often not kept. Often this happens due to the fact that the customer and the contractor evaluate the quality level according to different criteria, and they do not have agreement on this issue, and the approach to assessing the quality of programs is not formalized enough. In addition, sometimes there is a lack of ability to properly assess the resources needed to achieve high quality programs. As a result, the quality of software products often remains low, unreliable and uncompetitive on the market. international market. Therefore, the most important problem in the development and application of many modern systems is the training and education of specialists in the field of software engineering, the use of international standards that contribute to the high quality of the software and its reliable evaluation with the main goal - to make project processes manageable, and the results are predictable. It is necessary to be able to formalize the requirements and achieve specific values ​​of the characteristics of the quality of functioning and application of complex software packages, taking into account the resources that are available to ensure and improve this quality.

Model CMM maturity I-1.1 refines and improves previous models CMM(see ), and also partially takes into account the basic requirements of existing international standards in the field of software management. Significant attention in CMMI is given to development processes and accounting for iterations when changing customer requirements, their traceability to functions, components, tests and project documents. Recently, information has appeared about the modernization of the SEI version of the 2003 version. CMMI-1.1 based on accumulated experience and feedback from enterprises. It is supposed to release in 2006 a new, significantly upgraded version of the model CMMI-1.2, after which version 1.1 should be phased out. Until the end of 2007, users should switch to the version CMMI-1.2, and in the future it will become mandatory for a formalized assessment of the quality (certification) of enterprise technology in the field of software engineering. The validity period of the certificate will be limited to three years. Customers and developers of large software systems should prepare for these changes before the official publication of version 1.2 by SEI.

Structure and content of the CMMI maturity model - 1.1

Two model options CMMI-1.1 designed to provide continuous evaluating a set of processes in a specific area of ​​software development or for phased evaluating and improving the maturity of the enterprise, as well as for organizing the life cycle of program complexes in general. Models CMMI provide assistance to specialists in organizing and improving their products, as well as in streamlining and servicing the processes of developing and maintaining PS. The concept of these models covers the management and evaluation of the maturity of complex systems, software engineering, as well as the processes of creating integrated software products and improving their development. Components of continuous and staged models in to a large extent similar, can be selected and used in different composition and sequence of use depending on the properties and characteristics of specific projects.

Model description options are built according to a single scheme, which includes general sections:

• foreword;
• 1 section - introduction;
• Section 2 - component model;
• Section 3 - terminology;
• Section 4 - the content of the levels and the main components of each version of the model (development of goals and procedures);
• Section 5 - the structure of the interaction of processes; the four categories of processes in section 7 are annotated, their general overview and the interaction schemes of CMMI processes:
  • process management;
  • management - project management;
  • engineering (technology);
  • support;
• Section 6 - Using the Model CMMI- brief recommendations for users on the application of the model and training; the compatibility and compliance of the model processes with the regulated processes of the previous CMM model in parts 2 and 3 of the standard are noted ISO 15504.
• Section 7 is the last, largest in each standard, it takes up about 500 pages of the total document, which is over 700 pages. This section provides detailed recommendations for the implementation of each of the processes listed in it, which take into account the characteristics of a particular model.

First option(continuous) model reflects the document: Capability Maturity Model Integration (CMMI) for Systems Engineering/ Software Engineering/Integrated Prod-uct and Process Development, Version 1.1, Continuous Representation (CMMI-SE/SW/IPPD, V1.1, Continuous). Integrated System Engineering/Software Engineering/Integrated Products and Development Process Maturity Assessment Model - continuous view. In this model, the seventh section consists of processes:

• process management:
  • organization of training;
  • organization of transformation (changes) of processes;
  • organizing innovations and expansions;
• project management:
  • project planning;
  • monitoring and control of project processes;
  • Management of risks;
  • quantitative project management;
• engineering (technology):
  • requirements management;
  • requirements development;
  • technical solutions;
  • product integration;
  • verification;
  • validation (attestation, approval);
• support:
  • configuration management;
  • analysis and decision-making on changes;
  • root cause analysis and problem resolution (defect elimination).

The five appendices provide:

BUT- the composition of the used literary sources and documents, which, however, does not mention standards ISO;

AT- abbreviations;

FROM- glossary based terminology ISO used in only four standards ISO 9000, ISO 12207, ISO 15504:1-9, ISO 15288;

D - descriptions of requirements and proposals for the formation of model components by maturity levels;

E - list of development participants CMMI- project.

In this model, attention is focused on organizational processes, on planning, managing and controlling the processes for implementing software projects, on developing and managing requirements for software products. The following are examples of details in CMMI some of them.

Project Planning in this as well as in the second model includes:

• assessment of the possible size (scale) of the software product;
• assessment of the complexity of the functions and characteristics of the PS project;
• definition of the model and stages of the life cycle of the software package;
• feasibility study of the project - determination of the cost, labor intensity and duration of the life cycle of the substation;
• development of a phased work schedule and project budget;
• analysis, identification and assessment of project risks;
• planning and managing documentation of processes and products in the life cycle of the PS project;
• planning and distribution of technical and human resources by stages of the life cycle of the PS;
• planning the provision of knowledge and qualifications of a team of specialists for the implementation of the project;
• generalization and analysis of the set of plans for the PS project;
• coordination of works and resources for the stages of the life cycle by the developer with the customer of the PS project;
• documenting the work plan and its approval by the project developer manager.

Requirements Development Processes to a software product are similar to the processes in both models and include:

• identification of the real needs of the customer and users for the functions and characteristics of the software product;
• development and coordination between the customer and the developer of the initial, basic requirements for the functions of the software product;
• determination of available resources and limitations of the software suite project;
• decomposition of the basic initial requirements for the functions of the PS into a set of requirements for the components and tests of the software package;
• formalization of requirements for interfaces between components, with the operating and external environment;
• development of the concept of a software product and scenarios for its use;
• development of requirements for the generalized characteristics of functional suitability and the use of the functions of the software product for its intended purpose.

Requirements Management both models include:

• achieving an unambiguous understanding of the requirements for the PS project by the customer and developers;
• obtaining by the customer from the developers of obligations to fulfill all its requirements for the software product;
• management of changes in the requirements for the PS project agreed between the customer and the developer;
• ensuring the correctness of changes from the general requirements for the PS project to the requirements for components and particular processes;
• identifying and identifying discrepancies between project development processes and customer requirements.

Second option presents the document: Capability Maturity Model Integration (CMMI) for Systems Engineering/Software Engineering/Integrated Product and Process Development, Version 1.1, Staged Representation (CMMI-SE/SW/IPPD, V1.1, Staged). Integrated System Engineering/Software Engineering/Integrated Products and Development Process Maturity Assessment Model - phased introduction. The model is based on maintaining the concept of five levels of maturity CMM[ , ]. The composition of the processes practically repeats the one given above for the first version of the model, but in a slightly different sequence and with relatively small additions.

First level is characterized by significant uncertainty in the composition and content of processes in various relatively simple projects, therefore it is not commented on in the document. Therefore, when clarifying and detailing the content of processes in a phased version CMMI recommended to be limited four main levels:

• second level- formalizes basic management projects:
  • requirements management;
  • project planning;
  • project monitoring and control;
  • management of agreements with suppliers;
  • measurement and analysis of processes and products;
  • ensuring the quality of processes and products;
  • configuration management;
• third level- contains the standardization of the main processes:
  • requirements development;
  • technical solutions;
  • product integration;
  • verification;
  • validation (certification);
  • the content of organizational processes;
  • definition of organizational processes;
  • organization of training;
  • integrated management of project processes and products;
  • Management of risks;
  • integration of the development team;
  • integrated supplier management;
  • analysis and resolution of problems (elimination of defects);
  • organization of the environment for integration;
• fourth level- defines quantitative management:
  • organization of representation of the quality of processes;
  • quantitative management of the entire project and resources;
• fifth level- optimization, continuous improvement:
  • organization, innovation, quantitative management of processes and provision of resources;
  • analysis of the causes of defects, improvement of quality and management of processes and products.

Applications in the second version of the model are similar in composition to the above applications for the first model. It is recommended at each higher level of maturity to apply all processes previous lower levels. In both versions of the model, each basic process highlighted above is commented on with detailed recommendations for its practical implementation, which contain descriptions of about 20–30 pages unified in structure:

• the overall objectives of the process to be achieved;
• introductory remarks and general description process functions;
• specific process objectives;
• process management;
• development of process requirements;
• interaction and interfaces with other processes;
• practical goals - the required results of the process activities;
• planning actions in a particular process;
• analysis and validation (approval) of the results of the process implementation;
• monitoring and control of the process.

These recommendations in terms of the scope, content and completeness of descriptions of basic processes are similar to a number of standards for the PS Life Cycle Profile presented in. Ordering and evaluating the completeness of the processes used in accordance with the levels of maturity, allows you to establish the production potential of enterprises - developers of software products in terms of the predicted quality of processes and the results of their activities and readiness for certification for compliance with a certain level of maturity of the model CMMI - 1.1.

Emphasis on models CMMI is given to the management processes of the PS project. These requirements and processes of the models practically correspond to the regulated and detailed recommendations in the standards. ISO 9001:2000 and the main components of the profile of complex PS life cycle standards [ , ]. Requirements for processes in functional sections 4-8 of the standards ISO 9001, ISO 9004, ISO 90003 a number of sections adequate in content can be compared in the model CMMI(in Fig. 1, the content overlap zone). The commonality of processes and requirements consists in the similarity: composition, terminology, structure, list of recommended management processes, planning, accounting for available resources, implementation of software engineering processes, evaluation and organization of specialists.

Figure 1. Commonality of processes and requirements of standards and maturity models

From the point of view of supporting and regulating the full life cycle of large software projects, the shortcomings of models CMMI regarding the profile of existing standards ISO can include the following:

An extensive technical report was developed and initially approved in 1998 to determine the levels of maturity of the processes for ensuring the life cycle of the PS presented above. ISO 15504, consisting of nine parts and many applications. It outlines the maturity model CMM and eight basic principles of software engineering based on the standard ISO 9000:2000. Then in ISO this document has undergone a radical revision, reduction, simplification of the structure and content, while fully maintaining the goals and concept, and approved as standard in five parts.

Standard ISO 15504:1-5:2003-2006 regulates the assessment and certification of the maturity of the processes of creating, maintaining and improving software tools and systems performed by enterprises:

• to establish the state of one's own technological processes and their improvement;
• to determine the suitability of own processes to meet specific requirements or classes of customer requirements;
• for the purpose of its suitability for the performance of certain contracts with customers of PS and systems.

The standard contributes to: self-assessment of the maturity of enterprises, ensuring adequate management of attested processes, determining the profile of process ratings, and also fits any scope and size of OS and systems. The application of the standard is aimed at developing enterprises and specialists culture of continuous improvement technology maturity ensuring the life cycle of the PS that meets the business goals of the projects and optimizing the use of available resources. Enterprise process maturity assessment provides an opportunity to compare and select them, which are preferable for certain projects:

• for customers, buyers, users of software products and systems: the ability to determine the current and potential maturity of the supplier's life cycle processes;
• for vendors and developers: the ability to determine the current and potential maturity of their own software and systems life cycle processes, areas and priorities for process improvement;
• for matriculation assessors: a framework for conducting and improving assessment processes.

Approval in the standard is dealt with in two aspects: to improve the life cycle processes of the PS and systems of a particular enterprise and to determine whether the declared maturity of the project or enterprise support processes corresponds to the actual processes used. This is reflected in the following five parts of the standard. ISO 15504:1-5:2003-2006.

Part 1 - Concept and vocabulary. Contains general information about the processes for certification of the maturity of software and systems and recommendations for the use of parts of the standard. The general requirements for certification, terminology, structure are outlined, the scope of the remaining parts of the standard is determined.

Part 2 - Performance (production) of certification. It includes detailed requirements for conducting certification processes as a basis for improving and determining the level of maturity of technological processes for ensuring the life cycle of the PS and systems. The document defines processes for performing attestation, models for recommended processes for attestation and verification of processes so that they are objective, meaningful and representative.

Part 3 - Guidance for the production of certification. Provides an overview of the technology for performing maturity assessment processes and interpreting requirements implementation. It reflects: performance of certification; measuring tools for determining maturity processes; selection and application of certification tools; assessing the competence of certifiers; verification of conformity of attestation to the declared requirements. Validation tools can be used by enterprises in planning, managing, monitoring, controlling and improving software products and systems, in their acquisition, development, application and maintenance.

Part 4 - User guidance for process improvement and process maturity in these two aspects. A number of steps are recommended which include: applying the results of the validation processes; setting goals for maturity assessment; determination of initial data for certification; assessment of the possible reduction of the resulting risks; steps to improve processes; steps to determine the level of maturity; comparing the results of the qualification analysis with the requirements.

Part 5 - Sample model of attestation processes for compliance with the requirements presented in Part 2. Extensive document (162 pages) with examples practical application previous parts of the standard for organizing, evaluating, and improving life cycle process maturity assessments for various application areas, software projects, and enterprises.

In the practical implementation of projects and ensuring the life cycle of complex software, it is sometimes difficult for developers and suppliers to identify and highlight the advantages of models for application. CMMI. Depending on the traditions of the enterprise and the characteristics of a large project, it is often advisable to use the PS as the main full standards profileISO, and for customer evaluation maturity level management, organizational and technological support of PS projects apply specific recommendations CMMI. These guidelines can be used effectively in process quality certification at enterprises providing the life cycle of the PS, as an alternative or along with certification according to a set of management standards ISO 9000, depending on the specifics of the project and the requirements of the applicant for certification of a software product or technology to ensure its life cycle.

Organization of certification of software products

Certification consists of a series of organizational processes that make up certification system, these processes are supported by regulated procedures and documents and must be carried out by qualified, certified experts - inspectors. For certification of the enterprise-developer and the results of its activities - software products, models CMMI or standards profiles ISO[ , ] a certain discipline is recommended, which should be adapted to the specific characteristics of the objects and the environment of the life cycle of the PS. The processes and documents listed below are designed for large projects and may be reduced by agreement between developers, customers, and certifiers in simpler cases.

Certification work begins with the accreditation of a body or a testing laboratory, the formation and submission of an application and a set of documents to the Central Certification Body for a decision on the feasibility of accreditation. If the test results are positive, an accreditation certificate is drawn up and issued.

Regulations on the certification body or laboratory is the main document that establishes the subject area of ​​accreditation, legal status, functions, structure, rights and obligations, methods, means and organization of tests. The passport of the certification laboratory (center) must contain information about the equipment computer science necessary for testing, on personnel and personnel, equipment with testing tools, provision of regulatory, technical and methodological documents, as well as other resources necessary for testing.

Quality quide contains a statement of the principles, a description of the methods and procedures associated with the implementation of the main functions and tasks of the certification body or laboratory, ensuring the quality of the tests and confidence in the results of assessments, tests and examinations. The quality manual usually includes sections [ TWLSC$

• policy in the field of quality assurance of testing and examinations;
• equipping the center with relevant methodological materials and software and testing tools;
• formalization of requirements for test objects;
• policy in the field of technical equipment of the center and staff development;
• archiving and safety control of documentation of certification results.

For evaluation of products or processes subject to certification, the applicant sends an application to the certification body in the form adopted in the certification system. The certification body carries out work on the preparation and organization of product certification upon application. This work includes:

• selection of a certification scheme taking into account the specifics of products (volume, technology, requirements of regulatory documents, etc.) and the developer's proposals;
• determination of the number and order of sampling and components to be tested, if this is not specified in the standards;
• selection and identification of an accredited testing laboratory to carry out the tests;
• preparation of a draft contract for the performance of work.

The preparatory part of the work on certification ends with the release of a decision in the form adopted in the certification system. The decision, together with the draft contract for the performance of work, is sent to the applicant. When organizing certification tests, selection and study of the current regulatory documents for products declared for certification, methods of testing and evaluating the results are carried out.

The applicant makes the final decisions which elements of the quality system, areas and types of organizational and technical activities are subject to verification during certification in a given time interval. The applicant must create conditions and submit documents to ensure the verification processes. He can submit to the certification body test reports carried out during the development and production of products, documents on tests performed by third-party testing laboratories and other documents indicating the conformity of the technology or products established requirements. Based on the analysis of the documented evidence of the conformity of its products with the established requirements, presented with the application, the certification body may decide to reduce the scope of tests or to issue a certificate.

Tests are carried out by testing laboratories accredited to conduct only those tests that are provided for in their regulatory, accreditation documents. If it is not possible to conduct tests at the testing facility of an accredited laboratory, tests can be carried out by the personnel of this laboratory at the manufacturer or consumer of this product using the testing laboratory's own facilities or the test facilities available from the supplier.

The process of certification of software products and enterprise quality systems includes:

• analysis and selection by the developer or customer (applicant) of a body competent in this field and a certified laboratory to perform certification tests;
• submission by the applicant of an application for testing to the certification body and the adoption by the certifiers of a decision on the application, the choice of a certification scheme, the conclusion of a certification agreement;
• identification of requirements for the enterprise's quality system and / or for the version of the software product to be tested;
• performance of certification tests of the company's quality system or version of the software product by the certification laboratory;
• analysis of the results obtained and decision-making by the laboratory and/or certification body on the possibility of issuing a certificate of conformity to the applicant;
• issue by the certification body to the applicant - a certificate and a license to use the mark of conformity and to issue certified products - versions of the software product;
• implementation of inspection control by the certification body of the certified quality system of the enterprise and / or products;
• carrying out by the applicant of corrective measures in case of violation of the conformity of the processes of the quality system and / or products with the established requirements and in case of incorrect application of the mark of conformity.

When checking the responsibility of the developer's management for product quality, it should be determined that the enterprise or project has a documented policy, goals and obligations in the field of quality, as well as the degree to which this policy is understood, implemented and maintained in working condition at all levels of the organization. It should be established that the enterprise has a management representative who, regardless of other duties, has the authority and responsibility for the continuous implementation of the requirements of the standards and regulatory documents of the quality system. The availability of requirements, procedures, tools and trained personnel for the practical implementation of the quality system processes should be checked, as well as the relevance and systematic documentation of all components, requirements and provisions of the quality system, which is an integrated process throughout the life cycle of the PS. Quality system checks should include a definition:

• availability and completeness of technological documentation and compliance with its requirements in practice;
• the state of technological equipment and the availability of a system for their maintenance;
• the existence and effectiveness of the control and testing system;
• state of measuring and testing instruments;
• availability of a system for identifying and eliminating identified deficiencies in products or technology.

Based on the tests, the results obtained are evaluated and conclusions about the compliance or non-compliance of products or processes with the requirements of regulatory documents are substantiated. Test reports are submitted to the certification body, as well as to the applicant at his request. Test reports are subject to storage for the periods established in the rules of product certification systems and in the documents of the testing laboratory, but not less than three years.

After receiving and checking the completeness and quality of the documentation, the specialists of the testing laboratory should carry out examination of the degree of actual application of the quality system at the enterprise. Testing begins with the development of a quality system test program, which should serve as a working plan for subsequent work. The program is an internal working document of the testing laboratory and should contain a list of works, detailed in accordance with the specifics of the developer enterprise and including an analysis of the completeness and quality of the submitted source documents and the degree of their practical application in the design, development and delivery of the software. Examination of the application of the quality system procedures is carried out by the testing laboratory at the workplaces of the enterprise that provides the life cycle of the PS. Checks are carried out on the presence of specialists-developers of relevant documents at the workplace and on the completeness of the use of their provisions and recommendations. Reviews of the status of the project and internal audits of the quality system, processes and/or products should be carried out by personnel independent of those directly responsible for the implementation of these works.

Methods for checking the quality of development must be provided with the necessary resources for the implementation of the test program, methods for planning and developing private test procedures. Methods should contain: objects and goals of testing; assessed quality indicators; conditions and procedure for testing; methods of processing, analysis and evaluation of test results; technical support for testing and reporting. The hardware and software used during the tests and the test procedure, as well as the expected results of the checks, should be indicated. Methods should be developed to control corrections, actions to correct defects, if such a request is received by the audit management service. The test program management service should establish procedures for maintaining the confidentiality of any test information, as well as data held by assessors.

Test reports submitted to the applicant and to the certification body. The applicant may submit to the certification body test reports, taking into account the terms of their validity, carried out during the development and production of products, or documents on tests performed by domestic or foreign testing laboratories accredited or recognized in the certification system. On the basis of certification test protocols, the results obtained are evaluated and the conclusions drawn about the compliance or non-compliance of products with the requirements of regulatory documents are substantiated.

Conclusion on the results of certification tests is developed by certifiers and contains generalized information about the test results and the rationale for issuing a certificate. In case of obtaining negative results of certification tests, a decision is made to refuse to issue a certificate of conformity. After completion of the certified product or quality system, the tests can be repeated. The results of the analysis of the state of technology or product quality are drawn up by an act, which gives estimates for all positions of the Test Program and contains conclusions, including a general assessment of the state of production and products, the need for corrective measures. The act is used by the certification body along with test reports, an application for issuing and determining the validity period of a certificate for a software product, the frequency of inspection control, and also for drawing up corrective measures.

Based on the results of certification tests and examination of documentation, a decision is made to issue a certificate. In case of obtaining negative results of certification tests, a decision is made to refusal to issue a certificate compliance. In addition, proposals can be sent to the applicant enterprise to eliminate the alleged causes of negative test results, after the completion of the certified products, the tests can be repeated.

The certification body, after analyzing the test reports, evaluating production, certifying the quality system, analyzing the documentation specified in the decision on the application, assesses the conformity of the products with the established requirements, draws up a certificate based on the opinion of experts and registers it. When making changes to the design or operational documentation that may affect the quality of the system or software product certified during certification, the applicant must notify the certification body about this in order to decide on the need for additional tests. After registration, the certificate comes into force and is sent to the applicant enterprise. Simultaneously with the issuance of a certificate, the applicant enterprise may be issued license for the right to use the mark of conformity.

For certified software products during their operation during the entire period of validity of the certificate of conformity, inspection control. Inspection control is carried out in the form of periodic and unscheduled inspections compliance with the requirements for the quality of technology and certified products. The objects of control, depending on the certification scheme, are certified products, the quality system or the stability of the production of the developer. When determining the frequency and extent of the inspection, the following factors are taken into account: the degree of potential danger of the software product, the stability of production, the volume of release, the availability and application of a quality system during development, information on the results of tests of the product and its production carried out by the manufacturer, state control and supervision bodies.

Results of inspection control are drawn up by an act, which evaluates the results of sample tests and other checks, makes a general conclusion about the state of production of certified products and the possibility of maintaining the validity of the issued certificate. The act is stored in the certification body, and its copies are sent to the developer and to the organizations that took part in the inspection control. Based on the results of inspection control, the certification body may suspend or cancel the validity of the certificate and revoke the license for the right to use the mark of conformity in case of non-compliance of products with the requirements of regulatory documents controlled during certification, as well as in the following cases:

• fundamental changes in the maturity model, standards profile, product regulations or test method;
• changes in the design (composition), completeness of products;
• changes in the organization or technology of development and production;
• non-compliance with the requirements of technology, methods of control and testing, quality system, if the listed changes may cause non-compliance of products with the requirements controlled during certification.

The decision to suspend the validity of the certificate and license for the right to use the mark of conformity is not taken if, through corrective measures agreed with the certification body that issued it, the applicant can eliminate the detected causes of non-compliance and confirm, without re-testing in an accredited laboratory, the conformity of the product or processes to normative documents. If this cannot be done, then the validity of the certificate is canceled and the license for the right to use the mark of conformity is cancelled. Information about the suspension or cancellation of the certificate is brought by the certification body that issued it to the applicant, consumers and other interested organizations. The validity of the certificate and the right to label products with the mark of conformity can be renewed if the developer enterprise fulfills the following conditions:

• identifying the causes of non-compliance and eliminating them;
• submission to the certification body of a report on the work done to improve and ensure product quality;
• conducting additional tests of products according to the methods and under the control of the certification body and obtaining positive results.

Documentation of processes and results of certification of software products

Composition and content of documentation for quality system certification enterprises depend on the characteristics of the design, development and modification of software, as well as on the requirements for their quality and the characteristics of the technological environment. Therefore, the necessary set of documents for each enterprise or project should be selected and adapted in relation to these characteristics. The indicators of the quality system evaluated during certification are the availability of relevant documents and the practical fulfillment of the requirements of a certain level of the maturity model. SMMI or an adapted standards profile based on ISO 9000:2000, as well as created on their basis, job descriptions specialists of the enterprise-developer. The applicant must prepare and submit to the testing laboratory a set of documents agreed between the customer and the developer and approved to verify their reliability, sufficiency of composition and workmanship in accordance with regulatory documents.

An indicative set of basic documents for certification consists of three groups:

• basic regulations quality systems in accordance with the nomenclature and content of the profile of standards based on ISO 9000:2000 or maturity models SMMI, as well as the program, manual and instructions prepared by the developers on their basis, presented to the testers (experts) of the quality system or products of the audited enterprise;
• source documents characterizing a particular enterprise or project, as well as the life cycle of a software tool, prepared by the project management for certification of its quality;
• testers' reporting documents reflecting the results of the audit (certification) of the enterprise's quality system and / or software product, submitted to the certification body, the applicant and the management of the audited enterprise.

The software product or enterprise quality system submitted for certification must be submitted together with the relevant documentation. The list and approximate content of the groups of these documents are focused on the general case of checking the quality systems of enterprises that ensure the life cycle of large software products. The set of documents may be reduced and adapted as agreed between the applicant, the certifier and the management of the audited enterprise in accordance with the characteristics of software projects. Some documents can be combined into integrated reports with a clear responsibility of certain specialists for their implementation.

Basic documents of the enterprise quality system and software life cycle

1. Concept, terminology, requirements and guidance for performance improvement - quality management systems - ISO 9000:2000 or a version of the CMMI maturity model.
2. Adapted versions or list of sections and recommendations of the standards ISO 12207, ISO 15504, their changes and application guidelines, selected during adaptation and mandatory for use in the quality system of a particular enterprise or software product project.
3. Adapted version or list of sections and recommendations of the standard ISO 900003, selected during adaptation and mandatory for use in the quality system of an enterprise producing a software product.
4. Basic characteristics and quality attributes of the PS project, identified, adapted and specified on the basis of standards ISO 12182, ISO 9126, ISO 14598, ISO 25000.
5. Adapted version and approved edition of the maintenance and configuration management manual based on the recommendations of the standards ISO 14764, ISO 10007, ISO 15846.
6. A set of job descriptions that define the responsibility, authority and procedure for interaction of all the managerial, performing and checking the work of the personnel involved in the procedures of the enterprise quality system for a specific PS project.

Source documents reflecting the features of the life cycle of a particular software tool

1. Description of the characteristics of software products created at the enterprise, the system and the external environment of their life cycle, necessary for the adaptation and preparation of working versions of the standards and requirements of the PS project and the enterprise quality system in accordance with the recommendations of the standards ISO 12207, ISO 15504, ISO 90003 and ISO 9126.
2. Description of the goals, requirements and obligations of the enterprise-developer in the field of the quality system, quality criteria for the processes and products of development, delivery and support of the entire life cycle of the software.
3. A set of operational documents supplied to the customer and users to ensure the life cycle and use of a specific version of the software product based on adapted standards ISO 9294, ISO 15910, ISO 18019.
4. Documentation and automation tools for design, development, modification, control and testing used to ensure the life cycle of a software product.
5. Plans and methods for testing the application and evaluating the effectiveness of the processes of the quality system of the enterprise and the software product.
6. Methods of maintenance, identification of software product components and documentation, analysis and approval of versions of software and data complexes.
7. The methodology for configuration management, approval, storage, protection, copying of software product versions and accompanying documents, as well as the accumulation and storage of data on quality characteristics registered in the enterprise archive during the life cycle of software product versions.

The resulting test documents - certification of the quality system of the enterprise and / or software product

1. A report on the availability, relevance and systematic execution of documentation adapted to the requirements and provisions of the enterprise quality system, which provides an integrated quality assurance process throughout the entire life cycle of the software product.
2. The results of monitoring and testing the condition and application of the quality system, carried out periodically to determine its suitability and effectiveness.
3. Report on the availability and maintenance of methods for conducting inspections and documented reports on the results of the achieved quality of fulfilling the requirements of the certification agreement with the customer.
4. The results of registration of the achieved quality characteristics of the software package: identification, accumulation, storage of registered data on the characteristics and attributes of the quality of the software product and its components.
5. The results of the implementation of the development plan, documented input and output data of the development stages and protocols for checking the implementation of the life cycle of the PS.
6. results practical implementation quality programs and the implementation of regulated activities in the field of quality at all stages of the life cycle of the PS.
7. The results of certification of environment simulators and test generators, as well as an assessment of their sufficiency for performing certification tests of a software product.
8. The results of the analysis of the implementation of plans and test methods, test reports, assessments of the compliance of test results with the requirements, as well as test results approved by representatives of the applicant, customer and supplier.
9. The act of the results of verification of the real characteristics of the life cycle of the software system and the quality system of the enterprise, conclusions about their compliance with the requirements for certification of the production of a software product.
10. Certificate of the quality system of the enterprise and / or software product and ensuring its life cycle, license for the use of conformity marks.

Literature

V.V. Lipaev -- Software Life Cycle Standards Profiles. -- Jet Info, Newsletter, N 12 , 2005

K. Milman, S. Milman -- SMMI is a step into the future. -- open systems., N 5-6. (2005), N2. (2006) , 2005, 2006

Assessment and certification of the maturity of the processes for creating and maintaining software tools and information systems ISO IEC TR 15504-CMMI. Per. from English -- M.: Book and business, 2001

V.V. Lipaev -- Processes and standards of the life cycle of complex software. Directory.-- M.: SINTEG, 2006

V.V. Lipaev -- Methods for ensuring the quality of large-scale software.-- M.: RFBR. SINTEG, 2003

"; antisource: "Software products are now used to solve management problems in almost all areas of human activity: in the economy, social, military and other areas. Ensuring the high quality of domestic software products during their mass development and delivery for various applications in the country and on the world market has become a strategic task."; condition: 1]$

In addition to national and international standards, there are several approaches to certification of the development process. The most famous of them in Russia are, apparently, CMM and CMMI.

CMM (Capability Maturity Model) is a maturity model of software development processes, which is designed to assess the level of maturity of the development process in a particular company. According to this model, there are five levels of development process maturity. The first level corresponds to “how it goes” development, when developers go to each project as a feat. The second corresponds to more or less well-established processes, when it is possible with sufficient confidence to count on a positive outcome of the project. The third corresponds to the presence of developed and well-described processes used in the development, and the fourth corresponds to the active use of metrics in the management process to set goals and monitor their achievement. Finally, the fifth level refers to the company's ability to optimize the process as needed.

CMM and CMMI requirements

After the advent of CMM, specialized maturity models began to be developed for creating information systems, for the process of selecting suppliers, and some others. Based on them, an integrated CMMI model (Capability Maturity Model Integration) was developed. In addition, an attempt was made in CMMI to overcome the shortcomings of CMM that had manifested by that time - an exaggeration of the role of formal descriptions of processes, when the presence of certain documentation was valued much higher than just a well-established, but not described process. However, CMMI is also focused on using a highly formalized process.

Thus, the basis of the CMM and CMMI models is the formalization of the development process. They aim developers at the implementation of a process described in detail in the regulations and instructions, which, in turn, cannot but require the development of a large amount of project documentation for appropriate control and reporting.

The relationship of CMM and CMMI to iterative development is more indirect. Formally, neither of them put forward specific requirements for adhering to a waterfall or iterative approach. However, according to some experts, CMM is more compatible with the waterfall approach, while CMMI also allows for an iterative approach.

Of course, RUP is an iterative methodology. Although formally the mandatory execution of all phases or some minimum number of iterations is not indicated anywhere in RUP, the whole approach is focused on the fact that there are quite a lot of them. The limited number of iterations does not allow you to take full advantage of RUP. At the same time, RUP can also be used in almost cascading projects, which really include only a couple of iterations: one in the Build phase and the other in the Transfer phase. By the way, this is the number of iterations that is actually used in waterfall projects. After all, testing and trial operation of the system involves making corrections, which may involve certain actions related to analysis, design and development, that is, in fact, they are one more pass through all phases of development.

RUP Methodology

With regard to the formality of the methodology, RUP presents the user with a very wide range of possibilities. If you do all the work and tasks, create all the artifacts, and fairly formally (with an official reviewer, with the preparation of a full review in the form of an electronic or paper document, etc.) conduct all reviews, RUP can turn out to be an extremely formal, ponderous methodology. At the same time, RUP allows you to develop only those artifacts and perform only those works and tasks that are necessary in a particular project. And selected artifacts can be executed and reviewed with an arbitrary degree of formality. It is possible to require a detailed study and careful execution of each document, the provision of an equally carefully executed and formalized review, and even, following the old practice, to approve each such review at the scientific and technical council of the enterprise. Or you can limit yourself to an email or a sketch on paper. In addition, there is always one more possibility: to form a document in your head, that is, to think about the relevant issue and make a constructive decision. And if this decision concerns only you, then limit yourself, for example, to a comment in the program code.

Thus, RUP is an iterative methodology with a very wide range of possible solutions in terms of formalizing the development process.

Let's summarize the second part of the article. RUP, unlike most other methodologies, allows you to choose the degree of formalization and iteration of the development process in a wide range, depending on the characteristics of projects and the developing organization.

And why this is so important - we will discuss in the next part.