Critical success factors for software projects: A comparative study

,


INTRODUCTION
The term 'Software Engineering' was first coined at the first NATO Software Engineering Conference in Germany in 1968 (Naur and Randell, 1969), and there was widespread consensus that there were problems with software development and maintenance.These problems were later discussed by Brooks (1975Brooks ( , 1987Brooks ( , 1995)), and he concluded that there is no silver bullet to overcome this problem.The term 'Software Crisis' emerged to describe the software industry's inability to provide customers with high quality products within schedule and under budget.Hardware costs were dropping while software costs were rising rapidly.Major computer system projects were sometimes years late, and the resulting software was unreliable, hard to maintain and performed poorly.
Since the 1980s, in the medical field for example, computers have been designed to help people, and most *Corresponding author.E-mail: hairulnizam@um.edu.my. of the time they do.However, in the case of Therac-25, computer errors could be fatal.Between 1985 and1987, two people died and four others were seriously injured after they received massive radiation beamed via Therac-25 radiation therapy machines.Successful investigations revealed that defective software was among the various factors leading to this accident (Leveson and Turner, 1993).Another example is the delay of over 16 months in the opening of Denver International Airport and the over 100 million dollars in excess of the budget in the airport's construction cost (Swartz, 1996)."One main reason for the delay and overrun was the presence of major bugs in the baggage handling control software" (Glass, 1998).The explosion of the European Space Agency rocket, Ariane 5, 40 seconds after lift-off in June 1996, is another example of an accident caused by software defects.The rocket and the four scientific satellites on board, which total cost about 2.5 billion dollars, were completely destroyed.In-depth investigation uncovered that "specification and design errors in the software, and poor analysis and testing of the failed subsystems, were to blame" (Ariane 501 Inquiry Board, 1996).Even today, this situation has not changed much.Software development projects are known for being completed far over budget and behind schedule (Gray and Larson, 2008).In the United States (US), a survey conducted by The Standish Group (1995) in 1994, which reported data from several thousand IT projects, revealed a success rate of only 16% of software projects.Meanwhile, 31% of projects failed while the remaining 53% had cost overruns, time overruns and impaired functionality.Of these, the average cost overrun was 189%, and the average time overrun was 222%.A recent report by The Standish Group showed a slight improvement, yet the figure remained troublesome, with a success rate of less than 40%.Table 1 tracks the progress of Standish software project performance over a decade.These figures were published by a number of different authors in various journals and white papers (Standish Group, 1995, 1999, 2001, 2010;Othman et al., 2010;Hass, 2007;Eveleens and Verhoef, 2010;Humphrey, 2005;Hartmann, 2006).It is important to note, however, that some researchers have argued and expressed concerns regarding the reliability of the figures (Jorgensen and Molokken, 1994), some felt that the figures do not reflect reality (Glass, 2005(Glass, , 2006)), and some questioned their validity (Eveleens and Verhoef, 2010).
Another study, conducted in the United Kingdom (UK) by Oxford University in collaboration with Computer Weekly in 2003, reported that only 16% of the 421 IT projects examined were finished on time within the estimated budget and with the agreed functionalities, only 55% of projects were completed on time, and 41% were completed within the agreed budget (Sauer and Cuthbertson, 2003).Similarly, another UK survey published by the Royal Academy of Engineering and British Computer Society (Taylor, 2000) reported that only three out of the more than 500 development projects assessed met the survey's criteria for success.Although the success rate of software projects is improving each year, the general trend still suggests that software projects are hard to manage.This is also supported by the research conducted by Simons et al. (2003), where they found that the issues surrounding software engineering identified over 40 years ago, during the NATO Software Engineering Conference, remain unresolved today.
Critical success factors are factors that, if addressed appropriately, will significantly improve the chances of project success (Pinto and Rouhiainen, 2001).Over the past several decades, numerous research studies (Pinto and Mantel, 1990;Belassi and Tukel, 1996;Tukel and Rom, 2001;White and Fortune, 2002) have been performed in the area of project management to identify critical factors that influence the success and/or failure of projects.However, the critical factors are usually identified in general for projects in various industries, such as engineering, manufacturing, construction and training rather than being focused on software development or IT projects.Software has certain unique characteristics (Brooks, 1995;Galin, 2004;Jain, 2008;Fairley, 2009) that make software development projects differ from other typical engineering projects.Also, most researchers agree that there are differences in project management among different industry types (Cooke-Davies and Arzymanow, 2002;Ibbs and Kwak, 2000;Zwikael and Globerson, 2006), Dvir et al. (1998) suggest that project success factors are not universal to all projects.Thus, the critical success factors identified in other industries cannot be used as valid critical factors for software projects.In this research study, we did not differentiate between IT projects and software development projects, because IT projects involve software (Royal Academy of Engineering and the British Computer Society, 2004).
Some research studies and articles have reported on the critical success factors specific to software and IT projects (Standish Group, 2010;Sauer and Cuthbertson, 2003;Taylor, 2000) however, these studies are specific to one particular country.There has been no comprehensive study reported on different project sizes in various domains and in multiple countries.Such a thorough analysis is important in order to identify critical factors that are applicable for software projects.Three research questions have motivated the investigation reported here: 1. Research question 1 (RQ1): What factors, as identified in the study, have a critical impact in determining software project success?
Based on the findings from RQ1, we formulated other research questions as follows: 1. Research question 2 (RQ2): After over 20 years, are non-technical factors still a major contributor in determining software project success, as claimed by DeMarco (1988)? 2. Research question 3 (RQ3): Are there any clear and significant differences between the critical success factors of software projects identified in this study and with those for general projects identified in other, related study?

MATERIALS AND METHODS
In this research study, the four well-known online journal databases: www.Sciencedirect.com,www.ieeexplore.ieee.org,www.springerlink.com and www.Emeraldinsight.comwere extensively searched.As a result, we found 76 articles consisting of case studies, surveys and views of practitioners and experts.Of these 76 articles, only 43 articles were related to software projects, and 33 non-software projects were excluded.The remaining articles were analysed to develop a list of critical factors that specifically affect the success of software projects.The occurrences of each factor in the literature have been identified to determine the relative importance of each factor.
In this research study, each article was carefully reviewed, and a list of factors was compiled.There were three types of articles.First, in the articles describing the results of empirical study (that is, surveys and case studies), it was easy to identify the factors because the authors often provided a summary of success or failure factors in the form of a list or a separate paragraph.The wellknown surveys, such as reports published by The Standish Group and the British Computer Society, belong to this category.Second, in the articles in which the authors (that is, experts and practitioners) have described the success and/or failure factors based on their wide range of experiences, it was also quite easy to identify the factors.Third, there were a few articles in which software or IT project failure was discussed, but the authors did not provide a summary of success or failure factors; in this case, each article was read carefully to avoid misunderstandings and misinterpretations.This research study covers 29 published sets of empirical data from case studies, 9 published empirical data sets from surveys and 5 articles written by experts and practitioners from 1990 to 2010.Only 2 publications were found from before 1990, namely those published by Schmitt and Kozar (1978) and Wingrove (1986), which are not covered in this analysis.
In order to analyse and produce a list of critical success factors from the extensive literature, the content analysis method was adopted.Content analysis is an approach to the quantification of qualitative data (Holsti, 1969).Although it was originally developed for the analysis of human communication in social sciences, several empirical software engineering studies, such as (Rainer et al., 2003;Rainer and Hall, 2003), have adopted this method as part of their research methodology.Babbie (2010) defined content analysis as the "study of recorded human communications" including various forms of communications such as books, magazine, web-pages and letters.In this research study, the communications to be analysed were the published articles.Seaman (1999) describes the method as follows: an "analysis method based on counting the frequency of occurrence of some meaningful lexical phenomenon in a textual data set."Meanwhile, Weber (1990) described the method of measurement in content analysis by "counting the occurrences or calculating percentages of meaning units such as specific words, phrases, content categories and themes, and later transfer to control document."This method later enables the application of frequency analysis by extracting quantitative data from qualitative data in the article and further recording it in frequency tables for the purposes of analysis.
Prior to performing the frequency analysis, the articles were read to generate appropriate categories for responses.Different factors that contributed to the same meaning were grouped into one category.For example, focused and hardworking staff, team commitment, team morale and motivated personnel were grouped together in the 'committed and motivated team' category.This process was repeated until distinct sets of categories were obtained.Each category represents a critical success factor for software projects.
The method of content analysis was adopted in this study, rather than the data extraction method or the frequency analysis method alone, because some of the factors described by the authors in the articles were not explicitly clear and required careful reading, understanding and interpretation to produce accurate findings.Because different authors use different terms to identify the same factors, it is quite complicated to determine to which category a given factor belongs.It is not enough to simply count the occurrence of words (factors) as is done in the data extraction or frequency analysis methods.It is revealing that no study has used content analysis in the articles published from 1978 to 2010.One article by Wateridge (1998) produces a list of success criteria using only frequency analysis.Meanwhile, White and Fortune (2006), produce a list of success factors using only frequency analysis.
The inter-rater reliability was verified to ensure that there was no substantial bias or subjectivity in the identification and grouping process.Another researcher, who was not familiar with the current issues being discussed, was asked to identify factors that appeared in all the articles.The results were compared to those of the previous lists, and no major disagreements were found among the results.
In order to perform the frequency analysis, the occurrence of each success factor in each article in the literature search was recorded.The numbers and percentages of occurrences of each factor were then transferred and tabulated in a frequency table.By comparing the occurrences of a critical success factor in a number of articles against the occurrences of other factors in the same articles, the relative importance of each critical success factor could be calculated, and furthermore, the success factors could be compared and ranked.Finally, analysis and discussions were performed regarding the factors that were found to be critical in the literature.

Validity
The findings of the critical success factors in this research study were extracted from multiple empirical data and expert views reported in eight countries (that is, Finland, the United States, the United Kingdom, Hong Kong, Singapore, Belgium, Australia, and Canada), concerning small to large software projects in various domains; however, the findings are applicable only to software projects.Although a sample of eight countries is small and generalisability to the whole software engineering community worldwide is problematic, we have high confidence in our research findings because most of the articles included were taken from established scientific research journals and had a minimum of 11 citations, and a few of them were from well-known survey reports and articles written by experts and practitioners who had wide experience in software-related industries from 1990 until 2010.Often, the factors reported in books are based on the previous work of others and do not cover the latest research findings, so we did not consider books.We also decided not to include conference and workshop proceedings because it is hard to determine the quality of the articles in such publications.Note that this research study was not intended to localise the findings; thus, we considered it irrelevant to conduct an empirical study in any particular country.

RESULTS
Here, we present the findings with regard to the research questions presented at the end of the introduction.

Critical success factors
To answer RQ1, Table 2 shows the list of critical success factors of software projects identified throughout 43 publications.Based on the analysis of our extensive literature search, 26 critical success factors were found to be related to project success.The total frequency of occurrences was 372.Although some of the factors have a low frequency, we still decided to treat them as critical factors because the criticality depends not on the frequency but on the literature in which the critical success factors were highlighted by the articles.
There is a good opportunity, if an organisation or project manager is attentive, to control the top 5 critical factors to drive towards project success since the percentage of frequency of occurrences for each is more than 50%.The extensive literature shows that most of the practitioners consider clear requirements and specifications; clear objectives and goals; and a realistic schedule to be the three most critical success factors that can contribute to project success.These three critical success factors can be considered to be pre-project execution aspects that need to be made clear and solidified before commencing and executing software projects.Although 88% of the publications include at least one of these three factors, only 26% cite all three.This finding is very much in line with the research studies conducted by Wateridge (1995) and Fortune and White (2006), which reflect that there is no broad consensus among researchers and practitioners in determining project critical success factors.
The estimations of schedule and budget, which are the third and eighth highest factors, respectively, show that arbitrary and illogical estimation due to schedule pressure, or sometimes to a business strategy in order to win a tender, can contribute to project failure.The common problem of early estimation, without fully defined requirements, has troubled the software industries for more than fifty years (Boehm, 1981;Jones, 1998).Accurate estimation based on proven methods is needed to address these factors.Several methods, such as the Constructive Cost Model (COCOMO) and Constructive Cost Model-II (COCOMO-II), proposed by Boehm (1981) and Boehm et al. (1996), respectively; the PROxy-Based Estimation (PROBE) method through the adoption and utilisation of the Personal Software Process (PSP) by Humphrey (1995); Function Point Analysis; expert judgment method; Delphi technique; and also Wide Band Delphi technique by Boehm (1981), have been proposed and developed to support estimation procedures.
The results also suggest that, in the opinion of many practitioners, the top three most important human factors are effective project management skills and methodologies applied by the project manager, support from top management and user involvement.The critical success factor 'effective communication and feedback' (between stakeholders) was cited in 46.6% of the Nasir and Sahibuddin 2177 literature, and was ranked at seventh place.
In our previous analysis, we combined skilled and sufficient staff with good performance vendors into one category, that is, 'skilled and sufficient staff/contractors/vendors', and this category falls into the fourth rank.However, we split this category into two, that is, 'skilled and sufficient staff' and 'good performance by vendors/contractors/consultants', once we realised that both are acquired and handled by different units in organisations, as outlined in the Project Management Body of Knowledge (PMBOK) (2008), and Projects In Control Environment 2 (PRINCE2) (2009).Staff-members are handled by the human resource department, while the vendors and consultants are acquired by the procurement department.Both departments handle, manage, acquire and assess these resources differently.Splitting this into two makes 'skilled and sufficient staff' become ranking number nine, while 'good performance by vendors/contractors/consultants' becomes ranking number twenty-five.
We also considered frozen requirements to be another group of factors, different from clear requirements and specifications, because sometimes the sponsors or endusers are very clear about what they want in the first stage, but as the project progresses through subsequent phases or stages, they might see the need to introduce new or changing requirements (or sometimes features that are not required).This also affects the project schedule and budget.

DISCUSSION
Here, we discuss the findings with regard to the research questions presented at the end of introduction particularly for RQ2 and RQ3.

Non-technical factors versus technical factors
According to Zahran (1998), software development has three main aspects, which are people, process and technology.Thus, in order to answer RQ2, the critical success factors have been grouped into these three general categories, that is, people factor, process factor and technical factor, as shown in Table 3.Based on our examination, from the 26 critical project success factors listed in Table 2, it appears that 7 critical success factors (27%) belong to the people factor category, 16 success factors (62%) belong to the process factor category and the remaining 3 critical factors (12%) belong to the technical factor category.This is not surprising, as software projects almost never fail because of technical reasons, despite the fact that people and process problems may manifest technically.Yet, these technical factors can be alleviated with proper management of people and process practise.Based on our findings, the

Comparison with previous study
For comparison, we have chosen a research study, conducted by Fortune and White (2006), which took a quite similar approach to that of our study in deriving project critical success factors.In their study, 63 publications were reviewed, consisting of case studies, surveys and theoretical studies.The method used was quite similar to ours; it covers different project sizes, various domains and multiple countries.We also found that this research study is the latest publication, reported up to September 2010, to derive critical success factors based on comprehensive literature.The only major difference is that our research study only considered and opted for software projects, but they included both: 36 nonsoftware project articles and 27 software project articles, with the proportions of 57 and 43%, respectively.
In order to answer RQ3, Table 4 highlights a number of significant differences between the findings (that is, p value highlighted for significant differences).18 critical success factors (51%) have significant differences while 17 critical success factors (49%) have no significant differences between the studies.For example, 'realistic schedule', 'effective project management skills/methodologies (competent project manager)' and 'appropriate development processes/ methodologies (process)' were cited by both; however, the analysis shows that statistical differences exist between the studies.Another example: 'clear requirements and specifications', 'frozen requirements' and 'realistic budget' were not cited in (Fortune and White, 2006) but these critical factors were present in our study.Similarly, 'strong business case/sound basis for project', 'project sponsor/champion' and 'adequate budget' were present in (Fortune and White, 2006) but were not identified in our research study.
Figure 1 illustrates and indicates that there were some The results suggest that the factors of clear and frozen requirements; realistic estimation of schedule and budget; and effective project management skills and methodologies applied by the project manager are the five most critical success factors of software projects, illustrating clear and significant differences between our study and the research study in (Fortune and White, 2006).

Conclusion
We have reported on our extensive literature survey of critical success factors that impact software projects.In this research study, 43 articles were found to be significant contributions that can be analysed to develop a list of critical factors that specifically affect the success of software projects.The 43 articles consist of 9 published sets of empirical data from case studies, 9 published empirical data sets from surveys and 5 articles written by experts and practitioners from 1990 to 2010.The method of content analysis was adopted in this study, rather than the data extraction method or the frequency analysis method alone, because some of the factors described by the authors in the articles were not explicitly clear and required careful reading, understanding and interpretation to produce accurate findings.
Based on the analysis of our extensive literature search, 26 critical success factors were found to be related to project success.There is a good opportunity, if an organisation or project manager is attentive, to control the top five critical factors to drive towards project success since the percentage of frequency of occurrences for each is more than 50%.All these critical success factors were then grouped into three general categories, namely, people factor, process factor and technical factor.Based on our examination, from 26 critical success factors, it appears that the non-technical factors dominated over technical factors, with 23 critical success factors and 3 critical success factors,   , 8, 10, 12, 21, 22, 23, 24 27, 28, 29, 30, 31, 32, 33, 34, 35 2, 3, 4, 5, 6, 7, 9, 11, 13, 14, 15, 16, 17, 18, 19, 20, 25, 26 Our research study Research study in Fortune and White ( 2006) respectively.Thus, our findings were in line with the claim made by DeMarco (1988) over 20 years ago.We also found that the factors of clear and frozen requirements; realistic estimation of schedule and budget; and effective project management skills and methodologies applied by the project manager; are the five most critical success factors of software projects, illustrating clear and significant differences between our study and the research study by Fortune and White (2006) that focuses on general projects.
We believe that our findings will assist researchers in selecting a reliable set of critical success factors of software projects for empirical studies.One might continue to survey other countries to build a body of evidence to support the findings presented here.A similar study approach can be replicated in other fields such as construction, engineering, manufacturing, research and development, etc., in order to identify the critical success factors for that particular field.
For future work, the list of critical factors can be used as a valid factor to determine the degree to which a project management framework and methodology such as PMBOK or PRINCE2; software process improvement standards and methods such as the Capability Maturity Model Integration (CMM-I), International Standard for Software Process Improvement and Capability dEtermination (SPICE), International Organization for Standardization (ISO) 9000, or TSP; well-known software development processes such as agile processes, Rational Unified Process (RUP), etc. can address all these critical factors.Based on our initial research study, for example, PMBOK and Team Software Process (TSP) contribute a certain degree to addressing the identified critical success factors (Nasir and Sahibuddin, 2011).To highlight 'realistic schedule' as an example, PMBOK uses formal processes for estimating and scheduling that are capable of producing a realistic schedule when suitable historical data is available.However, nothing specific is included in the process to address political pressure to produce lower cost estimates and shorter schedules during the negotiations of scope, budget, and delivery date that are private between the project manager and sponsoring management.This can be a much bigger problem than the lack of capability of the estimating process.TSP, in contrast, uses a highly capable estimation process based on historical data.The entire team produces and reviews the plan together, identifies any proposed exceptions to management goals and constraints, and participates in the out-brief to sponsoring management.This helps catch omissions and inconsistencies that could result in an unrealistic plan.Any negotiations concerning scope, budget, staffing, etc. are performed in public as part of the out brief meeting.Team and management commitments are documented in the minutes of the meeting.This is explicitly included to prevent the project manager from privately acquiescing to unrealistic goals or constraints under pressure from management or the client.We can integrate both models together to complement each other to more effectively address the critical success factors for software projects.
Each framework or methodology on its own could not perfectly address all the identified critical success factors.By blending a project management framework and software process improvement, or other excellent software development processes, we believe one can address all the critical success factors to a better degree.This is supported by several research efforts that seek integration between a few areas, such as traditional project management and agile (Hass, 2007), CMMI for Development (CMMI-Dev) and PMBOK (von Wangenheim et al., 2010), PMBOK and Rational Unified Process (Callegari and Bastos, 2007), agile and PRINCE (Nawrocki et al., 2006), CMMI and PMBOK (Jenkins, 2005) and many more in order to ensure better control in managing software projects.A study conducted by Bayo et al. (2007) showed that there has been increasing demand in the knowledge and application of quality software to improve the country socio-economic growth.Thus, a comprehensive model is needed in ensuring the way to produce high quality software.

Figure 1 .
Figure 1.Comparison of two data sets.

Table 1 .
Standish software project performance over a decade.

Table 2 .
Critical success factors identified across 43 publications.

Table 3 .
Critical success factors based on categories.

Table 4 .
Significant differences between the two studies.