Verifying Theoretical Concepts of Performance Management Framework

Purpose – This paper aims to discuss the verification process of the theoretical concepts of the proposed performance management (PM) framework in practice. Design/methodology/approach – A case study based on a focus group discussion (FGD) method is used to describe the application a PM framework and the implementation of a PM system in a case organisation. Findings – The findings show that the case organisation has been applying the Balanced Scorecard framework and show that it needs to add some important aspects to the framework to support the better implementation of its PM system. Research limitations/implications – This paper is based on a single case study due to the need for an effective FGD in a selected organisation. Originality/value – The study drives the development of PM research in the use of a theoretical verification method to confirm the application of the theoretical concepts of PM framework in practice.


Introduction
A research process on the development of a performance management system (PMS) at an eye hospital (EH) in Indonesia, as a case organisation, has been undertaken gradually. It consecutively begins with a preliminary study, verification of the theoretical concepts of performance management (PM) framework, data collection to validate the theoretical model, and ends up with the development of a PMS.
A preliminary study in the form of discussion on the implementation of a PMS was successfully conducted. The discussion involved several organisational officials who were assigned by the President Director. From this process, it was known that the case organisation has been using the Balanced Scorecard (BSC) framework as the basis for the development of its PMS. The verification process begins by identifying some theoretical concepts of PM framework. Subsequently, they are compared with the concepts identified in the "extended" BSC. Those concepts are vision and strategy, financial perspective, customer perspective, internal business process perspective, learning and growth perspective (Kaplan and Norton, 1996); leadership, strategic planning, data integration infrastructure, business intelligence includes reporting, analytics, a visual representation of data, broader stakeholders, and benchmarking (Kaplan and Norton, 2008).
To ensure that the theoretical concepts are also needed or practically used by the case organisation, a focused discussion on this subject is necessary to be conducted. The discussion process that had been successfully done was in the form of a focus group discussion (FGD) with the top management and some officials of working units who were responsible for the successful implementation of a PMS at the case organisation. The FGD is used as a method for conducting a verification of theoretical concepts of a proposed PM framework.
This study aims to verify the theoretical concepts of PM framework in practice. This was confirmed by an FGD that was able to identify some important aspects needed for enhancing the applied PM framework. The FGD results are expected to provide a better understanding of the required aspects for developing a better PMS.

The Balanced Scorecard
The BSC is a PM framework (DeAngelo et al., 2014;Otley, 1999;Vukomanovic and Radujkovic, 2013) The BSC measures organisational performance over four balanced perspectives, i.e., financial, customers, internal business processes, and learning and growth (Kaplan and Norton, 1996). Generally, the financial perspective is set as the ultimate goal of the organisational performance. The BSC is developed to minimize the use of a number of measures (Kaplan and Norton, 1992). It is used by many types of healthcare organisations (Yap et al., 2005). (Kaplan and Norton, 2008) have identified some aspects that can be integrated into the BSC framework to form a comprehensive management system that links strategy and operations. Those concepts are leadership, reporting capability, analytics as part of management innovation, data integration infrastructure, a visual representation of data, broader stakeholders, as well as internal and external benchmarking.

Leadership
Leadership means that senior management in an organisation must provide clear direction, commitment, and support (Atkinson, 2012;DeAngelo et al., 2014) to the processes of PM. The senior management leadership comprises each phase of the management system. Without a strong leadership, even the comprehensive management system cannot deliver dramatic performance (Kaplan and Norton, 2008). The lack of leadership can cause a problem to the PMS implementation.

Planning
Planning is the management effort to set an organisation's strategic direction, vision, mission, and resources needed to achieve the defined strategic objectives (Atkinson, 2012;DeAngelo et al., 2014;Kaplan and Norton, 1996). Likewise, critical success factors, key performance indicators (KPIs), and performance targets can be defined in the planning stage.

Performance Targets
The element of the planning activities in the PM framework includes performance targets (Atkinson, 2012;DeAngelo et al., 2014;Kaplan and Norton, 1996) that need to be achieved and monitored.

Execution
Execution refers to the implementation of the strategy (Atkinson, 2012;Kaplan and Norton, 1996). A good strategy without execution is worthless. Thus the implementation of the strategy is needed to effectively measure the organisational performance.

Monitoring
Monitoring indicates the process to keep track of the implementation of the strategy (Kaplan and Norton, 1996) and the achievement of the strategic objectives (Atkinson, 2012). An organisation can monitor its performance information to know how it performs. It includes a regular reporting of KPIs.

Communication
Communication refers to the process of delivering the required performance information. To successfully developing and implementing a PM framework, consistent communication and demonstration of the framework value are considered as part of the key success factors (Atkinson, 2012). The successful PM implementation considers the communication of progress and expectations (DeAngelo et al., 2014). The BSC should be practiced as a communication, informing, and learning system, not a controlling system (Kaplan and Norton, 1996). A regular communication is important to ascertain that the organisational performance meets the stakeholders' needs.

Learning
Learning is the process of taking up the experience of developing and implementing a PM framework (Atkinson, 2012;DeAngelo et al., 2014). However, Kaplan and Norton (1996) have focused on the strategic learning to provide the capability of organisational learning for the executives. The learning activity is also used by an organisation to improve or enhance its PMS.

Evaluation
Evaluation means the process to review the achievement of the performance targets and organisational strategic objectives (Atkinson, 2012) or measures and scorecard objectives (Kaplan and Norton, 1996). The performance evaluation compares the actual performance with the performance target and identifies factors that contribute to the success or failure of achieving the defined performance targets. The evaluation results are important for executing performance improvement activities.

Improvement
Improvement is the continuous effort to ascertain that an organisation achieves the expected performance results (DeAngelo et al., 2014). The activities are vital for the betterment of the services an organisation delivers (Atkinson, 2012). The performance improvement activities can be focused on increasing business outputs, such as the number of patient visits, and improving efficiency for certain business process, such as reducing the waiting time of outpatient.

Data Sourcing
Data sourcing is the process of identifying the data systems (DeAngelo et al., 2014) or source systems (Nantiyakul and Meredith, 2008) and collecting the required performance data. The prevalent data sources in a hospital organisation are finance, medical treatment, medical services, and human resources. The data must be effortlessly retrievable and usable (Gupta, 2004). The identification of the data source is to ascertain the availability, accessibility, and adequacy of performance data. Furthermore, the collected data must follow a transformation process to provide an integrated and reliable performance information.

Data Quality
Data quality refers to the level of data usability and validity to support the delivery of reliable performance information. The process of data collection and data quality assurance are inseparably linked (Ward et al., 2014). Nantiyakul and Meredith (2008) have suggested the use of a data quality framework that consists of these components: quality factors, stakeholders, quality metrics, and improvement strategies. The data and information quality may cover accuracy, validity, consistency, and completeness aspects.

Integrated Information Processing
The integrated information processing is important to satisfy information needs (Winter et al., 2011). The data integration infrastructure facilitates the visual representation (Kaplan and Norton, 2008) of performance information. The data integrity must be maintained (Gupta, 2004).

Categorised Measures
Performance measures can be differentiated into three categories, which are business results, internal processes, and resource capabilities (Wibisono and Khan, 2010); or into four perspectives of the BSC (Kaplan and Norton, 1996). This grouping is to simplify the way to manage organisational performance measures.

Reporting Capability
Reporting capability refers to the regular reporting on corporate performance (Atkinson, 2012). It is the ability of the PM process to provide the required performance information for particular users. It can be implemented in a system of operational dashboards and strategy scorecards to support the need for operational review and strategy review (Kaplan and Norton, 2008).

Analytical Capability
The analytical capability denotes the ability to provide a means for analyzing performance information by way of descriptive, prescriptive, predictive, and other interactive information analyses. The analytics capability can be integrated into PM framework (Klatt et al., 2011;Schläfke et al., 2013). An organisation can interactively perform in-depth data drilling using a provided analytical capability (Kaplan and Norton, 2008).

Information Visualisation
The reporting mechanism can practice a visualisation method to present the performance information. The automated dashboards make possible the visual representation of the underlying data (Kaplan and Norton, 2008). One of the differentiating features of many recent dashboards and analytical systems is the use of advanced visualisation techniques (Ward et al., 2014).

Stakeholders
The implementation of the framework is subject to continuing engagement and consultation with stakeholders (Atkinson, 2012). Businesses must regularly communicate an undeviating message to all stakeholders (Gupta, 2004). The main goal of an organisation is to create long-term value for stakeholders (Kaplan and Norton, 2008). Stakeholders can be internal or external (Gupta, 2004;Kaplan and Norton, 2008). The internal stakeholders may encompass employee, medical staff, and board of executives while the external stakeholders may cover patients, society, business partners, shareholders, and government.

Benchmarking
Benchmarking supports competitive positioning by identifying best practices and assessing comparative operating performance (Gupta, 2004). External benchmarks for performance measures can be beneficial (Kaplan and Norton, 2008). The performance assessment can use benchmarking in the search for continuous improvement (Atkinson, 2012).

The Proposed PM Framework
A PM framework based on the theoretical concepts is proposed, as shown in Figure I. The theoretical concepts of the proposed framework need to be verified in a field research. The proposed PM framework encompasses three core processes, i.e., planning and definition, execution, and monitoring. These processes indicate the need for translating the defined strategy into the execution phase (execution and monitoring processes). The performance measurement performs several tasks that include categorising performance data (measures), formulating PIs, and refining indicators used by the organisation. The framework is completed by three supporting processes which are leadership, communication and learning, and evaluation and improvement.

Verification of Theoretical Concepts
Some scholars address the need for a verification of the theoretical concepts identified from the literature. A developed theory is subjected to empirical verification (Jaccard and Jacoby, 2010). The developed theory needs to be compared to the theoretical framework identified in the literature review stage (Amaratunga and Baldry, 2001). In the context of PM studies, theoretical verification is used as one of the available methods of investigation (Neely, 2005). The method can be used to identify the application of the theoretical concepts in practice. In a qualitative study, the researcher looks for the confidence of theoretical variables through a verification process (Creswell, 2009).

Methodology
The following steps were applied to verify the theoretical concepts of PM framework: 1. Define the purpose of study; 2. Review relevant literature; 3. Set the baseline of comparison between the theoretical concepts of the proposed PM framework and the dimensions or aspects of the extended BSC framework; 4. Select an EH as a place to carry out a case study; 5. Conduct an FGD as an appropriate method to collect sufficient data that describe the application of a PM framework and the implementation of a PMS at the case organisation; 6. Analyse the data that may reflect comprehensive aspects needed by a PM framework to support a PMS; 7. Findings are discussed to ensure the support of evidence to the need for the theoretical concepts of the proposed PM framework; 8. Suggest undertaking quantitative research for finding the relationships amongst variables of PM framework.

Theoretical Concepts and Dimensions of the Extended BSC
To convince that the theoretical concepts can be found in practice, they need to be initially compared to the dimensions of the BSC framework with some complementing aspects. The complementing aspects are leadership,

Planning and Definition
Leadership reporting, analytics, data integration infrastructure, a visual representation of data, stakeholders, and benchmarking (Kaplan and Norton, 2008). The integration between the dimensions of the BSC and their complementing aspects can be called as the dimensions of the extended BSC. In this paper, the theoretical framework identified in the literature review stage is the extended BSC while the developed theory is based on the theoretical concepts of PM framework. Therefore, the theoretical concepts can be specifically compared to the dimensions of the extended BSC by following the suggestion of Amaratunga and Baldry (2001); as summarized in Table I. This baseline comparison was used in the theoretical verification process.

Case Study
A case study was carried out at an EH in Indonesia. Some considerations were taken as criteria, why the research was conducted using a case study approach in this organisation, as follows: 1. Focus efforts on theoretically useful cases, i.e., an opportunity to replace or extend the developed theory by filling conceptual categories (Eisenhardt, 1989); 2. Know well about the underlying issues of PM framework application in the organisation; 3. Learn the benefits and limitations of the PMS implementation within the case organisation which has been practicing the BSC framework for several years; 4. Conduct a gradual study in developing a better PMS for the case organisation.

Focus Group Discussion
An FGD was conducted with the top management and some officials of working units, as participants. The participants were selected based on the criteria of the involvement of each participant in the initial study and some appropriate officials who are responsible for the successful implementation of a PMS at the case organisation. A formal permission request was sent to ask six officials to be involved in the FGD. However, in the execution phase, there were eight officials and three staffs who involved in the FGD process.
This FGD was conducted to find out the participants' views on the important aspects of the PM framework. The top management comprises the Board of Directors (BOD, key participants 1 through 4) and the Quality Committee (key participant 5) while the officials of working units consist of some heads of department, i.e., Information Technology (IT, key participant 6), Research and Development (R&D, key participant 7), and Budgeting and Planning (BP, key participant 8). Several staffs from the Human Resources (HR) Section, BP Section, and R&D Unit, as supporting participants, also got involved in providing data for the FGD process.
To get insight about the application of PM framework and the implementation of PMS, some questions grouped in two different subtopics were provided as follows: (PM framework) 1. What do you think about the PM framework currently used to support a PMS in your organisation? 2. What factors need to be added or are considered to be lacking in the framework currently used by your organisation? 3. What are your opinions regarding the use of the PM framework as a guideline for implementing a PMS with the support of information systems? 4. What factors are needed by a PM framework that is considered to be suitably developed and used by your organisation? (Indicators and benchmarking) 1. What are your opinions regarding the use of criteria to determine KPIs in your organisation? 2. What do you think about the need to use other decision-making methods, such as developing priorities for the criteria and alternatives to determine KPIs for your organisation? 3. What are your opinions regarding the need for benchmarking of performance indicators (PIs) with similar organisations, both at national and international levels? 4. What do you think about the organisation's need to collaborate with or involve in the associations of similar organisations at the international level, such as the ASEAN Association of Eye Hospitals (AAEH) and the World Association of Eye Hospitals (WAEH), to get the benefits of benchmarking of PIs and knowing best practices for improving performance? As planned, the first group of questions needed to be answered by all key participants while the second had be discussed and answered by key participants 1 through 4. Similar FGD processes were also planned to be carried out in six other similar organisations. Unfortunately, the number of responses to the delivered formal requests was not as expected. Only two similar organisations agreed to give permissions for conducting the same study. However, these organisations have not practically used a complete PM framework nor implemented a PMS to manage their organisational performance. Therefore, the FGD was only conducted in one organisation.

Important Notes of FGD
The FGD resulted in some important notes for the process of developing a PMS. The identified notes are related to the subject matter of each subtopic in the discussion, as summarized in Table II. The conducted FGD resulted in three important issues that need attention in developing better PMS at the case organisation as follows: 1. How to measure the achievement of the organisation's vision through the success or failure of KPIs targets fulfillment? 2. How to conduct a structured data analysis on the causes of not achieving KPIs targets that affect the achievement of the organisation's strategic objectives? 3. How to ensure the data quality so that the validity of information can be accounted to the users?

Important Aspects of PMS
The identification of the above notes of FGD results shows some important aspects of PMS, i.e., vision and mission, strategic objectives, strategy and programme, performance targets, perspective, indicators and KPI, data source, data quality, data validity, data integration, integrated database for reports, reporting of information, data/information analysis, information visualisation, monitor the achievement of KPIs, evaluate the achievement of strategic objectives and vision, stakeholder, and benchmarking, as shown in Figure II. These aspects are considered for developing a PMS at the case organisation.

The Desired PMS
The conducted FGD described the PMS to be developed in the case organisation. The following is a comprehensive opinion of the FGD participants regarding some important aspects of a PMS. The opinion was to describe the needs for simple and easy to manage the PM framework to support the desired PMS.
"The consideration of using the BSC is because the PM framework is sufficiently directed, simple and easier to understand. The BSC applied in the organisation is a modified version, which makes customer satisfaction (stakeholder) as the main goal of the organisational performance achievement, besides utilizing learning and growth, business process, and financial perspectives."

Figure-II. Important aspects of PMS
The opinion reflects the readiness and sufficient satisfaction of the case organisation for using the BSC framework. As previously explained, the BSC emphasizes the importance of the strategic plan. The plan needs to be translated into action or execution to ensure the achievement of the organisational performance. It accommodates the definition and formulation of KPIs used by the organisation. However, the applied framework did not provide any suggestion on how to select appropriate KPIs to track the achievement of strategic objectives. It was confirmed by the following description: "The use of BSC has been directed since developing the organisation's strategic plan that is cascaded from vision, mission, strategic objectives, and implementation plan in the form of strategies and programmes. However, the selection criteria have not been used in determining the right KPIs to achieve the strategic objectives. At this time, the selection of KPIs is done through an FGD to make the agreed choices of appropriate KPIs by looking at strategic objectives." Hence, the selection of KPIs did not use any selection criteria. The use of criteria is important since it can help the selection of KPIs be more manageable and be more accounted for. It is supported by this opinion: "However, for the next, we need to use it, for example by choosing priorities with the criteria of urgency, impact, or others...." In the current business environment, performing information analysis becomes a necessity since it can give many benefits to the management to track the progress of achieving the organisation's strategic objectives. However, the BOD realised that it is difficult to perform the appropriate performance information analysis and evaluation. This was due to the fact that: "....data used to support KPIs are already a lot but the data cannot be used as sources for conducting performance data analysis based on their importance. Especially for the purpose of analysis and evaluation of the success or failure of the achievement of performance targets, strategic objectives, and vision of the organisation...it cannot be done in the BSC...." Although KP 1 and 6 realised the importance of data as sources for performance data analysis, they commented that the data quality and validity are still outstanding issues. It was reflected in the following opinion: "....the use of a PMS cannot guarantee the validity of data (e.g., service data, quality indicators) because once the data are processed into information the value can be different…The data validity is important because, prior to any reporting and analysis of information, the data quality must also be maintained." Furthermore, the KP 6 mentioned that the data integration is important to be considered for having a reliable information that must be provided by his department. He expressed his feeling by issuing the following comments: "....data are not currently integrated as a whole. In terms of IT function, the expectation is that all information is generated from one system, which is the hospital information system (HIS), so that the report produced can be accounted for. However, not all data is entered into and processed through HIS...there are data that have not been accommodated in HIS...When a report that includes information on PIs is required, its provisioning process will be done manually." Complementing the previous argument, the KP 6 continued to express his opinion on the data validity issue: "The risk of data discrepancies can occur in a series of interrelated processes (input-process-output)...If one does not do so, the difference in data will be known during the recapitulation process. The IT function will recognise the data recorded in the system. However, this will cause data validity issues, because there are incomplete data...." Those opinions are to make sure that it is important to aware of data quality and validity. Another concern about information visualisation was raised by KP 1. She confirmed the requirement of information presentation as follows: "....the use of internally standardized dashboard is a necessity for the organisational level or management interest...Hence, an application is required to visualise the information." Two directors considered to extend the involvement of organisation in benchmarking practice. The efforts were mainly aimed at learning the best practices to improve the performance of the organisation. Both of them considered the case organisation to join a regional association of EHs which has been established for years as a non-profit organisation. They explained their concerns in the following comments: "With regards to benchmarking, currently the organisation does it with an EH from India, which is considered better. However, benchmarking is not to look at equality but it is the effort to reach a level like the compared organisation...Similar activities are also conducted with an EH in Thailand, as well as other EHs in Dundee, Scotland. Benchmarking is continuously done only with the EH in India because the operation pattern and the patient demography are similar. At the national level, benchmarking is conducted on almost equal organisations, which manage quality indicators...Participation in international EHs associations, such as AAEH, is deemed necessary to develop cooperative relations." The KP 1 had a long explanation of the expectation on the conducted study. She looked enthusiastic in explaining the expectation, but in essence, it was an indication that she has a good commitment to the successful effort of PM improvement. Therefore, she had reflected the leadership role in this case. The explanation is as follows: "We hope to get a better PMS for our organisation. What we want most is when we analyse, monitor, and evaluate quality improvement (data and processes) on a continuous basis, such as how to be practical to improve the quality of data obtained. The data collected are a lot but it needs the right way to process them into useful information. If the current information system is used as an example, then the system may not only be showing performance measures, but it must be able to provide inputs to what will happen to the performance achievement (predictive). Thus the system has the analytical ability to know what data/information to be fixed…The evaluation process of how we can minimize the shortcomings and maximise the advantages of the system also needs to be done. Our hope is that this research can benefit us to improve our PMS."

Findings
From the point of view of the management process, the findings show that the case organisation has been practicing a thorough concept of PM framework, which encompasses planning, execution, monitoring, learning, as well as evaluation and improvement. The performance measurement (data sourcing, data/information processing) has also been clearly practiced. The leadership to enhance the PM framework and to support a better PMS has been practiced by the top management.
The findings show that the case organisation wants to keep using the BSC as the framework underlying the development of its PMS by adding or strengthening some important aspects aimed at answering the three important issues described earlier. The linkage amongst aspects of a PMS is shown in Figure III.
The findings show that the PMS needs to be supported by a directed-simple-easy to understand PM framework, quality and integrated data, valid information that can be presented in reports and visualised in dashboards, analysis capabilities of the achievement of vision and strategic objectives of the organisation through the success or failure of KPI targets fulfillment, structured analysis of the causes of non-achievement of KPI targets, and the use of information for stakeholder interests and for the support of the benchmarking purposes.

Conclusion
This study has confirmed the need for theoretical verification (Neely, 2005) of the theoretical concepts. The verification process of the theoretical variables in a qualitative study (Creswell, 2009) was done using an FGD method. The proposed PM framework has been compared to the extended BSC that is identified in the literature review stage (Amaratunga and Baldry, 2001).
From the carried out FGD, it can be concluded that the discussion was effectively running in accordance with the expectation to meet the specified objective. This is indicated by an in-depth knowledge level of participants on the importance of using a PM framework, a constructive view of key aspects that support the development of a PMS, as well as a comprehensive discussion of better PMS, from collecting data to providing information.
The proposed PM framework can be used as an alternative to the BSC framework to guide the implementation of a better PMS for the case organisation. It is supported by the confirmation of theoretical concepts in practice. Communicating performance information to stakeholders is the only aspect that needs to be improved in practice at the case organisation. Furthermore, in order to have a robust theory, further research needs to be carried out to validate the relationships amongst variables of PM framework.