Jusletter IT

Quality, innovation and technology are one of the most important factors for a company to stay competitive, especially in today’s economic environment, where markets, technologies and processes are changing fast. Business requirements and related processes will change based on future technologies.¹ Today’s complex and highly automated business processes require the right people with the right skills at the right place, as data and related information are a major part of companies’ future competitive advantages.² Unqualified resources cause lot of efforts within organizations as well as limitations in innovation, research and technology.³

A transformation towards end-to-end business process service delivery models is required considering important factors like security.⁴ Questions, such as,

• how markets and business requirements will change or
• if today’s services will also be required in the future or
• in which strategic direction the organization will develop

are difficult to answer as there are too many assumptions. From technology and process point of view predictions are a little bit easier, as there are facts and roadmaps existing. In the context of digitalization, internet of things (IoT) and industry 4.0, information technology (IT) becomes a more and more integrative part of core business processes.⁵ One major success factor are the resources operating the services independently of companies IT sourcing model.⁶ The difficulty lies within an objective assessment of the IT service quality as well as the identification of qualified resources.⁷ This paper compared – based on a real case example – the major differences of two suppliers as part of the supplier selection and evaluation process, with the goal to identify objective IT service quality indicators, hidden cost drivers, proper service level agreements (SLAs) as well as further critical success factors. Therefore a metadata analysis of communication items of the IT service delivery process has been performed.

Nowadays literature describes the importance of several different supplier evaluation criteria, where quality is still one of the most important factors. ⁸ The expectation of service quality is determined by factors such as recommendations, personal needs and experiences. Parasuraman, Zeithaml and Berry⁹ ‎described in their SERVQUAL model ten (mainly subjective) determinants that may influence how customers experience services and – if not fulfilled – might result in a gap between experienced and expected service quality and related customer satisfaction.¹⁰ In most of the cases such subjective measures are assessed based on surveys, with the challenge to avoid low participation rates and bad data quality, which makes an objective assessment of service quality difficult.

In this real case an existing service provider (supplier 1) has been replaced by another service provider (supplier 2), with the goal to optimize costs and to bundle IT services. In general, the transformation worked fine except in one area. Therefore, management decided to perform a detailed analysis, to better identify the influencing factors and to define proper counter measures. A comparison of the two service providers in this area has shown differences in terms of:

• Service delivery model: capacity service vs. managed service (partially).
• Language of services changed to pure English.
• Flexibility: ability to short term adjust scope, resources and delivery times.
• Service delivery location: on-site, which makes tracking of communication efforts more difficult vs. nearshore, which has direct impact on the collaboration and communication between the service provider and the customer.
• Required skills and process know how: a temporary factor considering that over a certain period of time the new supplier will have sufficient company specific end-to-end business process expertise. This is depending on a stable resource situation to avoid continuous efforts for knowledge transfer.
• Resources being able to cover multiple required skills: This as a temporary factor, as resources will reach a sufficient level of knowledge after a certain period. This is also depending on a stable resource situation to avoid continuous efforts for knowledge transfer. Therefore, the volume of the scope has to be big enough to allow the service provider building up sufficient backup resources.
• Employee turnover
• Costs
• Company size

Based on these comparison major questions are:

• Is it possible to identify criteria affecting the IT service quality?
• Is it possible to identify a combination of criteria causing difficulties in terms of qualitative IT service delivery?
• Is it possible to build an (automatic) SLA configuration tool creating best-fit SLAs and KPIs depending on specific supplier criteria?
• Is it possible to include related information into an automated decision support system that will make use of domain expertise and historical data of suppliers to reduce supplier evaluation costs? ¹¹
• Is it possible to optimize service transition costs and time?
• How to interpret data considering those differences?

Answering to all those questions requires further research in a larger scale to proof significance of certain criteria.

Employees spend a lot of time answering emails, participating meetings and phone calls during the IT service delivery process. A proper documentation of the business processes as well as the incident resolution process in terms of number and content are important quality factors and ensure independency from certain suppliers. The hypothesis is that if many communication items (e.g. emails, communication items within an incident management tool, etc.) for a specific scope of work exist over a longer period, indicates missing knowledge and skills. This leads to bad quality, additional effort as well as dissatisfaction on the customer’s side. Due to legal requirements and compliance procedures, the customer has to evaluate and – in case of successful service delivery – provide the necessary approvals, which requires qualified internal resources.

Figure 1: Communication between customer and service provider

The main driver for a number of interchanges are both how well initial requests (t1) are formulated and how good the knowledge of the service provider is (t2), so especially the transition period is crucial. This exchange of communication items (ping-pong between supplier and customer) allows calculating the average number of communication items per scope of work (SoW). Below described methods, enable a direct comparison of different service providers in terms of communication efforts being flexible enough to be enhanced with further data like phone calls, meetings, etc.

The advantage of such an analysis is that existing data can be used. Usually service management and incident management tools are used to track and document related tasks.

The data of this example is based on records of an incident management tool between 2010 and 2016 for a specific area to ensure comparability of SoW. Year 2014 has not been considered as this was the year of the service transition from supplier 1 to supplier 2. Over the years 712 incidents in average were handled by external suppliers for this specific SoW – see Figure 2.

Figure 2: Incidents per year handled by external suppliers for specific SoW

Furthermore, it was possible to extract the yearly incident related communication items exchanged with externals suppliers out of the incident management tool. As supplier 1 was providing its services on-site, tracking of communication items becomes more difficult as verbal communication is usually not recorded in any tool, which needs to be considered when interpreting the results. Figure 3 shows an increase of communication items of supplier 2 also after the transition period. How the graph of supplier 2 will develop (further decrease, stabilize or increase) is difficult to predict, as at the moment of the analysis too less data for the year 2017 existed. This shows that the transition period to reach the same level like before took longer than expected and provides room for optimizations.

Figure 3: Yearly incident related communication items exchanged with externals suppliers

Calculating the average communication items per SoW based on incidents by using the formula on page 3 results in the yearly average supplier communication items per incident. Comparing the average of 5.1 communication items per incident of supplier 1 with 12.4 communication items in average of supplier 2, shows a difference of 7,3 communication items per incident in average – see Figure 4. As mentioned above the move from a capacity service to partially managed service explains a limited part of the increase.

Figure 4: Yearly average supplier communication items per incident

This means each incident handled by supplier 2 requires ~7,3 communication items more in average compared to supplier 1, or ~2,4 times more communication effort per incident compared to supplier 1 – see Figure 5.

Figure 5: Average supplier communication items per incident (data out of incident management tool)

Assuming a time effort of 5 minutes per communication item results in ~36 minutes (7.3 x 5) more communication effort for every incident handled by supplier 2 compared to supplier 1. For instance using the yearly average of 712 incidents from Figure 2 will result in ~435 hours more communication effort ((712 x 36) / 60) when using supplier 2. Multiplying this effort with an hourly rate will raise the question if the service price of supplier 2 is still at optimum? As there was a change from a capacity based service model to a partially managed service model the interpretation of the results has to be done carefully, as not recorded communication items – due to direct verbal communication – might influence the result. The question of further research will also be how much will be the difference when just changing a supplier without also changing the operating model (e.g. compare managed service with another managed service supplier for the same scope)? The resulting opportunity costs could be used for other value adding activities (e.g. research and innovation).

This approach also allows a direct objective comparison of resources. Resources with high average communication items per incident over a longer period of time for the same SoW are most probably not qualified enough, as this is an indication of missing know-how. Figure 6 shows a resource comparison of supplier 2.

Figure 6: Yearly average communication items per incident per resource (sample resources of supplier 2)

Another data source can be sent and received emails. Of course emails often contain personal and sensitive information which could be filtered based on machine learning and big data algorithms to ensure that only relevant data are analyzed. Another possibility could be to setup algorithms on the central email infrastructure collecting the required data e.g. sent to a certain domain or certain user groups. Both approaches have direct relation to big data analytics with the focus to measure vendor performance.

When using emails for comparison it is important to have a harmonized baseline for the workload. E.g. if using own emails also a basis for the own workload should be used (e.g. only own incidents). Figure 7 shows that supplier 2 requires in average 710 emails more compared to supplier 1 for the same workload. The email communication effort of supplier 2 is in average ~2.7 times higher compared to supplier 1. Comparing this with Figure 5 (~2.4 times more communication effort) shows a similar result.

Figure 7: Average yearly supplier email communication (data out of email conversations)

The approach can be enhanced with further parameters to get an even more detailed picture and enables to drill down to specific cases. To optimize the results machine learning algorithms can be used. The following example shows the communication efforts for a detailed case including phone calls and calendar (meeting) data:

Data:	Case 1	Unit of measure
Communication items – number of mails (#Mails):	25	quantity
Estimated time effort per communication item:	5	min
Number of phone calls (#PhoneCalls):	6	quantity
Estimated average time to prepare for a phone call:	15	min
Average duration of phone calls:	30	min
Number of meetings (#Meetings):	3	quantity
Estimated average time to prepare for a meeting (without time for traveling and slide preparation):	30	min
Average duration of meetings:	60	min
Communication effort:	665	min
Communication effort:	11	h

Table 1: Calculation of communication effort of detailed case incl. mails, phone calls and meetings

The advantage of this approach is that no additional data needs to be maintained or surveys do be filled out, as most of the data already exist (created during the service delivery process).

The objective assessment of qualified resources, required skills and offered service quality is difficult to be assessed. A methodology is researched which has the target to give a structured approach to evaluate service quality and efficiency based on available metadata. A comparison between two suppliers delivering IT services has shown differences in terms of delivery model, language location, resources, knowledge, costs and skills. Especially when changing supplier and operating model in one-step caused difficulties related to service quality and additional communication efforts. By analyzing metadata of communication items, which created during the IT service delivery process, it was possible to identify significant differences in the number of communication items. Based on this analysis management was able to setup the necessary counter measures. When interpreting the results it is important to consider the mentioned supplier differences. In case of an on-site service delivery model verbal communication is usually not recorded in a tool and therefore causes deviations within the results. This method fits perfectly for a supplier comparison using the same operating model. Nevertheless, the hypothesis  if many communication items exist over a longer period of time for a specific scope of work that is an indicator that used resources do not have the required know-how, skillsets and cause additional efforts  has been confirmed. This approach allows a direct comparison of resources. Resources with high average communication items over a longer period for the same SoW are most probably not qualified enough. While considering the different operating models and supplier differences these additional efforts have to be considered as part of the supplier evaluation process, as those efforts could be used for other value adding activities (e.g. research and innovation).

The future digitalization and automation in context of internet of things (IoT) and industry 4.0 requires an increased availability of data, information, IT systems, hardware as well as optimized processes, considering service quality and security aspects.¹² Further research activities in a larger scale are required to prove significance of objective measureable criteria influencing the IT service quality. One approach could be big data analysis by using machine-learning algorithms to identify the corresponding parameters. Critical factors, which are influencing the service transition efforts, have to be identified. In addition, the question – how does bad IT service quality and low qualified resources influence the risk of cyber attacks – has to be assessed. This requires the development of a new scalable and flexible evaluation model, to identify criteria and critical combinations, which make a qualitative delivery of IT services difficult or even impossible. Results from (automatic) testing procedures (e.g. software testing, security vulnerability checks, etc.) have to be considered as additional input. Combining all related information such as domain expertise and historical data of suppliers into an automated decision support system will reduce supplier evaluation costs and enables the automatic generation of best-fit SLAs and KPIs. Depending on the volume and scope, an optimized service delivery model can be proposed. Finally, it is possible to establish an IoT platform for objective IT service quality evaluation by using big data analytics and machine learning algorithms to enable global supplier comparison. Such a platform could offer services to analyze companies systems, processes and data. This will open the possibility to create individual quality assessments of certain processes and systems or even predict the future effort. In addition, the impact of factors like the European Union General Data Protection Regulation (GDPR)¹³ as well as tax constraints have to be considered.

Abdolshah, A Review of Quality Criteria Supporting Supplier Selection, Journal of Quality and Reliability Engineering, Article ID 621073, 2013.

Aubert/Rivarda/Patrya, A transaction cost model of IT outsourcing, 2003.

Bharadwaj, A Resource-Based Perspective on Information Technology Capability and Firm Performance: An Empirical Investigation, MIS Quarterly, Vol. 24, No. 1, pp. 169–196, 2000.

Brody/Pureswaran, The next digital gold rush: how the internet of things will create liquid, transparent markets, Strategy & Leadership, Vol. 43 Issue: 1, pp. 36–41, 2015.

Cater-Steel/Lepmets, Measuring IT Service Quality: Evaluation of IT Service Quality Measurement Framework in Industry, Journal of Service Science Research, 2014.

Cleven/Mettler/Rohner/Winter, Healthcare quality innovation and performance through process orientation: Evidence from general hospitals in Switzerland, Institute of Information Management, University of St. Gallen, Switzerland, 2016.

Daugherty/Banerjee/Negm/Alter, Driving Unconventional Growth through the Industrial Internet of Things, Accenture technology, 2014.

Dickson, An Analysis of Vendor Selection Systems and Decisions, Journal of Purchasing, Vol. 2, No. 1, pp. 5-20, 1966.

Gallego, Review of existing methods, models and tools for supplier evaluation, Dissertation Linköpings University, Madrid, 2011.

Hartmann/Halecker, Management of Innovation in the Industrial Internet of Things, ISPIM Conference Proceedings, Manchester: The International Society for Professional Innovation Management, 2015.

Harvard Business Review, Internet of Things: Science Fiction or Business Fact?, 2014.

Kagermann/Wahlster/Helbig, Securing the future of German manufacturing industry: Recommendations for implementing the strategic initiative INDUSTRIE 4.0: Final report of the Industrie 4.0 Working Group, 2013.

Kohlbacher/Gruenwald, Process orientation: conceptualization and measurement, Business Process Management Journal Vol. 17 No. 2, pp. 267–283, 2011.

Kohlbacher/Reijers, The effects of process-oriented organizational design on firm performance, Business Pro-cess Management Journal Vol. 19 No. 2, pp. 245–262, 2013.

Lee/Bagheri/Kao, Recent Advances and Trends of Cyber-Physical Systems and Big Data Analytics in Industrial Informatics, IEEE Int. Conference on Industrial Informatics, 2014.

Lee D./Lee T./Lee S./Jeong/Eom, BestChoice: A Decision Support System for Supplier Selection in e-Marketplaces, Data Engineering Issues in E-Commerce and Services, Lecture Notes in Computer Science, vol. 4055, Springer, Berlin, Heidelberg, 2006.

Lockamy/McCormack, The development of a supply chain management process maturity model using the concepts of business process orientation, Supply Chain Management: An International Journal Volume 9 Number 4, pp. 272–278, 2004.

Mattern/Friedemann/Floerkemeier, From the Internet of Computers to the Internet of Things, ETH Zurich, 2016.

Mukherjee/Krishnendu, Supplier selection criteria and methods: past, present and future, International Journal of Operational Research, 2014.

Parasuraman/Zeithaml/Berry, A Conceptual Model of Service Quality and Its Implications for Future Research, Journal of Marketing, vol. 49, pp. 41–50, 1985.

Siponena/Baskervilleb/Kuivalainena, Integrating Security into Agile Development Methods, Proceedings of the 38th Hawaii International Conference on System Sciences, 2005.

Skrinjar/Bosilj-Vukssic/Indihar-Stemberger, The impact of business process orientation on financial and non-financial performance, Business Process Management, Journal Vol. 14 No. 5, pp. 738–754, 2008.

Sureshchandar/Rajendran/Anantharaman, The relationship between service quality and customer satisfaction – a factor specific approach, Journal of Services Marketing, vol. 16, pp. 363–379, 2002.

Tang/Pee/Iijima, The Effects Of Business Process Orientation On Innovation, 2012.

Turber/Vom Brocke/Gassmann/Fleisch, Designing Business Models in the Era of Internet of Things: Towards a Reference Framework, Book Advancing the Impact of Design Science: Moving from Theory to Practice, 2014.

Vermesan/Friess, Internet of Things: Converging Technologies for Smart Environments and Integrated Ecosystems, Aalborg, River Publishers, 2013.

Vital/Aubert, A Resource-Based Analysis of IT Sourcing, The DATA BASE for Advances in Information Systems, Vol. 33, No. 2, 2002.

Zope/Anand/Lokku, Reviewing Service Quality for IT Services Offerings: Observations in the light of Service Quality Models & Determinants, 2014.