Does sacking a coach really help? Evidence from a Difference-in-Differences approach

i Does sacking a coach really help? Evidence from a Difference-in-Differences approach Gustavo de Souza Machado Fabrício Dissertation presented as partial requirement for obtaining the master’s degree in Advanced Analytics iii NOVA Information Management School Instituto Superior de Estatística e Gestão de Informação Universidade Nova de Lisboa Does sacking a coach really help? Evidence from a Difference-in-Differences approach By Gustavo de Souza Machado Fabrício This project is presented as partial requirement for obtaining the master’s degree in Data Science and Advanced Analytics Supervisor: Bruno Damásio November 2021 iv Acknowledgements I would like to express my gratitude to my supervisor, professor Bruno Damásio who gave me all the support and attention during the research and analysis, providing materials, his time to discuss about the methods, his patience, his vision that helped me to make this thesis a reality. I am extremely grateful to my parents Mr. Jose Carlos Fabrício and Mrs. Solange Fabrício for their trust in me, for the time and effort invested in my education, without them none of this would be possible. Also, a special thank you to my life partner Danielle Newton for her patience and support during my master’s. Finally, my thanks to my master’s friends that made this journey more interesting than it already was, with special mention to Victor Gonzalez my partner in most of the projects during the master’s. Thanks for your support, and partnership. To all of you, without whom I would not be who I am and what I have achieved, Thank You. v Abstract This project looks to evaluate if football clubs should or should not change their coach in order to improve their performance in the national league. For this analysis I selected, three of the most important European football leagues, La Liga (Spain), Serie A (Italy) and Premier League (England). The data used in this project was taken from the transfermarkt website, a large football platform. The data period is from season 2005-06 to season 2019-20 and has information about individual games results and squad value by player. The steps before the analysis were a data cleaning and consolidation of the information, creation of new features as a performance measure and selection of cases of interest for this analysis based on club and coach profile. Numeric variables were standardized to be on the same scale and make different seasons comparable. A K-means was applied to identify clubs according to their investments which has a proportional correlation with performance. Finally, a difference in differences analysis was applied to evaluate if a club would obtain a performance gain if they decided to sack their coach between game twelve and twenty-six of the season after a poor performance in consideration to squad price. As a general conclusion, it is possible to consider that on average the clubs in the treatment group and comparison group recover their performance after a period of underperforming, but the recovery of the clubs that sack their coach is lower compared with the clubs that keep them. Keywords Difference in Differences; Statistics; Football; Data Scraping; Clustering vi Index 1. Introduction ............................................................................................................. 1 1.1 Thesis Objective .............................................................................................. 2 2. Literature Review .................................................................................................... 2 3. Theoretical background ........................................................................................... 4 3.1 K-Means ........................................................................................................... 4 3.2 Backward Stepwise .......................................................................................... 5 3.3 Differences in Differences ............................................................................... 5 3.3.1 Notation and modelling ............................................................................ 6 3.3.2 Unbiased estimator assumptions .............................................................. 7 4. Methodology ........................................................................................................... 8 4.1 Data Scraping ................................................................................................... 8 4.2 Data preparation ............................................................................................... 9 4.3 Clubs’ segmentation ...................................................................................... 12 4.4 Filtering cases of interest and data analysis ................................................... 15 4.5 Difference in differences model..................................................................... 17 5. Result Presentation ................................................................................................ 18 5.1 Serie A ........................................................................................................ 18 5.2 La Liga ....................................................................................................... 19 5.3 Premier League .......................................................................................... 20 5.4 Results Overview ........................................................................................... 21 6. Conclusion and future work .................................................................................. 22 7. Limitations ............................................................................................................ 23 8. References ............................................................................................................. 24 9. Appendix ............................................................................................................... 26 vii List of Figures Figure 1 - Difference in diferences estimation ............................................................................ 6 Figure 2 - Scrapping and data preparation .................................................................................. 9 Figure 3Average number of clubs by First coach quantity of games. .................................... 10 Figure 4 - Percentage of sacked coaches in the final data set by league and cluster. ................ 16 Figure 5 - Boxplot – Performance distribution for Série A by time and Cluster ...................... 16 Figure 6 - Boxplot – Performance distribution for La Liga by time and Cluster ...................... 17 Figure 7 - Boxplot – Performance distribution for Premier League by time and Cluster. ........ 17 viii List of Tables Table 1Variables for K-means segmentation. ......................................................................... 13 Table 2 - K-means segmentation for Série A ............................................................................ 13 Table 3 - K-means segmentation for La Liga............................................................................ 13 Table 4 - K-means segmentation for Premier League ............................................................... 13 Table 5 - Performance of treated group in 𝑡0 by League and Cluster. ..................................... 14 Table 6 - Quantity of observations by league, cluster and treated ............................................ 15 Table 7DiD Regression Results for Serie A. .......................................................................... 18 Table 8 - DiD Regression Results for La Liga .......................................................................... 19 Table 9 - DiD Regression Results for Premier League ............................................................. 20 Table 10 - Match data set example ............................................................................................ 26 Table 11 - Players data set example .......................................................................................... 26 Table 12 - Coach consolidation data set .................................................................................... 26 ix List of abbreviations and acronyms DiD Difference in Differences Performance Total points won divided by total possible points Comparison Group Group of observations without intervention / treatment Treatment Group Group of observations that suffered intervention Warning Performance A performance percentage that clubs usually starts to sack their coach. 1 1. Introduction In recent decades, in collective sports, the importance of the coach has been highlighted. The coach is the person who oversees training and team formation, preparing it for a good performance to obtain positive results. The coach is an expert in technical and tactical direction and in the player’s psychological and physical development. In football this is no different, part of a successful season for a football club involves hiring the ideal coach to manage the team, forming the best squad, and frequently indicating players to hire. In football history we have a few examples of coaches that stayed for decades at the same club, such as Guy Roux, who stayed for forty-four years (1961 - 2005) in charge of Auxerre-FRA, leading the club from the third division to the title of French League in 1995. In England, the most successful manager in the football history, Alex Ferguson, led Manchester United for twenty-seven years (1986 - 2013) winning a total of fortynine titles. But the history has more cases of clubs sacking their coaches during the season than examples of longevity. For example, in the Italian first division during the last five seasons (2015-16 – 2019-20) an average of twelve clubs per season kept their coaches for the duration of the season and in the same period in eleven occasions clubs there were three or more coaches during the same season. When a club decides to sack its coach before the end of their contract it causes some inconveniences. One of them is a termination fine, that depends on the contract between both parties but when we talk about the richest leagues in the world this usually involves a large amount of money. As an example, when José Mourinho was sacked from Tottenham in April 2021, he had a contract until the end of the season 2022-23, and as a consequence of this breach of contract Tottenham had to pay a severance package of around fifteen million pounds to Mourinho according to the website football.london [13]. This project will analyse data from the top three leagues in Europe, La Liga – Spain, Serie A – Italy and Premier League - England between seasons 2005-06 and 2019-20. All leagues have almost the same structure. The entire season runs from August through to May and the league consists of twenty clubs playing against each other both at home 8 4. Methodology This chapter focuses on explaining in depth the whole analysis process to answer the proposed questions in this thesis, starting from data scraping, data preparation, club’s segmentation, filter cases of interest, descriptive analysis and DiD model application. 4.1 Data Scraping The data used in this analysis was scraped from the web site transfermarkt.co.uk/ Transfermarkt offers the world's largest football database with all information on players, clubs, and competitions as well as one of the largest football communities and playing areas for anyone who wants to share and exchange ideas about football. An overview of player market values in addition to news, statistics, consultant information and fan insights. The platform was founded in May 2000 by Matthias Seidel. The scraping process was made with R using the library rvest, who efficiently scraped the data from seasons 2005-06 to 2019-2020 of the English, Spanish and Italian first division leagues. For each league there was a two-scrap process, one for the game results and one for player’s information’s. The game results data consists of information such as date of the match, time, home team and away team, their position on the league table, coach and formation and attendance and result. Each line represents one game, an example of this table is available in the appendix at table 10. This information is fundamental to identify when a club sacked a coach during the season and calculate the performance. The player’s information consists of the players name, club, age, position, market value, best foot, and height. Each line represents one player during a specific season. This information will be helpful to identify club performance expectations based on the squad value and to create independent variables to explain performance. An example of this table is available in the appendix at table 11. The diagram bellow represents the process of scrapping the data from transfermarkt using R, data cleaning and transformation to build the data tables for each league in this thesis. 9 Figure 2 - Scrapping and data preparation Step 1: Using the library rvest, access the transfermarkt web site in a specific URL to access the information of interest from two main URL structures, one for matches information that was composed by four variables that changes according with club, season, and league. The second was to extract the players information and was composed by tree variables as season and two related to the club. Step 2: Store that information into lists, where each list represents one specific information e.g. the score, name of the club playing home, name of the club playing away. Step 3: Using the library tidyr clean variable out of the format and transforming lists into data tables, as mentioned before, one for matches and other for players. Step 4: Export those data tables to txt file that going to be used on Python for the analysis 4.2 Data preparation This step is essential to the project, has all consolidations, validations and criteria definitions were made to be used in the analysis. The idea is to aggregate the information by Season - Club – Manager to calculate performance, evaluate the cases of interest and define the three variables to apply DiD regression, Treated, Time and Treated x Time. As described before, in the table each line represents a game of a specific club, that aggregation was made to create a table with club, season, coach, number of games, date of first game and last game of the coach. As an example, the table 12 in the appendix, 10 shows that Alaves had three coaches during the season 2005-2006. That means each line represents a work period of a coach in a club during a specific season. From the number of games in the aggregate table, it is identified which coach was sacked or not during the season and for how long they managed the club. The criteria are, if the coach had a work period of thirty-four games or more, it is considered that he stayed for the whole season, between twenty-seven and thirty-three games, the coach stayed for more than half of the season, between twelve and twenty-six games he stayed for half of the season and less than twelve games he stayed for less than half of the season. According to this information, each coach's work period will be divided into two different groups, the ones whose work period lasted at least thirty-four games, these observations will represent the comparison group, which means observations without treatment during the season, and the work period that was interrupted during the season or started in some point after the start of the season, will be representing the treated group, observations that suffered an intervention during the season. Figure 3Average number of clubs by First coach quantity of games. Note: Figure 3 represents the average number of clubs over the fifteen seasons that kept their coach by four different periods of time and league. As we can see in figure 2, during a season in Serie A, only eleven clubs on average will maintain their coach until the end of the season, and seven clubs on average will sack 11 their couch halfway through the season or before. Between the three leagues, Italy has the highest number of changes. Evaluating the group with treatment, the ideal homogeneous behaviour for all clubs with treatment would be, club A during season one had coach X for the first nineteen games and then coach Y for the last nineteen games, where coach X represents 𝑡 and coach Y represents 𝑡. But in general, this does not happen, the data base has several cases of clubs with more than two coaches during the same season by consequence coaches with less than ten games in 𝑡 which will be considered a low number of games to evaluate performance. To make a fair evaluation, this thesis will only consider cases where the coach that started the season (𝑡) had at least between twelve and twenty-six games, for the cases that the club had two or more coaches after the first coach, it will be considered one single work period, in other words, it will evaluate the performance of the coach that started the season and the performance of the club after the coach was sacked. The comparison group also need to be split in 𝑡 and 𝑡, in order to compare performance in two different moments, in this thesis for the coaches who stayed the entire season this break will be the game with lowest cumulative performance between fifth and twentythird game. With these criteria the season will be divided in the most critical moment that is a similar condition to what happens with the treated group. The coach performance is calculated dividing the total points won by total possible points, that means if the coach won twenty-one points in fourteen games, his total possible points will be fourteen times three equals to forty-two, so his performance is twenty-one divided by forty-two equals to fifty percent. This metric will be the outcome of the model. The other data set that will be used in this project is related to squad, each line has information about one player in a specific season and club like age, estimated value and position. This data set will be consolidated by season and club, with information of total, average and highest value by player’s position, those new features will be used as independent variables to explain performance and create cluster of clubs/ season. Analysing the evolution of the estimated market value of the player season over season we notice that this value has been rising over time, comparing the average estimated value of a player in season 2005-2006 against season 2019-2020 we see Serie A with a jump from 2.36M to 6.44M, with more than 170% growth, La Liga from 3.02M to 12 7.72M with 153% growth and Premiere League from 3.7M to 10.73M with 189% growth. In this scenario the price variables by position would not be effective to predict performance once performance is a fixed range between zero and one and the price have been growing season over season, the correlation between price and performance would depend on the year. A solution for this issue was, once the data was consolidated, standardise the values for each season individually, in this way it is possible to have the same scale over the seasons and express the difference between clubs inside the season. The standardise technique was the standard score or Z - Score scale, this technique is better explained in the Theoretical Background. With this transformation the correlation between performance and total value of the squad standardized is 0.69 for Serie A, 0.74 for Premier League and 0.78 for La Liga, as expected a more valuable squad tends to have better results. 4.3 Clubs’ segmentation As was commented before, a more valuable squad tends to have better results, and consequently it implies greater pressure for coaches to have better results. Nobody expects that SPAL with an estimated squad value of ninety-seven million and four hundred thousand euros would have the same performance of Juventus with a squad value nine times higher in season 2018-19. In other words, if Massimiliano Allegri (Juventus coach) had the same performance of Leonardo Semplici (SPAL coach) of 37% probably he would be sacked during the season, whereas SPAL kept Semplici until the end of the season. This segmentation is for the purpose of dividing clubs into clusters according to their squad value and after, identifying the critical percentage of performance of the clubsseason by cluster. The four most correlated variables with performance will be used to perform a K-means segmentation. The K-means was applied for three hundred observations (twenty clubs per season, times fifteen seasons.) The Elbow graph was used to determine the number of clusters, minimizing the variance inside the groups and maximizing the variance between group in order to have the most efficient number of cluster. 13 Table 1Variables for K-means segmentation. The segmentation generated the following results: Table 2 - K-means segmentation for Série A As showed in the previous table, in Série A Cluster 1 has clubs with low investments e.g. Brescia, Pescara and Treviso. Cluster 3 has clubs with medium investments e.g. Udinese, Fiorentina, and Torino. Clusters 0 and 2 Clubs with high investments e.g Juventus, AC Milan, and Inter. Those two clusters have almost the same clubs in different seasons, for that reason Cluster 2 will be incorporated to Cluster 0. That happens because in some seasons those clubs had a large difference of investments compare to the others. Table 3 - K-means segmentation for La Liga. Cluster 0 has clubs with low investment e.g. Las Palmas, Levante, Maiorca and Numância. Cluster 2 has clubs with medium investment e.g. Sevilha, Valencia and Villarreal. Cluster 1 has clubs with high investment and is composed just with Barcelona, Real Madrid, and Atletico Madrid in some seasons. Table 4 - K-means segmentation for Premier League League Serie A Totat Value Mean Value Mean Value Defence Total Value Attack LA Liga Mean Value Totat Value Mean Value Attack Max Value Defence Premier League Totat Value Mean Value Total Value Defence Mean Value Attack K-Means Variables - Standardized Cluster Totat Value Mean Value Total Value Defence Mean Value Attack N 03,05 2,85 2,63 2,83 43 10,38 0,39 0,36 0,42 156 24,40 4,42 4,40 4,26 35 31,42 1,30 1,22 1,37 66 Cluster Totat Value Mean Value Max Value Defence Mean Value Attack N 0 0,29 0,32 0,43 0,26 202 1 4,71 4,65 4,63 4,45 32 2 1,56 1,58 1,80 1,43 66 Cluster Totat Value Mean Value Total Value Defence Mean Value Attack N 01,24 1,33 1,17 1,17 64 14,38 4,24 4,43 4,25 44 20,41 0,42 0,46 0,41 152 33,03 2,69 2,71 2,73 40 14 The Premier League is the league with more intersection of clubs between clusters, as an example Manchester City appears in all clusters, the reason is the growth of financial potential with investments over the years. Cluster 2, clubs with low investments e.g. Birmingham, Sheffield United and West Brom. Cluster 0, clubs with medium investments e.g. West Ham, Southampton and Wolverhampton; and Cluster 1 and 3 with clubs with high investments e.g. Manchester United, Liverpool and Chelsea. Those two clusters were grouped together due to the similarity of the clubs. The next step is to merge the cluster information with the treated dataset, in doing that, it is possible to evaluate the warning performance percentage for each cluster by analysing the performance of the observations in 𝑡. Once the warning performance for each cluster is known these values will be used to select the observations to be used in the DiD regression. For those coaches whose performance is equal to or below the warning performance in 𝑡 means that they were with their position in risk, so it is reasonable to compare them with those who, in the same level of investments were sacked. Table 5 - Performance of treated group in 𝑡 by League and Cluster. As was expected, the clubs with high investments, had a higher standard for performance. On table 5 it is possible to notice that La Liga has a higher average performance for the treated observations when compared to the other leagues, which explains the difference of investments between the clubs. Just out of curiosity, between the seasons 2005-06 and 2019-20 only five different clubs finished the league in the top three, and the league had only three different champions. Max P 90 Mean Min High 60% 57% 50% 39% 9 Medium 46% 44% 38% 21% 16 Low 37% 33% 25% 13% 44 High 70% 70% 65% 60% 4 Medium 57% 51% 42% 26% 11 Low 38% 33% 28% 14% 38 High 67% 66% 50% 31% 8 Medium 56% 43% 33% 21% 10 Low 41% 35% 28% 11% 32 Serie A Premier League La Liga Performance ClusterLeague Qtt 15 4.4 Filtering cases of interest and data analysis Once the treated club/season cases are already prepared and the performance by segment is known, the next step is filtering the non-treated observations in 𝑡 with a similar performance by cluster in the treated group in 𝑡 in order to have a fair comparison. For this filter the percentile 90% will be used. That means if a non-treated club A in season X of the Premier League that belongs to cluster medium investment with a performance equal or lower to 43% in 𝑡, this observation will be selected to the analysis. In the same example above, it would not be fair to compare a non-treated observation that have 70% performance in 𝑡, because in normal conditions this performance level is outside of the performance range that clubs use to sack their coach. The final database for the analysis is composed by non-treated cases, comparison group, that performance in 𝑡 by cluster is lower than percentile 90% for the same cluster in the treated cases plus the treated cases, treatment group, that the manager at 𝑡 had between twelve and twenty-six games. Table 6 - Quantity of observations by league, cluster and treated The table above shows that in the cluster high the number of observations in nontreated is higher than treated, indicating that clubs with high investments and consequently high performance, tend to keep their coach when the performance hits the warning percentage. On the other side, clubs with low investments are more likely to change their coach. A hypothesis for that behaviour can be related to the relegation, clubs with low investments and performance bellow the expectations tends to fight against relegation and sack a coach expecting a better performance is one of the most common effects. The plot below represents the percentage of sacked coaches with performance below the warning percentage of the cluster. Serie A La Liga Premier League High Yes 9 4 8 High No 17 10 38 Medium Yes 16 11 10 Medium No 5 16 24 Low Yes 44 42 33 Low No 26 28 49 117 111 162 Qtt Observations in Total Cluster Treated 𝑡  16 Figure 4 - Percentage of sacked coaches in the final data set by league and cluster. Note: Figure 4 represents the percentage of sacked coaches in the final dataset by League and cluster. Comparing leagues Serie A are more likely to sack their coach in relation to the other leagues in every cluster. Premier League has the lowest percentage with 17% for the clubs in the cluster High value. Also, can be notice that the clubs with lower investments are more likely to sack their coaches. A possible explanation is clubs with higher investments tends to hit their performance goals easier in comparison with clubs with lower investments. The Boxplot below shows the distribution of performance by treated/non-treated case and time divided by cluster. Figure 5 - Boxplot – Performance distribution for Série A by time and Cluster 17 Figure 6 - Boxplot – Performance distribution for La Liga by time and Cluster Figure 7 - Boxplot – Performance distribution for Premier League by time and Cluster. Note: Figures 5, 6 and 7 represents the distribution of performance of club/season by treated and comparison group in 𝑡 and 𝑡 by cluster. By the distribution above, it is easy to notice that time has a big influence for both cases treated and non-treated. When a club has a bad start to the season according to their expectations and capabilities, they tend to recover after a certain point. For every league and cluster, the median is higher in 𝑡, comparing with 𝑡. DiD will help to understand if there is performance gain by changing the coach. 4.5 Difference in differences model After preparing the data in the previous steps a difference in differences model was applied for each league and the feature selection process was Backward Stepwise 24 8. References [1] D. Bloyce et al.(December 2008), Playing the Game (Plan): A Figurational Analysis of Organizational Change in Sports Development in EnglandEuropean Sport Management Quarterly, Vol. 8, No. 4, 359 - 378 [2] By RT, Diefenbach T, Klarner P (2008). Getting Organizational Change Right in Public Services: The Case of European Higher Education. J Change Manag. 2008;8(1):2 1-35. [3] Besters, Lucas M., Jan C. van Ours, and Martin A. van Tuijl. (2016), Effectiveness of in‐season manager changes in English Premier League football. De Economist 164: 335–56. [4] González‐Gómez, F., Picazo‐Tadeo, A.J. and García‐Rubio, M.Á. (2011), The impact of a mid‐season change of manager on sporting performance, Sport, Business and Management, Vol. 1 No. 1, pp. 28-42. [5] Martínez, J. A. (2012), Entrenador nuevo, ¿victoria segura? Evidencia en baloncesto /Chaning a coach, guarantee the win? Revista Internacional de Medicina y Ciencias de la Actividad Física y el Deporte 12 (48), 663-679. [6] David Card & Alan B. Krueger (1994), Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania; The American Economic Review, Volume 84, Issue 4 (Sep. 1994), 772-793 [7] Callaway, B., & Sant’Anna, P. H. C. (2020), Difference-in-differences with multiple time periods. Journal of Econometrics. [8] Book. Advances in K-means Clustering - A Data Mining Thinking - Hardback – 2012. [9] Desboulets Loann (November 2018), A Review on Variable Selection in Regression Analysis Econometrics 6(4):45 [10] Book. Impact Evaluation in Practice (Second Edition) Paul J. Gertler, Sebastian Martinez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch - 2012 [11] Anders Fredriksson, Gustavo de Oliveira (October 2019), Impact evaluation using Difference-in-Differences RAUSP Management Journal 54(4):519-532 25 [12] Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics [13] https://www.football.london/tottenham-hotspur-fc/news/cost-of-sacking-josemourinho-20416106 [14] https://www.transfermarkt.co.uk/ 26 9. Appendix Table 10 - Match data set example Table 11 - Player’s data set example Table 12 - Coach consolidation data set Date HomeTeam HomeTeamPos AwayTeam AwayTeamPos Formation Manager Attendence Result TeamMatches Season 01/10/2000 AC Milan 6.0 Vicenza 16.0 3-4-3 Alberto Zaccheroni 46.836 02:00 AC Milan 2000 15/10/2000 Bolonha 17.0 AC Milan 2.0 3-4-3 Alberto Zaccheroni 34.631 02:01 AC Milan 2000 21/10/2000 AC Milan 8.0 Juventus 2.0 3-4-1-2 Alberto Zaccheroni 81.954 02:02 AC Milan 2000 01/11/2000 AC Parma 15.0 AC Milan 8.0 3-4-3 Alberto Zaccheroni 21.572 02:00 AC Milan 2000 05/11/2000 AC Milan 12.0 Atalanta 2.0 3-4-1-2 Alberto Zaccheroni 54.641 03:03 AC Milan 2000 PlayerNames Position Number Birth Height Foot Joined Value Season Team Diego CavalieriDiego Cavalieri GuardaRedes 28 01/12/1982 (27) 1.89 esquerdo 01/08/2010 600 mil 2010 Cesana Alex TeodoraniA. Teodorani GuardaRedes 91 21/09/1991 (18) 1.91 direito 01/07/2009 200 mil 2010 Cesana Francesco AntonioliF. Antonioli GuardaRedes 1 14/09/1969 (40) 1.87 direito 07/07/2009 100 mil 2010 Cesana Alex CalderoniA. Calderoni GuardaRedes 33 31/05/1976 (34) 1.82 esquerdo 07/01/2011 100 mil 2010 Cesana Aldo SimonciniA. Simoncini GuardaRedes 86 30/08/1986 (23) 1.84 esquerdo 01/01/2011 100 mil 2010 Cesana TeamMatches Season Manager Match_qtt FirstMatch LastMatch Alavés 2005 Chuchi Cos 18 27/08/2005 08/01/2006 Alavés 2005 Juan Carlos Oliva 5 15/01/2006 12/02/2006 Alavés 2005 Mario Luna 15 18/02/2006 13/05/2006 Alavés 2016 Mauricio Pellegrino 38 21/08/2016 20/05/2017 Alavés 2017 Luis Zubeldía 4 18/08/2017 17/09/2017 Page | i