A New Hybrid CNN-LSTM for Wind Power Forecasting in Ethiopia

A New Hyb id CNN-LSTM o Wind Powe Fo ecas ing in E hiopia E. Te e a1,M.Ma ´ınez-Balles e os2(B), A. T oncoso3, and F. Ma ´ınez-´ Al a ez3 1Depa men o So wa e Enginee ing, Addis Ababa Science and Technology Uni e si y, Addis Ababa, E hiopia [email p o ec ed] 2Depa men o Compu e Science, Uni e si y o Se ille, 41012 Se ille, Spain [email p o ec ed] 3Da a Science and Big Da a Lab, Pablo de Ola ide Uni e si y, 41013 Se ille, Spain {a olo , ma al }@upo.es Abs ac . Renewable ene gies a e cu en ly expe iencing p omising g ow h as an al e na i e solu ion o minimize he emission o pollu an gases om he use o ossil uels, which con ibu e o global wa ming. To in eg a e hese enewable ene gies sa ely wi h he g id sys em and make he elec ic g id sys em mo e s able, i is i ally impo an o accu a ely o ecas he amoun o wind powe gene a ed a specific wind powe gene a ion si es and he iming o his gene a ion. Deep lea ning app oaches ha e shown good o ecas ing pe o mance o complex and nonlinea p oblems, such as ime se ies wind powe da a. Howe e , u - he s udy is needed o op imize deep lea ning models by in eg a ing mul iple models wi h hype pa ame e op imiza ion, o a ain op imal pe o mance om hese indi idual models. In his pape , we p opose a hyb id CNN-LSTM model o wind powe o ecas ing in E hiopia. Bayesian op imiza ion is applied o une he hype pa ame e s o he indi- idual lea ne s, including 1D-CNN and LSTM models, be o e building he hyb id CNN-LSTM model. The p oposed model is es ed on h ee case s udy wind powe da ase s ob ained om he E hiopian Elec ic Powe Co po a ion. Acco ding o he MAE, RMSE, and MAPE e alua- ion me ics, he hyb id model pe o ms significan ly be e han bench- ma k models, including ANN, RNN, BiLSTM, CNN, and LSTM models, o all case s udy da a. Keywo ds: hyb id models · ime se ies · enewable ene gies · o ecas ing 1 In oduc ion Renewable ene gy has shown p omising g ow h in ecen yea s due o i s sus ain- abili y, en i onmen ally iendly na u e, and abundan a ailabili y as a sou ce o elec ic ene gy [22]. Among a ious ypes o enewable ene gy, wind powe has h ps://doi.o g/10.1007/978-3-031-40725-3_18 208 E. Te e a e al. demons a ed ema kable g ow h as one o he mos effec i e s a egies o com- ba clima e change and mee g eenhouse gas emission a ge s in many coun ies. Go e nmen s and esea che s s ongly encou age he p oduc ion and consump- ion o wind ene gy [22]. Accu a ely quan i ying he amoun o enewable ene gy p oduc ion, pa icula ly wind ene gy gene a ion, is c ucial o he sa e in eg a- ion o enewable ene gy in o he g id sys em and o enhance efficien powe g id ope a ion [3]. Howe e , wind powe gene a ion is inhe en ly andom, non- linea , non-s a iona y, and highly in e mi en , making i s in eg a ion wi h he g id sys em challenging. Despi e significan enewable ene gy po en ial in E hiopia, including hyd o- elec ic, wind, and geo he mal ene gy, cu en ene gy p oduc ion is limi ed, and he ene gy supply alls sho o ising ene gy consump ion demands [24]. Fu he - mo e, he absence o elec ic load es ima ion and modeling me hods con ibu es o ene gy fluc ua ions and powe in e up ions ha affec elec ic ene gy ans- mission and dis ibu ion sys ems [15]. As a esul , ene gy ou ages and powe in e up ions affec all cus ome ca ego ies, inc easing de ensi e expendi u es due o un eliable and uns able ene gy supply. The e o e, accu a e p edic ion o wind powe gene a ion can play a key ole in imp o ing he eliabili y and s abil- i y o he powe sys em [7] and enable sa e in eg a ion o p oduced wind powe in o he g id sys em [4]. Recen ly, deep lea ning has shown ema kable pe o mance in a ious appli- ca ions, including enewable ene gy o ecas ing, due o i s abili y o handle nonlinea , non-s a iona y, spa io empo al da a gene a ed om ene gy sys ems. Recu en Neu al Ne wo ks (RNN), Con olu ional Neu al Ne wo ks (CNN), Deep Belie Ne wo ks (DBN), and Mul ilaye Pe cep on (MLP) a e among he well-known and widely used deep lea ning algo i hms, wi h Long Sho -Te m Memo y (LSTM) s anding ou in he con ex o ime se ies o ecas ing [6]. To imp o e he accu acy o ene gy o ecas ing in he enewable ene gy sec o , a ious a ificial in elligence echniques ha e been applied. An example o his is he Bayesian op imiza ion-based a ificial neu al ne wo k model de eloped in [19]. Addi ionally, he p ecision o sho - e m o ecas ing has been enhanced by u ilizing hyb id deep lea ning models ha in eg a e a ious neu al ne wo k a chi ec u es. Se e al s udies ha e shown ha hese hyb id models ou pe o m single models in a ious applica ions [11,19], and [20], and ha e been success ul in accu a ely p edic ing wind speed and e apo anspi a ion [7,27]. This pape aims o e alua e he effec i eness o a hyb id model o 1D-CNN and LSTM o o ecas ing wind powe ime se ies da a in E hiopia o he fi s ime. The model le e ages CNN’s ea u e ex ac ion capabili y om nonlinea wind powe ime se ies da a and he po en ial o LSTM in lea ning high empo al ime se ies da a. The pape ’s con ibu ions can be summa ized as ollows: 1. The op imal hype pa ame e s we e de e mined using Bayesian op imiza ion algo i hm o ob ain he op imal pe o mance o he p oposed model. 2. A CNN-LSTM hyb id model was de eloped o day-ahead wind powe o e- cas ing by using he effec i e ea u e ex ac ion capabili ies o 1D-CNN and o ecas gene aliza ion o LSTM models A New Hyb id CNN-LSTM o Wind Powe Fo ecas ing in E hiopia 209 3. The effec i eness o a hyb id o CNN-LSTM model agains base line models such as 1D-CNN, LSTM, ANN and BiLSTM is e ified o wind powe o e- cas ing using he me ics o mean absolu e e o (MAE), oo mean squa e e o (RMSE), and mean absolu e pe cen age e o (MAPE). 2 Rela ed Wo ks Indus ies and ins i u ions can gene a e a conside able olume o da a on hei day- o-day ope a ions by in oducing senso de ices [18]. The ene gy sec o , including enewable ene gy, is among he ew ha p oduces da a on a imely basis ega ding cus ome ene gy consump ion, such as minu e, hou ly, and daily usage. Fu he mo e, wind a ms’ Supe iso y Con ol and Da a Acquisi ion (SCADA) sys ems collec da a ela ed o wind powe a specified ime in e als. This ype o da a is known as ime-se ies da a, ep esen ing a se ies o pe iodic measu emen s o a a iable. Specifically, ime se ies da a gene a ed om ene gy sys ems is nonlinea and non-s a iona y, exhibi ing no only empo al co ela ion bu also spa ial pa e ns [8]. The ene gy sec o , pa icula ly in he field o enewable ene gy o ecas ing, has made significan ad ancemen s wi h he u iliza ion o A ificial In elligence (AI) me hods such as machine lea ning and deep lea ning echniques [1,21]. These algo i hms ha e been ex ensi ely used o o ecas a ious wea he pa am- e e s. Fu he mo e, a combina ion o diffe en AI echniques has eme ged as a p e e ed app oach in ecen imes o de elop models ha pe o m be e han indi idual models. Se e al s udies ha e shown ha hyb id deep lea ning models ou pe o m indi idual o single models [10]. To p o e his, Goh e al. [5] in es iga ed a hyb id o a con olu ional neu al ne wo k (1D-CNN) and a long sho memo y ne wo k (LSTM). They ob ained an imp o emen o 16.73% o single-s ep p edic ion and 20.33% o 24-s ep load p edic ion. Addi ionally, [11] implemen ed a hyb id 1D-CNN and BiLSTM model o enhance wind speed p edic ion accu acy and add ess unce ain y modeling issues. Resul s indica ed ha he p oposed hyb id app oach achie ed a 42% imp o emen o e e e ence app oaches. Ano he s udy by [9] p oposed he combina ion o Ensemble Empi ical Mode Decomposi ion (EEMD) and BiDLSTM sys em o accu a e wind speed o ecas ing. Fu he mo e, Wang e al. [25] p oposed a 3-hou ahead a e age wind powe p edic ion me hod based on a con olu ional neu al ne wo k. The au ho s in [26] in oduced a deep lea ning app oach based on a pooling long sho - e m mem- o y (LSTM) based con olu ional neu al ne wo k o p edic sho - and medium- e m elec ic consump ion. Resul s e ealed ha he p oposed me hod imp o ed sho - and medium- e m load o ecas ing pe o mance. Au ho s in [20] de el- oped he CNN-LSTM-Ligh GBM-based sho - e m wind powe p edic ion model by conside ing a ious en i onmen al ac o s. Mo eo e , a hyb id deep lea ning model o accu a ely o ecas he e y sho - e m (5-min and 10-min) wind powe gene a ion o he Boco Rock wind a m in Aus alia was p oposed by Hossain e al. [7]. Howe e , he au ho s used he Ha is Hawks Op imiza ion algo i hm o imp o e he p oposed model. 210 E. Te e a e al. Ano he hyb id model was in oduced by J. Yin e al. [27] o o ecas sho - e m (1–7-day lead ime) e apo anspi a ion (ET0). The au ho s used a hyb id Bi-LSTM ha combines BiLSTM and ANN using h ee me eo ological da a (maximum empe a u e, minimum empe a u e, and sunshine du a ion). The bes o ecas pe o mance o sho - e m daily ET0 was ound. Lu e al. [14] p oposed a hyb id model based on a con olu ional neu al ne - wo k and long sho - e m memo y ne wo k (CNN-LSTM) o sho - e m load o ecas ing (STLF). The au ho s no ed ha o ecas ing accu acy can be no ably imp o ed. T. Li e al. [13] in oduced a hyb id CNN-LSTM model by in eg a - ing he con olu ion neu al ne wo k (CNN) wi h he long sho - e m memo y (LSTM) neu al ne wo k o o ecas he nex 24-hou PM2.5 concen a ion in Beijing, China. Resul s indica ed ha he p oposed mul i a ia e CNN-LSTM model achie ed he bes esul s due o low e o and sho e aining ime. 3 Me hods The sec ion p esen s he main pa s o he me hodology ca ied ou in he p o- posed hyb id deep lea ning model, he CNN-LSTM model, o wind powe o e- cas ing. The main ea u es o he me hodology a e p esen ed in Fig. 1. In pa icula , a desc ip ion o he selec ed deep lea ning algo i hms, CNN and LSTM, used o build he hyb id model, da a p ep ocessing, hype pa ame e uning, model aining, and e alua ion a e desc ibed in his sec ion. The s udy uses h ee wind powe da ase s gene a ed om diffe en wind a m si es as case s udy da a. Each da ase is di ided in o aining and es se s, while main aining he o de o he ime se ies da a. The fi s h ee yea s o da a a e used o build and fi he model, and he final one-yea da a is used o assess he model’s pe o mance. Hype pa ame e uning is pe o med on he aining da a using a k- old c oss- alida ion app oach. 3.1 Deep Lea ning Models Deep lea ning algo i hms ha e eme ged as one o he mos widely used app oaches in a ificial in elligence in he las yea s. One o he key ad an- ages o deep lea ning is i s abili y o au oma ically lea n ea u es and ex ac mul i-le el abs ac ep esen a ions om complex da a se s, se ing i apa om o he machine lea ning models. In pa icula , deep lea ning models such as hose discussed in Sec . 3.1 ha e demons a ed be e pe o mance in handling la ge and complex da a, including image p ocessing, pa e n ex ac ion, classifica ion, and ime se ies o ecas ing [12]. RNNs a e a ype o deep lea ning me hod ha is pa icula ly effec i e in han- dling la ge da ase s con aining empo al dependencies. RNNs can lea n sequen- ial da a by ecu si ely applying ope a ions du ing he o wa d pass and using backp opaga ion h ough ime o lea ning. As such, RNNs ha e been s udied o many eal-wo ld applica ions ha gene a e sequen ial and ime se ies da a, including speech syn hesis, na u al language p ocessing, and image cap ioning A New Hyb id CNN-LSTM o Wind Powe Fo ecas ing in E hiopia 211 [17]. Howe e , a challenge o RNNs is long- e m dependency, which leads o he anishing g adien p oblem as he gap be ween ele an in o ma ion and he poin whe e i is needed g ows [17]. To add ess he limi a ions o RNNs, LSTM and Ga ed Recu en Uni (GRU) echniques we e in oduced. These me hods can e ain in o ma ion o e long pe iods, allowing o p ocessing complex and sequen ial da a. LSTM is especially no able o i s abili y o handle ime se ies o ecas ing. The main deep lea ning algo i hms in ol ed in he hyb id model CNN-LSTM, CNN and LSTM espec i ely, a e b iefly desc ibed in he ollowing subsec ions. Fig. 1. Gene al scheme o he me hodology including CNN-LSTM hyb id model. CNN. This ype o deep lea ning algo i hm mimics humans’ isual pe cep ion p ocessing sys ems. They ha e become he mos widely used and ex ensi ely s udied deep lea ning me hod o asks such as compu e ision, image segmen- a ion, classifica ion, and na u al language p ocessing, demons a ing ema k- able pe o mance [5]. Addi ionally, CNNs ha e ecen ly gained a en ion om esea che s as a solu ion o ime se ies o ecas ing p oblems such as wind powe and sola adia ion [8]. LSTM. This ype o deep lea ning algo i hm was de eloped o sol e he an- ishing g adien p oblem in RNN by in oducing an efficien memo y cell ha can handle long- e m dependencies [2]. The memo y cells in LSTM ne wo ks can e ain he p e ious in o ma ion o he nex lea ning s ep. In addi ion o he cell uni , LSTM includes h ee ga e s uc u es: he inpu ga e, o ge ga e, and ou pu ga e [16]. The main unc ion o hese ga es in LSTM laye s is o con ol he flow o da a in o and ou o he cell s a e. The o ge ga e, which consis s o 212 E. Te e a e al. sigmoid ac i a ion nodes, de e mines which p e ious s a es should be e ained and which ones should be disca ded [23]. This pape p oposes a hyb id model ha combines a one-dimensional con- olu ional neu al ne wo k (1D-CNN) and LSTM, as shown in Fig. 2. The 1D- CNN is capable o ex ac ing meaning ul ea u es om wind powe ime se ies da a, and he LSTM ne wo k can le e age long- e m dependencies among he ex ac ed ea u es o p oduce imp o ed p edic ion esul s. Fig. 2. Hyb id o 1D-CNN-LSTM a chi ec u e. 3.2 Da a P ep ocessing Fo his s udy, he da ase was ob ained wi h he pe mission o he E hiopian Elec ic Powe Co po a ion. The sou ce da ase was collec ed om h ee g oups o wind powe gene a ion plan s managed by he co po a ion. Fo all g oups o wind powe da a, he ime esolu ion is he same anging om 9 Feb ua y 2019 o 25 July 2022, e e ed by Da ase 1, Da ase 2, and Da ase 3. The wind powe plan s om which da a is gene a ed is loca ed jus ou side Adama Town in O omia Regional S a e o E hiopia, which is 95 km sou heas o Addis Ababa, he capi al ci y o E hiopia. Fo u he in o ma ion, he da ase used o es ab- lish he findings o his pape is a ailable a h ps://gi hub.com/Da aLabUPO/ WindPowe HAIS23. To imp o e he pe o mance o he p oposed model, we pe o med da a p e- p ocessing, which included handling missing alues and emo ing duplica es. The da ase mus be ans o med in o a o able ange alues o deep lea ning model aining o effec i ely lea n he inpu da a. In his case, he da ase is scaled in o (0,1), which will imp o e compu a ion and model con e gence speed. The min-max no maliza ion me hod was used o ans o m he da a in o he ange (0,1), as exp essed by Eq. 1. A New Hyb id CNN-LSTM o Wind Powe Fo ecas ing in E hiopia 213 n=X0−Xmin Xmax −Xmin (1) whe e n ep esen s he no malized alues o X, while X0 ep esen s he cu en alue o he a iable X.Xmin and Xmax e e o he minimum and maximum da a poin s o he a iable Xin he inpu da ase . 3.3 Pe o mance E alua ion Diffe en e alua ion echniques, such as MAE (Eq. 2), RMSE (Eq. 3), MSE (Eq. 4), and MAPE (Eq. 5), ha e been used o de e mine he p edic ion pe - o mance o ained models. MAE =1 n n  i=1 |y−ˆy|(2) RMSE =    1 n n  i=1 y−ˆy2(3) MSE =1 n n  i=1 y−ˆy2(4) MAPE =1 n n  i=1 |y−ˆy| y∗100 (5) whe e yand ˆy ep esen he ac ual and p edic ed alues, espec i ely. In addi ion, n ep esen s he o al numbe o obse a ions used o ain he model. 3.4 Analysis and Resul Discussion In o de o de elop he hyb id CNN-LSTM model o wind powe o ecas ing, i is c ucial o de e mine he op imal hype pa ame e s. In his s udy, we define he ange and ype o hype pa ame e s o each deep model including 1D-CNN and LSTM models wi hin hei espec i e hype pa ame e spaces. The bes com- bina ion o op imal hype pa ame e s was de e mined using he Bayesian op i- miza ion algo i hm and some o he op imal alues ound o each model a e no he same despi e we define he same hype pa ame e space and ypes such as lea ning a e, ac i a ion unc ion, e c. Fo example, hype pa ame e s space, pa ame e ype, and op imal alues sea ched o 1D-CNN and LSTM models a e shown in Table 1. A e sea ching o he op imal pa ame e s o each deep lea ning model, 1D- CNN and LSTM models we e defined in in e co ela ed sequence laye s. In he fi s lea ning phase, he ex ac ion o ime se ies ea u es was achie ed using he 1D-CNN con olu ion laye . Lea ning he empo al co ela ion o he ime se ies 214 E. Te e a e al. Table 1. Hype pa ame e s used o he p oposed hyb id CNN-LSTM model. Model Pa ame e name Types/Range alues Op imal alue selec ed 1D-CNN Numbe o il e s [32,64,128,256] 64 Ke nel size [2,3,4,5] 3 D opou a e [0.1,0.2,0.4,0.5] 0.1 Ac i a ion unc ion [ elu, anh, Linea ] elu pool ype [MaxP ooling1D, A e ageP ooling1D] MaxPooling1D nneu ons a dense laye [20,30,40,50] 40 Epoch [32,64,128,260] 64 Ba ch size [20,30,40,50] 40 Lea ning a e [0.0001,0.001,0.01,0.1] 0.001 Op imize [RMSP op, Adam, Adadal a] Adam LSTM Ac i a ion unc ion [ elu, anh, Linea ] anh D opou a e [0.1,0.2,0.4,0.5] 0.1 nneu ons a hidden laye [20,30,40,50] 40 Op imize [RMSP op, Adam, Adadal a]RMSP op Epoch [32,64,128,260] 64 Ba ch size [20,30,40,50] 40 Lea ning a e [0.0001,0.001,0.01,0.1] 0.01 Table 2. Fo ecas ing pe o mance o deep lea ning models using MAE, RMSE, and MAPE me ics on aining and es da a. Da ase Model T aining Tes ing MAE RMSE MAPE (%) MAE RMSE MAPE (%) Da ase 1 ANN 0.4791 0.5453 4.531 0.5462 0.6167 2.6550 RNN 0.0679 0.0914 1.3477 0.0818 0.1031 1.1363 1D-CNN 0.0596 0.0774 1.3740 0.0679 0.0853 1.1737 LSTM 0.0586 0.0792 1.2552 0.0671 0.0845 1.2552 BiLSTM 0.0589 0.0791 1.1188 0.0655 0.0835 1.0726 CNN-LSTM 0.0507 0.0685 1.1262 0.0634 0.0809 1.1409 Da ase 2 ANN 0.1899 0.2361 1.9730 0.1998 0.2396 1.7481 RNN 0.0967 0.1216 1.5542 0.1051 0.1247 1.8984 1D-CNN 0.0663 0.0900 1.0412 0.0761 0.0970 1.1553 LSTM 0.0681 0.0922 1.0427 0.0690 0.0883 1.1358 BiLSTM 0.0683 0.0925 1.1188 0.0693 0.0889 1.1302 CNN-LSTM 0.0572 0.0771 1.0850 0.0678 0.0882 1.0482 Da ase 3 ANN 0.2639 0.3390 2.3053 0.1349 0.1708 1.6200 RNN 0.1161 0.1526 1.4745 0.1135 0.1416 1.4356 1D-CNN 0.1026 0.1336 1.3590 0.1048 0.1327 1.1482 LSTM 0.1098 0.1434 1.4502 0.1057 0.1311 1.2454 BiLSTM 0.1106 0.1406 1.2073 0.1006 0.1284 1.1405 CNN-LSTM 0.0874 0.1133 1.2107 0.0991 0.1264 1.1003 A New Hyb id CNN-LSTM o Wind Powe Fo ecas ing in E hiopia 215 da a was pe o med in he second lea ning phase. Finally, he ully connec ed laye p oduced he p edic ed ou pu , as defined in he las laye . Table 2p esen s he esul s o a day ahead wind powe p edic ion using he hyb id deep lea ning model ha combines 1D-CNN and LSTM models o he h ee wind powe da ase s. Using op imal hype pa ame e configu a ions, he p oposed CNN-LSTM model was compa ed agains ou indi idual deep lea ning models, including simple RNN, LSTM, 1D-CNN, and BiLSTM. Based on he e alua ion esul s o MAE, RMSE, and MAPE p esen ed in Table2,i canbe obse ed ha he shallow ANN exhibi ed he wo s pe o mance wi h he highes MAE and RMSE e o alues o 0.4791 and 0.5451, espec i ely, on he aining da a o all h ee cases. Simila ly, he ANN pe o med poo ly on he es da a wi h he la ges MAE and RMSE e o alues o 0.5461 and 0.6167, espec i ely, ollowed by he in e io pe o mance o he RNN models on bo h he aining and es da a compa ed o he es o he deep lea ning models (LSTM, BiLSTM, 1D-CNN) and CNN-LSTM, as summa ized in Table 2. Mo e impo an ly, based on he MAE and RMSE e alua ion me ics, he CNN-LSTM model exhibi s he lowes e o on bo h he aining and es da a o all case s udy da ase s used in his pape . F om his, we can conclude ha he combina ion o CNN- LSTM enhanced by he hype pa ame e uning app oach ou pe o ms he single op imized deep lea ning models and achie es excellen pe o mance o he non- linea and highly in e mi en wind powe o ecas ing p oblems. Addi ionally, Fig. 3displays he a e age MSE e o alues o all models on he h ee wind powe da ase s analyzed in his s udy. The esul s indica e ha he ANN model had significan ly la ge e o alues o he MSE me ic com- pa ed o he o he deep lea ning models. The esul s demons a e he capabili y o deep lea ning models in lea ning he nonlinea and complex wind powe da a as compa ed o he shallow ANN a chi ec u e. On he o he hand, he hyb idiz- ing o deep lea ning LSTM and 1D-CNN models wi h he use o an au oma ic hype pa ame e op imiza ion app oach yields he lowes MSE e o and exhibi s imp o ed o ecas ing pe o mance. Fu he mo e, he CNN-LSTM model exhib- i ed he bes pe o mance on he es da a, wi h he smalles MSE e o , as depic ed in Fig. 4, while he shallow ANN was he poo es model, ollowed by he RNN model, o all h ee wind powe da ase s analyzed. Fig. 3. MSE e o on he aining se o diffe en models on h ee da ase s.