jMe alPy: a Py hon F amewo k o Mul i-Objec i e Op imiza ion wi h Me aheu is ics
An onio Ben´
ı ez-Hidalgoa, An onio J. Neb oa, Jos´
e Ga c´
ıa-Nie oa, Izaskun O egib, Ja ie Del Se b,c,d
aDepa amen o de Lenguajes y Ciencias de la Compu aci´on, Ada By on Resea ch Building, Uni e si y o M´alaga, 29071 M´alaga, Spain
bTECNALIA, 48160 De io, Spain
cUni e si y o he Basque Coun y (UPV/EHU), 48013 Bilbao, Spain
dBasque Cen e o Applied Ma hema ics (BCAM), 48009 Bilbao, Spain
Abs ac
This pape desc ibes jMe alPy, an objec -o ien ed Py hon-based amewo k o mul i-objec i e op imiza ion wi h me aheu is ic
echniques. Building upon ou expe iences wi h he well-known jMe al amewo k, we ha e de eloped a new mul i-objec i e
op imiza ion so wa e pla o m aiming no only a eplica ing he o me one in a di e en p og amming language, bu also a aking
ad an age o he ull ea u e se o Py hon, including i s acili ies o as p o o yping and he la ge amoun o a ailable lib a ies o
da a p ocessing, da a analysis, da a isualiza ion, and high-pe o mance compu ing. As a esul , jMe alPy p o ides an en i onmen
o sol ing mul i-objec i e op imiza ion p oblems ocused no only on adi ional me aheu is ics, bu also on echniques suppo ing
p e e ence a icula ion and dynamic p oblems, along wi h a ich se o ea u es ela ed o he au oma ic gene a ion o s a is ical da a
om he esul s gene a ed, as well as he eal- ime and in e ac i e isualiza ion o he Pa e o on app oxima ions p oduced by
he algo i hms. jMe alPy o e s addi ionally suppo o pa allel compu ing in mul ico e and clus e sys ems. We include some use
cases o explo e he main ea u es o jMe alPy and o illus a e how o wo k wi h i .
Keywo ds: Mul i-Objec i e Op imiza ion, Me aheu is ics, So wa e F amewo k, Py hon, S a is ical Analysis, Visualiza ion
1. In oduc ion
Mul i-objec i e op imiza ion p oblems a e widely ound in
many disciplines [1, 2], including enginee ing, economics, lo-
gis ics, anspo a ion o ene gy, among o he s. They a e cha -
ac e ized by ha ing wo o mo e con lic ing objec i e unc ions
ha ha e o be maximized o minimized a he same ime, wi h
hei op imum composed by a se o ade-o solu ions known
as Pa e o op imal se . Besides ha ing se e al objec i es, o he
ac o s can make his amily o op imiza ion p oblems pa icu-
la ly di icul o ackle and sol e wi h exac echniques, such as
decep i eness, epis asis, NP-ha d complexi y, o high dimen-
sionali y [3]. As a consequence, he mos popula echniques o
deal wi h complex mul i-objec i e op imiza ion p oblems a e
me aheu is ics [4], a amily o non-exac algo i hms including
e olu iona y algo i hms and swa m in elligence me hods (e.g.
an colony op imiza ion o pa icle swa m op imiza ion).
An impo an ac o ha has igni ed he widesp ead adop-
ion o me aheu is ics is he a ailabili y o so wa e ools eas-
ing hei implemen a ion, execu ion and deploymen in p ac i-
cal se ups. In he con ex o mul i-objec i e op imiza ion, one
o he mos acknowledged amewo ks is jMe al [5], a p ojec
s a ed in 2006 ha has been con inuously e ol ing since hen,
including a ull edesign om sc a ch in 2015 [6]. jMe al is im-
plemen ed in Ja a unde he MIT licence, and i s sou ce code is
Email add esses: [email p o ec ed] (An onio Ben´
ı ez-Hidalgo),
[email p o ec ed] (An onio J. Neb o), [email p o ec ed] (Jos´
e
Ga c´
ıa-Nie o), [email p o ec ed] (Izaskun O egi),
[email p o ec ed] (Ja ie Del Se )
publicly a ailable in Gi Hub1.
In his pape , we p esen jMe alPy, a new mul i-objec i e
op imiza ion amewo k w i en in Py hon. Ou mo i a ion o
de eloping jMe alPy s ems om ou pas expe ience wi h jMe al
and om he ac ha nowadays Py hon has become a e y
p ominen p og amming language wi h a ple ho a o in e es -
ing ea u es, which enables as p o o yping ueled by i s la ge
ecosys em o lib a ies o nume ical and scien i ic compu ing
(NumPy [7], Scipy [8]), da a analysis (Pandas), machine lea n-
ing (Sciki -lea n [9]), isualiza ion (Ma plo lib [10], Holo iews
[11], Plo ly [12]), la ge-scale p ocessing (Dask [13], PySpa k
[14]) and so o h. Ou goal is no only o ew i e jMe al in
Py hon, bu o ocus mainly on aspec s whe e Py hon can help
ill he gaps no co e ed by Ja a. In pa icula , we place ou
in e es in he analysis o esul s p o ided by he op imiza ion
algo i hms, eal- ime and in e ac i e isualiza ion, p e e ence
a icula ion o suppo ing decision making, and sol ing dy-
namic p oblems. Fu he mo e, since Py hon can be hough o
as a mo e agile p og amming en i onmen o p o o yping new
mul i-objec i e sol e s, jMe alPy also inco po a es a ull sui e
o s a is ical signi icance es s and ela ed ools o he sake o
a p incipled compa ison among mul i-objec i e me aheu is ics.
jMe alPy has been de eloped by Compu e Science engi-
nee s and scien is s o suppo esea ch in mul i-objec i e op-
imiza ion wi h me aheu is ics, and o u ilize he p o ided al-
go i hms o sol ing eal-wo d p oblems. Following he same
1jMe al: h ps://gi hub.com/jMe al/jMe al. As o Ap il 18, 2019,
he pape s abou jMe al had accumula ed mo e han 1280 ci a ions (sou ce:
Google Schola )
P ep in submi ed o jou nal (unde e iew) Ap il 18, 2019
open sou ce philosophy as in jMe al, jMe alPy is eleased un-
de he MIT license. The p ojec is in con inuous de elopmen ,
wi h i s sou ce code hos ed in Gi Hub2, whe e he las s able
and cu en de elopmen e sions can be eely ob ained.
The main ea u es o jMe alPy a e summa ized as ollows:
•jMe alPy is implemen ed in Py hon ( e sion 3.6+), and i s
objec -o ien ed a chi ec u e makes i lexible and ex ensible.
•I p o ides a se o classical mul i-objec i e me aheu is ics
(NSGA-II [15], GDE3 [16], SMPSO [17], OMOPSO [18],
MOEA/D [19]) and s anda d amilies o p oblems o bench-
ma king (ZDT, DTLZ, WFG [2], and LZ09 [20]).
•Dynamic mul i-objec i e op imiza ion is suppo ed, includ-
ing he implemen a ion o dynamic e sions o NSGA-II and
SMPSO, as well as he FDA [21] p oblem amily.
•Re e ence poin based p e e ence a icula ion algo i hms, such
as SMPSO/RP [22] and e sions o NSGA-II and GDE3, a e
also p o ided.
•I implemen s quali y indica o s o mul i-objec i e op imiza-
ion, such as Hype olume [23], Addi i e Epsilon [24] and
In e ed Gene a ional Dis ance [25].
•I p o ides isualiza ion componen s o display he Pa e o
on app oxima ions when sol ing p oblems wi h wo ob-
jec i es (sca e plo ), h ee objec i es (sca e plo 3D), and
many-objec i e p oblems (pa allel coo dina es g aph and a
ailo ed e sion o Cho d diag ams).
•Suppo o compa a i e s udies, including a wide numbe o
s a is ical es s and u ili ies (e.g. non-pa ame ic es , pos -
hoc es s, boxplo s, CD plo ), including he au oma ic gen-
e a ion o L
A
T
EX ables (mean, s anda d de ia ion, median,
in e qua ile ange) and igu es in di e en o ma s.
•jMe alPy can coope a i ely wo k alongside wi h jMe al. The
la e can be used o un algo i hms and compu e he qual-
i y indica o s, while he pos -p ocessing da a analysis can be
ca ied ou wi h jMe alPy.
•Pa allel compu ing is suppo ed based on Apache Spa k [26]
and Dask [13]. This includes an e alua o componen ha
can be used by gene a ional me aheu is ics o e alua e so-
lu ions in pa allel wi h Spa k (synch onous pa allelism), as
well as a pa allel e sion o NSGA-II based on Dask (asyn-
ch onous pa allelism).
•Suppo ing documen a ion. A websi e3is main ained wi h
use manuals and API speci ica ion o de elope s. This si e
also con ains a se ies o Jupy e no ebooks4wi h use cases
and examples o expe imen s and isualiza ions.
2jMe alPy: h ps://gi hub.com/jMe al/jMe alPy
3jMe alPy documen a ion: h ps://jme alpy. ead hedocs.io
4Jupy e : h ps://jupy e .o g
Ou pu pose o his pape is o desc ibe jMe alPy, and o
illus a e how i can be used by membe s o he communi y in-
e es ed in expe imen ing wi h me aheu is ics o sol ing mul i-
objec i e op imiza ion p oblems. To his end, we include some
implemen a ion use cases based on NSGA-II o explo e he
main a ian s conside ed in jMe alPy, om s anda d e sions
(gene a ional and s eady s a e), o dynamic, e e ence-poin based,
pa allel and dis ibu ed la o s o his sol e . A expe imen al
use case is also desc ibed o exempli y how he s a is ical es s
and isualiza ion ools included in jMe alPy can be used o
pos -p ocessing and analyzing he ob ained esul s in dep h. Fo
backg ound concep s and o mal de ini ions o mul i-objec i e
op imiza ion, we e e o ou p e ious wo k in [5].
The emaining o his pape is o ganized as ollows. In Sec-
ion 2, a e iew o ele an ela ed algo i hmic so wa e pla -
o ms is conduc ed o gi e an insigh and a ionale o he main
di e ences and con ibu ion o jMe alPy. Sec ion 3 del es in o
he jMe alPy a chi ec u e and i s main componen s. Sec ion 4
explains a use case o implemen a ion. Visualiza ion acili ies
a e desc ibed in Sec ion 5, while a use case o expe imen a ion
wi h s a is ical p ocedu es is explained in Sec ion 6. Finally,
Sec ion 7 p esen s he conclusions and ou lines u he ela ed
wo k planned o he nea u u e.
2. Rela ed Wo ks
In he las wo decades, a numbe o so wa e amewo ks
de o ed o he implemen a ion o mul i-objec i e me aheu is-
ics has been con ibu ed o he communi y, such as ECJ [33],
E A [34], JCLEC-MO [35], jMe al [5, 6], MOEA F amewo k
[36], and Op 4J [37], which a e w i en in Ja a; Pa adisEO-
MOEO [38], and PISA [39], de eloped in C/C++; and Pla EMO
[40], implemen ed in Ma lab. They all ha e in common he in-
clusion o ep esen a i e algo i hms om he he s a e o he a ,
benchma k p oblems and quali y indica o s o pe o mance as-
sessmen .
As has been men ioned in he in oduc ion, he e is a g ow-
ing in e es wi hin he scien i ic communi y in so wa e ame-
wo ks implemen ed in Py hon, since his language o e s a la ge
ecosys em o lib a ies, mos o hem de o ed o da a analysis,
da a p ocessing and isualiza ion. When i comes o op imiza-
ion algo i hms, a se o ep esen a i e Py hon amewo ks is
lis ed in Table 1, whe e hey a e analyzed acco ding o hei
algo i hmic domains, main enance s a us, Py hon e sion and
licensing, as well as he ea u ed a ian s, pos -p ocessing a-
cili ies and algo i hms hey cu en ly o e . Wi h he excep ion
o he Inspy ed amewo k, hey a e all ac i e p ojec s (i.e., hei
public sou ce code ha e been upda ed a leas one ime wi hin
he las six mon hs) and wo k ou -o - he-box wi h a simple pip
command. All o hese amewo ks suppo Py hon 3.x.
DEAP and Inspy ed a e no cen e ed in mul i-objec i e op i-
miza ion, and hey include a sho e numbe o implemen ed al-
go i hms. Pagmo/PyGMO, Pla ypus and Pymoo o e a highe
numbe o ea u es and algo i hmic a ian s, including me hods
o s a is ical pos -p ocessing and isualiza ion o esul s. In
pa icula , Pagmo/PyGMO con ains implemen a ions o a num-
be o single/mul i-objec i e algo i hms, including hyb id a i-
2
Table 1: Mos popula op imiza ion amewo ks w i en in Py hon.
Name S a us Py hon License Pa allel Dynamic Decision Pos -p ocessing Algo i hms
e sion p ocessing op imiza ion making acili ies
DEAP 1.2.2 [27] Ac i e ≥2.7 LGPL-3.0 XS a is ics GA, GP, CMA-ES, NSGA-II, SPEA2, MO-CMA-ES
Gea py 1.1.5 [28] Ac i e ≥3.5 MIT GA, MOEA
Inspy ed 1.0.1 [29] Inac i e ≥2.6 MIT GA, ES, PSO, ACO, SA, PAES, NSGA-II
PyGMO 2.10 [30] Ac i e 3.x GPL-3.0 XVisualiza ion,
s a is ics
GA, DE, PSO, SA, ABC, IHS, MC,
CMA-ES, NSGA-II, MOEA/D
Pla ypus 1.0.3 [31] Ac i e 3.6 GPL-3.0 XVisualiza ion,
s a is ics
CMA-ES, NSGA-II, NSGA-III,
GDE3, IBEA, MOEA/D,
OMOPSO, EpsMOEA, SPEA2
Pymoo 0.2.4 [32] Ac i e 3.6 Apache 2.0 XVisualiza ion,
s a is ics
GA, DE, NSGA-II, NSGA-III,
U-NSGA-III, e e ence poin (R-NSGA-III)
jMe alPy 1.0.0 Ac i e ≥3.6 MIT X X X Visualiza ion,
s a is ics
GA, EA, NSGA-II, NSGA-III,
SMPSO, GDE3, OMOPSO, MOEA/D,
e e ence poin (G-NSGA-II, SMPSO/RP, G-GDE3),
dynamic (NSGA-II, SMPSO, GDE3)
an s, wi h s a is ical me hods o acing algo i hms, quali y in-
dica o s and i ness landscape analysis. Pla ypus suppo s pa -
allel p ocessing in solu ion e alua ion phase, whe eas Pymoo is
a he ocused on o e ing me hods o p e e ence a icula ion
based on e e ence poin s.
The jMe alPy amewo k we p oposed in his pape is also
an ac i e open sou ce p ojec , which is ocused mainly on mul i-
objec i e op imiza ion (al hough a numbe o single-objec i e
algo i hms a e included) p o iding an inc easing numbe o al-
go i hms and mode n me hods o s a is ical pos -p ocessing
and isualiza ion o esul s. I o e s algo i hmic a ian s wi h
me hods o pa allel p ocessing and p e e ence a icula ion based
on e e ence poin s o p o ide decision making suppo . Mo e-
o e , jMe alPy inco po a es algo i hms and mechanisms o dy-
namic p oblem op imiza ion, which is an addi ional ea u e no
p esen in he o he ela ed amewo ks. In his way, he p o-
posed amewo k a emp s a co e ing as many enhancing ea-
u es in op imiza ion as possible o suppo expe imen a ion and
decision making in bo h esea ch and indus y communi ies.
Besides hese ea u es, an impo an design goal in jMe alPy
has been o make he code easy o unde s and (in pa icula , he
implemen a ion o he algo i hms), o euse and o ex end, as is
illus a ed in he nex wo sec ions.
3. A chi ec u e o jMe alPy
The a chi ec u e o jMe alPy has an objec -o ien ed design
o make i lexible and ex ensible (see Figu e 1). The co e
classes de ine he basic unc ionali y o jMe alPy: an Algo i hm
sol es a P oblem by using some Ope a o en i ies which ma-
nipula e a se o Solu ion objec s. We de ail hese classes nex .
3.1. Co e A chi ec u e
Class Algo i hm con ains a lis o solu ions (i.e. popula ion
in E olu iona y Algo i hms o swa m in Swa m In elligence
echniques) and a un() me hod ha implemen s he beha io
o a gene ic me aheu is ic ( o he sake o simplici y, ull de-
ails o he codes a e omi ed):
1class Algo i hm(ABC):
de __ini __(sel ):
3sel .e alua ions = 0
sel .solu ions = Lis []
5sel .obse able = De aul Obse able()
7de un(sel ):
sel .solu ions = sel .c ea e_ini ial_solu ions()
9sel .solu ions = sel .e alua e(sel .solu ions)
sel .ini _p og ess()
11 while no sel .s opping_condi ion_is_me ():
sel .s ep()
13 sel .upda e_p og ess()
In he abo e code we no e he s eps o c ea ing he ini ial
se o solu ions, hei e alua ion, and he main loop o he algo-
i hm, which pe o ms a numbe o s eps un il a s opping condi-
ion is me . The ini ializa ion o s a e a iables o an algo i hm
and hei upda e a he end o each s ep a e ca ied ou in he
ini p og ess() and upda e p og ess() me hods, espec i ely. In
o de o allow he communica ion o he s a us o an algo i hm
while unning we ha e adop ed he obse e pa e n [41], so
ha any algo i hm is an obse able en i y which no i ies o eg-
is e ed obse e s some in o ma ion speci ied in ad ance (e.g.,
he cu en e alua ion numbe , unning ime, o he cu en so-
lu ion lis ), ypically in he upda e p og ess() me hod. In his
way we p o ide a s uc u ed me hod, o example, o display in
eal- ime he cu en Pa e o on app oxima ion o o s o e i in
a ile.
A p oblem is esponsible o c ea ing and e alua ing solu-
ions, and i is cha ac e ized by i s numbe o decision a i-
ables, objec i es and cons ain s. In case o he numbe o
cons ain s be g ea e han 0, i is assumed ha he e alua e()
me hod also assesses whe he he cons ain s a e ul illed. Sub-
classes o P oblem include addi ional in o ma ion depending o
he assumed solu ion encoding; hus, a Floa P oblem ( o nu-
me ical op imiza ion) o an In ege P oblem ( o combina o ial
op imiza ion) equi es he speci ica ion o he lowe and uppe
bounds o he decision a iables.
Ope a o s such as Mu a ion,C osso e , and Selec ion, ha e
an execu e(sou ce) me hod which, gi en a sou ce objec , p o-
duces a esul . Mu a ions ope a e on a solu ion and e u n a new
one esul ing om modi ying he o iginal one. On he con a y,
c osso e ope a o s ake a lis o solu ions (namely, he pa en s)
and p oduce ano he lis o solu ions (co espondingly, he o -
sp ing). Selec ion ope a o s usually ecei e a lis o solu ions
3
Manage
1
Sol e
1
C ea e/E alua e
*
Manipula e
*
Use
*
in e ace
Algo i hm
+ e alua ions: in = 0
+ solu ions: Lis [Solu ion] = lis ()
+ obse able = De aul Obse able()
+ini p og ess()
+s ep()
+upda e p og ess()
+ un()
+ ge esul ()
in e ace
Solu ion
+ numbe o objec i es: in
+ numbe o a iables: in
in e ace
P oblem
+ numbe o objec i es: in
+ numbe o a iables: in
+ numbe o cons ain s: in
+c ea e solu ion()
+e alua e(solu ion: Solu ion)
in e ace
Ope a o
+execu e(solu ion: Solu ion)
Figu e 1: UML class diag am o jMe alPy.
and e u ns one o hem o a sublis o hem.
The Solu ion class is a key componen in jMe alPy because
i is used o ep esen he a ailable solu ion encodings, which
a e linked o he p oblem ype and he ope a o s ha can be used
o sol e i . E e y solu ion is composed by a lis o a iables, a
lis o objec i e alues, and a se o a ibu es implemen ed as
a dic iona y o key- alue pai s. A ibu es can be used o as-
sign, o example, a ank o he solu ions o popula ion o a
cons ain iola ion deg ee. Depending on he ype o he a i-
ables, we ha e subclasses o Solu ion such as Floa Solu ion,
In ege Solu ion,Bina ySolu ion o Pe mu a ionSolu ion.
3.2. Classes o Dynamic Op imiza ion
jMe alPy suppo s dealing wi h dynamic op imiza ion p ob-
lems, i.e., p oblems ha change o e ime. Fo his pu pose, i
con ains wo abs ac classes named DynamicP oblem and Dy-
namicAlgo i hm.
A dynamic algo i hm is de ined as an algo i hm wi h a es a -
ing me hod, which is called whene e a change in he p oblem
being sol ed is de ec ed. The code o he DynamicAlgo i hm
class is as ollows:
1class DynamicAlgo i hm(Algo i hm, ABC):
3@abs ac me hod
de es a (sel ) -> None:
5pass
The DynamicP oblem class ex ends P oblem wi h me hods
o que y whe he he p oblem has changed wha soe e , and o
clea ha s a us:
1class DynamicP oblem(P oblem, Obse e , ABC):
3@abs ac me hod
de he_p oblem_has_changed(sel ) -> bool:
5pass
7@abs ac me hod
de clea _changed(sel ) -> None:
9pass
I is wo h men ioning ha a dynamic p oblem is also an ob-
se e en i y acco ding o he obse e pa e n. The unde lying
idea is ha in jMe alPy i is assumed ha changes in a dynamic
p oblem a e p oduced by ex e nal en i ies, i.e, obse able ob-
jec s whe e he p oblem is egis e ed.
4. Implemen a ion Use Case: NSGA-II and Va ian s
Wi h he aim o illus a ing he basic usages o jMe alPy, in
his sec ion we desc ibe he implemen a ion o he well-known
NSGA-II algo i hm [15], as well as some o i s a ian s (s eady-
s a e, dynamic, wi h p e e ence a icula ion, pa allel, and dis-
ibu ed).
NSGA-II is a gene ic algo i hm, which is a subclass o E o-
lu iona y Algo i hms. In jMe alPy we include an abs ac class
o he la e , and a de aul implemen a ion o he o me . An
E olu iona y Algo i hm is a me aheu is ic whe e he s ep() me hod
consis s o applying a sequence o selec ion, ep oduc ion, and
eplacemen me hods, as illus a ed in he code snippe below:
1class E olu iona yAlgo i hm(Algo i hm, ABC):
de __ini __(sel ,
3p oblem: P oblem,
popula ion_size: in ,
5o sp ing_size: in ):
supe (E olu iona yAlgo i hm, sel ).__ini __()
7sel .p oblem = p oblem
sel .popula ion_size = popula ion_size
9sel .o sp ing_size = o sp ing_size
11 @abs ac me hod
de selec ion(sel , popula ion):
13 pass
15 @abs ac me hod
de ep oduc ion(sel , popula ion):
17 pass
19 @abs ac me hod
de eplacemen (sel , popula ion, o sp ing):
21 pass
23 de ini _p og ess(sel ):
sel .e alua ions = sel .popula ion_size
25
de s ep(sel ):
27 ma ing_pool = sel .selec ion(sel .solu ions)
o sp ing = sel . ep oduc ion(ma ing_pool)
29 o sp ing = sel .e alua e(o sp ing)
sel .solu ions = sel . eplacemen (sel .solu ions, o sp ing)
31
de upda e_p og ess(sel ):
33 sel .e alua ions += sel .o sp ing_size
On e e y s ep, he selec ion ope a o is used (line 27) o e-
ie e he ma ing pool om he solu ion lis ( he popula ion) o
he algo i hm. Solu ions o he ma ing pool a e aken o e-
p oduc ion (line 28), which yields a new lis o solu ions called
o sp ing. Solu ions o his o sp ing popula ion mus be e alu-
a ed (line 29), and he ea e a eplacemen s a egy is applied o
upda e he popula ion (line 30). We can obse e ha he e al-
ua ion coun e is ini ialized and upda ed in he ini p og ess()
(line 23) and upda e p og ess (line 32), espec i ely.
4
The E olu iona yAlgo i hm class is e y gene ic. We p o-
ide a comple e implemen a ion o a Gene ic Algo i hm, which
is an e olu iona y algo i hm whe e he ep oduc ion is com-
posed by combining a c osso e and mu a ion ope a o . We
pa ially illus a e his implemen a ion nex :
1class Gene icAlgo i hm(E olu iona yAlgo i hm):
de __ini __(sel ,
3p oblem: P oblem[Solu ion],
popula ion_size: in ,
5o sp ing_popula ion_size: in ,
mu a ion: Mu a ion,
7c osso e : C osso e ,
selec ion: Selec ion,
9 e mina ion_c i e ion: Te mina ionC i e ion,
popula ion_gene a o =RandomGene a o (),
11 popula ion_e alua o =Sequen ialE alua o ()):
...
13
de c ea e_ini ial_solu ions(sel ):
15 e u n [sel .popula ion_gene a o .new(sel .p oblem)
o _in ange(sel .popula ion_size)]
17
de e alua e(sel , solu ions):
19 e u n sel .popula ion_e alua o .e alua e(solu ions, sel .p oblem)
21 de s opping_condi ion_is_me (sel ):
e u n sel . e mina ion_c i e ion.is_me
23
de selec ion(sel , popula ion: Lis [Solu ion]):
25 # selec solu ions o ge he ma ing pool
27 de ep oduc ion(sel , ma ing_pool):
# apply c osso e and mu a ion
29
de eplacemen (sel , popula ion, o sp ing):
31 # combine he popula ion and o sp ing popula ions
The e a e some in e es ing ea u es o poin ou he e. Fi s ,
he ini ial solu ion lis is c ea ed om a Gene a o objec (line
14), which, gi en a p oblem, e u ns a numbe o new solu ions
acco ding o some s a egy implemen ed in he gene a o ; by
de aul , a RandomGene a o () is chosen o p oduce a numbe
o solu ions uni o mly d awn a andom om he alue ange
speci ied o he decision a iables. Second, an E alua o ob-
jec is used o e alua e all p oduced solu ions (line 19); he de-
aul one e alua es he solu ions sequen ially. Thi d, a Te mi-
na ionC i e ion objec is used o check he s opping condi ion
(line 21), which allows deciding among se e al s opping c i-
e ia when con igu ed. The p o ided implemen a ions include:
s opping a e making a maximum numbe o e alua ions, com-
pu ing o a maximum ime, a key has been p essed, o he cu -
en popula ion achie es a minimum le el o quali y acco ding
o some indica o . Fou h, he ep oduc ion me hod applies he
c osso e and mu a ion ope a o s o e he ma ing pool o gen-
e a e he o sp ing popula ion. Finally, he eplacemen me hod
combines he popula ion and he o sp ing popula ion o p o-
duce a new popula ion.
Depa ing om he implemen ed Gene icAlgo i hm class,
we a e eady o implemen he s anda d NSGA-II algo i hm and
some a ian s, which will be desc ibed in he nex subsec ions.
Compu ing imes will be epo ed when unning he algo i hm
o sol e he ZDT1 benchma k p oblem [42] on a MacBook P o
wi h macOS Moja e, 2.2 GHz In el Co e i7 p ocesso (Tu bo
boos up o 3.4GHz), 16 GB 1600 MHz DDR3 RAM, Py hon
3.6.7 :: Anaconda.
4.1. S anda d Gene a ional NSGA-II
NSGA-II is a gene a ional gene ic algo i hm, so he popu-
la ion and he o sp ing popula ion ha e he same size. I s main
ea u e is he use o a non-domina ed so ing o anking he
solu ions in a popula ion o os e con e gence, and a c owd-
ing dis ance densi y es ima o o p omo e di e si y [15]. These
mechanisms a e applied in he eplacemen me hod, as shown
in he ollowing snippe :
1class NSGAII(Gene icAlgo i hm):
de __ini __(sel ,
3p oblem: P oblem,
popula ion_size,
5o sp ing_size,
mu a ion: Mu a ion,
7c osso e : C osso e ,
selec ion: Selec ion,
9 e mina ion_c i e ion: Te mina ionC i e ion,
popula ion_gene a o =RandomGene a o (),
11 popula ion_e alua o =Sequen ialE alua o ()
dominance_compa a o =DominanceCompa a o ()):
13 ...
de eplacemen (sel , popula ion, o sp ing):
15 join_popula ion = popula ion + o sp ing
17 e u n RankingAndC owdingDis anceSelec ion(
sel .popula ion_size, sel .dominance_compa a o ).execu e(
join_popula ion)
No mo e code is needed. To con igu e and un he algo i hm
we include some examples, such as he ollowing code:
# S anda d gene a ional NSGAII unne
2p oblem = ZDT1()
4max_e alua ions = 25000
algo i hm = NSGAII(
6p oblem=p oblem,
popula ion_size=100,
8o sp ing_popula ion_size=100,
mu a ion=PolynomialMu a ion(...),
10 c osso e =SBXC osso e (...),
selec ion=Bina yTou namen Selec ion(...),
12 e mina ion_c i e ion=S oppingByE alua ions(max=max_e alua ions),
dominance_compa a o =DominanceCompa a o ()
14 )
16 p og ess_ba = P og essBa Obse e (max=max_e als)
algo i hm.obse able. egis e (obse e =p og ess_ba )
18
eal_ ime = Visualize Obse e ()
20 algo i hm.obse able. egis e (obse e = eal_ ime)
22 algo i hm. un()
on = algo i hm.ge _ esul ()
24
# Sa e esul s o ile
26 p in _ unc ion_ alues_ o_ ile( on , ‘FUN’)
p in _ a iables_ o_ ile( on , ‘VAR’)
This code snippe depic s a s anda d con igu a ion o NSGA-
II o sol e he ZDT1 benchma k p oblem. No e ha we can de-
ine a dominance compa a o (line 13), which by de aul is he
one used in he s anda d implemen a ion o NSGA-II.
Figu e 2: Sc eensho o jMe alPy unning a NSGA-II o he ZDT1 benchma k
p oblem showing he p og ess and he Pa e o on app oxima ion.
5

0.0 0.2 0.4 0.6 0.8 1.0
x
0.0
0.2
0.4
0.6
0.8
1.0
y
NSGAII-ZDT1
Pa e o on app oxima ion
0.0 0.2 0.4 0.6 0.8 1.0
x
0.0
0.2
0.4
0.6
0.8
1.0
y
ssNSGAII-ZDT1
Pa e o on app oxima ion
0.0 0.2 0.4 0.6 0.8 1.0
x
0.0
0.2
0.4
0.6
0.8
1.0
y
gNSGAII-ZDT1
Pa e o on app oxima ion
Figu e 3: Pa e o on app oxima ions when sol ing he ZDT1 p oduced by he s anda d NSGA-II algo i hm (le ), a s eady-s a e e sion (cen e ), and G-NSGA-II
(using he e e ence poin [ 1, 2]=[0.5,0.5], shown in ed).
As commen ed p e iously, any algo i hm is an obse able
en i y, so obse e s can egis e in o i . In his code, we egis e
a p og ess ba obse e (shows a ba in he e minal indica ing
he p og ess o he algo i hm) and a isualize obse e (shows
a g aph plo ing he cu en popula ion, i.e., he cu en Pa e o
on app oxima ion). A sc een cap u e o NSGA-II unning in
included in Figu e 2. The compu ing ime o NSGA-II wi h his
con igu a ion in ou a ge lap op is a ound 9.2 seconds.
4.2. S eady-S a e NSGA-II
A s eady-s a e e sion o NSGA-II can be con igu ed by
eso ing o he same code, bu jus se ing he o sp ing pop-
ula ion size o one. This e sion yielded a be e pe o mance
in e ms o he p oduced Pa e o on app oxima ion compa ed
wi h he s anda d NSGA-II as epo ed in a p e ious s udy [43],
bu a a cos o a highe compu ing ime, which aises up o 190
seconds.
An example o Pa e o on app oxima ion ound by his
e sion o NSGA-II when sol ing he ZDT1 benchma k p ob-
lem is shown in Figu e 3-cen e . As expec ed gi en he li e -
a u e, i compa es a o ably agains he one gene a ed by he
s anda d NSGA-II (Figu e 3-le ).
4.3. NSGA-II wi h P e e ence A icula ion
The NSGA-II implemen a ion in jMe alPy can be easily ex-
ended o inco po a e a p e e ence a icula ion scheme. Con-
c e ely, we ha e de eloped a g-dominance based compa a o
conside ing he g-dominance concep desc ibed in [44], whe e
a egion o in e es can be delimi ed by de ining a e e ence
poin . I we desi e o ocus he sea ch in he in e es egion de-
limi ed by he e e ence poin , say e.g. [ 1, 2]=[0.5,0.5], we
can con igu e NSGA-II wi h his compa a o as ollows:
1 e e ence_poin = [0.5, 0.5]
algo i hm = NSGAII(
3...
dominance_compa a o =GDominanceCompa a o ( e e ence_poin )
5)
The esul ing on is show in Figu e 3- igh .
4.4. Dynamic NSGA-II
The app oach adop ed in jMe alPy o p o ide suppo o
dynamic p oblem sol ing is as ollows: Fi s , we ha e de el-
oped a TimeCoun e class (which is an Obse able en i y) which,
gi en a delay, inc emen s con inuously a coun e and no i ies
he egis e ed obse e s he new coun e alues; second, we
need o de ine an ins ance o DynamicP oblem, which mus
implemen he me hods o checking whe he he p oblem has
changed and o clea he changed s a e. As DynamicP oblem
inhe i s om Obse e , ins ances o his class can egis e in a
TimeCoun e objec . Finally, i is equi ed o ex end Dynami-
cAlgo i hm wi h a class de ining he es a () me hod ha will
be called when he algo i hm de ec s a change in a dynamic
p oblem. The ollowing code snippe shows he implemen a-
ion o he DynamicNSGAII class:
1class DynamicNSGAII(NSGAII, DynamicAlgo i hm):
de __ini __(sel , ...):
3...
sel .comple ed_i e a ions = 0
5
de es a (sel ) -> None
7# es a s a egy
9de upda e_p og ess(sel ):
i sel .p oblem. he_p oblem_has_changed():
11 sel . es a ()
sel .e alua o .e alua e(sel .solu ions, p oblem)
13 sel .p oblem.clea _changed()
sel .e alua ions += sel .o sp ing_size
15
de s opping_condi ion_is_me (sel ):
17 i sel . e mina ion_c i e ion.is_me :
sel . es a ()
19 sel .e alua o .e alua e(sel .solu ions, p oblem)
sel .ini _p og ess()
21 sel .comple ed_i e a ions += 1
As shown abo e, a he end o each i e a ion a check is
made abou a change in he p oblem. I a change has occu ed,
he es a me hod is in oked which, depending on he imple-
men ed s a egy, will emo e some solu ions om he popula-
ion and new ones will be c ea ed o eplace hem. The esul ing
popula ion will be e alua ed and he clea changed() me hod
o he p oblem objec will be called. As opposed o he s an-
da d NSGA-II, he s opping condi ion me hod is no in oked
o hal he algo i hm, bu ins ead o no i y egis e ed obse e s
(e.g., a isualize ) ha a new esul ing popula ion has been p o-
duced. Then, he algo i hm s a s again by in oking he es a ()
and ini p og ess() me hods. I is wo h no ing ha mos o
he code o he o iginal NSGA-II implemen a ion is eused and
only some me hods need o be ew i en.
To illus a e he implemen a ion a dynamic p oblem, we
nex show code o he FDA abs ac class, which is he base
class o he i e p oblems composing he FDA benchma k:
1class FDA(DynamicP oblem, Floa P oblem, ABC):
de __ini __(sel ):
3supe (FDA, sel ).__ini __()
6
sel . au_T = 5
5sel .nT = 10
sel . ime = 1.0
7sel .p oblem_modi ied = False
9de upda e(sel , *a gs, **kwa gs):
coun e = kwa gs[’COUNTER’]
11 sel . ime = (1.0 / sel .nT) * loo (coun e * 1.0 / sel . au_T)
sel .p oblem_modi ied = T ue
13
de he_p oblem_has_changed(sel ) -> bool:
15 e u n sel .p oblem_modi ied
17 de clea _changed(sel ) -> None:
sel .p oblem_modi ied = False
The key poin in his class is he upda e() me hod which,
when in oked by an obse able en i y (e.g., an ins ance o he
a o emen ioned TimeCoun e class), se s he p oblem modi ied
lag o T ue. We can obse e ha his lag can be que ied and
ese .
The code p esen ed nex shows how o con igu e and un
he dynamic NSGA-II algo i hm:
# Dynamic NSGAII unne
2p oblem = FDA2()
ime_coun e = TimeCoun e (delay=1))
4 ime_coun e .obse able. egis e (p oblem)
ime_coun e .s a ()
6
algo i hm = DynamicNSGAII(
8...
e mina ion_c i e ion=S oppingByE alua ions(max=
10 max_e als)
)
12 algo i hm. un()
A e c ea ing he ins ances o he FDA2 benchma k p ob-
lem [21] and he ime coun e class, he o me is egis e ed
in he la e , which uns in a concu en h ead. The dynamic
NSGA-II is se wi h s opping condi ion which e u ns a Pa e o
on app oxima ion e e y 25,000 unc ion e alua ions. An ex-
ample o unning o he dynamic NSGA-II algo i hm when sol -
ing he FDA2 p oblem is shown in Figu e 4.
0.0 0.2 0.4 0.6 0.8 1.0
0
0.0
0.2
0.4
0.6
0.8
1.0
1
FDA2
Pa e o on app oxima ion
Figu e 4: Pa e o on app oxima ions when sol ing he dynamic FDA2 p ob-
lem p oduced by he dynamic e sion o NSGA-II.
4.5. Pa allel NSGA-II wi h Apache Spa k
In o de o e alua e a popula ion, NSGA-II (and in gene al,
any gene a ional algo i hms in jMe alPy) can use an e alua o
objec . The de aul e alua o uns in a sequen ial ashion bu ,
should he e alua e me hod o he p oblem be h ead-sa e, so-
lu ions can be e alua ed in pa allel. jMe alPy includes an e al-
ua o based on Apache Spa k, so he solu ions can be e alua ed
in a a ie y o pa allel sys ems (mul ico es, clus e s) ollowing
he scheme p esen ed in [45]. This e alua o can be used as
exempli ied nex :
# NSGAII unne using he Spa k e alua o
2algo i hm = NSGAII(
...
4e alua o =Spa kE alua o ()
)
The esul ing pa allel NSGA-II algo i m combines pa allel
wi h sequen ial phases, so speed imp o emen s canno be ex-
pec ed o scale linea ly. A pilo es on ou a ge lap op indi-
ca es speedup ac o s in he o de o 2.7. Howe e , wha is in-
e es ing o no e he e is ha no changes a e equi ed in NSGA-
II, which has he same beha io as i s sequen ial e sion, so he
ob ained ime educ ions a e o ee.
4.6. Dis ibu ed NSGA-II wi h Dask
The las a ian o NSGA-II we p esen in his pape is a
dis ibu ed e sion based on an asynch onous pa allel model
implemen ed wi h Dask [13], a pa allel and dis ibu ed Py hon
sys em including a b oad se o pa allel p og amming models,
including asynch onous pa allelism using u u es.
The dis ibu ed NSGA-II adop s a pa allel scheme s udied
in [43]. The scheme is based on a s eady-s a e NSGA-II and he
use o Dask’s u u es, in such a way ha whene e a new solu-
ion has o e alua ed, a ask is c ea ed and submi ed o Dask,
which e u ns a u u e. When a ask is comple ed, i s co e-
sponding u u e e u ns an e alua ed solu ion, which is inse ed
in o he o sp ing popula ion. Then, a new solu ion is p oduced
a e pe o ming he eplacemen , selec ion, and ep oduc ion
s ages, o be sen again o e alua ion. This way, all he p oces-
so s/co es o he a ge clus e will be busy mos o he ime.
P elimina y esul s on ou a ge mul ico e lap op indica e
ha speedups a ound 5.45 can ob ained wi h he 8 co es o he
sys em whe e simula ions we e pe o med. We will discuss on
his lack o scalabili y and o he aspec s o his use case in he
nex subsec ion.
4.7. Discussion
In his sec ion we ha e p esen ed i e di e en e sions o
NSGA-II , mos o hem (excep o he dis ibu ed a ian ) e-
qui ing mino changes on he base class implemen ing NSGA-
II. No all algo i hms can be adap ed in he same way, bu some
o he a ia ions o NSGA-II can be implemen ed in a s aigh -
o wa d manne . Thus, we include in jMe alPy examples o dy-
namic, p e e ence-based, and pa allel e sions o some o he
included algo i hms, such as SMPSO, GDE3, and OMOPSO.
We would like o again s ess on he eadabili y o he codes,
by i ue o which all he s eps o he algo i hms can be clea ly
iden i ied. Some use s may ind he class hie a chy E olu ion-
a yAlgo i hm →Gene icAlgo i hm →NSGAII cumbe some,
and p e e o ha e all he code o NSGA-II in a single class.
Howe e , his al e na i e design app oach would hinde he lex-
ibili y o he cu en implemen a ion, and would equi e o epli-
ca e mos o he code when de eloping algo i hmic a ian s.
In he case o pa allel algo i hms, an exhaus i e pe o mance
assessmen is beyond he scope o his pape . The epo ed
7
speedups a e no ema kable due o he Tu bo Boos ea u e
o he p ocesso o he lap op used o pe o ming he expe i-
men s, bu hey gi e an idea o he ime educ ions ha can be
achie ed when using a mode n mul ico e compu e .
5. Visualiza ion
An ad an age o using Py hon (ins ead o Ja a) is i s powe
ela ed o isualiza ion ea u es hanks o he a ailabili y o g aphic
plo ing lib a ies, such as: Ma plo lib, Holo iews o Plo ly.
jMe alPy ha nesses hese lib a ies o include h ee ypes o
isualiza ion cha s: s a ic, in e ac i e and s eaming. Table 2
summa izes hese implemen a ions. S a ic cha s can be shown
in he sc een, s o ed in a ile, o included in a Jupy e no ebook
( ypically used a he end o he execu ion o an algo i hm).
Simila ly, in e ac i e cha s a e gene a ed when an algo i hm
e u ns a Pa e o on app oxima ion bu , unlike he s a ic ones,
he use can manipula e hem in e ac i ely. The e a e wo kinds
o in e ac i e cha s: hose ha p oduce an HTML page includ-
ing a cha (allowing o apply ac ions such as zooming, selec -
ing pa o he g aph, o clicking in a poin o see i s objec i e
alues a e allowed) and cha s such as he Cho d diag am ha
allows ho e ing he mouse o e he cha and isualizing ela-
ionships among objec i e alues. Finally, s eaming cha s de-
pic g aphs in eal ime, du ing he execu ion o he algo i hms
(and hey can also be included in a Jupy e no ebook); his can
be use ul o obse e he e olu ion o he cu en Pa e o on
app oxima ion p oduced by he algo i hm.
Table 2: Main isualiza ions included in jMe alPy.
Name Type Backend Desc ip ion
Plo S a ic Ma plo lib 2D, 3D, p-coo ds
In e ac i e Plo ly 2D, 3D, p-coo ds
S eaming
plo
S eaming Ma plo lib 2D, 3D
S eaming HoloViews 2D, 3D ( o Jupy e )
Cho d plo In e ac i e Ma plo lib Fo s a is ical pu poses
Box plo In e ac i e Ma plo lib Fo s a is ical pu poses
CD plo S a ic Ma plo lib Demsa ’s c i ical dis-
ance plo
Pos e io
plo
S a ic Ma plo lib Bayesian pos e io analy-
sis
Figu e 5 shows h ee examples o in e ac i e plo s based
on Plo ly. The a ge p oblem is DTLZ1 [46], which is sol ed
wi h he SMPSO algo i hm when he p oblem is de ined wi h
2, 3 and 5 objec i es. Fo any p oblem wi h mo e han 3 objec-
i es, a pa allel coo dina es g aph is gene a ed. An example o
Cho d diag am o a p oblem wi h 5 objec i es is shown in Fig-
u e 6; each depic ed cho d ep esen s a solu ion o he ob ained
Pa e o on , and ies oge he i s objec i e alues. When ho -
e ing o e a sec o box o a ce ain objec i e i, his cha only
ende s hose solu ions whose i alues all wi hin he alue
suppo o his objec i e delimi ed by he ex emes o he sec o
box. Finally, he ou e pa i ioned o us o he cha ep esen s
a his og am o he alues co e ed in he ob ained Pa e o on
o e e y objec i e.
Figu e 5: Examples o in e ac i e plo s p oduced when using SMPSO o sol e
he DTLZ1 p oblem wi h 2 ( op), 3 (middle), and 5 (bo om) objec i es.
8
Figu e 6: Example o Cho d diag am o he on ob ained by SMPSO when
sol ing a p oblem wi h 5 objec i es.
6. Expe imen al Use Case
In p e ious sec ions, we ha e shown examples o Pa e o
on app oxima ions p oduced by some o he me aheu is ics
included in jMe alPy. In his sec ion, we desc ibe how ou
amewo k can be used o ca y ou igo ous expe imen al s ud-
ies based on compa ing a numbe o algo i hms o de e mine
which o hem p esen s he bes o e all pe o mance.
6.1. Expe imen a ion Me hodology
An expe imen al compa ison equi es a numbe o s eps:
1. De e mine he algo i hms o be compa ed and he bench-
ma k p oblems o be used.
2. Run a numbe o independen uns pe algo i hm-p oblem
con igu a ion and ge he p oduced on s.
3. Apply quali y indica o s o he on s (e.g., Hype olume,
Epsilon, e c.).
4. Apply a numbe o s a is ical es o assess he s a is ical sig-
ni icance o he pe o mance di e ences ound among he
algo i hms conside ed in he benchma k.
The i s h ee s eps can be done wi h jMe alPy, bu also
wi h jMe al o e en manually (e.g., unning algo i hms using a
sc ip ). The poin whe e jMe alPy s ands ou is he ou h one,
as i con ains a la ge amoun o s a is ical ea u es o p o ide
he use wi h a b oad se o ools o analyze he esul s gene -
a ed by a compa a i e s udy. All hese unc ionali ies ha e been
p og ammed om sc a ch and embedded in o he co e o jMe -
alPy. Speci ically, he s a is ical es s included in jMe alPy a e
lis ed nex :
•A di e se se o non-pa ame ic null hypo hesis signi icance
es s, namely, he Wilcoxon ank sum es , Sign es , F ied-
man es , F iedman aligned ank es and Quade es . These
es s ha e been adi ionally used by he communi y o shed
ligh on hei compa a i e pe o mance by inspec ing a s a is-
ic compu ed om hei sco es.
•Bayesian es s (sign es and signed ank es ), which ha e
been ecen ly pos ula ed o o e come he sho comings o
null hypo hesis signi icance es ing o pe o mance assess-
men [47]. These es s a e complemen ed by a pos e io plo
in ba ycen ic coo dina es o compa e pai s o algo i hms un-
de a Bayesian app oach by also accoun ing o possible s a-
is ical ies.
•Pos hoc es s o compa e among mul iple algo i hms, ei he
one- s-all (Bon e oni-Dunn, Holland, Finne , and Hochbe g)
o all- s-all (Li, Holm, Sha e ).
The esul s o hese es s a e displayed by de aul in he
sc een and mos o hem can be expo ed o L
A
T
EX ables. Fu -
he mo e, boxplo diag ams can be also gene a ed. Finally,
L
A
T
EX ables con aining means and medians (and hei co e-
sponding s anda d de ia ion and in e qua ile ange dispe sion
measu es, espec i ely) a e au oma ically gene a ed.
6.2. Implemen a ion De ails
jMe alPy has a labo a o y module con aining u ili ies o
de ining expe imen s, which equi e h ee lis s: he algo i hms
o be compa ed (which mus be p ope ly con igu ed), he bench-
ma k p oblems o be sol ed, and he quali y indica o s o be
applied o pe o mance assessmen . Addi ional pa ame e s a e
he numbe o independen uns and he ou pu di ec o y.
Once he expe imen is execu ed, a summa y in he o m
o a CSV ile is gene a ed. This ile con ains all he in o ma-
ion o he quali y indica o alues, o each con igu a ion and
un. Each line o his ile has he ollowing schema: Algo i hm,
P oblem, Indica o , Execu ionId, Indica o Value. An example
o i s con en s ollows:
1Algo i hm,P oblem,Indica o ,Execu ionId,Indica o Value
NSGAII,ZDT1,EP,0,0.015705992620067832
3NSGAII,ZDT1,EP,1,0.012832504015918067
NSGAII,ZDT1,EP,2,0.01071189935186434
5...
MOCell,ZDT6,IGD+,22,0.0047265135903854704
7MOCell,ZDT6,IGD+,23,0.004496215669027173
MOCell,ZDT6,IGD+,24,0.005483899232523609
whe e we can see he heade wi h he column names, ollowed
by ou lines co esponding o he alues o he Epsilon in-
dica o o h ee uns o he NSGA-II algo i hm when sol ing
he ZDT1 p oblem. The end o he ile shows he alue o
he IGD+indica o o h ee uns o MOCell when sol ing he
ZDT6 p oblem. The ile con ains as many lines as he p oduc
o he numbe s o algo i hms, p oblems, quali y indica o s, and
independen uns.
The summa y ile is he inpu o all he s a is ical es s, so
ha hey can be applied o any alid ile ha ing he p ope
o ma . This is pa icula ly in e es ing o combine jMe al and
jMe alPy. The las e sions o jMe al gene a es a summa y ile
a e unning a se o algo i hms in an expe imen al s udy, so
hen we can ake ad an age o he ea u es o jMe al (p o iding
9