Creating synthetic datasets for collaborative filtering recommender systems using generative adversarial networks.

Creating Synthetic Datasets for Collaborative Filtering Recommender Systems using Generative Adversarial Networks Jes´us Bobadillaa , Abraham Guti´erreza , Raciel Yerab , Luis Mart´ınezb,∗ a Departamento de Sistemas Inform´aticos, ETSI Sistemas Inform´aticos, Universidad Polit´ecnica de Madrid, C/ Alan Turing s/n, 28031, Madrid, Spain b Departamento de Inform´atica, Universidad of Ja´en, Ja´en,Spain Abstract Research and education in machine learning requires diverse, representative, and open datasets that contain sufficient samples to handle the necessary training, validation, and testing tasks. Currently, the Recommender Systems area includes a large number of subfields in which accuracy and beyond-accuracy quality measures are continuously being improved. To feed this research variety, it is both necessary and convenient to reinforce the existing datasets with synthetic ones. This paper proposes a Generative Adversarial Network (GAN)-based method to generate collaborative filtering datasets in a parameterized way by selecting their preferred number of users, items, samples, and stochastic variability. This parameterization cannot be performed using regular GANs. Our GAN model is fed with dense, short, and continuous embedding representations of items and users, instead of sparse, large, and discrete vectors, to ensure fast and accurate learning, as compared to the traditional approach based on large and sparse input vectors. The proposed architecture includes a DeepMF model to extract the dense user and item embeddings and a clustering process to convert the dense GAN generated samples to the discrete and sparse samples necessary to create each required synthetic dataset. The results from three different source datasets show adequate distributions and expected quality values and evolutions in the generated datasets compared to the source datasets. Synthetic datasets and source codes are available to researchers. Keywords: Recommender Systems,Generative Adversarial Networks,Deep Learning,Collaborative Filtering 1. Introduction Recommender systems (RS) are a relevant area in artificial intelligence due to the growing popularity of social networks. The big companies that extensively use RSs are ∗ Corresponding author Email addresses: [email protected] (Jes´us Bobadilla), [email protected] (Abraham Guti´errez), [email protected] (Raciel Yera), [email protected] (Luis Mart´ınez) Preprint submitted to Knowledge-Based Systems August 30, 2023 Originally published in: Bobadilla, J., Gutiérrez, A., Yera, R., & Martínez, L. (2023). Creating synthetic datasets for collaborative filtering recommender systems using generative adversarial networks. Knowledge-Based Systems, 280, 111016. TripAdvisor, Netflix, Spotify, YouTube Music, TikTok, YouTube and Amazon [19]. These companies make use of the RS models to recommend to users similar items (music, videos, trips, news) to those that they have already consumed; some other companies, such as Facebook, work hard to collect customer activity to provide personalized advertising rather than personalized products or services. RSs are usually classified according to their filtering approach [54]; content-based RSs select the recommended items by looking for similar content [53]; since most item contents is text, natural language processing models are used. Reviews [59] and tweets [22] are two common types of content-based filtered data. Product images can also be processed to make recommendations; convolutional neural networks are the most commonly used models to perform this task [29]. Social filtering has been extensively used to improve social-based recommendations. This type of filtering uses data such as tags, followers, and being followed, and makes use of the concepts of reputation and trust [33]. Geographic information, such as GPS coordinates and POI, is mainly used to support context-aware filtering [46]. Demographic filtering (age, gender, country, etc.) is commonly combined with other types of filtering, implementing recommendation ensembles [35]. Beyond the previous filtering strategies, collaborative filtering (CF) [28] is the most important approach for implementing RSs, since it provides superior accuracy, particularly when combined with some other types of filtering. Effective RS research makes use of innovative models, adequate quality measures, and representative datasets. The historical evolution of CF begins with the use of memory-based models, mainly the K-Nearest Neighbors algorithm [60]. Memory-based approaches were replaced by model-based machine learning approaches due to their overall performance: they are superior in accuracy of results, also in time to obtain predictions (once the model has learned); and their output is capable of being explained through post-hoc techniques [52]. Matrix Factorization (MF) [18] is the most widely used machine learning model to implement collaborative filtering; it performs a dimensional reduction of users and items, capturing the main patterns that relate them to the votes cast. Additionally, by using Non-Negative Matrix Factorization (NFM) [1], semantic meanings can be assigned to latent factors. Bayesian NMF [3] allows clustering users and making predictions simultaneously, which opens the door to effective recommendations to user groups and social clustering applications [5]. Nowadays, CF research is mainly developed by deep learning models, where DeepMF [51] is the basis for modern approaches. DeepMF is the model that we use in this paper, in which users are coded in a latent space by means of an embedding layer, whereas items are coded in a different latent space by means of a second embedding layer; finally, predictions are made by making the dot product of both, item and user embeddings. DeepMF improves MF due to the inherent competence of neural networks to capture the non-linear relations hips between samples. Neural Collaborative Filtering (NCF) [26] is extensively used to implement CF; this model replaces the DeepMF dot layer with a Multi-Layer Perceptron (MLP) and outperforms DeepMF when applied to large and complex datasets. Beyond accuracy, deep learning models are emerging to perform some innovative tasks, such as improving fairness, where the DeepFair model [9] achieves a trade-off between equity and precision; green computing [27]; results explanation via latent space visualization [6] and efficient neighborhood identification [7]. The adversarial network-based recom2 mendation has recently been introduced in the RS area [56] and we will focus on it in the ’Related work’ section. Generative Adversarial Networks (GAN) [23] are responsible for the popular fake faces and fake videos that flood social networks. Their architecture has two separate neural networks that compete against each other (’adversarial’), such as an art forger competing against an art expert, ensuring that both improve their work. The GAN ’forger’ is a generator model that creates fake samples from random noise vectors, while the GAN ’expert’ is a discriminator model implemented as a simple binary classifier: fake, non-fake. However, while RS research is mainly focused on proposing novel recommendation models, this paper tries to make progress in CF datasets. In this respect, it is essential to identify quality measures as a key element to carry out adequate research, since they allow the baselines of the state of the art to be compared with the proposed algorithms, methods and models. Beyond the usual prediction and recommendation quality measures (MAE, MSD, precision, recall, F1, NDCG, etc.), some other measures, such as novelty and diversity [40], have recently acquired growing importance. Of these, diversity is currently the main focus of researchers’ attention, due to the risks of inappropriate recommendations in social networks, such as those that exhibit a lack of variability and promote prefixed ideas and behaviors. Diversity and reliability in RS have been improved by introducing diversity-enhancing constraints in the MF model [21]; additionally, a deep learning classification model [8] is proposed to obtain the recommendation reliability values from the softmax output layer of the neural network. Quality values are obtained when a model or method is tested on balanced CF datasets. To obtain balanced training and testing sets, with respect to their user and item distributions, deterministic strategies are proposed in [38]. Most of the RS research makes use of popular CF datasets such as MovieLens, FilmTrust, MyAnimeList or CiteSeer; CF datasets include different domains such as music, movies, POIs, tourism, news, research papers, tagged data, etc. Some of these datasets have been filled with explicit votes from users, while others contain implicit interactions between users and systems. There are also datasets filled with crawled Web pages or academic PDFs [10] and some others are enriched with social tags that researchers add to the articles [14]. A selection of relevant social CF datasets is provided in [43] and related to some articles using them. Recently, an educational news dataset [50] was released; which included contextualized information: time and location. Finally, an RS dataset has also been provided that contains artificial intelligence research data [36] to obtain segmented information, clustering, and geographical locations. Beyond these works, it is particularly relevant that parameterized synthetic datasets have not yet been used, so consequently the CF research does not benefit from the flexibility that parameterization provides in the experiment design: different dataset sizes, number of users and items, and so on. This paper aims to fill the gap by proposing a procedure, coined as GANRS, which focuses on the use of GANs to generate collaborative filtering recommender systems datasets in a parameterized way. Please note that current RS GAN-based models cannot simultaneously set the number of generated users, items, and rating distributions. Regarding our contribution, two main overall approaches can be identified in the state-of-art: statistical and generative. The main advantage of the statistical approach is that several relevant parameters can be simultaneously set: number of users, number of 3 items, dataset size, etc. The main drawback of this approach is its poor accuracy. On the other hand, current model-based generative approaches improve accuracy compared to statistical frameworks, but they lack flexibility, since parameterization is very limited. In fact, current GAN designs are focused on user profiles, and they can generate as many new fake users as required, but other relevant parameters cannot be set, such as the number of items that is fixed in the source set of user vectors and then, also, in the fake generated set of user vectors. This can be explained with an example: when we run a regular GAN to generate fake images, the synthetic images have the same shape (resolution and number of channels) as the source images. In the RS field, the synthetic user vectors contain the same number of items as the real user vectors. Following the example, there are some specific GAN designs that return ‘super-resolution’ images (they can increase resolution), but to our knowledge there are no RS GANs designed to generate fake users containing more items (or fewer items). Our proposed method is designed to simultaneously set some CF relevant parameters, such as the number of users and items. The rest of the paper has been structured as follows: related work is introduced in Section 2, focusing on the most recent uses of the GAN models applied to RS. Section 3 explains the proposed model and its formalization. Section 4 presents the design, result, and discussion of the experiments. Finally, Section 5 contains the main conclusions of the article and discusses future work. 2. Background 2.1. Basics on Generative Adversarial Networks GANs are designed to generate data from scratch [23]. They have been commonly used to create fake images, although their use has been spreading to many other domains: music, medicine, financial data, etc. The GAN architecture composes of two deep network models: generator and discriminator. The generator model learns to create samples as similar as possible to those in a dataset (e.g., a dataset with human faces images), whereas the discriminator model learns to detect fake samples (those samples created by the generator). To better understand the GANs we can can consider the example of a painting forger and a forgery expert: the more imitations the forger paints, the better their results, and the better the expert’s ability to detect fake paintings. Both people successively improve their abilities. When the learning begins, the GAN discriminator (the forgery expert, in our example) has an easy job, since the generator does not have the painting patterns. After thousands of learning epochs, the generator has learnt the patterns well enough to confuse the discriminator, who is forced to tune their weights. If the learning loop iterates enough times, both the generator and the discriminator models are well designed and the painting in the dataset contains suitable patterns, the generator will be able to create synthetic (fake) samples that are difficult to distinguish from the originals. Figure 1 shows the GAN architecture [23]; the discriminator model makes a binary classification between fake and real samples. The generator model updates its weights (learns) when the discriminator correctly classifies a fake sample. The discriminator model updates its weights when it incorrectly classifies a sample. Note that the generator takes a random noise distribution as input to generate samples; then, once it has 4 Figure 1: Generative Adversarial Networks architecture learnt, for each input random noise vector that feeds the generator a sample is created with the patterns of the dataset samples. For this reason, by providing random noise vectors we can create as many samples as required, which means that, in our context, we can create fake CF datasets of any size by creating fake profiles. To measure GAN loss, we use cross-entropy. The discriminator (D) loss can be expressed as the sum of the expectations: maxDV(D) = Ex∼pdata(x)[logD(x)]+Ez∼pz(z)[log(1−D(G(z)))] (1) Where the first term of the equation is used to recognize real images, and the second term recognizes generated images. Zrepresents the noisy vector, Gis the generator, and G(Z)is the generated sample. D(G(z)) is the classification result of the discriminator when its input is a fake sample. D(x)is the classification result of the discriminator when its input is a real sample. The generator loss is designed to learn when the discriminator correctly classifies (the true label is 1, and the fake label is 0). Its equation is: minGV(G) = Ex∼pdata(x)[log(1−D(G(z)))] (2) The GAN is a minimax in which Gwants to minimize Vwhile Dwants to maximize it: minGmaxDV(D,G) = Ex∼pdata(x)[logD(x)]+Ez∼pz(z)[log(1−D(G(z)))] (3) As in the previous example, GAN models act on non sparse values (e.g., pixels in a picture), but they are not designed to work with sparse vectors or matrices. Our problem here is that CF datasets contain extraordinarily sparse matrices of ratings (users only vote or consume a very limited number of the available items). Using the regular GAN architecture is not an adequate approach to addressing CF-based RS. This paper proposes an extended GAN architecture where embeddings are introduced to code the 5 sparse and discrete vectors of votes to dense and continuous vectors. This innovation makes it possible to use a regular GAN to generate dense and continuous vectors efficiently and accurately. This compression stage forces us to design the corresponding stage to decompress the generated dense vectors. The adopted solution makes it possible to set both the number of users and items in the generated dataset, which is a relevant innovation in the state-of-art. 2.2. Related works Generative deep learning is an innovative field in the CF RS area. Although some variational autoencoder approaches have been published [30, 55], current research is mainly focused on GAN models [20]. A CF subfield where GANs are used is the attack/defense strategies [16], where these models can reinforce security in RS. Nevertheless, the most extended uses of CF GANs are: a) to solve the issue of noisy data, and b) to tackle the data sparsity problem, and implement a data augmentation framework by capturing the distribution of real data. CFGAN [12] is a model that generates purchase vectors rather than the IDs of items and then uses the generated fake purchase vectors to augment the real vectors. The Wasserstein version of CFGAN is the unified GAN (UGAN) [48] and reports improvements compared to CFGAN. To prioritize long and short-term RS information (interactions between users and items that change quickly or slowly), the PLASTIC [58] model trains a generator and uses it as a reinforcement learning agent. The recurrent GAN: RecGAN [4], learns temporal patterns in ratings; combining GAN and recurrent neural networks (RNNs) models. To capture negative sampling information in the CF datasets, IPGAN [24] implements two different generative models: one for positive instances and another for negative instances. IPGAN considers the relations between the positive ratings sampled and the negative ones selected. Currently, the DCGAN model [57] combines GAN and reinforcement learning models to catch the information of the RS sessions, rather than the traditional historical matrices of votes from users to items. Session information includes the responses of users to current recommendations. The user’s immediate feedback is managed by the reinforcement learning model combined with the GAN. The NCGAN [44] incorporates a neural network to extract nonlinear features from users, and a GAN to guide the recommendation training; the generator model makes user recommendations, whereas the discriminator model measures distances between real and generated distributions. An innovative method to improve the information flow from generator to discriminator [31] reduces the discrepancies between both models in the CF GAN. A regularization Wasserstein GAN model is used in [47], combined with an autoencoder acting as a generator, reporting accuracy improvement when applied to high-dimensional and sparse CF matrices. A CGAN (Conditional GAN) is used [49] to improve CF recommendations, and the sizes of the rating vectors can be set, simplifying the generator and discriminator tasks. Additionally, it allows conditional rating generation to be established. For datasets that do not follow standard Gaussian distributions, a missing data imputation based on GAN [17] is proposed; results show improved quality in several representative classification data sets. Trust information is used in [13] to make effective recommendations. They propose a GAN where the discriminator is an MLP model, and the generator is a long-short term memory network (LSTM) model [45]. Finally, 6 CF datasets are usually imbalanced due to their social data collection (e.g: more young people than old people). To address this limitation, [41] proposes a Wasserstein GAN model in the generator, and the PacGAN concept in the discriminator [32], to minimize the mode collapse problem. A platform for multi-agent RS simulation is the probabilistic-based RecSim [34], which generates synthetic profiles of users and items, and uses Markov chains and recurrent neural networks. The Virtual-Taobao [42] is a multiagent reinforcement learning system designed to improve search in the social Taobao website; it makes use of a GAN to simulate internal distributions. A simple matrix factorization is used [61] to inject topic diversification into the recommendation process. The DataGenCars [11] is a Java-based generator of RS synthetic data; it contains a statistical basement that provides flexibility, but it returns low accuracy compared to deep learning generative models. Finally, the SynEvaRec framework [39] provides the generation of synthetic RS datasets using the Synthetic Data Vault (SVD) library. This library models multivariate distributions using copula functions; its CTGAN sub-library includes GAN models. The main advantage of SynEvaRec, compared to previous frameworks, is that it can use different RSs as a source; its main drawbacks are the poor quality of the results in most of the cases, and the excessive time it takes to perform the training stage. Previous research mainly focuses on improving different objectives such as noise reduction, recommendation quality, prediction values, defense against attacks, or balancing data. To make this happen, many different approaches and information sources have been combined: the use of GAN, CGAN, Wasserstein GAN, etc. GAN models have been combined with Recurrent Neural Networks [15] and LSTM networks [45], and reinforcement learning has been introduced in the GAN-based architectures. Long and short data have been introduced to the proposed models, in addition to trust information, session logs, including responses of the users to previous recommendations, and inferred negative votes. The pure generation of synthetic datasets does not seem to be a goal in this novel field of GAN applied to CF RS, which is currently focused on improving prediction and recommendation quality results by means of data augmentation based on the inherent ability of the GAN model to capture the complex nonlinear patterns of high-dimensional and sparse CF datasets. The innovation of our proposal is to generate representative and useful CF synthetic datasets, rather than to improve the existing results that are of varying quality. Additionally, it allows representative parameters to be set and a whole ‘family’ of synthetic datasets to be obtained, taking real datasets as a source, such as Movielens, Netflix, or MyAnimeList. These parameters are the number of users, the number of items, the number of samples and the variability of the generated data. By varying the parameter values, we can generate different versions of the same CF pattern, such as a Movielens-based dataset containing 8000 users and 3000 items, or another that contains 2000 users and 1000 items, among others. In this way, we can test the accuracy and performance impact of the dataset size, its sparsity, its number of users and items, as well as check the improvement of the MAE when the number of users increases. As far as we know, there are no published methods or models for creating, in a parameterized deep learning model, accurate and scalable synthetic datasets from diverse sources. 7 3. The Generative Adversarial Networks-based approach for datasets building in collaborative filtering As previously mentioned in the Introduction section, our research problem is defined as obtaining a larger, scalable synthetic dataset from an original RS dataset that synthesizes similar user behavior and valuation patterns in relation to the original dataset. In addition, it is desirable that such generation be parameterized, allowing the number of users, items, samples and variability of the distribution to be controlled. Next, this section proposes the GANRS method, which uses a GAN network to generate synthetic CF datasets; the GAN is fed with a real CF dataset and the model learns its internal patterns. The most innovative contribution is to feed the GAN with dense and small embedding representations of users and items, instead of the traditional approach where the GAN inputs are large and comprise sparse vectors containing the votes cast for each user. The main advantage of the GANRS method is that it greatly reduces the complexity of the GAN architecture, its convergence speed, and its performance. The traditional and sparse-based GAN architectures deal with very large input vectors: as large as the number of items in the dataset, which can be in the tens of thousands, and require a very large dense layer in the model to hold this huge amount of data. What is more, between 97% to 99% of the data is usually missing, since users only vote for or consume a tiny proportion of the available products or services, hence the extraordinary sparsity in the CF datasets. Following the huge dense layer, in classical GAN architectures, it is necessary to stack a large multilayer perceptron to reduce dimensionality. By comparison, the proposed model replaces the large dense layer with two embeddings, one to code users and the other to code items (borrowed from the DeepMF model in the first stage of the proposed method). Embedding layers are specifically designed to deal with sparse data; they receive integer values (user and item IDs, in our case), and they provide small embedding representations (typically 5 to 15 float values in the CF scenarios). Related users or items share similar embedding representations, and this feature allows for extraordinarily simplification of the model. Overall, the proposed architecture is much smaller than traditional architectures, it contains far fewer parameters, and consequently, learns faster. Additionally, it better captures the complex nonlinear relations between items and users, in the same way that non-GAN RS models do to improve predictions. The formalization of the GANRS method is presented and structured according to the following seven stages, also illustrated in Figure 2: • Stage 0. CF definitions 1 Let Ube the set of users who make use of a CF RS. 2 Let Ibe the set of items available for voting in the CF RS. 3 Let Vbe the range of allowed votes; usually V={1,2,3,4,5}. 4 Let Sbe the set of samples contained in the CF dataset; in which N=|S|=the total number o f votes cast. 5S={<u,i,v>1,< u,i,v>2,. . . .,< u,i,v>N}; where each u∈ {1,. . . ,|U|}, each i∈ {1,. . . ,|I|}, and each v∈ {1,. . . ,|V|}. 8 • Stage 1. DeepMF training 6 Let E be the size of two neural layer embeddings used to vectorize each user and each item belonging to Uand I, respectively. 7 Let feu(u) = [eu 0,eu 1,. . . ,eu E], where feu is the embedding layer output of the users, where u∈ {1,. . . ,|U|}. 8 Let fei(i)=[ei 0,ei 1,. . . ,ei E], where fei is the embedding layer output of the items, where i∈ {1,. . . ,|I|}. By combining both dense vectors of user and item embeddings: ([eu 0,eu 1,. . . ,eu E]and [ei 0,ei 1,. . . ,ei E]), we can make rating predictions in the DeepMF training stage. The dot product of the user embedding and the item embedding in each <u,i,v>j∈Sprovides its rating prediction: 9 ˆyj=feu(u)·fei(i) = [eu 0,eu 1,. . . ,eu E]·[ei 0,ei 1,. . . ,ei E] 10 1 2(yj−ˆyj)2is the output error used in the DeepMF neural network to start the backpropagation algorithm, where the neural weights are iteratively improved from the δjvalues: △wji =αyjf′(Neti)∑kwikδk, when kis a hidden layer, and △wji =αyif′(Neti)1 2(yk−ˆyk)2, if kis the output layer. i, j, and k are successive sequential layers. Netirepresents the cumulative input received for an artificial neuron, Neti=∑jyj∗wj, where jis the index of the neurons in the layer preceding the current neuron. • Stage 2. DeepFM feedforward Once the DeepMF has learned, we can collect the embedding representation of each user and each item in the CF RS. 11 Let E∗={<u,[eu 0,eu 1,. . . ,eu E]>,∀u∈U}, be the set of embeddings for all the RS users. (u∈[1...#U], one to u) 12 Let E∗(u) = [eu 0,eu 1,. . . ,eu E] 13 Let E∗∗ ={<i,[ei 0,ei 1,. . . ,ei E]>,∀i∈I}, be the set of embeddings for all the RS items. (i∈[1...#I], one to i) 14 Let E∗∗(i) = [ei 0,ei 1,. . . ,ei E] • Stage 3. Setting the dataset of embeddings 15 Let R= [<E∗(u),E∗∗(i),v>],∀<u,i,v>j∈Sbe the embedding-based dataset of real samples. • Stage 4. GAN training 16 Let fDbe the discriminator D model belonging to a GAN model. 17 Let fGbe the generator G model belonging to a GAN model. 18 Let fGD be the optimization function of the GAN model; fGD =MinGMaxDf(D,G) = ER[log(D(R))]+Ez[log(1−D(G(z)))], where ERis the expected value for real samples, zis the random noise that feeds the generator G, and Ezis the expected value for the generated fake profiles G(z). Note that Rrefers to [15]. 9 train the model and test the results is the same for both the real and generated datasets; ensuring the consistency of the graphs in the comparative figures (Figures 4a, 4b, 7b, 7e, 8b and 7e). Table 3 shows the GAN generated synthetic datasets used to test the proposed GANRS method, using Netflix* as source data. The ’#’ columns show the number of the generated datasets; ’std’ is the standard deviation used in the random noise Gaussian distribution; #users and #items are the total number of users and items chosen to generate each dataset; #samples is the number of fake samples created by the GAN generator. Please note that the final number of samples contained in each of the datasets is lower than #samples, due to the removing process of repeated samples. Cases 1 to 15 in Table 3 are used to test the effect of changing standard deviation and number of users. Cases 16 and 17 test the consequences of increasing the number of items. Finally, cases 18, 19 and 20 test the behavior of the synthetic datasets when they have different sizes (number of samples). All generated datasets and the source code of the proposed GANRS method are fully available in http://suleiman.ujaen.es: 8061/gitlab-instance-981c80cc/ganrs. Additionally, Appendix B (Figure .9) shows an example of the distribution graphs obtained for each of the synthetic datasets. Following the link provided, each generated dataset is located in its specific directory where a ’readme.txt’ file is provided along the synthetic dataset distribution graphs. # std #users #items # std #users #items # std #users #items #samples 1 2.0 100 4000 6 2.5 100 4000 11 3.0 100 4000 1.5M 2 2.0 1000 4000 7 2.5 1000 4000 12 3.0 1000 4000 3 2.0 2000 4000 8 2.5 2000 4000 13 3.0 2000 4000 4 2.0 4000 4000 9 2.5 4000 4000 14 3.0 4000 4000 5 2.0 8000 4000 10 2.5 8000 4000 15 3.0 8000 4000 16 1.5 4000 2000 17 1.5 4000 8000 18 1.2 2000 4000 150K 19 1.2 2000 4000 500K 20 1.2 2000 4000 1M 21 1.2 2000 4000 3M Table 3: Parameter values of the synthetic datasets generated by GAN. Source: Netflix*. Using the parameter values of Table 3, a variety of experiments have been conducted. The classification of the experiments is as follows: 1. Number of users (a) Distribution of users versus ratings (b) Distribution of the user ratings (c) Number of repeated samples (d) Proportion of samples with the same user and item (e) MAE and accuracy of the data set (f) Users’ precision and recall 2. Number of items 16 (a) MAE and accuracy of the dataset (b) Item’s precision and recall 3. Number of samples (a) Number of samples generated (b) Precision and Recall These experiments refer to well-known metrics in collaborative filtering. Precision is focused on measuring the proportion of relevant recommendations (i.e. the user rated the item with a rating value equal or greater than a threshold θ) among the top N items recommended to the user u, collected in the list TN u(Equation 4). On the other hand, Recall measures the proportion of correctly predicted relevant recommendations among the total number of relevant votes of each user; therefore, recall is sensitive to the existing proportions of relevant ratings (Equation 5). Precision =1 #U∑ u∈U |{i∈TN u|rui ≥θ}| N(4) Recall =1 #U∑ u∈U |{i∈TN u|rui ≥θ}| |{i∈TN u|rui ≥θ}|+|{i/∈TN u|rui ≥θ}| (5) Where Uis the set of training users, rui is the rating of the training user ufor the item i,Nis the number of recommendations, and TN uis the set of Nrecommendations for the test user u:Nhighest predictions of the user uabove the relevancy threshold θ. Please note that Precision measures the proportion of recommendation hits (hits with respect to number of recommendations), whereas Recall measures the proportion of recommendation hits with respect to the total number of relevant items). Precision takes into consideration the number of true positives, whereas Recall combines both the true positives and the false negatives. The importance of the precision and the recall quality measures largely depends on the scenario in which they are applied, e.g. Recall seems to be crucial in medicine, where a false negative is a serious mistake (i.e. not detecting cancer). Nevertheless, Recall is less important in RS since missing a relevant film (false negative) is not serious; the objective is to maximize a correctly recommended film (true positives). The F1 quality measure combines both Precision and Recall (Equation 6). F1=2∗Precision ∗Recall Precision +Recall (6) Finally, this paper also tests accuracy (Equation 7), where true negatives are also considered. In this case both the positive and the negative hits contribute to the results (to positively recommend relevant items and to negatively recommend non relevant items). Accuracy =|{i∈St|pui ≥θ∧rui ≥θ}|+|{i∈St|pui <θ∧rui <θ}| |St|(7) 17 Where Stis the set of test samples, and each sample is the tuple <u,i,r>containing the user ID, item ID and rating of the user u for the item i(rui). The model prediction of the rating is pui. Two values of the threshold θwill be explored across the experimental scenario when precision, recall, F1, and accuracy quality measures are tested Note that the accuracy quality measure does not use the term TN usince the typical RS does not include negative recommendations. Accordingly, this accuracy formulation does not average users’ results and acts on the entire training data, such as we have done with the Mean Absolute Error (Equation 8). MAE =1 |St|∑ s∈St |pui −rui|,u,i∈S(8) 4.2. Results This subsection shows the graphs obtained when the designed experiments (previous subsection) are run. The synthetic datasets described in Table 3 are used to obtain results that allow us: 1) to compare the distributions of users, items and ratings belonging to the source datasets, in relation to those obtained using the synthetic datasets, 2) to measure the number of repeated samples returned in the clustering stage, and 3) to test the prediction and recommendation qualities and trends obtained by running the proposed RSGAN method and comparing them to those shown by the source datasets. 4.2.1. Experiment 1a. Number of users: Distribution of users versus ratings Figure 4a shows the density of users (y-axis) that have cast different numbers of votes (x-axis). (Selected datasets: 8 and 10 in Table 3). As expected, for a fixed number of ratings in the dataset, we can observe that the higher the number of users, the lower the number of ratings. If the fixed number of samples in the dataset is distributed among a high number of users, each user centroid in the clustering stage receives a lower number of samples. Please, note that Netflix* contains around 23000 users. 4.2.2. Experiment 1b. Distribution of the user ratings Figure 4b shows the proportion of each rating 1,. . . ,5 (x-axis) when different random noise Gaussian distributions are applied. (Selected datasets: 3, 8 and 13). It can be observed that the standard deviation 2.5 generates a more similar distribution of votes, compared to the Netflix* original , than the adjacent standard distributions 2 and 2.5. Figure 4b also shows the impact of the Gaussian standard deviation in the layout of the individual values of the GAN-generated samples. 4.2.3. Experiment 1c. Number of repeated samples As explained in the ’Method’ section, the trained GAN generator predicts from random noise vectors as many dense samples as we want; all these samples are then converted from continuous dense values to discrete sparse ones. In the discretization process, repeated samples will appear that must be removed (Table 1 contains an example). Figure 4c shows the number of samples remaining in the dataset after the removal process. The lower the number of users, the higher the number of samples 18 Figure 4: a) Distribution of users versus ratings. Number of items: 4000. Datasets 8 and 10 in Table 3; b) Distribution of user ratings. Number of users: 2000; number of items: 4000. Datasets 3, 8 and 13 in Table 3; c) Number of samples remaining after removing the repeated ones. items: 4000. Datasets 6 to 10 in Table 3; d) Proportion of samples in which the same user has cast different votes for the same item. items: 4000. Datasets 6 to 10 in Table 3; e) MAE and accuracy. Number of users: 2000; number of items: 4000; ‘std’ is the standard deviation of the Gaussian random noise distribution. Datasets 1 to 15 in Table 3; f) Precision, recall, and F1. Standard deviation of the random noise Gaussian distribution: 2.5. Number of recommendations N = [2,4,6,8,10]. Datasets 6 to 10 in Table 3. assigned to each user (to its centroid in the clustering process), and therefore the higher the probability of repeating discrete samples. Overall, the smaller the number of users, the smaller the number of remaining samples. Selected datasets: 6 to 10 in Table 3. 4.2.4. Experiment 1d. Proportion of samples with the same user and item The GANRS generated datasets possess one attribute that does not exist in the source datasets (Movielens, etc.): they contain a proportion of samples where the same user has cast different votes for the same item; e.g.: <348,90,5>,<348,90,4>, as explained in the ’Method’ section. This can be seen as a mechanism to allow intermediate votes (4.5 in the example) or to allow users to change their minds. This makes sense if, the number of repeated votes is two or three. The rare cases of four or five repeated votes should be removed, just as we have done in all the generated datasets. From the standard quality metrics to measure the accuracy of predictions: Mean Absolute Error (MAE) and Root Mean Square Error (RMSE), we have chosen the former since it is the most widely used in RS state-of-art research. Some papers provide both measures, but experimental research shows that in the CF field, results for RMSE and MAE are very similar. This is because the distribution of the errors in the CF field usually has little variance. The MAE returns the absolute difference between the 19 predicted values and the real values in the testing set: MAE =1 n∑i|yi−ˆyi|. The lower the MAE, the better the model fits a dataset. The RMSE uses the square of the error instead of the absolute value: MAE =1 n∑i(yi−ˆyi)2; therefore, the RMSE is more sensitive to observations that are further from the mean, and this is not the case in CF. Figure 4d shows that for regular CF RS (1000 or more users), the proportion of four or five repetitions is not significant, and as the number of users increases, the proportion of repetitions drops very fast. 4.2.5. Experiment 1e. MAE and accuracy of the dataset Whereas the previous experiments analyze the internal composition and distribution of the synthetic datasets, this experiment and the following experiment test the behavior of the generated datasets on the prediction and recommendation tasks. Figure 4e shows the prediction quality (MAE) and the accuracy of the recommendation obtained from each set of individual samples in Datasets 1 to 15 in Table 3. Please note that these measures are not obtained by analyzing and averaging the results of users. The graphs in Figure 4e show an improvement in accuracy (and its corresponding decrease in MAE error) as the number of users increases. This behavior is expected in the CF RS, where a high number of users leads to better predictions, and it tells us that the GAN-generated samples follow a CF convenient pattern. The MAE values in the top graph of Figure 4e are closely related to the distribution of ratings for each of the standard deviations 2.0, 2.5 and 3.0. MAE/accuracy results can be used to select the most appropriate standard deviation; in this case: std = 2.5. 4.2.6. Experiment 1f. Users’ precision and recall This experiment provides the most significant results to test the generated datasets: we extract the values and evolutions of two representative recommendation quality measures: precision and recall. The top graphs in Figure 4f show the quality values obtained testing several numbers of recommendations N: [2,4,6,8,10] (x-axis), two different relevancy thresholds θ: [4,5], and two number of users: 2000 (green lines), and 8000 (blue lines). The standard deviation of the Gaussian random noise has been set to 2.5. Selected datasets: 6 to 10 in Table 3. The values and evolutions obtained from the synthetic datasets fit with the source dataset: Netflix* (black lines). Additionally, as expected, the overall results of the dataset generated by 8000 users outperform those of the 2000 users and are closer to the Netflix* reference (please note that Netflix* contains around 23000 users). The two bottom graphs in Figure 4f represent the F1 combination of precision and recall; they clearly show the similarity in the behavior of the generated datasets compared to the source dataset. 4.2.7. Experiment 2a. MAE and accuracy when the number of items varies. Experiment 1e tested MAE and accuracy quality measures on datasets with different numbers of users. Now we will test both quality measures on datasets with different numbers of items: [100, 1K, 2K, 4K, 8K]. The results in Figure 5 show adequate values for both MAE and accuracy, and consistent evolutions where accuracy increases and MAE decreases as the number of items (x-axis) increases. Thus, the higher the number of items, the better the accuracy: this shows that the GAN generator can enrich the data. The Netflix * source dataset contains 1,750 items and we can observe in Figure 20 Figure 5: a) MAE and accuracy obtained from the dataset samples when the number of items varies. Number of users: 4000. Standard deviation of the Gaussian random noise: 1.5. Datasets 16 and 17 in Table 3; b) Precision, recall, and F1 when the number of items varies. Standard deviation of the random noise Gaussian distribution: 1.5. Number of recommendations N = [2,4,6,8,10]. Datasets 16 and 17 in Table 3. 5a how the improvement slows down around this value (x-axis). Selected datasets: 16 and 17 in Table 3, and the 100, 1000, 4000 user versions not included in Table 3. 4.2.8. Experiment 2b. Items’ precision and recall Experiment 2b is similar to Experiment 1f; now we will test the behavior of datasets that contain different numbers of items (instead of different numbers of users). Figure 5b shows the performance of Netflix* (1750 items), represented using black lines, and compares it with the 2000 item dataset (green lines) and the 8000 item dataset (blue lines). We can observe that evolutions and values are consistent with the source datasets (black lines); furthermore, both the 2K and 4K items versions perform well: the first one conveniently captures the Neflix* patterns of items, since both contain a similar number of items. The dataset generated second (8K items) can enrich the data and 21 Figure 6: a) Number of generated samples using different number of users (x axis) and different number of GAN generated samples (legend). Standard deviation of the random noise Gaussian distribution: 1.2. Number of items: 4000. Datasets 18 to 21 in Table 3; b) Precision and recall using a different number of recommendations (x axis) and a different number of GAN generated samples (legend). Standard deviation of the random noise Gaussian distribution: 1.2. Datasets 18, 19 and 21 in Table 3. show better accuracy than the 2K items version. Selected datasets: 16 and 17 in Table 3, and the 100, 1000, 4000 user versions not included in Table 3. 4.2.9. Experiment 3a. Number of samples generated in datasets with different sizes Here we will test the number of samples that the GANRS method obtains when different numbers of generated samples and different numbers of users have been set. For this purpose, we define four different numbers of samples: 150K, 500K, 1M and 3M (Datasets 18 to 21 in Table 3, and their equivalent datasets for 100, 1000, 4000 and 8000 users) in the GAN generation process. The number of items is fixed at 4K for all experiments. In Figure 6a we can observe that the smaller the number of users, the smaller the number of generated samples; this is due to the fact that the smaller the number of users, the higher the number of samples assigned to each user (to each centroid in the clustering stage), and therefore the higher the probability of repeated samples that will be removed. As an example, Figure 6 shows that the 8K user dataset preserves, approximately, 1M samples from the GAN generated (version 3M), and 600K in version 1M. 4.2.10. Experiment 3b. Precision and recall on datasets with different sizes This experiment shows the impact of increasing the number of samples in datasets with fixed parameters, in this case: 2000 users, 4000 items, and a standard deviation of 1.2 (Table 3; Datasets 18, 19 and 21). It is important to realize that we are using the same source dataset Netflix* to generate the three cases shown in Figure 6b: 150K 22 samples (yellow lines), 500K samples (magenta lines), and 3M samples (red lines). Please note that 150K, 500K and 3M samples refer to the dense and continuously generated samples, prior to the removal stage to convert them into their sparse, discrete version. Figure 6a shows the final sizes of the datasets in the 2000 user data (x-axis). Figure 6b compares the precision and recall values obtained in the Netflix* dataset (black lines) with the generated values. Overall: 1) precision increases and recall decreases; 2) the bigger the generated dataset, the better its precision; 3) the higher the dataset, the lower its recall. Precision results improve when using large datasets, as there are more relevant samples to choose from, and therefore it is easier to succeed in the fixed number N of recommended predictions. On the other hand, recall gets worse using large datasets because they contain more variability in the samples, particularly when large standard deviations have been chosen for the random noise Gaussian distribution. Unlike precision, whose denominator is the constant N (number of recommendations), the recall quality measure depends on the variable: ‘number of relevant votes’ in the set of test items for each user tested. As the number of samples increases, the number of user votes also increases (and, from them, the number of relevant votes); this is the reason why recall is lower in the 3M synthetic dataset in Figure 6b, and higher in the 150K version. Figures 7 and 8 show, respectively, the results obtained from the MyAnimeList and Movielens 100K test datasets. Graph ’a)’ compares the rating distribution of each source dataset (in blue) with the generated rating distributions obtained by setting different values of the Gaussian random noise standard deviation. We have chosen the standard deviation value of 1.2 for MyAnimeList, and the standard deviation value of 2.5 for Movielens 100K, since the obtained distributions of ratings are closest to their respective baselines. Results ’b)’, ’c)’ and ’e)’ are obtained using the selected standard deviation values. Graph ’b)’ shows the distribution of users according to their number of casted ratings (x-axis). As expected, they follow the same pattern as the one in Netflix*. To compare results, please note that MyAnimelist dataset contains 19179 users, and Movielens 100K contains 943 users. Graph ’c)’ shows the number of samples left after removing repeated instances. The higher the number of users, the lower the probability of generating samples containing the same user ID, item ID, and rating. In the MyAnimeList case, we started with 1.5 million generated samples, whereas for Movielens we selected 1 million generated samples. Graph ’d)’ refers to MAE error and accuracy values obtained by processing the individual samples contained in each dataset. As usual in the CF context, the higher the number of users, the lower the error, and the higher the accuracy. Finally, Graphs ’e)’ tests the recommendations obtained by processing the users in each dataset. As is with Netflix*, compared to baselines, precision improves and recall gets worse. The results obtained in this section highlight the importance of those that test the performance of the synthetic datasets against the source datasets, particularly when specific RS metrics are used. To check the consistency between synthetic and real data, two types of experiments have been conducted: direct and indirect. In direct comparisons, rating distributions have been obtained and compared from both source datasets and their synthetic versions. Figures 4b and 4c show the Netflix* results by varying the number of generated users and the standard deviation of the random Gaussian distribution used to feed the proposed GAN. Figures 7b and 8b show, respectively, 23 Figure 7: MyAnimeList. 1.5 million generated samples, a) distribution of the MyAnimeList ratings 1 to 10, b) distribution of users according to their number of casted ratings, c) number of samples after the removing process of the repeated ones, d) error and accuracy by processing the samples of the dataset, e) CF precision and recall (by testing the dataset users). The GANRS std=1.2 value has been set to test experiments b) to e). comparison of the MyAnimeList and the Movielens 100K datasets, in this case by varying the number of users in the synthetic datasets versus their equivalent source counterparts. Indirect experiments tested and compared the recommendation performance on both the synthetic and the source datasets. We have chosen the recommendation quality measures of precision, recall and F1, obtained using the classical neural model DeepMF. Results can be found in Figure 4f (Netflix* vs. its synthetic version), Figure 7e (MyAnimeList vs. its synthetic version), and Figure 8e (Movielens 100K vs. its synthetic version). Overall, as expected, the results show that synthetic datasets behave like their source datasets. The more similar the results are, the more suitable the generated datasets will be, as this means that the original datasets can be effectively replaced by synthetic ones. 24 4.3. Comparison of the proposed framework with previous work The Related Works section identifies some previous work focused on data generation methods for recommender systems. In this subsection, a brief analysis will be performed, which will focus on showing how this previous work is not truly comparable with our current proposal in a fair way, since it is focused on different objectives and also generates data of a different nature. • Mladenov et al. [34] presented RecSim NG, an architecture centered on the generation of synthetic profiles of users and items as part of the recommendation environment. Overall, the goal of the work is the development of a configurable platform for both authoring and learning RS simulation environments. The aim of this simulation is to evaluate existing RS policies, or generate data to train new policies (in either a tightly coupled online fashion, or in batch mode). Furthermore, this paper lacks information about the presented method, and therefore does not allow reproducibility. • Shi et al. [42] introduce a multi-agent reinforcement learning architecture tailored to Taobao-specific website search improvement, and uses a GAN to simulate the internal rating distribution. Therefore, considering that it is focused on data generation for a specific context, it is not comparable with the framework proposed in the current paper. • Del Carmen et al. [11] introduce DataGenCars, a Java-based generator of RS synthetic data. Here it is important to remark that this work is specifically focused on the context-aware recommendation scenario. In this sense, even though the proposed tool supports the generation of synthetic datasets of users, items, contexts, and ratings; this generation always relies on context-related characteristics through criteria introduced throughout the work, such as the uncertainty of the content, the user’s expectations or the item’s attributes. As result, this work is not comparable with the methodology presented in our current paper, which mainly uses rating values as input and does not consider datasets with contextual information. • Provalov et al. [39] introduce the SynEvaRec framework, focused on the presentation of a novel paradigm for evaluating recommendations based on the generation of synthetic RS datasets. In contrast to our current paper, this approach is mainly focused on generating synthetic user and item profiles that are internally used by SynEvaRec to guarantee user privacy protection, mitigate the data insufficiency problem, and measure the effect of the no-free-lunch problem. Regarding the aim of the architecture proposed in Provalov et al. [39] is not the proper retrieval of the whole synthetic rating datasets to be used in further evaluations (i.e. an evaluation protocol is presented rather than a dataset generation method), a major transformation of their work is needed to make it comparable with this paper. A fair comparison is then not possible at this stage. 25 [51] Xue, H.-J., Dai, X., Zhang, J., Huang, S., & Chen, J. (2017). Deep matrix factorization models for recommender systems. In IJCAI (pp. 3203–3209). Melbourne, Australia volume 17. [52] Yera, R., Alzahrani, A. A., & Mart´ ınez, L. (2022). Exploring post-hoc agnostic models for explainable cooking recipe recommendations. Knowledge-Based Systems,251, 109216. [53] Yera, R., Alzahrani, A. A., & Mart´ ınez, L. (2022). A fuzzy content-based group recommender system with dynamic selection of the aggregation functions. International Journal of Approximate Reasoning,150, 273–296. [54] Yera, R., & Mart´ ınez, L. (2017). Fuzzy tools in recommender systems: A survey. International Journal of Computational Intelligence Systems,10, 776–803. [55] Zamany, S., Li, D., Fei, H., & Li, P. (2022). Towards deeper understanding of variational auto-encoders for binary collaborative filtering. In Proceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval (pp. 254–263). [56] Zhang, S., Yao, L., Sun, A., & Tay, Y. (2019). Deep learning based recommender system: A survey and new perspectives. ACM Computing Surveys (CSUR),52, 1–38. [57] Zhao, J., Li, H., Qu, L., Zhang, Q., Sun, Q., Huo, H., & Gong, M. (2022). Dcfgan: An adversarial deep reinforcement learning framework with improved negative sampling for session-based recommender systems. Information Sciences,596, 222–235. [58] Zhao, W., Wang, B., Ye, J., Gao, Y., Yang, M., & Chen, X. (2018). Plastic: Prioritize long and short-term information in top-n recommendation using adversarial training. In Ijcai (pp. 3676–3682). [59] Zheng, L., Noroozi, V., & Yu, P. S. (2017). Joint deep modeling of users and items using reviews for recommendation. In Proceedings of the tenth ACM international conference on web search and data mining (pp. 425–434). [60] Zhu, B., Hurtado, R., Bobadilla, J., & Ortega, F. (2018). An efficient recommender system method based on the numerical relevances and the non-numerical structures of the ratings. IEEE Access,6, 49935–49954. [61] Ziegler, C.-N., McNee, S. M., Konstan, J. A., & Lausen, G. (2005). Improving recommendation lists through topic diversification. In Proceedings of the 14th international conference on World Wide Web (pp. 22–32). Appendix A 32 DeepMF values Embedding size (both for users an items) 5 Optimizer Adam Loss function Mean squared error Epochs 20 GAN generator Input shape, noise vector size 100 Block 1 dense layer #neurons 10 Block 1 activation function LeakyRelu, alpha 0.2 Block 1 normalization BatchNormalization, momentum 0.8 Block 2 dense layer #neurons 20 Block 2 activation function LeakyRelu, alpha 0.2 Block 2 regularization Dropout 0.2 Block 3 dense layer #neurons 2 ∗embedding size +1 Block 3 activation function linear GAN discriminator Input: shape 2 ∗embedding size +1 Block 1 dense layer #neurons 6 Block 1 activation function LeakyRelu, alpha 0.2 Block 2 dense layer 1 Block 2 activation function Sigmoid GAN train Epochs 20 Batch size 64 Stochastic noise Gaussian (0,1) Loss function (real samples loss +f ake samples loss)/2 Table .4: Main parameter and hyperparameter values set for the neural models involved in the RSGAN method. Appendix B 33 Figure 8: Movielens 100K results. 1 million generated samples, a) Distribution of the Movielens 100K ratings 1 to 5, b) Distribution of users according to their number of casted ratings, c) Number of samples after the removal process of the repeated ones, d) error and accuracy by processing the samples of the dataset, e) CF precision and recall (by testing the dataset users). The GANRS std=2.5 value has been set to test experiments b) to e). 34 Figure .9: Main distributions of the data in the synthetic dataset generated from Movielens 100K compared to the distributions of the data in the source dataset. Number of users: 8000, number of items: 4000, initial number of samples: 800000, standard deviation of the gaussian noise: 2.5. Graph a) shows the distribution of the fake users (y axis) versus the number of ratings belonging to each of the users (x axis). Graph b) shows the distribution of the fake items (y axis) versus the number of ratings belonging to each of the items (x axis). Graph c) shows the percentage of ratings (y axis) for each of the available vote values 1, 2, 3, 4, 5 (x axis) in the dataset. 35