Deep reinforcement learning based medical supplies dispatching model for major infectious diseases: Case study of COVID-19

Zeng, Jia-Ying; Lu, Ping; Wei, Ying; Chen, Xin; Lin, Kai-Biao Article Deep reinforcement learning based medical supplies dispatching model for major infectious diseases: Case study of COVID-19 Operations Research Perspectives Provided in Cooperation with: Elsevier Suggested Citation: Zeng, Jia-Ying; Lu, Ping; Wei, Ying; Chen, Xin; Lin, Kai-Biao (2023) : Deep reinforcement learning based medical supplies dispatching model for major infectious diseases: Case study of COVID-19, Operations Research Perspectives, ISSN 2214-7160, Elsevier, Amsterdam, Vol. 11, pp. 1-19, https://doi.org/10.1016/j.orp.2023.100293 This Version is available at: https://hdl.handle.net/10419/325778 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ Operations Research Perspectives 11 (2023) 100293 Available online 23 November 2023 2214-7160/© 2023 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/bync-nd/4.0/). Deep reinforcement learning based medical supplies dispatching model for major infectious diseases: Case study of COVID-19 Jia-Ying Zeng a , Ping Lu a , * , Ying Wei b , Xin Chen c , Kai-Biao Lin b a School of Economic and Management, Xiamen University of Technology, Xiamen 361024, China b Department of Computer Science and Technology, Xiamen University of Technology, Xiamen 361024, China c School of Data Science and Computer Science, Xiamen Institute of Technology, Xiamen 361021, China ARTICLE INFO Keywords: Major public health event Medical supplies dispatching Deep reinforcement learning Epidemiological model ABSTRACT Stockpiling and scheduling plans for medical supplies represent essential preventive and control measures in major public health events. In the face of major infectious diseases, such as the novel coronavirus disease (COVID-19), the outbreak trend and variability of disease strains are often unpredictable. Hence, it is necessary to optimally adjust the prevention and control dispatching strategy according to the circumstances and outbreak locations to maintain economic development while ensuring the human health survival, however, many models in this scenario seldom consider the dynamic material prediction and the measurement of multiple costs at the same time. Taking the COVID-19 scenario as a case study, we establish a deep reinforcement learning (DRL)- based medical supplies dispatching (MSD) model for major infectious diseases, considering the volatility of the COVID-19 situation and the discrepancy between medical material demand and supply due to the high infectiousness of the Omicron series strains. The present model has three main components: 1) First, for the dynamic medical material prediction problem in complex infectious disease scenarios, taking the lifted COVID-19 lockdown scenario as an example, the modified susceptible-exposed-infected-recovered (SEIR) model was utilized to analyze the spread of the COVID-19, understand its characteristics, and map out the related medical supplies demand; 2) Second, to break away from the previous premise of only considering supply-demand, this study adds scheduling rules and cost function that weighs health and economic costs. An epidemic dispatching optimization model (Epi_DispatchOptim) was established using the OpenAI Gym toolkit to form an environment structure with virus transmission space, and emergency MSD while considering both human health and economic costs. This architecture interprets the balance between the supply-demand of medical supplies and reflects the importance of MSD in the balanced development of health and economy under the spread of infectious diseases; 3) Finally, the MSD strategy under the balance of health and economic cost is explored in Epi_DispatchOptim using reinforcement learning (RL) and the evolutionary algorithm (EA). Experiments conducted on two datasets indicate that the RL and EA reduce economic as well as health costs compared to the original environmental strategies. The above study illustrates how to use epidemiological models to predict the demand for healthcare supplies as the premise of scheduling models, and use Epi_DispatchOptim to explore the dynamic MSD decisions under mortality and economic equilibrium. In Shanghai, China, the economic cost of the exploration strategy is reduced by 27.36–27.07B compared to static scheduling, and deaths are reduced by 126–150 in 150 day compared to the no-intervention scenario. By integrating knowledge of epidemiology, optimal decision making, and economics, Epi_DispatchOptim further constructs epidemiological models, cost functions, state-action spaces, and other modules to assist public health decision makers in adopting appropriate MSD strategies for major public health event. 1. Introduction Novel coronavirus disease (COVID-19) was a public health emergency with rapid transmission, widespread infection, and difficulty in prevention and control in the last 100 years [1]. Compared to other coronavirus (CoV) strains, such as severe acute respiratory syndrome * Corresponding author. E-mail address: [email protected] (P. Lu). Contents lists available at ScienceDirect Operations Research Perspectives journal homepage: www.elsevier.com/locate/orp https://doi.org/10.1016/j.orp.2023.100293 Received 6 August 2023; Received in revised form 12 November 2023; Accepted 22 November 2023 Operations Research Perspectives 11 (2023) 100293 2 (SARS) and middle east respiratory syndrome (MERS), COVID-19 exhibits different infectious characteristics. As shown in Table 1, although the fatality rate of COVID-19 is lower than SARS and MERS, has wider range of infection, a large confirmed base, and a significantly higher death toll than both. With the advancement of knowledge about COVID-19 and the implementation of effective prevention and control measures, China has avoided large-scale epidemics of the more pathogenic original and Delta strains, and thus reduced severe illnesses and deaths [2]. However, the highly infectious Omicron strain, which is globally prevalent, still exerts a negative impact on human health. Between December 8th, 2022, and January 12th, 2023, the cumulative in-hospital COVID-19 infection-related deaths in China reached 54,435 [3]. The Omicron strain especially poses a threat to the lives of elder patients and those with underlying diseases. In major infectious disease outbreaks, the adequacy of medical supplies in each region directly affects the prevention and control of infectious diseases [4]. In early March 2022, the demand for COVID-19 emergency medical supplies in Shanghai, China, surged to more than four times its original demand. As regions adopt lifting lockdown strategies, confirmed diagnoses will surge in densely populated countries or cities without implemented non-pharmaceutical interventions (NPI), such as lockdown, isolation, and social distancing. This will be an unprecedented test of the medical supplies’ security system [5]. The shortage of medical supplies will affect the health of medical workers and patients, as well as the economic development and social stability of the country, and in some cases lead to a run on supplies and panic [4]. The timely supply and rational use of medical supplies is crucial for the prompt control of infectious diseases and public health [6], such that it is necessary to establish a special reserve plan, dynamic monitoring, and real-time dispatching of regular drugs related to infectious diseases [7]. Studies on MSD in COVID-19 were mainly performed using qualitative methods [8,9] in the early stage. Wang et al. [8] discussed the emergency dispatching model of medical human resources in Hunan Province, China, based on a single-case study, which inductively illustrated the importance of optimizing the medical resource classification system and improving emergency resource reserves. Yu et al. [9] emphasized that big data must be used to achieve information sharing and implement relief and reserve dispatching in different regions through production and reserve plans. Although qualitative and empirical methods [10] provide a reference for COVID-19 MSD, they only have a guiding role and assess the effect of influencing factors. For example, Wang et al. [10] based on the health production function perspective, showed that improving the level of medical resources reserves can reduce the infection rate of COVID-19 and shorten the prevention and control time. However, there are no material dispatching rules and dispatching process models proposed in these studies to meet realistic needs. As the understanding of COVID-19 and prevention and control measures in each region grew, quantitative methods such as system dynamics (SD) [11,12], operations research (OR) [13–15], and machine learning (ML) [6,16,17] took hold in the research: (1) SD method is mainly constructed the modified SEIR model to understand the relationship between the epidemic extent and material supply and its influencing factors [12]. At the same time, the importance of early planning and real-time adjustment of medical resources was illustrated [11], but the specific material dispatch process was still not established; (2) It is mainly realized by goal programming and dynamic programming in the OR method. Jana et al. [13] and Yang [15] proposed to establish a framework for critical medical hierarchical resource allocation using an objective planning mathematical model and to formulate a mixed integer linear programming model to solve the MSD prediction and planning problem under COVID-19, respectively. Both above established specific material scheduling processes; However, their scheduling environments are completely knowable environments that are not subjected dynamic real-time scheduling conditions and lack stochasticity. In contrast, Wang et al. [14] used a Markov decision processes (MDP) model to establish appropriate medical supply measures in COVID-19 and achieved dynamic dispatching, taking the lockdown scenario in Wuhan, China, as an example. Nevertheless, this model mainly utilizes the pre-defined state transfer matrix to express the environmental development dynamics, without considering the stochastic nature of environmental state transfer; (3) For an unknown environment with MDP properties, its transfer probability and influencing parameters can be explored using reinforcement learning (RL). Bednarski et al. [17] initially explored the optimal reallocation scheme for COVID-19 critical medical equipment (ventilators) by RL and deep learning (DL), which proved superior to heuristic algorithms, such as maximum demand priority, minimum demand priority, and random order, in solving the supply-demand problem. While Awasthi et al. [16] proposed to construct a COVID-19 vaccine allocation simulation model using deep reinforcement learning (DRL) and contextual bandits to solve the vaccine supply and demand scheduling balance problem. They further demonstrated that the vaccine distribution strategy is closely related to the susceptible population. The above model illustrates that DL and RL methods can optimize MSD through scheduling rules and reward function settings, providing a solution to the RL framework for MSD under major infectious diseases. Nevertheless, it mainly addresses the supply and demand allocation problem without considering economic factors. In major infectious diseases, it is necessary to consider not only the supply-demand of medical supplies, but also the economic cost in the distribution process to solve the problem of "open-source throttling" and improve the efficiency of resource utilization [6]. In summary, this study is divided into two parts to address the above issues. First, to address the dynamic prediction of medical demand in the evolution of epidemic, we construct a dynamic environment of epidemic based on the Susceptible-Infected-Recovered (SIR) series model, and make predictions of the relevant medical demand by understanding the dynamic of different entity. Second, we incorporate health and economic considerations in addition to focusing on supply-demand issues, and utilize the DRL method to build an auxiliary model for MSD that weighs public health and economic elasticity. The contributions of this study are as follows: (1) COVID-19 is adopted as an example of a major infectious disease, considering the first peak strain of the China-lifted lockdown as the Omicron BF.7 subtype, and the fact that the characteristic parameters of this strain are currently difficult to obtain [18]. To address this situation, in this study, the SIR series models are expanded into the SEIRAHD model using Omicron BF.2 for the analysis of COVID-19 spreading characteristics and medical supplies prediction; (2) Based on the premise of the above epidemiological modeling, this study constructs the epidemic dispatching optimization model (Epi_DispatchOptim) based on the Python toolbox of Colas et al. Table 1 Comparison of confirmed cases, fatality, and distribution of coronaviruses. Name COVID-19 MERS SARS Year 2019 2012 2003 Confirmed 675,053,734 2,494 8,096 Deaths 6,870,344 858 774 Fatality ( %) 1.02 34.1 9.6 Countries 222 27 26 J.-Y. Zeng et al. Operations Research Perspectives 11 (2023) 100293 3 [19], which breaks through the original framework to consider only the lockdown strategy to build an auxiliary model for optimizing the MSD under major infectious diseases, this model not only focuses on the supply-demand issue, but also weighs the health and economic costs; (3) To better adapt to the complexity of medical supply-demand under large-scale infectious diseases, we explore MSD strategies under economic efficiency and public health tradeoffs in Epi_DispatchOptim using DRL and EA, before validating them using two datasets. This study is structured as follows: Section 2 presents related work; Section 2.4 presents the main principles of the Epi_DispatchOptim model construction; Section 3 analyzes infectious disease dynamics and forecasts medical needs using the SEIRAHD; Section 4 introduces Epi_DispatchOptim for medical dispatching of infectious diseases with economic and public health benefit tradeoffs; Section 5 explores the optimal decision of MSD after lifting the lockdown in Epi_DispatchOptim using the deep Q-network (DQN) and NSGA-II, by evaluating and analyzing examples from France and Shanghai, China; Relevant recommendations, unresolved issues and future work are discussed in Section 6; Finally, to better explain the key variables in this study, variable descriptions are summarized in Appendix A. 2. Related work For the construction of overall model, we have the following arrangement to related work, firstly, to understand the development trend of epidemic and the forecast of related medical demand, Section 2.1 describes the role and related application of epidemiological models; Secondly, to understand the related research on COVID-19 prevention and control measures, Section 2.2 points out the role and related application of the AI method in the infectious diseases; Lastly, as there are fewer researches on the application of RL to the scheduling under the major infectious diseases, Section 2.3 introduces the application and advantage of the RL in the resource scheduling and get the inspirations. 2.1. Dynamic prediction of infectious diseases and simulation models Medical needs are closely related to various circumstances of infectious disease outbreaks, such as the population base, infection rate, and asymptomatic cases [12]; therefore, an accurate understanding of infectious disease transmission mechanisms is the basis for effective scheduling and planning of medical requirements. For example, since the COVID-19 outbreak, this virus with widespread transmission and high insidiousness has posed an unprecedented test to the emergency medical material security system in cities with high population density and mobility, as well as a major challenge to the normal economic development of society in general [5]. Therefore, the establishment of dynamic predictions of infectious diseases and simulation mechanisms is in demand to better handle future major infectious diseases like COVID-19. The SIR series models, representing a milestone in the establishment of mathematical models for infectious diseases, were proposed by Kermack and McKendrick [20] in 1927, and are still widely used today. The SIR model is a compartmental model that divides the population in different stages of the disease to predict and understand the transmission trend of infectious diseases. In major infectious diseases, especially in COVID-19, SIR models have been applied to realize dynamic prediction [21–23], mitigation measures [24–26], and gain information on infectious disease pathology [27,28]: (1) For the dynamic prediction of infectious disease trends and material demand, Hackl et al. [21] simulated the daily activities of different populations by combining the SIR model with an agent-based model (ABM) to predict the trends of infected, recovered, etc., populations during the pandemic. Zhang et al. [23] considered the assessment of basic clinical resource requirements as a need for infectious disease control and used SEIRH, an extension of the SIR model, to predict and assess critical clinical material requirements in different scenarios based on the proportion of material requirements with different personnel; (2) In terms of mitigation measures, Capobianco et al. [24] constructed an agent-based epidemic simulation model (PandemicSimulator) through SIR to explore government lockdown strategies by simulating fine-grained interactions among different populations in society. Feng et al. [25] further used ABM and the SIR series model to simulate the physical distance behavior between people on educational building and evaluate distance strategies; Kai et al. [26] used the SEIR model, which can interpret the dynamics of COVID-19, and the ABM, which is more stochastic, to collaboratively demonstrate how the masks strategy significantly reduces the infection rate of COVID-19. Colas et al. [19] expanded SIR to the SEIRAH and used RL to explore the optimal lockdown control strategy under health and economic cost equilibrium; (3) In infectious disease pathology research, to better understand COVID-19 development, Xu et al. [22] used the SEIR model to simulate the COVID-19 transmission process in order to predict confirmed cases and the inflection point of outbreak development based on the daily posted dataset of confirmed cases; Kissler et al. [23] used the SIR series model to explore important characteristics of COVID-19 transmission, such as seasonality, immunity, and cross-immunity. In COVID-19 studies, the SIR series models play their role in epidemiological control decisions, especially in infectious disease transmission and material prediction. Infectious disease simulations can provide some ideas for future prevention and control under outbreaks of new variant strains and variant subtypes; however, early studies focused on original strains [22,23,26–28] and Delta strains [25] and could not be directly applied to the Omicron strain [18]. 2.2. AI-based infectious disease detection and control model In previous studies, more attention has been paid to the pathology of infectious diseases and transmission mechanisms, and less to the study of epidemic prevention and control mechanisms through technological methods, such as big data and artificial intelligence (AI). Unlike SARS, MERS, influenza, and other viruses that infected small geographical areas, COVID-19 caused global mass infection, which delivered a blow to human health and economic development all over the world, and initiated a wave of research and consideration of control decisions under large-scale infectious diseases. Facing the complexity and randomness of infectious diseases, ML algorithms such as DL and RL help us enhance our understanding of COVID-19 or other infectious viruses. AI is currently addressing COVID-19 from a molecular, clinical, and societal perspective, as described: (1) From the molecular perspective, novel or existing therapeutic drugs and potential vaccine targets can be identified through biomedical knowledge mapping, protein-ligand binding affinity prediction, and modeling of molecular docking [29]; (2) From the clinical perspective, AI and ML have examined diseases through data such as clinical images, in which Cao et al. [30] utilized a shape-aware method based on DL for rib fracture detection and segmentation via computed tomography images on the RibFrac dataset, which has a large detection sensitivity of 0.926, and plays an auxiliary role in clinical assessment and treatment; To assist physicians in the diagnosis of skin diseases, Xiao et al. [31] constructed a dual-flow modal alignment module J.-Y. Zeng et al. Operations Research Perspectives 11 (2023) 100293 4 through skin images from consumer IoT and clinical, and utilized few-shot learning network to diagnose dermatological disease, which was comparable to that of a clinician and improved accuracy by 11.2 % over the baseline model. In addition, AI generates patient predictions mainly through the diagnosis of COVID-19 by medical imaging, based on data such as electronic health records, and using alternative methods of tracking the disease evolution with non-invasive devices [29]; (3) From the societal perspective, AI is mainly applied in the areas of dynamic prediction [21] and control measures [19,25,32]. In dynamic prediction, Kumar et al. [21] used an improved long short-term memory model to predict individual infections, deaths, and recoveries of COVID-19 in the coming days, which also suggested using RL to optimize the prediction results of COVID-19 based on symptom data. Regarding control measures, the studies focused on lockdown measures, Colas et al. [19] employed the EpidemiOptim model based on SEIRAH, and explored the optimal strategy for lockdown control under the equilibrium of mortality and economic recession. Capobianco et al. [25] proposed the PandemicSimulator model to explore mitigation strategies to reduce economic impact without exceeding hospital capacity using RL. Both EpidemiOptim and PandemicSimulator provide a relevant framework for the integration of epidemiology, economics, and AI methods, illustrating the importance and promise of ML for the optimization of infectious disease detection and control, but they both emphasize lockdown strategies and lack of extension to the MSD problem. 2.3. DRL-based material intelligent scheduling optimization model The RL algorithm finds the decision with the largest reward based on the trade-off between present and future reward under environmental rules by trial and error, which holds some advantages in decision problems. In resource scheduling, RL has been successfully applied to scenarios such as container allocation [33,34], network resource allocation [35–37] and MSD [16,17,38]. In container allocation, Shi et al. [34] used the local actor-critic to improve agents’ ability to understand complex representations of adjacency points messages, enabling them to extract useful information from observations of heterogeneous interaction graphs to optimize redistribution scheduling decisions for empty containers in shipping; In network resource allocation, Deng et al. [35] modelled the server allocation resource problem as an MDP and dynamically generated appropriate allocation strategies based on the system state through RL to maximize the trustworthiness of the service. Also RL plays a role as a decision aid in Internet of Vehicles, Li et al. [36] used Multi-action and Environment-adaptive Proximal Policy Optimization algorithm to generate offloading decisions and priority assignment decisions to reduce completion time of service requests and transmission energy consumption in Vehicular Edge Computing. In order to break through the limitations of heuristic algorithms that are difficult to obtain the global optimum, and meta-heuristic algorithms that have many parameters that are inconvenient to train, Gao et al. [37] used MARL to enhance offloading performance in mobile edge computing through the collaboration of multiple agents, and outperformed other baseline methods in terms of energy consumption, load status, and latency. The material scheduling environment in the above problem has complex industry rules and is subject to dynamic instability, such that traditional OR are no longer applicable to the existing scheduling environment. Meanwhile, RL can better solve the decision-making problem in stochastic events [33]. In large-scale public health events, especially for major infectious diseases, which are unstable due to the intervention of prevention and control strategies, cross-regional population movement, and multi-site outbreaks, and which are consistent with the nature of stochastic events, DRL can be adopted to better explore the optimal strategy for the intelligent dispatch of supplies. For MSD, Hyun-Rok et al. [38] viewed the patient admission problem in a major disaster as a decentralized partial observable problem and used behavioural cloning to construct an initial strategy based on the demonstration sample, thus enhancing the RL strategy search capability. The algorithm effectively combines OR and RL to improve patient survival rates while reducing patient transfer rates compared to heuristic strategies. In contrast, studies on MSD for COVID-19 include Bednarski and Awasthi et al. [16,17], both of whom proposed RL-based MSD frameworks for infectious diseases that shed light on research in this direction. However, they focused more on resource allocation meeting the demand, aspiring to save more lives initially, and did not consider economic factor trade-offs with health costs. 2.4. Conclusion First, the developmental dynamics of major infectious diseases are closely related to MSD decisions. The SIR series models are capable of interpreting pandemic developmental dynamics, and thus they are effectively applied to control strategy problems. For a better simulation of COVID-19 transmission, this study expands SIR model to susceptibleexposed-infected-recovered-asymptomatic-hospitalized-deceased (SEIRAHD) model by adding asymptomatic (A), hospitalized (H), and deceased (D) compartments, which contribute to the understanding of the COVID-19 development and predict the medical needs of different populations, as well as to prepare the environment for MSD. RL has some advantages in solving sequential decision problems for stochastic events, and although it has been successfully applied to different scheduling allocation domains and lockdown control for infectious diseases, it has been less frequently applied to MSD strategy for major infectious diseases. Therefore, in this study, SEIRAHD is integrated into Epi_DispatchOptim for COVID-19, and MSD strategies are explored. Finally, although some studies employ RL to solve the problem of MSD strategies for infectious diseases, they mainly focus on the objective of supply-demand without considering economic factors. In turn, Epi_DispatchOptim focuses on the trade-off between health and economic costs by construction of model modules. To summarize, this study integrates the epidemiological model into RL framework, uses the SEIRAHD model to understand the epidemic dynamic development and related medical needs, as the initial environment of Epi_DispatchOptim, and constructs modules such as stateaction space, cost function, and optimization algorithm, in addition to the consideration of supply-demand issue, but also add the consideration of the health and economic costs, to explore the MSD strategy under the balance of health and economic cost. 3. Theoretical foundations 3.1. SIR epidemiological model The generic SIR model originated from Kermack and McKendrick’s study of the Black Death in London in 1927 [20]. This series of SIR models employs compartmentalization, which divides the population into compartments representing different disease states to understand epidemic transmission. The basic structure of the SIR model is shown in Fig. 1. SIR Epidemiological Model. J.-Y. Zeng et al. Operations Research Perspectives 11 (2023) 100293 5 Fig. 1, and uses three compartments to represent states: susceptible (S), which is not yet infected; infected (I), which is already infected and infectious; and recovered (R), which indicates recovered from the infected state and acquired immunity. In SIR,S is population that is not infected but has β probability to be infected, where β is the infection rate; I recovers as R after σ days, where σ is the recovery time. Based on the above, the ordinary differential equation (ODE) of the original SIR model [20] is given as: dS dt = − βSI (1) dI dt =βSI −I σ (2) dR dt =I σ (3) Various scalable SIR models can be developed based on the characteristics of different epidemics, which have important reference values in the decision-making of epidemic prevention and control. 3.2. Principle of DQN DQN combines the DL and RL algorithms, and was developed to address the inability of Q-table of Q-Learning (QL) to store state-action space sets with exponentially growing dimensionality. Instead of generating a complete Q-table at initialization time, DQN utilizes the deep neural network (DNN) for value function approximation [39,40]. DQN takes an action (at) according to current state (st), and generates the state-action value function (Q-Function, Q(st,at)) for each observation environment (s1,a1,r1, ...st,at,rt)by DNN. Q(st,at)must trend to the immediate reward (rt+1) plus next state Q-function (Q(st+1,at+1)) with discount factor (γ) discounting, thus realizing the value function approximation and achieving the convergence effect, i.e., Q(st,at)←Q(st,at) + α Q[rt+1+γQ(st+1,a∗) − Q(st,at)] (4) where α Q is learning rate; a∗is the action that maximizes the Q-function, i.e., Q(st+1,a∗) = QMAX a(st+1,at+1); [rt+1+γQ(st+1,a∗) − Q(st,at)] is the temporal difference (TD) with rt+1+γQ(st+1,a∗)as the target. As shown in Fig. 2, DQN adds the experience buffer pool to record observation sequences (st,at,rt,st+1, ...)and thus facilitate repeated learning by agents. According to the current state (st), agents take certain action (at) and calculate prediction value (Q(st,at)) through the current network. When observing the next state (st+1), the target value (rt+1+γQ(st+1,a∗)) is computed by the target network, and parameters of the current network are updated by calculating the TD error between the predicted and target values. In this process, DQN selects action in accordance with the baseline of the current Q-function, which can be regarded as future expectation reward obtained by the agent through current action and following the optimal strategy thereafter. DQN utilizes DNN to train the Q-function to approximate the value of a given state-action set (s,a), thus effectively combining the RL and DL. 3.3. Principle of NSGA-II NSGA-II is a non-dominated sorting genetic algorithm based on the genetic algorithm (GA) and Pareto optimality concept, which exhibits better performance in terms of the optimization effect and computation time through an elitist strategy, fast non-dominated sorting, and crowding measure calculation compared to other multi-objective GA [41]. (1) The elite strategy retains excellent individuals in the parent population by mixing all the individuals of parent population (Pt) and offspring population (Qt) and then performing nondominated sorting to form a new parent population (Pt+1), which more effectively prevents the loss of the obtained Pareto optimal solution; (2) Fast non-dominated sorting is a cyclic hierarchical process (as shown in Fig. 3): first, the set of nondominated solutions is identified in the population (i.e., the set of solutions with np =0, where np is the number of solutions of dominating individual p), which is recorded as the first nondominated layer (L1). All p in L1 are assigned with the nondominated ordinal value (NOV, irank) equal to 1, and removed from the overall population. Then, the set of nondominated solutions is found in the remaining population, denoted as the second nondominated layer (L2), and the individuals in L2 are assigned the irank =2 by analogy, until the whole population is stratified, and the individuals in the same stratum have the same irank. The population is stratified according to the noninferior solution levels of the individuals, which guides the algorithmic search towards the Pareto-optimal solution set. (3) The crowding measure calculation serves to selectively sort within individuals with the same irank. Individuals with larger distances are prioritized by calculating the crowding distances of individual i, such that the calculation results are more evenly distributed in the target space to maintain the diversity of the population. To enhance the search space, NSGA-II merges the parent and offspring population and performs a non-dominated sorting, selecting consecutively the individuals with higher priority when generating the next generation of parent population. NSGA-II uses the crowding measure to make the selection among statistical individuals, ensuring that excellent individuals are retained to a greater probability degree. The process of NSGA-II is illustrated in Fig. 4: Fig. 2. Structure of DQN (whereQis the current network, and Qis the target network). J.-Y. Zeng et al. Operations Research Perspectives 11 (2023) 100293 6 (1) First, NSGA-II randomly generates an initial parent population (Pt) of size Np from the solution by means of epidemic data (θe) and cost parameter (ce) observations; (2) Second, the offspring population (Qt) is obtained by selection, crossover, and mutation, and Pt, Qt are united together to form a population Rt of size 2Np(i.e., Rt=Pt∪Qt); Fig. 3. Non-dominated sorting process (Pis the parent population,Spis the set of solutions dominated byp). Fig. 4. Structure of NSGA-II. J.-Y. Zeng et al. Operations Research Perspectives 11 (2023) 100293 7 (3) In addition, the objective function c in each individual of Rt is computed, and fast non-dominated ranking is performed, while the crowding measure is computed for each individual in the nondominated layer, and the appropriate individuals are selected to form a new parent population (pt+1) by the ranking of the Pareto front (fitness) and crowding measure (novelty) [19]; (4) Finally, a new offspring population Qt+1 is generated through the basic operations of EA, and Pt+1 is merged with Qt+1 to form a new population Rt. Steps 2 and 3 are repeated until the conditions for the end of algorithmic procedure are satisfied. 4. Material demand planning based on SEIRAHD model In infectious diseases, to measure medical demand in advance and provide estimates to relevant departments to address security and demand of medical supplies, it is necessary for each region to understand its population base, pandemic development situation, and case proportion of each type, among other factors [42]. Therefore, the premise of realizing reasonable and orderly accurate allocation of medical supplies is to understand the development of infectious diseases. Thus, Epi_DispatchOptim, as an auxiliary MSD simulation model, must establish a reasonable epidemic model and make corresponding predictions of future developments. 4.1. Constructing the SEIRAHD model The SEIRAHD model is a derivative of the SIR models. This experiment considers an asymptomatic increase and the mortality of different infected persons in the occurrence of Omicron BF.2, and suggests that entities in this scenario will evolve in the order ‘S-E-I(A)-H-R-D’. Therefore, two modifications were made to the SIR: the infected compartment in the SIR was classified into symptomatic (I) and asymptomatic (A) based on Omicron strain’s characteristic of increase in asymptomatic cases; A, I, and hospitalized (H) compartments were classified based on the characteristics of different recovery times and mortality rates for different cases. In SEIRAHD, entities will transfer states at different average transmission rates. Given the COVID-19 virus characteristics, this study makes following assumptions: •All entities are susceptible and vaccine factors is not considered due to use available clinical data; •Entities are infectious when they enter the exposed and infection phases; •It is assumed that there is only one strain effect; •An entity acquires immunity after recovery, regardless of reinfection; •The overall model does not account for population births and nonCOVID-19 deaths; •Due to the lack of data on underlying diseases, it is assumed that entities with underlying diseases enter the infected and hospitalized compartments according to clinical data. Based the need of the experiment, the entity transfer state and rules are as follows (e.g., Fig. 5): 1) S: susceptible, expressed as an uninfected entity in an exposed environment, S has probability β to become exposed, where β denotes the infection rate. According to symptomatic entity will choose isolation and other ways to be well-protected, while asymptomatic entities will continue to have social contact as their symptoms are not evident. Consider that have different contact rates R0 I, R0 A for symptomatic and asymptomatic, respectively. And α is the ascertainment rate, i.e., considering the situation of asymptomatic patients who cannot be monitored due to the detection technology or individual reasons. 2) E: exposed, indicates that the entity has been infected without symptoms of infection, and has tested negative in the PCR test. The entity becomes infectious and moves to the infectious state after τ days, where τ is the incubation period, and ν is the proportion of asymptomatic patients. 3) A: asymptomatic, indicating an entity with no symptoms of infection and a positive PCR test. Since asymptomatic infections are defined as no clinical symptoms, no therapeutic medication is required for the time being when there is no conversion to symptomatic infections [43]. According to the Chinese government, in the pandemic caused by Omicron strain, more than 95 % of patients had asymptomatic and mild cases, while 1 % were severely infected [44], and most of the severely infected were older, underlying diseases or not vaccinated, if the case is inoculated and there is no underlying disease, when prompt medical attention reduces mortality [44]. Due to the lack of worldwide statistics on underlying diseases with COVID-19, it was tentatively set that A recovered through σ i days without death. 4) I: infected, indicating an entity that has symptoms of infection and is infectious. Entities entering this state have a probability m of requiring hospital treatment and becoming hospitalized; where σ i, di are the recovery time and mortality rate of symptomatic individuals, respectively. Fig. 5. Entity state transformations of SEIRAHD. J.-Y. Zeng et al. Operations Research Perspectives 11 (2023) 100293 8 5) H: hospitalized, indicating the entity that required hospitalization for symptomatic infection, where σ h, dh are the recovery time and mortality rate for hospitalized individuals, respectively. 6) R: recovered, indicating a recovered healthy entity, when the entity gained antibodies to the virus and is no longer infected. 7) D: deceased, denoting an entity that has died as a result of infectious disease. In addition, the set of transfer equations between variables is shown below: dS dt = − Sβ( α R0 AA+R0 II) N(5) dE dt =Sβ( α R0 AA+R0 II) N−E τ (6) dI dt =(1− ν )E τ −mI − (1−m)diI−[1− (1−m)di−m]I σ i (7) dR dt =A σ i +[1− (1−m)di−m]I σ i +(1−dh)H σ h (8) dA dt = ν E τ −A σ i (9) dH dt =mI −(1−dh)H σ h −dhH(10) dD dt = (1−m)diI+dhH(11) Notably, N is the total population of a region, and each state variable is listed in Table 2. 4.2. Model parameter prediction and result analysis According to the SEIRAHD model, it is necessary to predict the transmission parameters through COVID-19 epidemiological characteristics, thus effectively expressing the trends of COVID-19 development. In contrast, Shanghai, China, experienced a whole process of COVID-19 prevention and control of small peaks from March 1st to June 30th, 2022. Therefore, by fitting the cumulative data of infected entities in Shanghai obtained from Our-WorldInData.org [45] and the Shanghai Municipal Health Commission report [46], more suitable transmission parameters for the Omicron BF.2 strain could be obtained. The experiment employs the least mean squared error (LMSE) on Anylogic 8.7.10 to predict more accurate transmission parameters (e.g., calibration for infection rate β, asymptomatic rate ν , and prediction for ascertainment rate α ) using initial characteristic parameters such as the incubation time [47], hospitalization rate [48], and mortality rate [49]. Here, the cumulative daily reported cases in Shanghai were compared with the confirmed cases estimated using the model, to validate its accuracy. The parameters involved in the model, their values and data sources are listed in Table 3, where literature indicates that the data were obtained directly from the literature, ‘Adjusted’ indicates that the prediction was Adjusted by data from that literature, and ‘prediction’ indicates that the value was predicted by fixing parameters. Notably, the pandemic control in Shanghai started with the areawide closure in Pudong on March 28th and city-wide closure from April 1st, such that the model was set up for population movement control according to the literature [18]. The gap between model data and actual reported cases in Shanghai is shown in Fig. 6; Fig. 6(a) displays the curve of symptomatic entities, which is slightly higher than actual symptomatic report during the simulation period. Nevertheless, the difference between the predicted data and actual reported cases in Shanghai in the LMSE evaluation is about 3861, This is due to the reduction of numerous NPI in the SEIRAHD model to simulate the state of COVID-19 lifting lockdown, and the citywide sealing of the pandemic control in Shanghai in the later stages, so the model data is slightly higher than the actual reported data, the error of predicted data is within an acceptable range; Fig. 6(b) displays the curve of asymptomatic entities, where the difference between the predicted and reported data is small for the first 50 days of the simulation time, and more in line with the actual development trend, Meanwhile, the predicted data is slightly higher than the reported data in the later period, with a gap of about 53, 801, and the above phenomenon may be related to the strengthening of prevention and control strategies in Shanghai in the later period. Overall, according to LMSE evaluation, the numerical difference between the SEIRAHD predicted and reported data is within reasonable limits, and the model provides a better explanation of the developmental trends of the Omicron BF.2 strain. In the SEIRAHD model, this experiment is based on the actual data report time, and the model simulation time is set to 150 days. As shown in Fig. 7, the COVID-19 development trend of different people in Shanghai can be observed. First, in Fig. 7 (a), it can be found that COVID-19 gradually returns to the regular state after about over 60 days, so the COVID-19 outbreak in a major city may take 2-3 months to recover, during which real-time emergency MSD should be carried out to mitigate the COVID-19 spread; Secondly, in Fig. 7, different personnel ’s peak can be observed with the corresponding rule, and the development trend in the SEIRAHD is in line with the principle of reality, from Fig. 7 (b), the peak of Exposed occurs the earliest, about 40 days after the simulation time, followed by the Asymptomatic, and the time is similar to the Exposed (as in Fig. 7 (e)); And followed by I, which appeared a few days later than the previous two (e.g., Fig. 7 (c)); Hospitalized personnel’s peak appears a little later than symptomatic personnel, and its peak is smoother (e.g., Fig. 7(f)), so the MSD should be considered in accordance with the trend of COVID-19, and the priorities of the materials required by different personnel. Table 2 State Variables in SEIRAHD. variable Interpretation S Susceptible E Exposed I Infected R Recovered A Asymptomatic H Hospitalized D Deceased Table 3 Parameters and their values in SEIRAHD. Parameter Interpretation Value Sources β Infection rate 0.091 Adjusted from (Zhuang et al., 2022) [50] τ Incubation time 3.4 (day) literature (Wu et al., 2022) [47] α Ascertainment rate 0.55 Prediction data ν Asymptomatic rate 0.912 Adjusted from (Huang et al., 2022) [49] m Hospitalization rate 0.0078 literature (Adjei et al., 2022) [48] σ h In-hospital cure time 18 (day) literature (Ward et al., 2022) [51] σ i Non-in-hospital cure time 7 (day) literature (Yin et al., 2022) [52] dh In-hospital mortality rate 0.0009 literature (Huang et al., 2022) [49] di Mortality rate of infected persons 0.00012 N Shanghai City population 24890000 China Statistical Yearbook (2022) [53] J.-Y. Zeng et al. Operations Research Perspectives 11 (2023) 100293 15 vaccine prevalence, aging population, and underlying diseases population, future work will use ABM for modeling and add the more construction of the rules and nature of human activities. (2) Epi_DispatchOptim breaks through the original consideration of only Lockdown strategy or supply-demand, and builds an infectious disease MSD optimization model that can explore the tradeoffs between economic cost and health into through various modules. And RL and EA were utilized to explore dynamic MSD strategies with economic and health cost trade-offs in two sets of data to obtain better results. Among them, the dynamic scheduling decision explored by the DQN in the Shanghai saved 37.07 B of economic cost over the fixed scheduling strategy and reduced 150 deaths over the no-intervention strategy. Scheduling strategies require more rules than a single reduction in infectious disease transmissibility, which can be modeled with reference to OR. Future work will consider more complex scenario building. Overall, the combination of the SEIRAHD model with Epi_DispatchOptim can be used in major infectious disease scenarios to solve the problem of forecasting the medical demand and the related MSD decisions, and is shown in this research to reduce deaths and economic costs. In future work more details of this model will be further constructed based on the above problems. CRediT authorship contribution statement Jia-Ying Zeng: Conceptualization, Methodology, Software, Investigation, Formal analysis, Writing – original draft. Ping Lu: Fig. 11. DQN scheduling status for the French scenario (βc=0.5). Fig. 12. NSGA-II scheduling status in China (βc=0.5). J.-Y. Zeng et al. Operations Research Perspectives 11 (2023) 100293 16 Conceptualization, Funding acquisition, Resources, Supervision, Writing – review & editing. Ying Wei: Data curation, Writing – review & editing. Xin Chen: Investigation, Resources, Visualization. Kai-Biao Lin: Supervision, Writing – review & editing. Declaration of Competing Interest We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and/or company that could be construed as influencing the position presented in, or the review of, the manuscript entitled. Data availability Data available within the article or its supplementary materials. Appendix A In order to understand the whole process of model construction, the main parameters of SEIRAHD model and Epi_DispatchOptim model are summarized in Table 7. Table 7 Parameters in The Experiments. SEIRAHD Model Epi_DispatchOptim Model Parameters Definition Parameters Definition S Susceptible β0 Initial infected rate E Exposed ζDi The scheduling effect in Di I Infected Di The ist schedule for 4 steps R Recovered cj jstcost A Asymptomatic Nc Cost types,j∈Nc H Hospitalized ch Health cost D Deceased ceoc Economic cost β Infection rate. βc Weight parameter forchandceoc R0 I Contact rates for I c Total cost function R0 A Contact rates for A Y0 Initial GDP α Ascertainment rate K0 Initial capital stock τ Incubation time L0 Employed individuals’ number ν Asymptomatic rate α k Capital elasticity σ i Non-in-hospital cure time Ak Exogenous technical progress di Mortality rate of I λe Employment rate σ h In-hospital cure time SMt Medication supply number dh Mortality rate of H DMt Medication demand number m Hospitalization rate ST Simulation time N Population t Training step (day) θpre Weekly dosage of preventive medicine Ch Cumulative health cost (CHC) θCure I Weekly dosage of Cure medicine for I Ce Cumulative economic cost (CEC) θCure H Weekly dosage of Cure medicine for H C Cumulative aggregated cost (CAC) Fig. 13. DQN Scheduling Status in China (βc=0.5). J.-Y. Zeng et al. Operations Research Perspectives 11 (2023) 100293 17 Appendix B Appendix B takes the NSGA-II in the French scenario as an example to analyze the interesting phenomenon of different weight parameters βc. In this experiment, βc is trained according to[0,0.25,0.5,0.75]with the c= (1−βc)ch+βcceoc formula, and βc is the weight ratio of the two costs. When the health cost is favored, the agent will favor the MSD strategy that reduces deaths; when the economic cost is favored, the agent will favor the MSD strategy that reduces the economic cost. The above phenomenon is also reflected in the experimental results. In Fig. 14, when βc is 0, the agent takes on weekly dispatching until the end of epidemic with the Eco cost as high as 86.81B under the premise of considering only the Death Cost. Meanwhile, when βc is 0.25 (as in Fig. 15), the situation is alleviated, and to weigh the balance of two costs and gradually prefer the consideration of economic costs, the agent will take on interval dispatching compared to the stage of βc=0, which reduces about 28.35B in Eco cost and increases deaths by about 25. When βc is 0.75, the agent will choose the behavior that favors economic cost, as shown in Fig. 16, the agent will dispatch supplies in the early stage of epidemic and not take on any dispatching behavior in the later stage due to the consideration of the Eco cost. This experiment also demonstrates the importance of exploring economic and health cost tradeoffs in MSD, and suggests that the model must add penalty rules for not taking any dispatching actions. Fig. 14. NSGA-II scheduling status for the French scenario (βc=0). Fig. 15. NSGA-II Scheduling status for the French scenario (βc=0.25). J.-Y. Zeng et al. Operations Research Perspectives 11 (2023) 100293 18 Fig. 16. NSGA-II scheduling status for the French scenario (βc=0.75). References [1] Gao W J Wang B, Lyu J, et al. Current status of COVID-19 pandemic and progress in response strategy. Zhonghua Liu Xing Bing Xue Za Zhi 2021;42(1):22–7. [2] 2023 GOV 2023. Press Conference of the Joint Prevention and Control Mechanism of the State Council[EB/OL]. Retrieved February 9, 2023 from https://www.gov. cn/xinwen/gwylflkjz235/index.htm. [3] 2023 GOV 2023. Press Conference of the Joint Prevention and Control Mechanism of the State Council[EB/OL]. Retrieved January 14, https://www.gov.cn/xinwen/ gwylflkjz231/wzsl.htm. [4] Wu H. Research on the path of emergency medical supplies for major emergencies ——taking the COVID-19 as an example. Logistics Sci Tech, 2022;45(17):20–2. [5] Xiao M S, Zhu W Q, Huang X Y, et al. Practice and thinking on construction of an emergency medical supply security system in megacities. Occup Health Emerg Rescue 2023;41(02):211–4. [6] Wu H, Zhang L, Li Y. Research on financing and accurate scheduling of emergency medical materials——take COVID-19 for example. Price:Theory Practice, 2021, 8790+173. 10.19851.j.cnki.cnll-1010/f.2021.01.09. [7] GOV 2023. Press Conference of the Joint Prevention and Control Mechanism of the State Council[EB/OL]. Retrieved February 23, https://www.gov.cn/xinwen/gwy lflkjz236/index.htm. [8] Wang D, Tian Y Q, Qian Z X, et al. Emergency dispatching of trans-regional medical human resources under COVID-19 epidemic. Chin J Infect Control 2021:742–7. [9] Yu H Y, Chen G L, Yu J M. The trend of industrial chain development and emergency management in the post-epidemic era. Stud Party Government 2021; (03):121–8. [10] Wang Z R, Zhang H L, Dong C, et al. Medical resource reserve and the effectiveness of COVID-19 prevention and control in China: based on the perspective of health production function. Finance Econ 2021:71–83. [11] Fan C J, Du Y Q, Liang D, et al. Modeling and simulation of emergency medical resources allocation in Shanghai during COVID-19. J Syst Simulation 2022:93–103. https://doi.org/10.16182/j.issn1004731x.joss.21-0320. [12] Fu Z T, Dong P W, Li Z Z, et al. Research on medical materials synergetic supports for epidemic controlling in public health emergency. Indus Eng Manage 2021. [13] Jana S, Majumder R, Ghose D. Critical Medical Resource Allocation during COVID19 Pandemic. arXiv. 2022. [14] Wang X, Liang Z, Zhu K. Markov decision model of emergency medical supply scheduling in public health emergencies of infectious diseases. Int J Comput Intell Syst 2021;14:1155–69. [15] Yang Y. A study on optimal dispatch of meical resources under the situation of COVID-19. J Phys Conf Ser 2021;1955(1):012105. [16] Awasthi R, Guliani K K, Khan S A, et al. VacSIM: Learning effective strategies for COVID-19 vaccine distribution using reinforcement learning. (2666-5212 (Electronic)). [17] Bednarski B P, Singh A D, Jones W M. On collaborative reinforcement learning to optimize the redistribution of critical medical supplies throughout the COVID-19 pandemic. J Am Med Inform Assoc 2021;28(4):874–8. [18] Zhao D H, Yang J. Dynamic zero-COVID" strategy for controlling the pandemic in Shanghai——simulation analysis using real data from march to may 2022. J Public Manage 2023:169–70. https://doi.org/10.16149/j.cnki.23-1523.20221206.001. 10-9+. [19] Colas C, Hejblum B, Rouillon S, et al. EpidemiOptim: A Toolbox for the Optimization of Control Policies in Epidemiological Models [M]. 2020. [20] Kermack W O, Mckendrick AG. A contribution to the mathematical theory of epidemics. In: Proceedings of the Royal Society of London Series A, Containing Papers of a Mathematical and Physical Character. 115; 1927. p. 700–21. [21] Hackl J, Dubernet T. Epidemic spreading in urban areas using agent-based transportation models. Future Internet 2019;11(4):92. [22] Kumar R L, Khan F, Din S, et al. Recurrent Neural Network and Reinforcement Learning Model for COVID-19 Prediction. (2296-2565 (Electronic)). [23] Zhang T, Wang Q, Leng Z, et al. A Scenario-Based Evaluation of COVID-19-Related Essential Clinical Resource Demands in China. (2095-8099 (Print)). [24] Capobianco R, Kompella V, Ault J, et al. Agent-Based Markov modeling for improved COVID-19 mitigation policies. J Artif Intell Res 2021;71:953–92. [25] Feng B, Smith K, Masoud S, et al. AGENT-BASED SIMULATION TO PREDICT OCCUPANTS’ PHYSICAL-DISTANCING BEHAVIORS IN EDUCATIONAL BUILDINGS, F, 2021 [C]. [26] Kai D, Goldstein G-P, Morgunov A, et al. Universal Masking is Urgent in the COVID19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations [M]. 2020. [27] Xu C J, Zhou Q K. Epidemic analysis of covid-19 based on seir model. Comput Appl Softw 2021:87–90. [28] Kissler S A-O, Tedijanto C A-O, Goldstein E A-O, et al. Projecting the transmission dynamics of SARS-CoV-2 through the postpandemic period. (1095-9203 (Electronic)). [29] Aylett-Bullock J, Luccioni A S, Pham K H, et al. Mapping the landscape of artificial intelligence applications against COVID-19. ArXiv 2020. abs/2003.11336. [30] Cao Z, Xu L, Chen D Z, et al. A robust shape-aware rib fracture detection and segmentation framework with contrastive learning. IEEE Trans Multimedia 2023; 25:1584–91. [31] Xiao J, Li J, Gao H. FS3DCIoT: a few-shot incremental learning network for skin disease differential diagnosis in the consumer IoT. IEEE Trans Consum Electron 2023:1. -. [32] Zong K, Luo C. Reinforcement learning based framework for COVID-19 resource allocation. (1879-0550 (Electronic)). [33] Li X, Zhang J, Bian J, et al. A Cooperative Multi-Agent Reinforcement Learning Framework for Resource Balancing in Complex Logistics Network [M]. 2019. [34] SHI W, WEI X, ZHANG J, et al. Cooperative Policy Learning with Pre-trained Heterogeneous Observation Representations [M]. 2020. [35] Deng S, Xiang Z, Zhao P, et al. Dynamical resource allocation in edge for trustable internet-of-things systems: a reinforcement learning method. IEEE Trans Ind Inf 2020;16:6103–13. [36] Li P, Xiao Z, Wang X, et al. EPtask: deep reinforcement learning based energyefficient and priority-aware task scheduling for dynamic vehicular edge computing. IEEE Trans Intell Vehicles 2023:1–17. [37] Gao H, Wang X, Wei W, et al. Com-DDPG: task offloading based on multiagent reinforcement learning for information-communication-enhanced mobile edge computing in the internet of vehicles. IEEE Trans Veh Technol 2023:1–14. J.-Y. Zeng et al. Operations Research Perspectives 11 (2023) 100293 19 [38] Lee H R, Lee T. Multi-agent reinforcement learning algorithm to solve a partiallyobservable multi-agent problem in disaster response. Eur J Oper Res 2020:291. [39] Watkins CJCH, Dayan P. Q-learning. Machine Learn 1992;8(3):279–92. [40] Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning. Nature 2015;518(7540):529–33. [41] Deb K, Pratap A, Agarwal S, et al. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans Evol Comput 2002;6(2):182–97. [42] GOV 2022. Overall Plan for the Implementation of "Class B and B Management" for Novel Coronavirus Infection [EB/OL]. Retrieved December 27, 2022 from https:// www.gov.cn/xinwen/2022-12/27/content_5733739.htm. [43] GOV 2022. What are the criteria for symptomatic infections? Do medications need to be taken? Experts answer top questions [EB/OL]. Retrieved December 19, 2022 from https://ws.cq.gov.cn/zwgk_165/hygq/202212/t20221219_11401378.html. [44] GOV 2022. Press Conference of the Joint Prevention and Control Mechanism of the State Council[EB/OL]. Retrieved March 19, 2022 from http://www.gov.cn/xinw en/gwylflkjz187/index.htm. [45] Edouard M, Hannah R, Lucas R, et al. Coronavirus Pandemic (COVID-19). Our World In Data.org. 2020 https://ourworldindata.org/coronavirus. [46] Shanghai Municipal Health Commission. Daily report on the outbreak of novel coronavirus pneumonia in Shanghai[EB/OL]. https://wsjkw.sh.gov.cn/yqtb/i ndex.html. [47] WU Y, KANG L, GUO Z, et al. Incubation Period of COVID-19 Caused by Unique SARS-CoV-2 Strains: A Systematic Review and Meta-analysis. (2574-3805 (Electronic)). [48] Adjei S Fau - Hong K, Hong K Fau - Molinari N A M, Molinari NM Fau, BullOtterson L, et al. Mortality Risk Among Patients Hospitalized Primarily for COVID19 During the Omicron and Delta Variant Pandemic Periods - United States, April 2020-June 2022. (1545-861X (Electronic)). [49] Huang Z, Xu S, Liu J, et al. Effectiveness of inactivated and Ad5-nCoV COVID-19 vaccines against SARS-CoV-2 Omicron BA. 2 variant infection, severe illness, and death [Z]. medRxiv 2022. https://doi.org/10.1101/2022.09.04.22279587. [50] Lei Z, Ruizhi Z, Weiyi G. Analysis on the epidemic situation of COVID-19 by Omicron variant in a single office building. J Diagnostics Concepts Practice 2022; 21(02):150–3. [51] Ward I L, Bermingham C, Ayoubkhani D, et al. Risk of covid-19 related deaths for SARS-CoV-2 omicron (B.1.1.529) compared with delta (B.1.617.2): retrospective cohort study. (1756-1833 (Electronic)). [52] Yin Y, Lin J, Yuan S, et al. The relationship between early isolation and the duration of viral shedding of mild and asymptomatic infection with SARS-CoV-2 Omicron BA.2 variant. (1532-2742 (Electronic)). [53] CSY. China Statistical Yearbook [M]. China Statistical Yearbook, 2022. [54] Xing R R. Research on Scheduling Models of Emergency Medical Supplies under the Major Infectious Diseases [D]. Beijing jiaotong University; 2021. [55] Yu C, Liu J, Nemati S, et al. Reinforcement learning in healthcare: a survey. ACM Comput Surv 2021;55(1):5. Article. [56] Douglas P H. The Cobb-Douglas production function once again: its history, its testing, and some new empirical values. J Polit Econ 1976;84(5):903–15. [57] Prague M, Wittkop L, Clairon Q, et al. Population modeling of early COVID-19 epidemic dynamics in French regions and estimation of the lockdown impact on infection rate. medRxiv 2020. [58] Sun J S, Li G, Wang Y. Estimation of China’s potential growth rate: from the perspective of changing human capital. China Population Resour Environ 2021: 127–37. [59] GOV 2023. Press Conference of the Joint Prevention and Control Mechanism of the State Council[EB/OL]. Retrieved May 8, https://www.gov.cn/yaowen/2023-05/ 08/content_5754595.htm. [60] Conference of National Health Commission of the People’s Republic of China. Press Conference of National Health Commission of the People’s Republic of China [EB/ OL]. Retrieved April 26, 2023. http://www.nhc.gov.cn/xcs/yqfkdt/202304 /2a7d547fb3174a6f8ce6814d5c56bb0c.shtml. J.-Y. Zeng et al.