Inference Using Synthetic Data: Balancing Privacy, Bias, and Variance in Modern Statistical Practice

1 Inference Using Synthetic Data: Balancing Privacy, Bias, and Variance in Modern Statistical Practice Tom Krenzke and Minsun Riddles, Westat Abstract As privacy concerns grow, synthetic data have emerged as a promising solution, yet ensuring proper inference remains a critical challenge. This roundtable session explored best practices for making valid inferences from synthetic datasets, with a focus on mitigating bias and accurately estimating variance (the uncertainty of results derived from synthetic data). Participants from diverse research areas shared insights, identified gaps in current practice, and discussed challenges related to the use and dissemination of synthetic data across different fields. The goal was to foster a shared understanding of these issues and identify key areas for further development in methods and tools as this field continues to evolve. Introduction The rising adoption of synthetic data reflects a growing need to balance data utility with privacy protection. As access to sensitive datasets becomes more restricted, synthetic data offer a promising alternative for enabling research, testing software, and supporting policy analysis. However, the ability to draw valid inferences from synthetic data remains a critical concern. Synthetic data are valued for their ability to preserve privacy, making them a vital tool for secure data sharing. Synthetic data offer a solution to privacy concerns by enabling broader data sharing without compromising confidentiality. Its utility is especially pronounced in situations where real data are limited, such as with rare subgroups that lack sufficient sample sizes, or expensive to collect. Common use cases include testing software products, generating analytical tables, supporting simulation studies, and creating microdata for public use. In each case, synthetic data provide a layer of security that facilitates safer dissemination. This paper summarizes insights from a roundtable session at the Joint Statistical Meetings in Nashville, TN, focused on inference challenges in synthetic data. It explores the types of synthetic data, methods of generation, and the statistical implications related to bias and variance. Drawing from practitioner experiences and recent innovations, we present practical strategies for mitigating disclosure risk while maintaining analytical utility in synthetic data. We further identify persistent challenges, also highlight gaps in current practices, and outline key directions for future research and tool development. This article focuses primarily on inference challenges and practical applications and is not intended as a comprehensive review of all aspects of synthetic data generation and use. Types of Synthetic Data Synthetic data can be categorized based on the scope and method of synthesis. Two commonly cited types are: • Full synthetic data (Rubin, 1993) are where a synthetic population is generated entirely from the original database using statistical models. Samples are drawn from this population, with no one-to-one correspondence with original records. • Partial synthetic data (Little, 1993) are where entire data vectors for select variables are synthesized while maintaining a one-to-one correspondence with original records. Our focus in this article is on a type of partially synthetic data, we refer to as “select synthetic” data (see Reiter and Kinney, 2012). In this approach, only specific values for select variables and records, where those identified as high-risk for disclosure are more likely to be synthesized. This approach is often used when only a subset of the data poses disclosure risks, allowing much of the original data to remain untouched and analytically useful. Each type offers different trade-offs in terms of privacy, utility, and inference complexity. 2 Our applications of generating select synthetic data target specific records and variables based on their risk profiles. Records with higher disclosure risk are assigned a greater probability of being synthesized. Records with lower risk are less likely to be altered, ensuring that a portion of the original data remains intact. This selective approach strives to maintain key statistical properties such as aggregates, marginal distributions, and inter-variable associations, while minimizing disclosure risk. Implementation may involve stratifying records by disclosure risk, defining a target treatment rate by risk stratum, and randomly selecting records for synthesis within each stratum based on the predetermined target treatment rate. Data Generation Methods Synthetic data can be generated using a variety of methods, broadly categorized into three groups: model-assisted, model-based, and AI-based approaches. These methods are increasingly applied in real-world settings, including synthetic health records and educational statistics. A model-assisted approach will rely on a model’s predictions to inform donor-based data replacement strategies using original data values. For example, the model-assisted constrained hotdeck (MACH) method builds upon a sequential imputation framework originally developed to address non-monotone missing data patterns in complex surveys (Judkins et al., 2007), particularly those with skip patterns and diverse variable types. As detailed in Krenzke, Li, and McKenna (2017), the MACH has been successfully applied to synthesize American Community Survey data for the Census Transportation Planning Products (CTPP). The procedure operates sequentially across target variables, forming hotdeck cells based on a combination of risk flags, constrained bins, model-based prediction groups, auxiliary variables, and sampling weight strata. Within each hotdeck cell, synthetic values are drawn with replacement from the empirical distribution of donor records. A key strength of the MACH lies in its ability to control the amount of noise introduced into the data while preserving associations between synthesized and non-synthesized variables through stepwise regression-based predictor selection. Model-based approaches typically involve sequential regression modeling, where each variable is synthesized conditionally based on previously synthesized variables. This approach is similar in sequencing to MACH but relies entirely on statistical models rather than donor records. Raghunathan, et al. (2001) introduced a sequential regression-based multiple imputation method designed for handling missing data under the assumption of missing at random, using regression models tailored to variable types such as linear, logistic, and Poisson. The approach employs logical bounds to improve the realism of imputed values. The imputation approach was later proposed as a strategy to create synthetic data (Raghunathan, et al. 2003). AI-based approaches to generating synthetic data have been on the rise recently. These include models such as Large Language Models (LLMs), Generative Adversarial Networks (GANs), and Variational Autoencoders (VAEs), which are well-suited for different data types. LLMs are effective for synthetic data generation based on text data and nominal categorical variables, using their capacity to capture semantic and contextual patterns. GANs can be useful for ordinal and continuous data. By leveraging latent representations and adversarial training, GANs can produce realistic synthetic samples that capture the underlying distribution of the source data. VAEs are a type of generative model that learn a compressed, lower-dimensional representation of data in a probabilistic manner, enabling them to generate new, realistic structured and semi-structured data that closely resemble the source data. Table 1 summarizes the applicability of different AI models across various criteria, including their suitability for specific data types. See Goyal and Mahmoud (2024) for more details. 3 Table 1. Most applicable scenarios for AI-based methods Criteria Large Language Models (LLMs) Generative Adversarial Networks (GANs) Variational Autoencoders (VAEs) Primary Data Types Text, nominal categorical data Continuous, tabular (structured) Tabular (structured), semi-structured data Training Stability Stable due to pretraining, fine-tuning may vary Unstable, requires careful tuning and large data More stable than GANs Computational Complexity Very high (due to scale of models) High Moderate Strengths - Captures complex language patterns - Versatile - High fidelity output - Works well with visual data - High fidelity output - Works well with visual data Limitations - Resource intensive - Less suited for numerical data - Training instability - Requires large data - May produce blurry outputs - Lower visual fidelity Inference Challenges We discuss the inference challenge in synthetic data analysis in terms of mean square error (MSE), which is equal to the bias squared plus the variance. These two components represent different dimensions of inferential accuracy: bias reflects the systematic deviation from the truth, while variance reflects uncertainty or precision. We consider each in turn. Bias refers to the systematic deviation of synthetic estimates from true population values. In the context of synthetic data, it is a measure of how accurately the synthetic data reflects the original data's statistical properties. When synthesizing data, it is important to investigate sources of bias. Some sources of bias may include: inherited bias from the original training data (e.g., missing or underreported subgroups), model design flaws in synthetic data generators, distorting effects of outliers, which can skew both marginal distributions and inter-variable relationships. In a project by the Lantana Consulting Group, simulation methods were developed for generating large-scale hierarchical healthcare patient-level data with two primary goals: (1) to conduct load testing of an application and (2) to provide data scientists and statisticians with realistic datasets for methodological exploration. The training data came from a pilot cohort of approximately 25,000 patients, with plans to scale to hundreds of millions of patients. Two approaches were implemented: an auto-encoded neural network and the open-source Python package Synthetic Data Vault (SDV), which leverages Gaussian copulas and supports hierarchical table structures. The simulation aimed to preserve both marginal distributions and inter-table correlations. Ongoing work includes integrating more conditional logic, such as constraining observation dates to occur within encounter start and end dates, to enhance realism and usability of the simulated datasets. However, outliers, particularly extreme outliers in the training data, proved challenging because they disrupted the simulated distributions. To address this, extreme outliers were removed from the training dataset before model fitting and then reintroducing a small percentage of outlier-like values into the simulated data. This approach improved distributional fidelity while still reflecting the occasional presence of unusual values. Once the sources of bias are identified, there are mitigation strategies to consider. For example, vetting and preprocessing training data will provide a good foundation for model development. Another consideration is balancing datasets to represent underrepresented groups. That is, suppose there is a subgroup in the original database that was underreported. Synthetic data can increase their representation to better reflect population distributions. In addition, Westat’s work with the Census Transportation Planning Products (CTPP) for the Census Bureau (see Section 5.1 for a brief description) demonstrates how weight calibration can better align synthetic estimates with original distributions. 4 Variance reflects the uncertainty in synthetic data estimates, or conversely, the precision of those estimates. It is important to identify the various components of variances. For example, in synthetic survey data derived from a stratified and clustered sample, the variance components may include variance inherent in the original survey data, such as sampling variance from the original data collection, and synthetic data variance introduced by the data generation process. Once the variance components are identified, we need to estimate those components appropriately. Each variance estimation method introduced below has implications for privacy and analytical accuracy. The following are three approaches, though others may also be appropriate depending on the context. One variance estimation approach is to create multiple dataset (implicates) to incorporate synthetic data uncertainty. This approach involves generating multiple implicates by repeating the synthesis with different random seeds, allowing for variance estimation that reflects the variability introduced by the synthetic data generation process. Formulas provided by Reiter and Kinney (2012) can be used to obtain proper variance estimates and inference when releasing multiple implicates of synthetic data. However, it is not hard to determine which variables and data records were synthesized. That is, records may have the same values for each implicate for a variable, and if so, then it is highly likely that is the original value for that record. Another approach is to create and use Variance Inflation Factors (VIFs) to account for the synthetic data component of variance. VIFs can be computed from implicates, modeled or averaged to derive adjustment factors, and then provided in user guides to help data users adjust variance estimates. Releasing one implicate protects privacy. That said, if the VIF = 1 for a variable, it indicates that the variable has not been synthesized. This may be acceptable but should be carefully considered and discussed by the project team. Using VIFs can adjust variance from synthetic data but limits the ability to estimate variance validly, especially for more complex analyses. In those cases, releasing multiple implicates would be preferable for analysis purposes, however, there is increased disclosure risk as mentioned above. One can also apply a fractional imputation approach when generating select synthetic data. This approach produces multiple synthetic records per original record, distributing the weight from the original record across the multiple records. If the dataset is released without an indicator of the original record associated with the synthetic record, then it may not be straightforward for a data intruder to reveal which data values were not synthesized. An advantage of this approach is that the variance estimation approach used for the original data can be used for the synthesized data, supporting both analytical validity and privacy protection. One potential drawback is that if data users do not apply the weights provided with the synthetic data, they may overestimate the precision of their results. This can occur when methods like fractional imputation increase the number of records, creating a false impression of a larger effective sample size. Real-World Applications Below are three applications that we have worked on recently or are currently working on. • Census Transportation Planning Products. For the Census Bureau, underlying American Community Survey (ACS) 2017–2021 microdata were synthesized, and over a billion tables were generated to support small-area analysis for transportation planning. The approach used ensures privacy by synthesizing a targeted percentage of records using a MACH method, with added noise and raking for weight calibration. Synthetic data were used for all Census Transportation Planning Product (CTPP) tables with Census tracts as the lowest geography. A variance estimation method was developed and implemented to account for synthesis-related variability. The total variance is essentially the sum of variance from the original ACS data and the synthetic data variance. The synthetic data variance is simply estimated as the squared difference between the point estimate derived from the original ACS data and the point estimate from the synthetic data. More details can be found in found in Krenzke, et al. (2017). 5 • NCSES SDR PUF. The Survey of Doctorate Recipients (SDR) collects longitudinal data on individuals with research doctorates in science, engineering, and health fields from U.S academic institutions. To expand access while protecting confidentiality, the National Center for Science and Engineering Statistics, in collaboration with Westat, explored creating a synthetic longitudinal public use file (L-PUF) using data from SDR 2015–2019. The project involved evaluating multiple data synthesis methods to balance privacy and analytical utility. Ultimately, to mitigate the risk in the longitudinal data, a select synthetic data approach was used to preserve relationships between variables, selectively synthesizing variables and records based on risk levels. Among several methods tested, the MACH approach was chosen for its balance of privacy protection and data utility. The resulting experimental synthetic L-PUF for 2015–2019 showed minimal disclosure risk and strong consistency with published tables and regression results, though some subgroup analyses revealed discrepancies. A variance estimation method was also developed to account for synthetic data variability, enhancing transparency and analytical precision. The results are under evaluation as to whether the longitudinal synthetic data supports science and engineering statistics. An NCSES working paper has been written and is under review. • Synthetic Data Generation with Large Real-World Data. The Synthetic Data Generation with Large Real-World Data (RWD) project, part of the National Secure Data Service (NSDS) Demonstration Project and National Artificial Intelligence Research Resource (NAIRR) Pilot, explores AI-driven synthetic data generation for secure, tiered access to large datasets. The project aims to improve understanding of how synthetic data generators work with large RWD to inform a synthetic data generator toolkit. Using the National Clinical Cohort Collaborative (N3C) as a foundation, the project evaluates open-source methods for generating synthetic data in a secure supercomputing environment. This ongoing project selected synthetic data generation techniques that rely on LLMs, and approaches that include domain reduction, patient sequencing, and conditional modeling, with the goal of producing synthetic health records that enable research while maintaining privacy. Future work will explore how well synthetic data preserve key distributional properties of RWD to support robust downstream analyses. Gaps in Practice and Future Directions There has been considerable progress in the development and use of synthetic data. For example, there have been tools developed to assist data users. One such tool is verification servers, where synthetic data are provided to users who can perform their analyses and submit their code to be run on the original data. A message is sent back to the user indicating whether the conclusions from the synthetic data are consistent with those from the original data. Another tool is flexible table generators that allow users to create customized statistical tables from microdata, selecting variables, geographies, and formats to suit specific analytical needs. To protect privacy, agencies apply confidentiality treatments such as data synthesis, variable suppression, recoding, and noise infusion, ensuring that individual identities cannot be inferred while preserving the utility of the data for meaningful analysis. Also, small area estimation (SAE) can be considered another relevant tool that generates model-based estimates for domains (e.g., geographic areas or demographic groups). SAE results can be considered synthetic, as they are model-based, and therefore allow for publication of statistics for small geographies that would otherwise be suppressed due to limited sample sizes. These tools help bridge the gap between synthetic data producers and users. However, despite these tools and progress, several gaps remain and present opportunities for future development. • Communicating inferences to data users. To ensure proper use of synthetic data, data producers need to clearly communicate which variables were synthesized, provide guidance on variance estimation methods, and offer tools and documentation to support bias mitigation. 6 • Evaluation metrics for bias. There is a need for more effective evaluation metrics for bias and better preprocessing techniques, including data cleaning and outlier detection. • Privacy preserving variance estimation. Some methods may not fully address privacy concerns and may inadvertently reduce the privacy protection intended by the synthetic data approach. There is also a need for clearer communication of uncertainty. Users must understand what variance estimates capture and what they do not. • Improved education and documentation. This roundtable included professionals who have just started to explore the generation of synthetic data. The discussion highlighted the need for more education and documentation, especially with regards to inference from synthetic data. Educational efforts can come in the form of short courses, seminars, and briefing documents, such as this article. Synthetic data offers a powerful solution for balancing privacy and utility, but its success depends on rigorous methods for inference. By addressing bias and variance thoughtfully, data users can ensure that synthetic data supports valid, trustworthy analysis. As the field continues to evolve, collaboration across disciplines will be essential to refine tools, share insights, and build confidence in synthetic data products. Acknowledgements The authors would like to thank the participants of the roundtable discussion, including Helio Arizono, Shannon Ferrante, Holmes Finch, and also Becky Lien, a Statistics and Data Architecture Manager within the Lantana Consulting Group, who graciously contributed helpful information to this article. References Goyal, M., & Mahmoud, Q. H. (2024). A Systematic Review of Synthetic Data Generation Techniques Using Generative AI. Electronics, 13(17), 3509. https://doi.org/10.3390/electronics13173509 Krenzke, T., Li, J., & McKenna, L. (2017). Producing multiple tables for small areas with confidentiality protection. Journal of Official Statistics, 33(2), 469– 485. https://doi.org/10.3233/SJI-160259 Li, J., & Krenzke, T. (2016). Confidentiality approaches for real-time systems generating aggregated results. ASA Proceedings. Little, R. J. A. (1993). Statistical analysis of masked data. Journal of Official Statistics. 9:407– 426. National Academies of Sciences, Engineering, and Medicine. 2024. A Roadmap for Disclosure Avoidance in the Survey of Income and Program Participation. Washington, DC: The National Academies Press. https://doi.org/10.17226/27169. NCSES. SDR Longitudinal Synthetic PUF. https://ncses.nsf.gov/pubs/nsf22326 Raghunathan, T. E., Lepkowski, J. M., Van Hoewyk, J., & Solenberger, P. (2001). A multivariate technique for multiply imputing missing values using a sequence of regression models. Survey Methodology, 27(1), 85–96. Raghunathan, T. E., Reiter, J. P., & Rubin, D. B. (2003). Multiple imputation for statistical disclosure limitation. Journal of Official Statistics, 19(1), 1. Reiter, J.P. and Kinney, S.K. (2012). Inferentially valid partially synthetic data: Generating from posterior predictive distributions not necessary. Journal of Official Statistics. 28:583-590. 7 Riddles, J., & Westat. (2020). SDCNway: Tools to Evaluate Disclosure Risk. https://cran.rproject.org/package=SDCNway. Rubin, D.B. (1993). Discussion: Statistical disclosure limitation. Journal of Official Statistics. 9:462–468. Shlomo, N., Krenzke, T., & Li, J. (2019). Comparison of three post-tabular confidentiality approaches. Transactions on Data Privacy, 12(3), 145–168.