Unmasking Deepfakes: CNNs and Vision Transformers for Cutting-Edge Detection

Unmasking Deepfakes: CNNs and Vision Transformers for Cutting-Edge Detection Abdelhalim Saadi1, Yacine Slimani2, Ridha Louze3, and Roufaida Hammadou4 1Faculty of technology Setif 1 University – Ferhat Abbas Setif, Algeria , [email protected] 2Faculty of technology Setif 1 University – Ferhat Abbas Setif, Algeria, [email protected] 3Faculty of NTIC University of Abdelhamid Mehri – Constantine 2, Algeria, [email protected] 4Faculty of NTIC University of Abdelhamid Mehri – Constantine 2, Algeria, [email protected] Abstract Deepfake technology, driven by Generative Adversarial Networks (GANs), poses challenges in digital security by enabling highly realistic synthetic media. This study proposes a detection framework combining Convolutional Neural Networks (CNNs) for local feature extraction and Vision Transformers (ViTs) for global analysis. Evaluated on two datasets, CNNs achieved 97.1 % accuracy on a 140K-image dataset, outperforming ViTs at 90.06%, though ViTs showed better generalization. Despite these advances, deepfake detection faces challenges like adversarial attacks and dataset biases. Future work will enhance real-time processing, robustness, and multi-modal approaches integrating audio and behavioral cues. Keywords: Deep Learning (DL), Deepfake Detection, Convolutional Neural Networks (CNN), Vision Transformers (ViT), Generative Adversarial Networks (GAN). 1 Introduction The rise of deepfake technology has brought significant challenges to digital media security, enabling the creation of highly realistic synthetic images and videos. While deepfakes have applications in entertainment and creative industries, they also pose serious threats, including misinformation, identity fraud, and political manipulation. The increasing sophistication of Generative Adversarial Networks (GANs) has made detecting manipulated media more complex, necessitating advanced detection techniques. This study proposes a deepfake detection framework leveraging Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) to enhance classification accuracy. Using two datasets—a 140K Real and Fake Faces dataset and a smaller Real and Fake Face Detection dataset—the models are trained and evaluated based on accuracy, loss, and robustness. The paper is structured as follows: Section 2 reviews deepfake generation and detection techniques, Section 3 presents the methodology, Section 4 discusses experiments and results, and Section 5 concludes with future perspectives. This research contributes to the ongoing efforts to strengthen digital media security and combat deepfake threats. [?]. 2 Review methodology and literature Deepfake technology, powered by deep learning and artificial intelligence, has rapidly evolved in recent years, leading to highly realistic synthetic media that are often indistinguishable from authentic content. While this techno-logy has numerous positive applications in entertainment and creative industries, it also poses significant risks, including misinformation, identity fraud, and political manipulation. This section provides a comprehensive overview of deepfake generation techniques, detection methodologies, and the current challenges in combating manipulated media. 2.1 Deepfake Generation Techniques Deepfake generation has rapidly evolved with advancements in deep learning, particularly through the use of Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs). These techniques allow the synthesis of highly realistic images, videos, and audio that are increasingly difficult to 38 distinguish from authentic media. This section provides an overview of the most prominent deepfake generation methods and their impact on digital media.  Generative Adversarial Networks (GANs) Generative Adversarial Networks (GANs) are among the most widely used architectures for deepfake generation. A GAN consists of two competing neural networks: a generator that produces synthetic media and a discriminator that attempts to distinguish between real and generated content. Through iterative training, the generator improves its ability to create highly realistic outputs. Advanced variations, such as StyleGAN and StyleGAN2, have significantly enhanced the quality of generated images by enabling fine-grained control over facial features, expressions, and lighting conditions [1]. Recent studies have also explored the potential of Latent Flow Diffusion (LFD), which incorporates optical flow sequences in the latent space to enhance temporal coherence in deepfake videos [11]. Compared to conventional GANs, LFD provides better preservation of spatial and motion consistency, making generated videos appear more authentic.  Variational Autoencoders (VAEs) and Hybrid Models Variational Autoencoders (VAEs) are another class of generative models used in deepfake generation. Unlike GANs, which rely on adversarial training, VAEs learn a probabilistic representation of data to generate realistic samples. They have been particularly effective in face-swapping applications, where they enable smooth blending of facial features while maintaining structural consistency [15]. Hybrid models combining GANs and VAEs have also gained traction. These models leverage the structured latent space of VAEs with the adversarial refinement of GANs to generate higher-quality deepfakes. The integration of attention mechanisms within these architectures has further improved the realism of generated media by focusing on fine details such as skin texture and micro-expressions [2].  Face manipulation techniques Face manipulation techniques in deepfake generation can be categorized into three main types: Face Swapping: This technique replaces the face of a person in a video with another person’s face while maintaining the original facial expressions and movements. It is commonly implemented using autoencoders and GANs. The DF-Platter dataset [12] demonstrates that face-swapping deepfakes can be generated at both high and low resolutions, highlighting the challenges in detection. Facial Attribute Manipulation: This method alters specific facial features such as age, gender, and expressions. It is achieved using models like StarGAN and AttGAN, which modify targeted attributes while preserving the overall identity of the subject [2]. Such manipulations are widely used in applications ranging from entertainment to identity anonymization. Lip-Sync Manipulation: This technique synchronizes lip movements with an audio track, making it appear as though a person is speaking words they never actually said. Models like Wav2Lip and SyncGAN have demonstrated impressive results in creating realistic lip-sync deepfakes, posing significant challenges in forensic detection [15].  Text-to-Image and Text-to-Video Synthesis With the advent of large-scale generative models, deepfake generation has extended beyond face manipulation to full-body synthesis. Text-toimage and text-to-video synthesis models, such as DALL  E and Stable Diffusion, enable the creation of highly realistic synthetic content based on textual descriptions. These models use diffusion processes to iteratively refine images, resulting in high-fidelity outputs that can be used for both benign and [14]. Furthermore, recent research has explored deepfake phylogeny, which examines how iterative manipulations can evolve deepfakes over multiple generations, leading to increasingly deceptive synthetic media [13]. The DeePhy dataset was developed to study the progression of deepfakes and their impact on detection algorithms.  Challenges in Deepfake Generation While deepfake generation techniques have significantly improved, they present substantial ethical and security concerns. The ability to create highly realistic synthetic media has raised issues related to misinformation, identity fraud, and political propaganda. The development of novel detection techniques must keep pace with advancements in generation methods to mitigate potential risks [20]. Moreover, existing deepfake generation models often suffer from limitations such as excessive computational requirements, data dependency, and difficulty in generating highly dynamic scenes. Researchers are exploring ways to enhance the efficiency and realism of these models while addressing concerns related to misuse and ethical responsibility [4]. Deepfake generation techniques have advanced rapidly with the integration of GANs, VAEs, and hybrid models. Face manipulation methods such as face swapping, attribute manipulation, and lip-syncing have reached new levels of realism, making detection increasingly 39 challenging. The emergence of text-to-image and text-to-video synthesis models has further expanded the capabilities of deepfake technology. However, as generation methods evolve, the need for robust and adaptive detection frameworks becomes more critical. Future research must focus on improving the interpretability of generative models, developing counter-measures against adversarial attacks, and ensuring the ethical use of deepfake technology. 2.2 Deepfake Detection Techniques As deepfake generation techniques continue to evolve, detecting these synthetic manipulations has become an essential challenge in digital media security. Various deep learning-based approaches, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformer-based models, have been developed to distinguish real content from manipulated media. This section provides an overview of state-of-the-art deepfake detection techniques and their effectiveness in different application domains.  Convolutional Neural Networks (CNNs) for Image and Video Detection Convolutional Neural Networks (CNNs) have been widely adopted for deepfake detection due to their ability to extract spatial features from images and videos. CNN-based models analyze inconsistencies in pixel distributions, texture artifacts, and facial asymmetries that may not be perceptible to the human eye. Studies have shown that CNNs, particularly Xception and MobileNet architectures, achieve high accuracy in detecting face-swapping deepfakes, with results ranging between 91% and 98% depending on the dataset used [7]. Despite their effectiveness, CNN-based models face challenges when applied to real-world deepfakes. These models often struggle with generalization across different datasets due to biases introduced during training. Additionally, CNNs primarily focus on spatial features, making them less effective in detecting temporal inconsistencies in deepfake videos [17].  Recurrent Neural Networks (RNNs) and Temporal Analysis To address the limitations of CNNs in video deepfake detection, Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks have been utilized for temporal analysis. These models analyze sequential frames in a video to detect unnatural facial movements, such as inconsistent blinking patterns or unnatural lip-syncing [9]. The use of spatiotemporal convolutional networks has further enhanced the capability of RNN-based approaches. For example, the Celeb-DF dataset benchmark demonstrated that incorporating temporal features significantly improves detection accuracy, outperforming frame-based detection models [10]. However, these approaches remain computationally expensive and require substantial processing power, limiting their feasibility for real-time applications.  Transformer-Based Models for Deepfake Detection Recent advancements in deep learning have led to the adoption of Transformer-based models for deepfake detection. Vision Transformers (ViTs) leverage self-attention mechanisms to capture both local and global dependencies in an image, making them highly effective in detecting subtle deepfake artifacts. Multi-modal Transformer architectures, such as M2TR, integrate RGB and frequency-domain features to improve detection accuracy [10]. Compared to CNNs and RNNs, Transformer-based models demonstrate superior generalization capabilities across different datasets. They are particularly effective in detecting complex deepfakes that incorporate high-quality synthesis techniques. However, their high computational cost remains a challenge, necessitating further research into optimization techniques for practical deployment [6].  Multi-Modal Deepfake Detection Approaches Multi-modal deepfake detection approaches integrate information from multiple sources, such as visual and auditory cues, to enhance detection robustness. Joint audio-visual deepfake detection has been proposed as an effective strategy, leveraging synchronization inconsistencies between speech and facial expressions [22]. These methods have shown promising results in identifying lip-sync deepfakes and voice-cloning manipulations. In addition to audio-visual synchronization, PRNU (Photo-Response Non-Uniformity)-based methods have been explored for deepfake detection. PRNU, commonly used in digital forensics, identifies unique device fingerprints left during the image capture process. Recent studies indicate that PRNU-based approaches can complement deep learning models in hybrid detection frameworks [8]. 40  Challenges in Deepfake Detection Despite advancements in deepfake detection, several challenges remain: Generalization Across Different Datasets: Most deepfake detection models struggle with dataset-specific biases. Methods trained on one dataset often fail to generalize well to unseen deepfakes generated by different techniques [3]. Adversarial Robustness: Adversarial attacks can be used to fool deepfake detection models by introducing imperceptible perturbations. This highlights the need for more robust adversarial training strategies [16]. Real-Time Processing Efficiency: Many state-of-the-art detection models are computationally intensive, making real-time deepfake detection a significant challenge [18].  Future Directions in Deepfake Detection To improve deepfake detection, future research should focus on: Hybrid Detection Models: Combining CNNs, RNNs, and Transformer-based models to leverage their respective strengths. Few-Shot and Zero-Shot Learning: Reducing reliance on large labeled datasets to enhance detection generalization [21]. Blockchain and Forensic Watermarking: Implementing digital watermarking techniques to verify content authenticity and track manipulations [5]. 3 Contribution This section presents the methodology adopted for deepfake detection. It begins with a description of the proposed project, followed by the system architecture and the development process of the models used. The chapter also includes details on the dataset, implementation, and performance evaluation of the deep learning models. 3.1 Project Description Our proposed project consists of two main phases: the generation phase using GANs and the detection phase, where we evaluate two efficient deep learning models—CNN and ViT—to differentiate between real and fake images.  Generation Phase (Using GANs) : In the generation phase, fake images are created using a GAN architecture, which comprises two adversarial neural networks: a generator and a discriminator. The generator takes a random latent vector as input and produces synthetic images, which are then passed to the discriminator. The discriminator, which has access to both real and generated images, is trained to distinguish between them, thereby forcing the generator to improve its ability to create realistic images. This generation process is crucial because deepfake detection models rely on deep learning, which requires large datasets for accurate predictions. However, many existing datasets suffer from low image resolution and are too small to effectively train advanced models like ViT.  Detection Phase (Using CNN & ViT Models) For the detection phase, both the CNN and ViT models receive an image as input. Before processing, the images go through a data preprocessing step to ensure optimal training and testing conditions. The dataset is then split into training and testing sets and fed into either the CNN or ViT model. Once training is complete, we evaluate the model’s performance and save the trained model for real-world predictions. The trained models can then analyze new images and determine whether they are real or fake. The proposed system follows a structured pipeline, as illustrated in the system architecture diagram, which includes both the generation and detection phases, ensuring a robust and efficient deepfake detection approach (See Figure 1). Figure 1. 3.2 Deep Convolutional GAN (DCGAN) Development The Deep Convolutional Generative Adversarial Network (DCGAN) is used for generating fake images. It consists of two main components:  The Discriminator Model The first step is to define the discriminator model. The model must take a sample image from our dataset as input and output a classification prediction as to whether the sample is real or fake. This is a binary classification problem: 41 Figure 1: Workflow of the proposed system 1. Inputs: An image with one channel and a resolution of 256 Ö 256 pixels. 2. Outputs: A binary classification, where the model predicts the likelihood that the input image is real or fake. The discriminator architecture consists of: 3. Five convolutional layers (Conv2D), each followed by: (a) LeakyReLU activation (instead of ReLU) to allow better gradient flow. (b) Batch normalization to stabilize training. (c) Dropout layers to prevent overfitting. 4. A final dense layer with a sigmoid activation function, which outputs a probability score. A final dense layer with a sigmoid activation function, which outputs a probability score. The model is trained using the binary cross-entropy loss function, with the Adam optimizer (learning rate = 0.00015, momentum = 0.5) to ensure stability.  The Generator Model The generator is responsible for creating fake images. It takes a latent vector (random noise) as input and transforms it into a realistic image through a series of upsampling layers. 1. Inputs: A 100-dimensional latent space vector sampled from a Gaussian distribution. 2. Outputs: A three-channel (RGB) image of 256 Ö 256 pixels with values normalized between [0,1]. The generator architecture consists of: 3. A Dense layer that expands the latent vector into a lower-resolution feature map. 4. Reshaping and upsampling layers to progressively increase the spatial resolution. 5. Several transposed convolutional layers (Conv2DTranspose), each followed by: (a) Batch normalization to improve stability. (b) LeakyReLU activation for non-linearity. 6. A final Conv2D layer with a sigmoid activation function, ensuring the output image values remain within the valid range.  GAN Model (Combining Generator & Discriminator) Once both the generator and discriminator are defined, they are combined to form a complete GAN model. The training process follows these steps: 1. The generator creates a batch of fake images from random latent vectors. 2. These fake images are passed to the discriminator, along with real images from the dataset. 3. The discriminator predicts whether each image is real or fake. 42 4. Backpropagation is applied, updating both the generator and discriminator weights to improve their respective performances. 5. This adversarial training continues until the generator produces highly realistic images that can fool the discriminator. By iteratively refining the generator and discriminator, the GAN model learns to generate increasingly convincing fake images, which are later used to train the deepfake detection models. A plot of the model is also created and we can see that the model expects a 100-element point in latent space as input and will predict a single output classification label. Figure 2: Plot of the Composite Generator and Discriminator model in the GAN 3.3 Process Development of DeiT (Data-efficient Image Transformer) The DeiT (Data-efficient Image Transformer) model is an optimized version of the Vision Transformer (ViT), designed for efficient training on smaller datasets. Unlike Convolutional Neural Networks (CNNs), which rely on convolutional layers to extract local features, DeiT leverages the self-attention mechanism to capture both local and global dependencies within an image. This characteristic enables it to recognize complex patterns and structural inconsistencies that may indicate deepfake manipulations. The development process of DeiT follows steps:  Linear Embedding Layer: -The input image is split into fixed-size patches (e.g., 16 Ö 16 pixels). -Each patch is flattened and mapped into a high-dimensional feature space through a learned embedding matrix. -A learnable classification token is added to the sequence, and positional encodings are introduced to preserve spatial relationships.  Transformer Encoder: -The sequence of image patches passes through L identical layers, each containing: -A Multi-Head Self-Attention (MSA) mechanism, which enables the model to analyze relationships between patches. -A Feed-Forward Network (MLP) that applies non-linear transformations for feature enhancement. -Layer Normalization (LN) and skip connections to stabilize training and improve information retention.  Multi-Head Self-Attention (MSA) Mechanism: -Computes attention between all patches, allowing the model to focus on important regions of the image. -Uses Query (Q), Key (K), and Value (V) matrices to determine the weight of each patch in the final representation. Classification and Output: -After passing through multiple transformer layers, the classification token is extracted. 43 -A fully connected layer is applied to classify the image as real or fake. DeiT offers a powerful alternative to CNNs for deepfake detection, particularly when dealing with large datasets. However, due to its reliance on large-scale training data, its performance can be impacted when applied to smaller datasets. In this study, CNN demonstrated higher accuracy on limited data, while DeiT showed better scalability and generalization potential for future deepfake detection improvements [19]. 4 Experiments and Results This section describes the implementation process and the experiments conducted to evaluate the proposed deepfake detection model. The implementation consists of dataset preparation, model training, performance evaluation, and final deployment. 1. Dataset The experimentation of the proposed technique is implemented by using the two datasets : The first one is the “140k real and fake faces” dataset contains 70k real faces from the Flickr dataset collected by Nvidia, as well as 70k fake faces sampled from 1 million fake faces (generated by style GAN) 1. The second is “Real and fake face detection” datasets contain two subfolders training real and training fake. Training real contains 1081 images and training fake contains 960 images, the total dataset is 2041 images 2 2. Parameter Settings The models were trained using the following hyperparameters: Table 1: PARAMETER SETTINGS Model Epochs Batch size Activation Optimizer CNN 20 64 Sigmoid Adam DeiT-Tiny 20 32 ReLU Adam 3. Performance Evaluation and Discution The performance evaluation of the CNN and DeiTTiny models was conducted using key metrics such as training accuracy, validation accuracy, training loss, and validation loss, as summarized in Table II. The results highlight that CNN outperformed DeiT-Tiny, especially on the smaller dataset, achieving 94.15% validation accuracy on the 140K dataset and 81.88% on the Real and Fake Face Detection dataset. In contrast, DeiT-Tiny reached 90.31% and 61.27%, respectively, indicating its difficulty in learning from limited data and reliance on larger training sets for optimal performance. Table 2: PERFORMANCE EVALUATION TABLE Model Dataset Train accuracy Validation accuracy Train loss Validation loss CNN 140K Real and Fake Faces 97.11 % 94.15% 7.47% 14.50% CNN Real and Fake Face Detection 94.58% 81.88% 22.72% 43.95% DeiT-Tiny 140K Real and Fake Faces 90.06% 90.31% 37.20% 37.01% DeiT-Tiny Real and Fake Face Detection 85.05% 61.27% 43.89% 68.44% Figure 4and 5illustrate the CNN model’s stability, with smooth loss curves and consistently high accuracy, confirming its reliability in deepfake detection. Figure 6and Figure 7depict the DeiT-Tiny model’s slower convergence and higher validation loss, suggesting greater training data requirements for stable results. Despite its generalization potential, DeiT-Tiny struggled with small datasets, whereas CNN demonstrated robust and reliable performance across both datasets. Overall, these findings confirm 1https://www.kaggle.com/datasets/ciplab/real-and-fake-face-detection 2https://www.kaggle.com/datasets/xhlulu/140k-real-and-fake-faces 44 Figure 3: Vision transformer architecture that CNN is better suited for real-world deepfake detection applications, particularly when dataset availability is limited. While DeiT-Tiny offers strong generalization, it requires extensive data and longer training times to match CNN’s performance, emphasizing the need for model selection based on dataset size and computational constraints. Figure 4: CNN Model Performance on Real and Fake Face Detection Dataset : (a) Loss function, (b) Accuracy Figure 5: CNN Model Performance on 140K Real and Fake Faces Dataset : (a) Loss function, (b) Accuracy 45 Figure 6: DeiT-Tiny Model Performance on Real and Fake Face Detection Dataset : (a) Loss function, (b) Accuracy Figure 7: DeiT-Tiny Model Performance on 140K Real and Fake Faces Dataset : (a) Loss function, (b) Accuracy 5 Conclusion Deepfake technology presents significant challenges in digital security and misinformation prevention, requiring robust detection mechanisms. This study explored CNN and DeiT-Tiny models for deepfake detection, demonstrating that CNN achieved higher accuracy and stability, especially on smaller datasets, while DeiT-Tiny required larger datasets for optimal performance. Despite advancements, deepfake detection remains complex due to adversarial attacks, dataset biases, and computational constraints. Future research should focus on real-time detection, improving model robustness, and integrating multimodal approaches such as audio and behavioral analysis. This study contributes to enhancing digital media security, emphasizing the need for continuous advancements in AI-driven detection frameworks to combat deepfake threats effectively. References [1] F. Abbas and A. Taeihagh. Unmasking deepfakes: A systematic review of deepfake detection and generation techniques using artificial intelligence. Expert Systems With Applications, 124260, 2024. [2] Z. Akhtar. Deepfakes generation and detection: a short survey. Journal of Imaging, 9(1):18, 2023. [3] A. Heidari et al. Deepfake detection using deep learning methods: A systematic and comprehensive review. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 14(2):e1520, 2024. [4] A. Kaushal et al. A review on deepfake generation and detection: bibliometric analysis. Multimedia Tools and Applications, pages 1–41, 2024. [5] B. Dolhansky et al. The deepfake detection challenge (dfdc) dataset. arXiv preprint, 2020. [6] C. Li et al. A continual deepfake detection benchmark: Dataset, methods, and essentials. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1339– 1349, 2023. 46