Synthetic Cognitive Feedback: Knowledge Erosion by Recursive Training of AI Generative Models

Synthetic Cognitive Feedback: Knowledge Erosion by Recursive Training of AI Generative Models Eric Guizzo∗ Independent Researcher [email protected] Abstract Synthetic Cognitive feedback is a recursive process in which a generative artificial intelligence (AI) model is trained on data it has produced itself. This loop can amplify internal biases, degrade output quality, and detach models from real-world data. As human-generated data becomes scarcer, such systems could increasingly rely on synthetic information, leading to possible scenarios where models are trained solely on outputs of other models. To reflect on this phenomenon, we present a musical piece in which, while a human performer plays, an AI is trained in real time on the performer’s past actions and recursively retrained on its own outputs. As the composition unfolds, the model gradually overrides human control and eventually takes full command of the execution. The work highlights the risks of over-relying on AI while neglecting the development of human knowledge, and encourages reflection on the shifting balance between authorship, originality, and machine-driven creation. 1 Introduction Generative machine learning models are becoming increasingly powerful, gradually replacing human labor across diverse fields and mutating the way knowledge and art are generated and consumed. This growing influence extends beyond automation, reshaping the very logic of cultural production and creative practice. As such tools grow in scale and capabilities, so does their demand for large volumes of training data. In response, researchers started using AI-generated content as input for training models (Schuhmann et al., 2022). However, this practice initiates a degenerative process: when a model’s output becomes the input for its own evolution, a form of synthetic cognitive feedback loop arises. This self-reference reinforces and amplifies models’ own artifacts, resulting in escalating distortions and a gradual loss of fidelity, causing them to lose touch with real-word data 2 (Alemohammad et al., 2023; Shumailov et al., 2024; Gibney, 2024). The use of feedback loops to reveal the inner properties of systems has deep roots in experimental music (Morris, 2007; Haworth, 2021). Among others, Alvin Lucier’s I Am Sitting in a Room (Lucier, 1969) revealed the acoustic fingerprint of a space by recursively re-recording a spoken text. In no-input music the noises produced by a self-patched mixing board are explored as an unpredictable sound palette (Valle, 2012). William Basinski’s Disintegration Loops captured the physical decay of magnetic tape as a powerful reflection on memory and loss (Basinski, 2002). In this work, we extend the feedback loop concept by placing a machine learning model within it, using it not only as a generative mechanism but as a means to interrogate its structural limitations. This approach highlights a broader issue: the consequences of relying exclusively on AI for task execution become increasingly evident as real-world data gradually becomes obsolete. Future models may be either trained on outdated information or on synthetic data produced by other models, creating a self-referential cycle in which defects accumulate and magnify. This perspective prompts reflection ∗Formerly affiliated to City St. George, University of London. 2Also referred as model collapse. Proceedings of the 6th Conference on AI Music Creativity (AIMC 2025), Brussels, Belgium, September 10th-12th Figure 1: Performance structure block diagram. Temporal progression from left to right. on the potential implications of neglecting the augmentation of our direct human knowledge in favor of an exclusive dependence on AI. 2 The Performance We present a musical piece that integrates a human performer with an AI model trained live and subjected to synthetic cognitive feedback. Figure 1 shows the performance’s structure. The performer plays a custom electronic instrument built in Max Msp, controlled via a tactile interface (as detailed in Section 2.1). Meanwhile, the AI model (described in Section 2.2) is simultaneously trained in real time on a dataset of performance scores previously recorded by the same musician. The model continuously receives its own outputs as new training data, creating a cognitive loop effect that gradually degrades the generated scores. Each time a new score is produced, one or more performer’s controls are locked and taken over by the AI, which begins manipulating it by reading the generated information. As the piece progresses, the AI gains control over more parameters, slowly overtaking the instrument’s interface. In the beginning, the AI extends the performer’s capabilities by controlling parameters in a way that complements the original material. However, as feedback artifacts accumulate, the model gradually diverges from the performer’s intent. Eventually, it takes full control of the performance, producing a score entirely disconnected from the human performer’s expressive logic. The piece ends automatically once the generated scores no longer show any meaningful relation to the performer’s original work, completing the transition from human expression to machine recursion. By structuring the piece so that each action corresponds to a distinct sonic output, the audience can immediately discern every movement of the musician and thus understand the machine’s subsequent responses. A visual rendering of the tactile interface is projected during the performance, offering a clear view of what the human and the AI are doing. 2.1 Performer’s Interface Figure 2: Pre-recorded performance scores. All parameters are sampled at fixed intervals and the resulting material is folded in 3-dimensional matrices. The instrument at the heart of the performer’s setup can be played using finger-drumming techniques, triggering both percussive and pitched sound sources and routing them through chains of spectral 2 processors, pitch-shifted delays, saturators, and reverbs for further shaping. The system offers up to 32 controllable parameters, enabling management of the source materials, the effects’ behaviour and their routing. Since some parameters can have contrasting effects, the performer retains the ability to counteract or balance the influence of a parameter determined by the AI 3 . To train the model, the performer recorded a dataset containing 20 8-minutes sessions. The recorded scores can be visualized as RGB images (see Figure 2). During the performance, the instrument continuously communicates with the training system. Each time a degraded score is generated, one or more random controls are locked and driven by the new matrix instead of the performer. 2.2 Self-feeding Model (a) (b) Figure 3: Visual effect of synthetic cognitive feedback training on (a) images and (b) performance scores. From left to right, the cumulative artifacts become increasingly pronounced. While previous works (Alemohammad et al., 2023; Shumailov et al., 2024; Gibney, 2024) demonstrated the effects of training generative models on their own outputs from a scientific perspective, the focus of this proposal is to explore this phenomenon from an artistic point of view. To this end, we developed a deliberately scaled-down setup designed to make feedback artifacts clearly perceptible, 3For example, low-pass filtering can temper the harshness introduced by high saturation. 3 while also allowing us to train models in real time. The scores are generated by a Variational Autoencoder (VAE) (Kingma et al., 2013) 4 , trained on the aforementioned performance matrices dataset. Structurally, the task resembles image generation: the model compresses and reconstructs inputs via an encoder–decoder architecture with four convolutional and transposed convolutional blocks, and a fully-connected bottleneck. Training uses a binary cross-entropy loss and a Kullback–Leibler (KL) divergence term, following standard VAE formulation. The artifacts generated by this network through synthetic cognitive feedback are visualized in Figure 3a using sample images 5 6 ,and in Figure 3b on an actual performance score. The degradation type and amount heavily depend on the network architecture and training hyperparameters (learning rate, epochs, KL weight etc.). These are tuned to ensure that a single training is completed within a few seconds 7 , while degradation unfolds over several minutes. The piece’s duration is calculated to match a target runtime, with the number of degradation steps kept close to the number of performer parameters, ensuring it ends shortly after the AI takes full control. 3 Final Remarks This work presents a musical experiment that reimagines the feedback loop as a critical lens on AI. By training a live generative model and subjecting it to synthetic cognitive feedback, the piece investigates the aesthetic and conceptual implications of recursive learning on synthetic data. As AI takes on an increasingly influential role, it highlights the tension between individual creativity and knowledge shaped by collective contributions, as encoded in AI systems. Beyond its artistic aims, the project prompts deeper reflection on the dangers of self-reinforcing algorithms, underscoring the need to anchor technological advancement in human insight and critical awareness. References Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R. G. (2023). Self-consuming generative models go mad. arXiv preprint arXiv:2307.01850, 4:14. Basinski, W. (2002). The disintegration loops. https://temporaryresidence.com/products/ trr166. Audio recording, originally released by 2062 Records. Gibney, E. (2024). Ai models fed ai-generated data quickly spew nonsense. Nature, 632(8023):18–19. Haworth, C. (2021). Music and cybernetics in historical perspective: Introduction to the special issue edited by christopher haworth and eric drott. Kingma, D. P., Welling, M., et al. (2013). Auto-encoding variational bayes. Lucier, A. (1969). I am sitting in a room. https://www.lovely.com/titles/cd1013.html . Audio recording, reissued by Lovely Music, Ltd., 1981/2001. Morris, J. M. (2007). Feedback instruments: Generating musical sounds, gestures, and textures in real time with complex feedback systems. In ICMC. Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022). Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278–25294. Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., and Gal, Y. (2024). Ai models collapse when trained on recursively generated data. Nature, 631(8022):755–759. Valle, A. (2012). Towards a Typology of feedback Systems. Ann Arbor, MI: Michigan Publishing, University of Michigan Library. 4Any other generative architecture can be used for the same purpose. 5Original input images created by Alice Lorenzon: https://www.alicelorenzon.com 6 Animated audiovisual versions of these processes are available at: https://drive.google.com/drive/ u/0/folders/1fmEzc6usBdIMOmdQBLaX6vxQwlAVboiX . Here the degraded audio is generated using the same synthetic cognitive feedback technique applied to spectrograms. 7With GPU acceleration enabled. 4