Music Buckets: A Novelty-Centric Recommendation Architecture for Infinite Music Discovery

Music Buckets: A Novelty-Centric Recommendation Architecture for Infinite Music Discovery Prithvi Vasireddy College of Engineering Northeastern University Boston, MA Abstract—Modern music streaming services, despite offering vast catalogs, often trap users in algorithmic echo chambers, reinforcing existing tastes rather than fostering genuine discovery. This phenomenon, known as the ”filter bubble,” results from recommendation engines optimizing for accuracy and engagement on familiar content, leading to habitual listening loops and diminished exposure to novel artists. This paper introduces ”Music Buckets,” a novel system paradigm designed to dismantle these bubbles by enforcing a core constraint: a user will never hear the same song twice. This 100% noveltycentric approach fundamentally reframes the objective of music recommendation from preference prediction to perpetual, personalized discovery. We present a comprehensive, scalable microservices architecture engineered to support this paradigm. Key technical contributions include the application of per-user probabilistic data structures (Bloom filters) for efficient, largescale listening history tracking, a robust persistence layer utilizing a polyglot approach with sharded NoSQL databases for high-throughput interaction logging, and a specialized threestage hybrid recommendation engine. This engine is optimized for a perpetual cold-start scenario, combining content-based and collaborative signals to recommend music that is not only novel but also highly relevant to a user’s evolving taste profile. The proposed system architecture and its underlying algorithms offer a viable blueprint for a new generation of music services dedicated to maximizing serendipity and redefining the landscape of music discovery. Index Terms—Recommender Systems, Music Discovery, Novelty, Serendipity, Filter Bubble, Microservices, Bloom Filter, Distributed Databases, Hybrid Recommendation. I. Introduction A. The Paradox of Choice in Music Streaming The digital music era has granted listeners unprecedented access to vast catalogs, with major streaming platforms like Spotify boasting over 100 million tracks. While this abundance presents immense opportunity, it also creates a significant ”paradox of choice”.[2] Users are often overwhelmed by the sheer volume of options, leading to decision fatigue and a tendency to retreat to familiar content. This behavior is a natural cognitive response to an environment of infinite choice, where the effort required to explore and evaluate new music outweighs the perceived benefit.[2] Consequently, the promise of limitless discovery is often unfulfilled, with users defaulting to a small, comfortable subset of the available musical universe. B. The Algorithmic Echo Chamber To combat this overload, streaming services have developed sophisticated recommender systems. These systems, however, are typically designed to optimize for engagement metrics by predicting and reinforcing established user preferences. This creates a powerful feedback loop: the system recommends what it thinks a user will like based on past behavior, and the user’s interaction with these recommendations further solidifies their taste profile in the eyes of the algorithm.[3] While effective at personalization, this process inadvertently constructs an algorithmic ”filter bubble” or ”echo chamber”.[2, 4, 5] Empirical data reveals the extent of this phenomenon. Listeners gravitate toward a core set of favorite songs, with studies indicating that users often cycle through the same 20-30 songs each week.[6] New music constitutes a surprisingly small fraction of consumption, with only 1015% of a typical listener’s weekly streams being tracks they have not heard before.[6] This habitual consumption is algorithmically reinforced, as personalized playlists, which account for over 60% of total streams on platforms like Spotify, are designed to resurface familiar favorites.[6] The result is an ”invisible ceiling of discoverability,” where emerging artists and experimental genres are filtered out, and the user’s musical horizons gradually narrow.[3] This system also has profound economic implications. Artist royalties are often tied to stream counts, with a play typically registered after 30 seconds.[7] The model incentivizes repeat listens, and in some cases, leads to ”playlist manipulation” to artificially inflate stream numbers.[7, 8] This concentrates revenue on a small number of popular, repeatable tracks, creating a challenging economic environment for a wider array of artists. C. Introducing the Music Buckets Paradigm To address these shortcomings, this paper proposes a radical alternative: the ”Music Buckets” paradigm. We define Music Buckets as a music streaming service architected around a single, inviolable constraint: the playback queue is populated exclusively with songs the user has never heard before. This system is a direct countersolution to the filter bubble, fundamentally shifting the recommendation objective from accuracy to discovery. This paradigm reframes the core problem of recommendation. It is no longer about predicting what a user will like from a set of known interactions, but rather predicting what a user will like from the infinite set of items they have *never experienced*. Every recommendation is, by definition, a ”cold-start” item for that user. Therefore, the system’s objective function cannot be simple prediction accuracy. It must be a proxy for ”successful discovery,” a metric that inherently combines relevance, novelty, and long-term user retention. This requires a groundup rethinking of the entire recommendation pipeline, from data structures to machine learning models and evaluation frameworks. By eliminating repeat plays, the Music Buckets model also alters the economic landscape, potentially fostering a more equitable system where a broader range of artists can earn revenue from each user, albeit only once per track. D. Paper Contributions and Structure This paper presents a comprehensive technical blueprint for the Music Buckets system. The primary contributions are: 1) A scalable, resilient microservices architecture designed for a high-throughput, novelty-centric music service. 2) The novel application of per-user probabilistic data structures (Bloom filters) to enforce the ”neverrepeat” constraint efficiently at a massive scale. 3) A detailed design for a polyglot persistence layer, combining relational and NoSQL databases with horizontal sharding to manage metadata and user interactions. 4) A specialized three-stage hybrid recommendation engine that perpetually operates in a cold-start context to deliver relevant and diverse novel content. 5) A robust evaluation framework based on metrics of novelty, diversity, and serendipity, coupled with a rigorous A/B testing protocol for online validation. The remainder of this paper is structured as follows. Section II provides a critical review of existing recommendation paradigms and metrics. Section III details the overall system architecture. Section IV describes the core data structures and persistence layer. Section V presents the design of the recommendation engine. Section VI outlines the evaluation framework. Finally, Section VII concludes the paper and discusses future directions. II. A Critical Review of Music Recommendation Paradigms To build a system capable of perpetual novelty, it is essential to understand the strengths and limitations of existing recommendation techniques. The Music Buckets paradigm places unique constraints that challenge traditional approaches and necessitate a focus on metrics beyond simple accuracy. A. Foundational Recommendation Techniques 1) Collaborative Filtering (CF): Collaborative filtering is the cornerstone of many recommender systems, operating on the principle of ”wisdom of the crowds”.[9, 10] It identifies patterns in a user-item interaction matrix to make predictions. •User-based CF finds users with similar taste profiles (e.g., users who have rated the same songs similarly) and recommends items that these ”neighbors” have liked but the target user has not yet encountered.[11, 12] •Item-based CF calculates similarities between items based on user interaction patterns (e.g., songs frequently listened to in the same session) and recommends items similar to those the user has previously enjoyed.[10] While powerful, CF suffers from critical weaknesses that are severely exacerbated in the Music Buckets model. The most significant are the cold-start problem and data sparsity.[13, 14, 15] The system cannot recommend new songs that have no interaction data, and it struggles with new users who have no history. In Music Buckets, *every* recommendation is a new song to the user, making the item cold-start problem a perpetual, rather than occasional, challenge. Furthermore, with a catalog of millions of songs, the user-item interaction matrix is extremely sparse, making it difficult to find meaningful overlaps between users. 2) Content-Based Filtering (CBF): Content-based filtering recommends items based on their intrinsic properties.[16, 17] For music, these properties include metadata like genre, artist, and release year, as well as audio features extracted directly from the signal, such as tempo, energy, danceability, and Mel-frequency cepstral coefficients (MFCCs).[18, 19] A user profile is built based on the features of items they have positively interacted with, and new items with similar feature profiles are recommended.[18] CBF’s main advantage is its ability to recommend new and unpopular items, as it does not rely on interaction data.[20] However, it has a tendency to create over-specialized recommendations, trapping users in a narrow content space and limiting the discovery of truly novel tastes.[21] B. The Necessity of Hybrid Models Given the limitations of standalone methods, a hybrid approach is essential for the success of Music Buckets.[15, 22, 23] Hybrid models combine CF and CBF to leverage their respective strengths and mitigate their weaknesses.[10, 13, 14] Common hybridization strategies include: •Weighted: Combining scores from separate CF and CBF models. •Switching: Using different models based on context (e.g., using CBF for new users). •Feature Combination: Feeding features from one model into another. This is the most suitable approach for Music Buckets. The core constraint of Music Buckets fundamentally alters the role of collaborative filtering. It cannot be used to directly recommend items, as a song liked by a similar user may have already been heard by the target user. Instead, CF’s primary function shifts to that of a feature engineering tool. The process becomes: 1) Use CF to identify a cohort of ”taste-twin” users. 2) Analyze the aggregate content features of the music this cohort enjoys (e.g., average tempo, common genres, dominant audio textures). 3) This analysis generates a dynamic, target ”taste profile” vector. 4) Use a content-based engine to find *novel* songs from the entire catalog that match this target profile. In this architecture, CF provides the directional signal (”what kind of new music to look for”), while CBF performs the search and retrieval. This synergistic relationship is central to solving the perpetual cold-start problem. C. Beyond Accuracy: Evaluating Modern Recommenders Traditional recommender systems are often evaluated on accuracy metrics like Precision, Recall, or Root Mean Squared Error (RMSE).[24, 25] These metrics measure how well a system predicts known user ratings or interactions. However, for a discovery-oriented system like Music Buckets, these metrics are insufficient and potentially misleading, as they do not measure the quality of the discovery experience.[26] Therefore, we must prioritize alternative evaluation criteria. •Novelty: This measures the system’s ability to recommend items that are unknown or unexpected to the user.[27, 28] In Music Buckets, novelty is a system-level guarantee, but it can be quantified by measuring how deep into the ”long tail” of popularity the recommendations are.[29] •Diversity: This refers to the variety within a single list of recommendations.[4] High intra-list diversity ensures the user is exposed to a range of artists, genres, or styles in a single session, preventing monotonous listening experiences.[30, 31] •Serendipity: This is arguably the most important metric for Music Buckets. Serendipity is defined as the experience of discovering something that is both novel and relevant—a ”happy accident”.[32, 33] A recommendation is serendipitous if it is unexpected but also genuinely enjoyed by the user.[24, 34] Because every recommendation in Music Buckets is guaranteed to be novel, the system’s success hinges entirely on the relevance of that novel content. Therefore, a successful recommendation in this system is, by definition, a serendipitous one. This elevates serendipity from a ”beyond-accuracy” metric to the central Key Performance Indicator (KPI) of the entire platform. The system’s ultimate goal is to maximize the rate of serendipitous discoveries for the user. III. The Music Buckets System Architecture A. Architectural Philosophy: Microservices and Decentralization To build a scalable and resilient platform capable of serving millions of users, a microservices architecture is the logical choice.[35, 36] This architectural style structures the application as a collection of loosely coupled, independently deployable services, each responsible for a specific business capability.[36] This approach offers several advantages over a traditional monolithic architecture: •Scalability: Each service can be scaled independently based on its specific load, allowing for efficient resource allocation.[37] For example, the Interaction Service, which handles a high volume of writes, can be scaled separately from the Metadata Service, which is read-heavy. •Fault Isolation: The failure of a single service does not cascade and bring down the entire application. This resilience is critical for maintaining high availability.[35] •Technological Flexibility: Teams can choose the most appropriate technology stack (databases, programming languages) for their specific service, enabling a polyglot persistence model.[36] •Independent Deployment: Services can be updated and deployed independently, accelerating development cycles and enabling continuous integration and deployment (CI/CD).[35] To visualize this architecture, we adopt principles from the C4 model, which provides a hierarchical way to describe a system at different levels of detail (Context, Containers, Components, Code), ensuring clarity for various stakeholders.[38] B. High-Level Architecture The overall system architecture is depicted in Fig. 1. Client applications (iOS, Android, Web) communicate with the backend through a centralized API Gateway. The gateway is responsible for request routing, authentication, rate limiting, and SSL termination, acting as a single entry point to the microservices ecosystem.[37] The backend itself is composed of several core services that communicate via synchronous (REST/gRPC) and asynchronous (event bus) protocols. C. Core Service Breakdown Each microservice encapsulates a distinct business capability: •User Service: Manages user identity, authentication (e.g., via OAuth2), profiles (username, preferences), iOS Client Android Client Web Client API Gateway User Service History Service Recommendation Service Interaction Service User DB (Postgres) History Cache (Redis) Interaction DB (Cassandra) Event Bus (Kafka) Metadata Service Streaming Service Metadata DB (Postgres) CDN Auth Get Recs Like/Skip Get Stream hasHeard? User Profile Features Publish Event Subscribe Fig. 1. High-Level Microservices Architecture for Music Buckets. Clients interact via an API Gateway, which routes requests to specialized backend services. Services communicate via synchronous REST/gRPC calls (dashed lines) and an asynchronous event bus (solid lines to/from Kafka) for real-time updates. and subscription status. It is the canonical source for user data.[39] •History Service: This is the cornerstone of the ”neverrepeat” rule. Its sole responsibility is to maintain the probabilistic data structure (detailed in Section IV) for each user’s listening history. It exposes a simple, high-performance API endpoint, e.g., /hasHeard?userId=<id>&songId=<id>, which returns a boolean. This service is designed for extremely high read throughput. •Recommendation Service: The most complex service, responsible for generating the personalized queue of novel music. Internally, it is structured as a threestage pipeline: Candidate Generation, Scoring, and Re-ranking, as detailed in Section V.[40, 41] It consumes events from the event bus to update user models in near real-time. •Interaction Service: A write-intensive service that ingests all real-time user feedback events, such as likes, dislikes, skips, and track completion signals (e.g., played > 90% of duration). It persists these raw interactions to a NoSQL database and publishes structured events (e.g., song_liked, song_disliked) to an event bus for downstream consumption by other services.[35] •Metadata Service: Acts as an abstraction layer over the music catalog database. It provides other services with clean, structured access to song and artist metadata, including genres, release dates, and precomputed audio features. •Streaming Service: Manages the technical aspects of audio delivery. This includes ingesting raw audio files, transcoding them into multiple adaptive bitrates (e.g., HLS or DASH formats), and delivering the audio segments to users via a Content Delivery Network (CDN) for low-latency global playback.[35, 37] D. Inter-Service Communication The communication patterns are chosen to match the needs of the interaction: •Synchronous (REST/gRPC): For request/response interactions where an immediate answer is required, such as the client fetching recommendations from the API Gateway, or the Recommendation Service querying the History Service, REST or gRPC protocols are used.[37] gRPC is often preferred for internal serviceto-service communication due to its performance benefits and strongly-typed schemas. •Asynchronous (Event Bus): For decoupling services and handling real-time data streams, an event bus like Apache Kafka is employed.[35] When the Interaction Service receives a ”dislike” event, it doesn’t need to know which services care about this information. It simply publishes a song_disliked event to a Kafka topic. The Recommendation Service, and potentially others (e.g., an analytics service), can subscribe to this topic and react accordingly. This event-driven approach enhances resilience and scalability. IV. Core Data Structures and Persistence Layer The data layer is the foundation of the Music Buckets system, engineered for massive scale, high throughput, and the unique constraints of the ”never-repeat” rule. This requires a carefully chosen combination of specialized data structures, databases, and scaling strategies. A. Enforcing the ”Never-Repeat” Constraint at Scale 1) The Challenge: The core functional requirement is to check, for every potential song recommendation, whether the user has heard it before. A naive approach would be to store every listening event (userId, songId) in a relational database and query it with SELECT 1 FROM listening_history WHERE userId =? AND songId =?. For a system with 100 million users, each having listened to an average of 5,000 songs, this table would contain 500 billion rows. Indexing and querying such a table for every single candidate in a recommendation list (potentially hundreds of checks per second per active user) is computationally and financially infeasible. 2) Proposed Solution: Per-User Bloom Filters: We propose using a Bloom filter, a space-efficient probabilistic data structure, to perform this membership test.[42] A Bloom filter represents a set using a bit array of size m and kindependent hash functions. To add an element, it is hashed ktimes, and the bits at the resulting indices in the array are set to 1. To query for an element, it is also hashed ktimes; if all corresponding bits are 1, the element is *probably* in the set. If at least one bit is 0, the element is *definitely not* in the set.[42, 43] The critical property of a Bloom filter is that it allows for false positives but guarantees no false negatives.[44] In our context: •False Positive: The filter reports a user has heard a song when they have not. The consequence is that one novel song is incorrectly removed from their potential recommendation pool. In a catalog of millions, this is an acceptable and minor degradation of the user experience. •False Negative: The filter reports a user has not heard a song when they have. This is not possible with a Bloom filter. This guarantee is what makes it suitable for our strict ”never-repeat” rule. The false positive probability can be tuned by adjusting the size of the bit array (m) and the number of hash functions (k) for an expected number of elements (n). This allows us to achieve a desired accuracy (e.g., < 0.1% false positive rate) with a memory footprint that is orders of magnitude smaller than storing the raw data. 3) Implementation: Each user is assigned their own Bloom filter. When a user completes a song, the songId is added to their filter. The filter itself, being a compact binary object, is stored in a distributed cache (like Redis) for microsecond-latency access by the History Service. This design transforms an expensive database query into a near-instantaneous in-memory computation. Table I compares this approach to alternatives. TABLE I Comparison of Membership Query Data Structures Data Structure Space Query Time Suitability Hash Set O(n)O(1) avg. Infeasible memory at scale. Database Index O(n)O(log n)High latency and I/O cost. Bloom Filter O(1)*O(k)** Optimal. Low FP rate for massive space savings. *Constant space for a given FP rate and capacity. **Constant time, where kis the number of hash functions. B. Database Design and Optimization A single database technology is insufficient for the varied workloads of the system. We adopt a polyglot persistence approach. 1) Metadata Store (PostgreSQL): For the music catalog, which consists of highly structured and relational data, a relational database like PostgreSQL is ideal. We leverage a schema inspired by the MusicBrainz project, which provides a comprehensive model for musical entities.[45, 46] The core entities include artists, release groups (albums), releases (specific versions of an album), recordings (audio), and works (compositions). A simplified Entity-Relationship Diagram is shown in Fig. 2. This data is read-heavy and updated infrequently (only when new music is added), making it a perfect fit for a traditional RDBMS. Artist Release Recording Work performs part of is a Fig. 2. Simplified ERD for the Metadata Store, inspired by the MusicBrainz schema, showing core entities and relationships. 2) Interaction Store (Apache Cassandra): User interactions (listens, likes, dislikes, skips) represent a massive, continuous stream of write operations. For this workload, a NoSQL, wide-column store like Apache Cassandra is superior.[47, 48] Cassandra is designed for high write throughput, horizontal scalability, and high availability with no single point of failure.[47] We design our Cassandra tables based on the queries they must support (”query-first design”). The primary use case is retrieving a user’s recent interaction history. A suitable schema is: CREATE TABLE user_interactions ( user_id uuid, event_time timestamp, song_id uuid, event_type text, -- ’like’, ’dislike’, ’skip’ PRIMARY KEY (user_id, event_time) ) WITH CLUSTERING ORDER BY (event_time DESC); This schema partitions the data across the cluster by user_id. Within each partition, rows are clustered and sorted by event_time, making time-series queries for a specific user extremely efficient.[49] 3) Scalability via Horizontal Sharding: As the user base grows, even a NoSQL database on a single cluster can become a bottleneck. We employ horizontal sharding, a technique that partitions the data across multiple independent database clusters.[50, 51] We use a hash-based sharding strategy with the user_id as the shard key.[52, 53] A hash function is applied to the user_id to determine which physical shard (database cluster) the user’s data resides on. This ensures an even distribution of data and load, allowing the system to scale near-linearly by adding more shards.[50, 54] C. High-Performance Caching Layer To minimize latency and reduce the load on our persistence layers, we implement a distributed in-memory caching layer using a technology like Redis or Memcached.[55] The cache stores frequently accessed data that changes infrequently. We primarily use the cache-aside (lazy loading) pattern.[56, 57, 58] As shown in Fig. 3, the application logic is as follows: 1) The application requests data from the cache. 2) If the data exists (a cache hit), it is returned immediately. 3) If the data does not exist (a cache miss), the application queries the primary database. 4) The application then populates the cache with the data retrieved from the database before returning it. This pattern ensures that the cache is only populated with data that is actively being used. Data ideal for this strategy includes user profiles, pre-computed user taste vectors for the recommendation model, and, most critically, each user’s Bloom filter. For critical data that must remain consistent, such as an updated user taste profile after a model retrain, a write-through strategy can be used, where the cache is updated simultaneously with the database.[56, 57] Application Cache Cache Hit? Database 1. Request Yes 2. Return Data No 3. Query DB 4. Populate 5. Return Data Fig. 3. Data Flow for a Cache-Aside (Lazy Loading) Read Operation. V. The Recommendation Engine for Perpetual Novelty The recommendation engine is the intelligent core of Music Buckets. It is designed to operate in a state of perpetual cold-start, where every item it recommends is new to the user. This requires a multi-stage, hybrid approach that balances relevance, diversity, and serendipity. The engine follows a standard three-stage architecture: Candidate Generation, Scoring, and Re-ranking [40, 41], as illustrated in Fig. 4. A. Stage 1: Candidate Generation The goal of this stage is to efficiently narrow down the millions of tracks in the catalog to a smaller, diverse set of a few hundred candidates that are highly likely to be relevant to the user. Critically, these candidates must also be novel. 1) Hybrid Candidate Sourcing: We employ multiple strategies in parallel to generate a rich candidate pool: 1) Content-Based Candidates: This is the primary source of novelty. We analyze the audio features Stage 1: Candidate Generation - Content-Based (Audio Similarity) - Collaborative (Taste-Twins) - Serendipity-Boosting Stage 2: Scoring - LightGBM / Neural Net Model - Predicts relevance score - Features: User, Item, Context Stage 3: Re-ranking & Filtering - Apply Business Logic - Boost Diversity/Novelty - History Filter User Profile & Music Catalog Final Top-N Playlist ∼500 Candidates Scored List History Service (Bloom Filter Query) For each candidate... Filter ’heard’ songs Fig. 4. Three-Stage Recommendation Engine Pipeline. A large pool of candidates is generated, scored for relevance, and finally re-ranked and filtered against the user’s history to produce the final playlist. of songs the user has recently liked. These features include low-level signal properties like MFCCs, spectral contrast, and tempo, as well as higher-level attributes like danceability, energy, and valence.[59, 60, 61] Using libraries like Librosa or audioflux, we can extract a feature vector for every song in the catalog.[1, 60, 62] A user’s ”taste vector” is computed as the centroid of the feature vectors of their liked songs. We then use an approximate nearest neighbor (ANN) search (e.g., using Faiss or ScaNN) to find songs in the catalog with feature vectors close to the user’s taste vector. 2) Collaborative-Based Candidates (Taste Profile Matching): As discussed in Section II, we use collaborative signals to guide content-based discovery. We first identify a cluster of ”taste-twin” users who have similar like/dislike patterns using techniques like MinHash or matrix factorization on the interaction data. We then analyze the listening histories of these taste-twins and source candidate songs that they have enjoyed but which are novel to the target user. This injects recommendations that the user’s own content profile might not have discovered.[1, 9] 3) Serendipity-Boosting Candidates: To prevent overspecialization, a small percentage of candidates are sourced from genres or artist clusters that are adjacent to, but not directly within, the user’s core taste profile. This can be achieved using a knowledge graph of artists and genres, finding nodes that are one or two hops away from the user’s most-liked entities. This controlled exploration is key to fostering long-term engagement and taste expansion.[34] B. Stage 2: Scoring Once a candidate set is generated, the scoring stage ranks these candidates based on their predicted relevance to the user. This is a classic machine learning ranking problem. 1) Machine Learning Model: We use a lightweight gradient-boosted decision tree model, such as LightGBM, or a shallow neural network for this task. These models are efficient enough to score hundreds of candidates in real-time while being powerful enough to capture complex interactions between features. The model is trained to predict a preference score (e.g., probability of a ’like’). 2) Feature Engineering: The model’s performance depends heavily on the quality of its input features. We construct a feature vector for each (user, song) pair, including: •User Features: A vector embedding representing the user’s historical likes and dislikes, distributions of preferred audio features (e.g., average energy, variance of tempo), and preferred genres. •Item Features: The song’s full audio feature vector, artist popularity, release recency, and genre information. •Interaction Features: The cosine similarity between the candidate song’s audio feature vector and the user’s current taste vector. This feature explicitly captures the content-based relevance. C. Stage 3: Re-ranking and History Filtering This final stage applies the system’s core business logic and constraints to the scored list of candidates. 1) History Filtering (The Critical Step): This is where the ”never-repeat” rule is enforced. The scored and ranked list of candidates is passed through a final filter. For each candidate song, the Recommendation Service makes a high-speed call to the History Service’s endpoint: /hasHeard(userId, songId). If the response is true (indicating a probable match in the user’s Bloom filter), the song is definitively removed from the list. This is an O(k) operation per candidate (where kis the number of hash functions), making it extremely fast and efficient. 2) Re-ranking Logic: After the hard constraint of history filtering is applied, a set of softer business rules can be used to re-rank the remaining candidates: •Diversity Promotion: Ensure that the top-N items in the final list are not all from the same artist or microgenre. An algorithm like Maximal Marginal Relevance (MMR) can be used to penalize similarity among topranked items. •Emerging Artist Boost: A small score bonus can be applied to songs from new or less popular artists to aid discoverability. •Real-time Feedback Integration: The system must be responsive to immediate user feedback.[63, 64] If a user dislikes or skips a song, the re-ranking logic can immediately down-rank or remove other songs in the upcoming queue that have a high audio similarity to the rejected track. This prevents a negative feedback loop within a single session. Algorithm 1 provides a pseudo-code representation of the end-to-end recommendation process. Algorithm 1 Music Buckets Hybrid Recommendation 1: procedure GetRecommendations(userId) 2: user_profile ←UserService.GetUserProfile(userId) 3: liked_songs ←InteractionService.GetLikes(userId) .Stage 1: Candidate Generation 4: candidates ←Set() 5: candidates.add(GetContentBasedCandidates(liked_songs)) 6: candidates.add(GetCollaborativeCandidates(userId)) 7: candidates.add(GetSerendipityCandidates(user_profile)) .Stage 2: Scoring 8: scored_candidates ←List() 9: for song in candidates do 10: score ←ScoringModel.predict(user_profile, song) 11: scored_candidates.add((song, score)) 12: scored_candidates.sort(by=score, reverse=True) .Stage 3: Re-ranking & Filtering 13: filtered_candidates ←List() 14: for (song, score)in scored_candidates do 15: if not HistoryService.HasHeard(userId, song.id)then 16: filtered_candidates.add((song, score)) 17: final_playlist ←ApplyRerankingLogic(filtered_candidates) 18: return final_playlist.top(N) VI. Evaluation Framework and Experimental Design Evaluating a system like Music Buckets requires a departure from traditional recommender system metrics. Success is not defined by predicting known preferences but by the quality and impact of the discovery experience. Therefore, our evaluation framework combines offline metrics tailored for novelty with a rigorous online A/B testing protocol to measure real-world user behavior. A. Offline Evaluation Metrics Offline evaluation is performed on historical data to iterate on models quickly without deploying them to users. Given that accuracy metrics like RMSE are unsuitable, we propose a suite of metrics focused on the characteristics of the generated recommendation lists.[13] These are summarized in Table II. B. Online A/B Testing Protocol While offline metrics are useful for model selection, the ultimate test of the system’s effectiveness is its impact on user behavior. We propose a rigorous online A/B testing framework to measure this impact directly.[65, 66] 1) Hypothesis and KPIs: A clear, quantifiable hypothesis is the foundation of a successful experiment.[67, 68] •Hypothesis: ”Switching from a traditional recommendation algorithm (Control) to the Music Buckets system (Variant) will lead to a 15% increase in 30day user retention, driven by higher engagement from continuous music discovery.” •Primary KPI: 30-day user retention rate. This is the ultimate measure of long-term user satisfaction. •Secondary KPIs: – Average session length – Number of positive interactions (likes) per session – User churn rate – Diversity of genres/artists in a user’s listening history over time These metrics provide a holistic view of user engagement, moving beyond simple click-through rates to measure deeper satisfaction and exploration.[69] 2) Experimental Design: The experiment must be designed to produce statistically significant and unbiased results. •Groups: Users will be randomly assigned to one of two groups upon signup. – Control Group: Receives recommendations from a baseline hybrid model that balances familiarity and novelty (e.g., a standard Spotify-like algorithm). – Variant Group: Experiences the full Music Buckets system. •Sample Size and Duration: A power analysis must be conducted beforehand to determine the minimum sample size required to detect the minimum sample size required to detect the hypothesized effect with a statistical significance level (alpha) of 0.05 and power (1-beta) of 0.8. The test should run for at least two full business cycles (e.g., 2-4 weeks) to average out weekly usage patterns and account for novelty effects wearing off.[67, 69] •Isolation and External Factors: It is crucial to isolate the groups to prevent network effects or crosscontamination.[70] The user experience (UI, latency) must be identical for both groups, with the only difference being the recommendation logic.[68] External events like holidays or marketing campaigns that could influence behavior must be documented and controlled for in the analysis.[69] By combining targeted offline metrics with a robust online testing framework, we can comprehensively evaluate the Music Buckets system and validate its potential to redefine user engagement in music streaming. TABLE II Evaluation Metrics for Novelty-Centric Systems Metric Definition Formula Rationale for Music Buckets Coverage The percentage of the total item catalog that appears in any user’s recommendation list over an evaluation period. Coverage =|Su∈URu| |I|Measures the system’s ability to expose the long tail of the catalog. A high coverage indicates the system is not just focusing on a small subset of items.[31] Intra-List Diversity (ILD) The average pairwise dissimilarity between all items in a recommended list, averaged over all users. ILD =1 |U|Pu∈U Pi,j∈Ru,i6=jd(i,j) |Ru|(|Ru|−1) Quantifies the variety within a single recommendation session. High ILD prevents monotony and exposes users to a broader range of content.[30, 31] d(i, j) is a dissimilarity metric like 1− cosine_similarity. Novelty Score The average self-information of recommended items, where an item’s self-information is inversely related to its global popularity. Novelty = −1 |U|Pu∈U 1 |Ru|Pi∈Rulog2p(i) Measures how surprising or nonobvious the recommendations are. p(i)is the probability of item iappearing, estimated from its global play count. This rewards recommending less popular ”long-tail” items.[29, 31] U: set of users, I: set of all items, Ru: list of recommended items for user u. VII. Conclusion and Future Directions A. Recapitulation This paper has introduced Music Buckets, a novel paradigm for music recommendation that prioritizes perpetual discovery by ensuring a user never hears the same song twice. We have presented a comprehensive blueprint for such a system, addressing the significant technical challenges it poses. The proposed solution is a scalable microservices architecture where each component is optimized for its specific task. The ”never-repeat” constraint is enforced at scale through the efficient use of per-user Bloom filters, a probabilistic data structure that offers massive space and time complexity advantages over traditional methods. The persistence layer employs a polyglot strategy, using a sharded NoSQL database for high-throughput interaction data and a relational database for structured music metadata. At the core of the system is a specialized three-stage hybrid recommendation engine, designed to operate in a continuous cold-start environment by synergistically combining collaborative signals with content-based features to generate a stream of relevant, diverse, and entirely novel recommendations. This architecture demonstrates that a system built on the principle of 100% novelty is not only theoretically conceivable but also technically feasible. B. Discussion of Limitations and Challenges Despite the robust design, the Music Buckets paradigm presents inherent challenges that must be acknowledged. •Catalog Exhaustion: While the vastness of modern music catalogs makes this a distant concern for most users, a hyper-active listener with niche tastes could theoretically exhaust a significant portion of their preferred genres over a long period. This would lead to a degradation in recommendation quality as the system is forced to suggest increasingly peripheral content. •The ”Bad First Date” Problem: The system’s success is highly dependent on its ability to quickly learn a new user’s taste profile. A series of poor initial recommendations—a ”bad first date”—could cause a user to churn before the system has gathered enough feedback (likes/dislikes) to form an accurate model of their preferences. The initial onboarding and feedback collection process is therefore critical to user retention. C. Future Work The Music Buckets framework opens several exciting avenues for future research and development. •Context-Aware Novelty: The current model primarily considers a user’s historical taste. A significant enhancement would be to incorporate real-time contextual signals, such as time of day, location, or user activity (e.g., ’workout’, ’focus’, ’relaxing’) inferred from device sensors or user input. The recommendation engine could then be tasked with finding novel music that not only matches the user’s general taste but also fits their immediate context, dramatically increasing the relevance and utility of the recommendations. •Dynamic Probabilistic Data Structures: While Bloom filters are highly effective, they are static. As a user’s listening history grows into the tens of thousands, the false positive rate will increase. Future work could explore the use of more advanced data structures like Cuckoo filters, which support deletions, or