A Practical Exploration of discv5 in Distributed Applications

A Practical Exploration of discv5 in Distributed Applications Bachelor Thesis to fulfill the requirements for the degree of Bachelor of Science (B.Sc.) at the Hochschule f¨ur Technik und Wirtschaft Berlin Faculty: Angewandte Informatik Program: Applied Computer Science First Examiner: Prof. Dr. Thomas Schwotzer Second Examiner: Prof. Adrianna Alexander Submitted by Youssef Chammam Submission Date: 11 August 2024 Abstract This thesis explores the discv5 discovery protocol, analyzing its role and implementation in decentralized network systems. Beginning with an overview of distributed networks and their architectural overlays, the study delves into the operational principles of the Kademlia Distributed Hash Table (DHT) as a foundational technology for discv5. The evolution from discv4 to discv5 is scrutinized, focusing on the enhancements and their implications for decentralized networking. The practical component of the thesis presents the integration of the discv5 Rust library into a prototype decentralized network. This network utilizes discv5 for node discovery and dynamic routing table management, coupled with a DHT for decentralized data storage, presenting a model for key-value storage in distributed environments. The implementation details are discussed, showcasing how the discv5 library can be adapted for real-world applications. Extensive testing of the library under various scenarios—including IP changes, ENR updates, and concurrent requests—provides insights into its resilience and e↵ectiveness against security threats, such as eclipse attacks. These tests highlight the library’s capabilities and limitations within a simulated decentralized social media platform, demonstrating its potential to enhance privacy and reduce reliance on central authorities. The findings underscore the viability of discv5 as a critical component for decentralized applications, emphasizing its security features, scalability, and adaptability. This thesis not only extends the theoretical understanding of distributed discovery protocols but also contributes practical insights into their application in enhancing decentralized digital ecosystems. Contents 1 Introduction 3 1.1 Motivation.................................. 3 1.2 Project Objective & Proceeding . . . . . . . . . . . . . . . . . . . . . . 4 2 State of the Art 4 2.1 DistributedSystems ............................ 5 2.1.1 Introduction to Distributed Systems . . . . . . . . . . . . . . . 5 2.1.2 Characteristics ........................... 5 2.1.3 Categories of Node Discovery in Distributed Systems . . . . . . 7 2.2 KademliaDHTs............................... 9 2.2.1 Introduction to DHTs . . . . . . . . . . . . . . . . . . . . . . . 9 2.2.2 Theoretical Foundations of Kademlia . . . . . . . . . . . . . . . 11 2.2.3 Distance Calculation using XOR . . . . . . . . . . . . . . . . . 11 2.2.4 Routing Table Structure . . . . . . . . . . . . . . . . . . . . . . 12 2.2.5 Bucket Management . . . . . . . . . . . . . . . . . . . . . . . . 12 2.2.6 Security and Efficiency in Kademlia . . . . . . . . . . . . . . . . 13 2.3 Discv5Protocol............................... 14 2.3.1 A brief overview of Discv4 . . . . . . . . . . . . . . . . . . . . . 15 2.3.2 Ethereum Node Records . . . . . . . . . . . . . . . . . . . . . . 15 2.3.3 Introduction to Discv5 . . . . . . . . . . . . . . . . . . . . . . . 18 2.3.4 Wire Protocol of Discv5 . . . . . . . . . . . . . . . . . . . . . . 19 2.3.5 Potential Goals of Discv5 . . . . . . . . . . . . . . . . . . . . . . 22 2.3.6 Features and Mechanics of Discv5 . . . . . . . . . . . . . . . . . 25 3 Discv5 in Practise 28 3.1 Introduction................................. 28 3.2 Exploring the Rust Discv5 Library . . . . . . . . . . . . . . . . . . . . 28 3.2.1 Overview .............................. 28 3.2.2 Discv5 API Overview . . . . . . . . . . . . . . . . . . . . . . . . 30 3.3 Analysis................................... 34 3.3.1 ProblemAnalysis.......................... 34 3.3.2 Application Environment . . . . . . . . . . . . . . . . . . . . . . 34 3.4 Solution Propositions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.5 Evaluation.................................. 36 3.5.1 Federated Server Model with discv5 . . . . . . . . . . . . . . . . 36 3.5.2 Topic-based Node Discovery & Clustering . . . . . . . . . . . . 36 3.6 Decision & Justification . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4 Example Application 37 4.1 Architecture................................. 37 4.2 Discv5 Functionalities Code Implementation . . . . . . . . . . . . . . . 39 1 4.2.1 Starting the Discv5 server . . . . . . . . . . . . . . . . . . . . . 39 4.2.2 Bootstrapping the Discv5 Server . . . . . . . . . . . . . . . . . . 42 4.2.3 NodeDiscovery........................... 44 4.2.4 Discv5 Event Streams . . . . . . . . . . . . . . . . . . . . . . . 46 4.2.5 TALK Request & TALK RESP .................. 48 4.2.6 DHT Interface Integration with Discv5 . . . . . . . . . . . . . . 51 4.3 Demonstration ............................... 54 5 Testing & Evaluation of Discv5 under di↵erent Scenarios 56 5.1 ToolsOverview ............................... 56 5.2 Setting up our Testground Environment . . . . . . . . . . . . . . . . . 56 5.3 FindNode.................................. 57 5.4 IPChange.................................. 59 5.5 ENRUpdate ................................ 62 5.6 Concurrent Requests in Discv5 . . . . . . . . . . . . . . . . . . . . . . . 63 5.6.1 General Concurrent Requests . . . . . . . . . . . . . . . . . . . 63 5.6.2 Concurrent Requests before establishing Secure Session . . . . . 65 5.7 EclipseAttack................................ 67 5.8 ResultsEvaluation ............................. 68 6 Outline 69 6.1 Conclusion.................................. 69 6.2 FutureWork................................. 70 List of Image Sources 74 Abbreviations 77 2 1 Introduction 1.1 Motivation Decentralization has become increasingly popular over the course of the recent years due to their potential of distributing the power by eliminating the role of a central entity to govern a system. By distributing the governance of software across several nodes, this approach to data storage and management has recently gained in popularity and have proven their potential and benefits through the lack of a central authority and the high increase in privacy and freedom. However, designing such models require sophisticated and careful measures to make sure it complies with its nature while avoiding breaches to the whole system and remaining scalable and fast. Over the last couple of months, X (also known as Twitter), has been taken over by Elon Musk in an attempt at ”freedom of speech”. His status and unconventional way of handling it, removing all rules, has made it increase in popularity drastically and is now considered the platform for freedom of speech by many, but it takes 5 minutes on the platform to see how it is nowhere near it. It is still the same centralized platform it used to be, except that it now has a new leader. Shadow-banning users, suppressing content and promoting other content is subjective to one’s power over the platform. The world is waking up to the need of a free platform, especially in unstable periods, and is coming to the realization that true freedom of speech is not a freedom that is decided by a single man, or a single community. The algorithm used is what governs the platform and rules it, and several attempts at decentralizing social media have risen. But the choice of technology used behind these platforms are not adequate for their purpose. Recognizing how unethical X has become with no other real competitor that is making a real attempt in this mission, this project is an attempt at showing how the discv5 (discovery V5) technology can be used to fulfill this purpose. Discovery V5 is an extension to the Kademlia discovery protocol, with the intention of preventing attacks that exploit unverified endpoint information [23]. This bachelor thesis aims to explore the Rust library of the discv5 protocol, by understanding its functionalities on the networking level and demonstrating its application through the development of a simple, practical data storage, as well as testing the library under circumstances to represent how it reacts to given scenarios. The research will assess the current landscape by investigating the existing frameworks and protocols underpinning distributed systems, with a particular emphasis on the role and significance of Distributed Hash Tables in enhancing privacy, security and distribution. It provides a comprehensive examination of the Discv5 protocol including its design and functionalities. Furthermore, it will bridge the gap between theoretical 3 understanding with a practical application o↵ering insights into the challenges and opportunities of working with these technologies. Through this work, we will explore how networking protocols like discv5 can lead the path for the next generation of decentralized applications. 1.2 Project Objective & Proceeding The objective of this bachelor thesis is to explore the Discv5 library practically as a mean to making a decentralized storage system for decentralized networks. The primary objective being how the discv5 protocol can be integrated on the networking layer, exploring its functionalities and usability for networking purposes. Once the discv5 is functional and well operating within our network, we will view how it can be integrated with another layer of key/value storage using Distributed Hash Tables across the nodes, for a fully decentralized network. Finally, we will run discv5 through several tests to analyze and assess how it handles several scenarios that can occur between nodes participating in the network. Meaning the final goal would be to have a deep exploration of discv5 theoretically, and to reinforce the theory learned by implementing it practically within an application to demonstrate its usability within a decentralized network. We will conclude with this paper on how it can serve as a laying foundation for networking over a decentralized network. 2 State of the Art This section is dedicated to exploring key concepts and techniques that form the foundation of the node discovery V5 protocol. Understanding these concepts is crucial for developing and implementing an application based on discv5. We begin by delving into the fundamentals of distributed systems in general, to assess where Distributed Hash Tables originate from and what problem they solve. Next, we dive into the fundamentals of Distributed Hash Tables (DHTs), which is a distributed system that provides a lookup service similar to a hash table. Key–value pairs are stored in a DHT, and any participating node can efficiently retrieve the value associated with a given key. [4]. DHTs, with their fast and reliable way of finding neighbouring nodes without the need of a central indexing node, are essential in building a decentralized application for data storage and retrieval. [4] This part will emphasize on Kademlia, an iteration over the concept of Distributed Hash Tables (DHTs). It is a specific type of DHT algorithm that was designed to improve the efficiency and performance of peer-to-peer networks by reducing the complexity of data look-ups. [30] This part will finally cover the Discv5 protocol and how it implements the Kademlia algorithms solely for node discovery, excluding data retrieval. Specifically, it will have an in-depth coverage of its functionalities, while moving within its Discv5 library to comprehend the theoretical concepts proposed by it. 4 By comprehending these fundamental concepts, we can have a deep knowledge of how discv5 works and we lay the groundwork for developing a decentralized application based on this protocol.These concepts serve as the building blocks for the subsequent stages of our work. 2.1 Distributed Systems 2.1.1 Introduction to Distributed Systems ”A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable.” -Leslie Lamport [34] (page 4). This observation humorously yet e↵ectively captures the interconnected and often complex nature of distributed systems, where components spread across multiple networked computers interact in ways that the failure of one can impact the functioning of others. These systems enhance performance, reliability and scalability by distributing tasks across multiple nodes, which can execute tasks simultaneously or sequentially. Distributed systems are networks of independent computers that work together forming a single coherent system in order to achieve a common goal [33] (page 2). These systems leverage multiple computers, also called nodes to build a network of nodes that execute common tasks. A big range of possibilities are open, including sharing data across computers, communicating between them through packets, or emitting computational work results as events to one another, sharing information about the current state of the whole ecosystem. However, these advantages of distributed networks can have potential underlying issues that need to be handled with careful consideration, including more complex software, degradation of performance and often weaker security [33] (page 31). They are nevertheless at the core of decentralization, since with the help of architectures, when the nodes involved are not held by the same entity, migrate the system from having one central authority over the network (A), Figure 1 and data shared to a more cohesive system (B & C), Figure 1. 2.1.2 Characteristics We can view a Decentralized Application as a sum of two important components: Data Storage and Node Discovery. Both Discv5 and Kademlia DHT o↵er frameworks that can be tuned to meet these requirements. The realization of a distributed system requires that we develop a piece of software that is placed on the nodes across the network. But these nodes still need to find each other and to maintain a connection.Therefore, the architecture of the system depends on the underlying software that helps the nodes communicate as well. Distributed Networking on the OSI Model On the OSI (Open Systems Interconnection) Model, these protocols are situated in the 3rd Layer. The Network Layer, is the third layer in the OSI model. This layer is crucial 5 Figure 1: Centralized, Decentralized & Distributed Networks for managing how data packets are sent and received across a network of computers. [13] The functions of the networking layer seen in figure 2 are: Figure 2: Layers of the Osi Model •Routing: The Network Layer determines the physical or logical path that data 6 should take based on network conditions, connectivity, and the routing table. •Logical Addressing: It assigns IP addresses to devices, which helps in identifying each device uniquely on the network. •Packet Handling: The Network Layer handles the packets, including packet forwarding and packet switching, across various network segments and points. •Error Handling and Diagnostics: It manages error reporting back to the source and network diagnostics. 2.1.3 Categories of Node Discovery in Distributed Systems An important consideration is, that we are dealing with a collection of nodes within the system. Therefore, the software will need to manage and organize the nodes belonging to the system, allowing them to join and leave, all while maintaining a record of the nodes in the network. [34] In practise, distributed systems are organized as an overlay network. Per definition, an overlay network is ”a computer network that is layered on top of another (logical as opposed to physical) network.” [6] A node is typically a software process equipped with a list of other processes it can directly send messages to. Messaging is sent in form of packets through TCP/IP protocol or UDP channels. There are two types of overlay networks : Unstructured overlay In these overlays, each node has a number of references to randomly selected other nodes. This paper has a focus on the next type [33] (page 47). Structured overlay In this case, each node has a well-defined set of neighbors with whom it can communicate. The nodes are organized in a tree or logical ring. This is the case of distributed hash tables. In both cases, an overlay network in general provides a very important principle: It should always be connected. Any node from the network can reach another node to pass a message, Which brings us to peer-to-peer (P2P) networking protocols [34] (page 4). Distributed systems can be classified into centralized or distributed computations (see figure 3). However for the sake of this study, we will focus solely on peer-to-peer networks. The p2p models can be divided into two categories : pure or hybrid. [35] Pure p2p networks need to be carefully designed so that removing any single node chosen randomly from the network, does not have any impact on the whole system. However, in hybrid p2p networks, there are nodes that have some central authority over the network. Let’s take the architecture of Napster, a renown peer to peer system for file sharing built in the 90s as an example to showcase the true benefits of pure p2p networks. In hybrid p2p systems, there is a central server that maintains directories of information about registered nodes in the network, in the form of meta-data. This meta-data is a very important trait in distributed systems, as it allows the nodes to reach for a specific node to communicate with, and it can contain some information such as the IP-address, the Port as well as some characteristics about the node like the processing 7 network, mitigating the impact of node failures on the network’s overall functionality. This limitation on the number of nodes per bucket is a critical feature designed to ensure network robustness and efficient management of node contacts. Since we start with the node’s position from left to right, with a limitation of k number of nodes per bucket, Kademlia is built in a way that nodes have more information about their neighboring. The arrangement of nodes within each k-bucket is also carefully managed: nodes are stored in a way that the ”least recently seen” node is at the head of the list, while the ”most recently contacted” node is at the tail. This method of organizing nodes within buckets ensures that more reliable and responsive nodes remain readily accessible, improving the efficiency of network queries and maintaining up-to-date information about the network’s topology. Scalability Moreover, the architecture of Kademlia ensures that while each node maintains more detailed information about its immediate network neighbors (nodes in closer buckets), it retains the ability to connect to any part of the network. This is accomplished through the design principle that each k-bucket should ideally contain at least one node—if not full—thus enabling a node to reach any part of the system through a series of steps, contacting nodes progressively closer to the target part of the key space. The duration of a search in Kademlia heavily depends on the correctness of a peer’s routing table. The most crucial being its kclosest neighbors in the overlay.[18] In the best case scenario, a peer should be able to return all of its kneighbors, in which case most operations take, following the big O notation, which is used to classify algorithms according to how their run time or space requirements grow as the input size grows [1], Olog(n) time to execute queries. [30] This structure makes Kademlia particularly adept at handling large-scale, decentralized networks by optimizing both the speed of query resolution and the robustness against node failures, maintaining network integrity even as nodes continuously join and leave the network. 2.3 Discv5 Protocol Having explored the intricacies of the Kademlia algorithm and its implementation within structured peer-to-peer systems, we now turn our attention to the evolution of these concepts within modern network protocols. In this chapter, we delve into discv5, the latest iteration of the Ethereum node discovery protocol. This protocol builds upon the foundational principles of Kademlia but introduces several enhancements aimed at improving security, scalability, and interoperability in increasingly complex network environments. 14 2.3.1 A brief overview of Discv4 Node Discovery V4, or discv4, is a kademlia-like DHT that stores information about nodes for networking purposes. It has been developed by the Ethereum foundation to implement it in Ethereum. Kademlia has been chosen mainly because of its efficient way at organizing a distributed index of nodes in a decentralized manner [27]. The discovery protocol itself is essentially based o↵of Kademlia, however, since it is purely for the node discovery, several aspects clarified in Kademlia are not regarded in it. Since it doesn’t use the value portion from Kademlia, discv4 (or discv5) does not use FIND VALUE as well as STORE RPCs [25]. Discv4 primarily uses Kademlia for the lookup method: A node contacts contacts some node and asks it for the nodes that are closest to itself, its neighboring nodes, in a recurring manner until it can no longer find new nodes. It does so by adding extra information that is communicated upon contacting the nodes, which every node is expected to maintain. Namely, the Ethereum Node Record, or ENR for short. We will delve deeper in what the ENR is and how it is structured in section 2.3.2. The ENR can be requested by a new Remote Call Procedure introduced in discv4, by any node, which is called ENR REQUEST. Discv4 has been replaced by Discv5 because it included a few limitations: •It is impossible to di↵erenciate between node sub-protocols. In discv4, nodes were being added to the DHT without consideration of their communicating protocols. They were then eliminated upon contacting them, which is when it is determined that the node is invalid [19]. •Another limitation was the use of a node’s local time clock to prevent Replay attacks, which is when data transmission is maliciously repeated or delayed. This attack involves the interception of a legitimate transaction that is then retransmitted to produce an unauthorized e↵ect by altering it. This has made a certain struggle in the Ethereum Network: When a machine’s clock was o↵by a short timeframe, for example 2 minutes, it simply threw an error to reject the node from joining the network, making it inefficient [20]. •There was no way of telling whether or not both peers have verified each other. A node could consider another one verified, while the recipient did not verify the emitter node, forcing the recipient node to drop the FIND NODE RPC [20]. All these issues makes Discv4 open for improvement. 2.3.2 Ethereum Node Records The Discovery V5 Protocol uses Ethereum Node Records (ENR), to provide p2p connectivity information between the nodes within the Ethereum network. Therefore, understanding how ENR works is a must. 15 The motivation behind developing the Ethereum Node Records is because of the restrictions presented in the node discovery protocol: Node identity is discovered by knowing the node’s identity public keys, their IP address and two port numbers. It is not possible to relay any extra information [28]. For that matter, the ENR proposes a more flexible format, the node record, for connectivity-related information, where it can hold not only the mentioned information (IP address, port...), but a record can also contain information about arbitrary transport protocols and public key material associated to them [22]. Why ? Two keywords come in handy: The Public Key Material as well as Arbitrary Transport Protocols. The Public Key Material within an ENR primarily pertains to cryptographic public keys associated with each node in the network. These keys serve several crucial purposes: •Identity Verification: Each node in the network has a unique identifier derived from its public key. Nodes are therefore self-certified, and if a node’s ID is known, the most recent version of its record can be retrieved. •Secure Communication: Public keys enable nodes to establish encrypted communication channels, ensuring that data exchanged between nodes is secure from attacks such as eavesdropping and tampering. •Signatures: ENRs are signed with the node’s private key corresponding to the public key included in the ENR. This signature confirms the authenticity of the information in the ENR, such as IP address and port, so that receiving nodes can trust that the data has not been tampered with. This allows a reliable and secure peer-to-peer networking. Arbitrary Transport Protocols: Transport protocols define the rules for data exchange over the network. By allowing ENRs to contain information about arbitrary transport protocols, Ethereum aims to achieve some goals towards a multi-chain network, the so-called Eth2.0 [19]. These include: •Flexibility and Extensibility / Interoperability: Nodes can support multiple transport protocols simultaneously. This allows diversification within the same network ecosystem where di↵erent nodes can have varying properties and capabilities allowing them to practise di↵erent protocols within the same network. •Future-proofing / Protocol Upgrades: As time moves on, new transport protocols can be developed, and integrated within the network with ease without the need of re-defining the discovery protocol. With ENRs, nodes can simply re-publish new ENRs that include these new protocols [22]. 16 Specifications A node record holds information that can be separated into 3: The signature, which is a cryptographic signature of record contents. The seq, which is a 64-bit unsigned integer representing the sequence number. This number should increase whenever the record is changed and re-published, so that contacting nodes are able to tell which record is the latest. And finally, the remainder consists of arbitrary key-value pairs [22]. The key/value pairs are sorted by their keys, and every key can only be present once. Key names have pre-defined meanings. We can find below a table of key value pairs that may be represented in an ENR. Only the id key is mandatory, and the rest are optional, even the endpoint, as long as the signature is valid. Beyond the standard keys represented in the table, developers can define additional keys to store information relevant to their specific applications or sub-protocols. This allows for the inclusion of new features or support for additional protocols without altering the core ENR specification [22]. The records are represented as a Recursive-Length Prefix Key Value id name of identity scheme, e.g. “v4” secp256k1 compressed secp256k1 public key, 33 bytes ip IPv4 address, 4 bytes tcp TCP port, big endian integer udp UDP port, big endian integer ip6 IPv6 address, 16 bytes tcp6 IPv6-specific TCP port, big endian integer udp6 IPv6-specific UDP port, big endian integer Table 2: Table of Key and Value pairs for ENR (RLP) Serialization. It is a standard format for the transfer of data between nodes that is efficient. The RLP cannot exceed 300 bytes, and discovery implementations should reject any node with an RLP exceeding that specific number [29]. The limitaion is there because records are relayed frequently, and they may be included in size-constrained protocols like DNS [28]. Records are signed and encoded as follows: content = [seq, k, v, ...] signature = sign(content) record = [signature, seq, k, v, ...] Text Encoding of the RLP Finally, the RLP can be then represented in a textual form, which is the base64 encoding of its RLP representation, prefixed by enr:. To put everything into an example and wrap up the Ethereum Node record, we have the following example from the official EIP proposal [28]: Information that is encoded: 17 •node ID: a448f24c6d18e575453db13171562b71999873db5b286df957af199ec94617f7 •IP address: 127.0.0.1 •port: 30303 The record is signed using the ”v4” scheme using the seq (sequence number) 1 and the private key : b71c71a67e1177ad4e901695e1b4b9ee17ae16c6668d313eac2f96dbcda3f291 The RLP structure of the record is : [ 7098ad865b00a582051940cb9cf36836572411a4727878...76f2635f4e234738f30..., 01, "id", "v4", "ip", 7f000001, "secp256k1", 03ca634cae0d49acb401d8a4c6b6fe8c55b70d115bf400769cc1400f3258cd3138, "udp", 765f, ] The Text encoding of the RLP is (dotted to shorten): enr:-IS4QHCYrYZbA..HcBFZntXNFrdvJjX04jRzjzCBO.._oxVtw0RW_QAdpzBQA8 2.3.3 Introduction to Discv5 Discv5 is a standalone protocol that operates on UDP over a dedicated port, designed to address the shortcomings of its predecessor, Discv4. This protocol introduces significant enhancements, primarily through the implementation of flexible peer records known as Ethereum Node Records (ENRs). The rationale behind Discv5 is to achieve a series of general and security-specific goals, enhancing reliability, security, and future scalability of node discovery mechanisms. [23] Why Discv5 ? According to the Ethereum Foundation [21], libp2p Kademlia DHT is a fully-fledged DHT protocol/implementation with content routing and storage capabilities, both of which are irrelevant in the context of Peer Discovery. 18 2.3.4 Wire Protocol of Discv5 Having discussed Ethereum Node Records (ENRs), which are pivotal in providing comprehensive and verifiable node information within the Ethereum network, we now transition to another critical component of Discv5: the wire protocol. ENRs serve as the backbone for node identity and metadata [28], but the practical application of these records is realized through the wire protocol, which handles the networking between nodes [26]. The wire protocol in Discv5 is the set of technical standards and procedures that dictate how nodes can exchange information. This protocol is crucial because it encapsulates the methods for initiating and maintaining communication, ensuring data integrity, and providing security through encryption [26]. The wire protocol’s design reflects an evolved understanding of network challenges and security threats, building on the shortcomings of previous versions [19]. UDP Communication Node discovery messages are sent as UDP datagrams [26]. As described in an example distributed System on the OSI Model in figure 6, the Wire protocol from the discv5 operating at the networking level (3), makes use of UDP for communication in the Transport Layer (4). The wire protocol mandates the use of UDP for no one big reason, but several smaller ones. These reasons come in 5 parts [31]: •NAT Traversal: UDP is chosen for its simplicity in handling NAT (Network Address Translation) issues. Unlike TCP, UDP doesn’t require complex setup processes for NAT traversal. UDP allows packets from any other node to reach the node behind the NAT without additional configuation, allowing more reliable communication across the system even with di↵erent network setups, unlike TCP. •Uniform implementation Across Nodes: UDP ensures that all nodes implement the same protocol of communication. This uniformity is crucial for discv5 because all nodes must be able to communicate with each other without any compatibility issues. •Reduced Communication Latency: UDP allows for faster communication, which is necessary in node discovery and maintenance. The protocol requires nodes to quickly interact with many others to update neighbor lists and verify the availability of other nodes. Also, as we will uncover shortly, packets are split into several ones because of size constraints. And UDP doesn’t require the receiver to actually confirm the reception of packets, unlike TCP. •Handling of High Connection Numbers: Using UDP avoids the overhead associated with managing numerous TCP connections. If TCP were used, nodes might either run out of system resources like file descriptors due to the need to maintain 19 connections with many nodes, or they would have to constantly establish and terminate TCP connections. UDP, by not requiring permanent connections, saves resources and simplified communication processes. •Resilience to Packet Loss: UDP’s tolerance for packet loss enhances the robustness of the protocol. In UDP, it is acceptable for packets to be lost and not reach their destination. This feature is beneficial as it forces the protocol to be resilient even in poor network conditions, allowing implementations to intentionally drop packets to manage traffic and computational loads without compromising the network’s functionality. Another strategic advantage is the fact it helps masking whether non-responses are due to intentional decisions or network issues, adding a layer of unpredictability. Figure 6: Integration of the Discv5 Wire Protocol with the OSI Model The Wire Protocol specifies that the maximum size of any packet is 1280 bytes; any implementation should not generate packets larger than this size. And the minimum packet should be 63 bytes. Packets lower than this size should be rejected. [26] In order to stay between these sizes, packets that exceed the upper-limit should be split into multiple smaller ones and send multiple messages all while specifying the total number of responses in the message. [26] Since the protocol expects low-latency communication, there is a short timeout of 500ms 20 for single request/response interactions (RPC interactions), and a 1 second limitation for handshakes. Furthermore, when responding to a request, the response should be sent to the UDP envelope address of the request. [26] Packet Encoding There are 3 distinct types of packets in the wire protocol, that are all 3 tied to each other in their logic [31]: •Ordinary message packets: Packets carrying an encrypted/authenticated message •WHOAREYOU packets: These packets are sent to a requester when a recipient of an ordinary message packet cannot decrypt or authenticate the packet’s message •Handshake message packets: These packets are always sent following the WHOAREYOU packets. They carry handshake-related data in addition to the encrypted/authenticated message first send by the ordinary message packet followed by the WHOAREYOU packet. Protocol Header Overview In Discv5, every discovery packet starts with a header, which is encrypted to prevent firewall detection. This header is constructed as follows: •Masking: The header information, including the protocol ID, version, and message metadata, is masked using AES/CTR encryption with a key derived from the destination node ID and a packet-specific random IV. Figure 7: AES/CTR Encryption AES is a widely used encryption standard that converts plain text into unreadable text (ciphertext) using a secret key. CTR (Counter) Mode turns AES into a stream cipher, which means it can encrypt data of any size, byte by byte. It does 21 this by combining the encryption of a counter value (the nonce) with the plain text to produce the ciphertext [2]. •Structure: The actual header consists of a static header and an authdata section. The static header includes the protocol ID (”discv5”), version (0x0001), packet type (flag), and a nonce. A nonce is an arbitrary number is only used one time in a cryptographic communication. [12] •Decryption: Recipients decrypt the masked header using their node ID and verify the protocol ID to process the packet correctly. Remote Procedure Calls Discv5 enhances node discovery and communication through several RPCs. It removes FIND VALUE & STORE commands introduced by Kademlia [19], leaving only necessary RPCs for node lookups and adding other Calls needed for communication between nodes [26]: •PING & PONG: Nodes periodically send PING messages to check the liveliness of other nodes, sharing their ENR sequence number. The PONG response confirms availability and shares the recipient’s current status and contact details. •FINDNODE: Requests information about nodes at specific distances in the network to maintain connectivity. •NODES: Responds to FINDNODE with a list of nodes that match the search criteria, aiding network discovery. •TALKREQ & TALKRESP: Facilitate custom data exchange between nodes for specific applications, with TALKRESP providing the required response or indicating unsupported protocols. There are other methods that have not been cited for the purpose of this thesis as it is a practical exploration, while these methods are yet to be implemented and only cited by Ethereum in theory. They are a work under development, but will enhance the protocol by providing a sub-protocol advertisement system through several RPCs such as REGTOPIC, TOPIC QUERY... [26] In essence, these RPCs would be interesting to implement in order to have a distributed network that is future-proof, allowing several sub-protocols to be under the same network, by making an advertisement mechanism to further propagate nodes depending on which sub-protocols they maintain for networking. 2.3.5 Potential Goals of Discv5 General Goals Discv5 introduces several solutions and key concepts that can be found under their official rationale as well [24]: 22 •Like stated earlier, the verification process proposed by discv4 was unreliable. The FIND NODE request may fail when node A assumes that node B already knows about a recent PING/PONG interaction. •In discv4, a node can provoke a response from any other node simply by knowing its ip address. This can be abused by malicious nodes and create DDOS (Distributed Denial Of Service,when multiple services flood a single targeted system with requests, forcing it to shutdown) attacks. Making it a requirement to know the node’s ID and making it expensive to obtaining it helps mitigating this sort of attack. •Allowing more node ID cryptosystems, which in turn will result in a more interoperable node discovery system that is future-proof and supports the idea of a multichain for Ethereum for instance. For now (discv4), only secp256k1/keccak256 is supported. •Replacing node information tuples with ENRs. As discussed earlier, ENRs can provide many arbitrary information about a node, which can permit several new abilities such as permitting capability advertisement to other nodes and transport negotiation, meaning that transport protocols can di↵er from node to another and be classified accordingly with the help of topic advertisement. •Guard against Kademlia implementation flaws: Since Kademlia operates based on distance metrics between nodes calculated following a distance metric, a misuse of this metric can result in a fragmentation of the network. •Support for finding nodes based on an arbitrary topic identifier. This, according to the rationale, should be as fast or even faster than the traditional approach to node querying. •While Discv4 relies on nodes joining the network with a clock on the machine that is very accurate to prevent replay attacks, where a node would receive requests and try to simulate false packets in the network, It is being prevented poorly in discv4, because it led to many complaints when joining the network or with connectivity problems. The Ethereum foundation took this chance of upgrading the discovery protocol to also introduce a solution to this common issue in Discv5 [20]. •The traffic between nodes should be obfuscated, meaning making it hard to identify/recognize patterns or read data. This is done to avoid sniffing (the process of ”hearing” packets that are being sent through the network). 23 3.2.2 Discv5 API Overview Following the overview of the library, we will now focus solely on the Discv5 application level documentation that is stated in 3.2.2. It can be found under [9]. We can dissect the main functionalities that are exposed to the application level as displayed in figure 9. The discv5::Discv5 is the main service struct, providing the user-level API for performing queries and interacting with the underlying service. It provides the main discv5 functionalities that we have covered in the Section 2.3.4. It takes the following parameters: •enr: An ENR generated from CombinedKey. There are two di↵erent signing keys that can be used. See figure 9. •enr key: The signing key that has been used to generate our ENR. •Config:aPre-setofconstantsthatisoftypeConfig.Thereisadefaultconfiguration that can be used as well, see 9. The Config is generated using ConfigBuilder, which allows us to read and write to our configuration. Methods The Discv5 struct methods cover all the primary RPC methods that we have cited in section 2.3.4. We can categorize these methods covered in our figure 9 as such: •Administrative functionalities – Start function: Starts the required tasks and begins listening on a given UDP SocketAddr. – Shutdown:Terminatestheservice. •Node Discovery & Management – add enr: Adds an ENR to the routing table. –remove node: Removes a node ID from the routing table. – disconnect node: Marks a node in the routing table as Disconnected. – ban node:Bansanodefromtheserverforaspecifieddurationorpermanently. – ban node remove: Removes a banned node from the banned list. –permit node: Permits a node, allowing it to bypass the packet filter. –permit node remove:Removesanodefromthepermitlist. •Informative functionalities – nodes by distance:Returnsavectorofnodesclosesttospecifieddistances. 30 – connected peers: Returns the number of connected peers in the routing table. – metrics:Retrievesmetricsassociatedwiththeserver. –raw metrics: Exposes the raw reference to the underlying internal metrics. – local enr: Returns the local ENR of the node. – external enr: Returns the external ENR, which is the local ENR exposed via an Arc. •ENR Manipulation – enr insert: Inserts or updates a field in the local ENR. – update local enr socket: Updates the local ENR TCP/UDP socket settings. •Routing Table Manipulation – kbuckets: Returns the routing table of the Discv5 service. – table entries id: Returns an iterator over all node IDs in the routing table. – table entries enr: Returns an iterator over all ENRs of nodes in the routing table. – table entries: Returns an iterator over all entries in the routing table. –with kbuckets: Uses a closure to interact with the kbuckets for potential viewing or modifying. •Network Security Policies – ban ip:BansanIPaddressfromtheserverforaspecifieddurationor permanently. – ban ip remove:RemovesabannedIPfromthebannedlist. –permit ip:PermitsanIP,allowingallpacketsfromittobypassthepacket filter. –permit ip remove: Removes an IP from the permit list. •Discv5 Remote Call Procedures (RPC) – send ping: Sends a PING request to a node. –talk req: Sends a TALK request to a node using specified protocol and request data. – find node designated peer: Sends a FINDNODE request to a designated peer based on specified distances. 31 – find node: Initiates an iterative FIND NODE request targeting a specific node ID. – find node predicate: Starts a FIND NODE request based on a given predicate. •Event Capturing –event stream:CreatesaneventstreamchannelforreceivingDiscv5events. The CombinedKey in 9 file provides currently only ‘secp256k1‘ and ‘ed25519‘ key types support, as per the API coverage in section 3.2.2. The file provides a keypair generation functionality for both key types, as well as a function to be able to encoding and decoding them from/into bytes. Finally, it also provides a function to derive the public key from it, as well as an essential function named verify v4() which takes as parameters a byte encoded message and a signature of the message, in order to verify that the message has been signed by the keypair itself. As we wrap up our detailed exploration of the Discv5 library and its API coverage, we’ve established a solid foundation for practical application. The insights gained provide us with the ncessarz knowledge to embark on developing an application with it. 32 Figure 9: Discv5 API Overview 33 3.3 Analysis Now that we have covered the API of Discv5, we set a clear stage on how we can make use of it to develop our application. We will start by uncovering problems that can come up when making a decentralized application, and that need to be generally conscious of. We will then present the necessities for our application in the application environment section. 3.3.1 Problem Analysis Distributed Applications present unique challenges that impact their design, efficiency, and security. Some of the primary challenges include: •Centralized risks: Despite the decentralized intent of p2p systems, improper implementation can inadvertently reintroduce centralization, diminishing the system’s resilience and control distribution. •Scalability vs speed: While distributed systems inherently o↵er excellent scalability, this can sometimes come at the cost of reduced performance, especially in terms of transaction or operation speed. •Security vulnerabilities The open nature of distributed systems can expose them to various security threats, making comprehensive security measures essential. These challenges are intrinsic with the core functionalities of Discv5 and the Kademlia Distributed Hash Table (DHT), as these technologies aim to optimize the balance between decentralization, scalability, and security. 3.3.2 Application Environment This thesis aims to showcase how discv5 can be implemented within the scope of an application. For those reasons, its focus will be more on the technicality of how discv5 can be utilized, rather than how the system actually works. To put our focus more on that point, we will be making a simple application as described in the motivation section, which is a simple data storage system that can serve for social networking purposes, in a decentralized manner. The applications consists of Posts and Comments. Under these topics, people are able to make posts where they write what they wish to. Every post can only be present under one single topic at a time. Posts can have comments, which present the same structure as the posts themselves. The di↵erence being, that a post is mapped to a topic, whereas a comment is mapped to a post / comment. In a coding perspective, posts and comments are the same. Except that their keys vary. This can be well represented in a key-value storage system. For the next chapter, we will discuss various solutions on how to distribute this data across the network, as well as how peers can find each other and maintain an overlay 34 network with discv5. Some architectures that can come to mind, based on the technologies that we have covered, will be presented. We will then decide on which one is better with a comparative analysis following a process of elimination, and proceed with the implementation of the better solution. 3.4 Solution Propositions Upon uncovering the necessities for our application, we will propose 3 solutions that leverage the use of Discv5 for the networking layer between nodes. We will then move along with a process of elimination to decide which system fits best and move along with its implementation, demonstrating how Discv5 can be utilized along the process. Topic-based Node Discovery & Clustering This solution organizes a network by clustering nodes around specific topics of interest to enhance content discovery efficiency and system responsiveness. Nodes indicate their interest in topics when they join or through dynamic updates, participating in a voting mechanism that determines new topic additions. Nodes are grouped into clusters based on shared interests, and queries or content are directed only towards relevant clusters, not the entire network. Topics are established through a governance consensus, allowing nodes to vote on potential new topics. Each node’s Ethereum Node Record (ENR) contains key-value pairs representing topics they’ve voted on, aiding in topic-based node discovery and efficient query responses. The TOPICQUERY RPC is used to connect nodes under the same topic, and nodes broadcast changes in their topic interests, facilitating real-time updates and maintaining a responsive and interconnected network. Federated Server Model with discv5 In this model, the application is powered by a set of federated servers, each serving di↵erent sectors of the application. In this system, any node can join with a chosen topic and maintain a governance over the topic it is hosting. For those reasons, they are called federated servers; hosting di↵erent parts of the application. Each server in turn maintains its own centralized database. All servers are inter-connected for peer discovery using the Discv5 routing table. Decentralized P2P Messaging System This system uses a peer-to-peer (P2P) architecture where each user operates as a node within the network. The nodes communicate directly with each other without a central server, using discv5 for node discovery and maintaining a dynamic, decentralized network. This distributed network maintains di↵erent layers that are partitioned across the entire system: 35 •Networking layer: The system maintains a networking layer using discv5 that is solely reserved for node discovery. It makes node lookups periodically to maintain fresh up to date routing tables with reachable nodes. •Storage layer: This layer will be in charge of handling the data. It utilizes an implementation of the Kademlia DHT. Discv5 will pass the discovered nodes to store them under the DHT, and external endpoints are exposed to the client in order to interact with the distributed network by storing and retrieving key-value pairs to the DHT. Content such as images and posts can be addressed by hashes, which ensures data integrity and aids in retrieving data from the DHT. To ensure fault tolerance, the routing table of the discv5 server is refreshed often, handing node information to the DHT. 3.5 Evaluation This section will detail each proposed solution, explaining how they address specific challenges of our decentralized application to deduct which solution fits best the incentive of this thesis. 3.5.1 Federated Server Model with discv5 Nodes are regrouped by the discv5 protocol, connecting them between each other and data can be fetched and passed from one node to another using the TALK request from the discv5 library, however, every node would represent a form of authority over a specific topic, which is not decentralization per definition. It presents high risks against fault tolerance: If a certain node is shutdown, a whole topic is shutdown and its data is forever lost. This system is only decentralized when it comes to networking, but the application layer itself is centralized. 3.5.2 Topic-based Node Discovery & Clustering This method organizes nodes into clusters based on the topics they are interested in. However, there are practical limitations in the current implementation of the discv5 protocol that a↵ect this method. Key functionalities such as the REGTOPIC RPC and the TOPIC management system are theoretically described but not currently recommended for use, as highlighted in the official Discv5 documentation (limitations referred in section 2.3.4 of our thesis). Additionally, this system extends the use of the Ethereum Name Record (ENR) beyond its typical networking functions, which could introduce complications. Since discv5 is designed primarily for networking, using it extensively for other purposes may lead to overhead issues and complicate the maintenance of routing tables. Ideally, a distributed network should treat all nodes uniformly, abstracting the application layer from the client. 36 3.6 Decision & Justification After careful consideration of the proposed solutions for implementing a decentralized application with discv5, and a careful review of how each solution can operate, it goes without saying that the latest solution, the Decentralized P2P Messaging system, suits best the narrative of a structured decentralized system for our application. This system optimally leverages the inherent features of discv5 and Distributed Hash Tables (DHT) to address fundamental requirements of decentralization. The system’s architecture is inherently decentralized with no single point of failure, as all nodes find each other and operate equally with no central authority. The use of Kademlia DHT enhances the system’s scalability. As the network grows, the DHT efficiently manages the increasing number of nodes and data entries through its proven distributed network algorithms. The introduction of discv5 as the networking layer to the system will ensure that we have secure sessions established between nodes, while having a fast-paced routing mechanism for nodes to discover each other and communicate between one another securely. This model not only meets the technical requirements of the system but also aligns closely with the ideological goals of maintaining network decentralization. This justification concludes the decision to adopt the Decentralized P2P Messaging system as the preferred architecture for the application, providing a solid foundation for future development. 4 Example Application This chapter describes the system design of our application presented in the latest solution, the goal being to showcase how we can utilize Discv5 and an implementation of its API. It has been decided that it will incorporate Discv5 for node discovery and for networking purposes only. Furthermore, it will make use of a DHT system that is in contact with disc to store the nodes across the network within it, as well as data storage management and retrieval across the nodes. We begin with an exploration of the foundational functionalities of Discv5, demonstrating how basic operations are seamlessly integrated to provide initial network discovery and connectivity. The discussion will then expand to encompass advanced features of Discv5, evaluating how it can be integrated with our Kademlia DHT. 4.1 Architecture The architecture of the decentralized data storage leverages a multi-layered approach. Each layer featuring specific functionalities to the application. As we can see in figure 10, These layers can be grouped into the following, following the Osi-Model 2.1.2: •Networking Layer: Utilizes the discv5 protocol for peer discovery and network 37 Figure 10: Overview of the Application Layers topology maintenance, ensuring secure node connections via a handshake protocol and propagating node information through Disc. •Application Layer: Employs a Kademlia Distributed Hash Table (DHT) for decentralized data storage and retrieval. This layer optimizes data routing by maintaining current node information from the Networking Layer, leveraging node ENRs for efficient data distribution across nodes. Handles data processing and management, incorporating Data Management components for hashing and data preparation prior to DHT storage. It o↵ers API Services that act as interfaces for client interactions, supporting operations like content posting and retrieval, thus abstracting system distribution complexities from end-users. Inter-layer communication within the architecture (see Figure 10) enhances system cohesion and functionality: •From Networking to Application Layer: Networking Layer supplies node updates to the Data Layer, facilitating DHT routing optimizations based on the live network state provided by the Discv5 protocol. 38 •APIs and Data Management: Application Layer’s APIs interface with external clients for data transactions, with the DHT. The separation into distinct layers allows for focused scalability and maintenance, with each layer designed to independently handle its specific set of responsibilities within the broader system context. Figure 11: Distributed Network Architecture This architecture as presented in figure 11, supports a decentralized model by eliminating central points of failure and distributing data and functionality across a wide network of nodes. It is designed as a distributed network per definition in section 2.1.1, as no complete reliance to a single point is required. Furthermore, we can classify this networking system as structured pure p2p as described in Section 2.1.3. No single node has any extra authority and all nodes are equal in regards to networking, having each a well established set of neighbouring nodes in their routing tables that are optimized with Discv5. 4.2 Discv5 Functionalities Code Implementation Upon covering the general architecture of our application, let’s have a deep dive into how it works by going through the functionalities of it, how the code works, describing its workflow and how it operates. We will begin by covering how the bootstrapping process described in figure 12 can be implemented practically, to then rotating over the RPC methods from discv5, as cited in section 2.3.4 one by one, with a thorough code explanation. 4.2.1 Starting the Discv5 server The server runs with the following command: cargo run -- --enr-ip4 <ip-address>--port <port> 39 info!( connected_peers, active_sessions =metrics.active_sessions, unsolicited_requests_per_second = format_args!("{:.2}",metrics.unsolicited_requests_per_second), "Current connected peers: " ); } As we can see, the lookup nodes function takes the discv5 server as a parameter. It then starts with a random node ID that is generated with the help of the provided function from the discv5 library, that will then be narrowed down with the help of the XOR metric by calculating distances away from the node, performing the node lookup iteratively. It Runs an iterative FIND NODE request that are sent through the secure connection already established between the known peers that have gone through the handshake process and are already stored under our local routing table with a secure established connection. Responding nodes send back a NODES message containing ENR (Ethereum Node Records) of nodes that are closer to the target ID. The local node updates its routing table with the new nodes it discovers from the NODES message. This will return peers containing contactable nodes of the discv5’s DHT closest to the requested NodeId. Upon finding these nodes, the library will handle the new connections by establishing the handshake process and securing the new communication channels with the newly discovered peers. the protocol automatically initiates the handshake to establish or refresh the session. It also handles encryption and decryption of messages as well as the verification of node identities. 4.2.4 Discv5 Event Streams Upon reading the discv5 library, one can see that it also includes the possibility of streaming the Events. It is therefore possible to create an event stream channel which can be polled to receive Discv5 events. Within our discovery loop function, we are also capturing all of the events that are being sent by discv5 to further optimize our application. A few of these events that can be captured are: •Discovered -Thiseventasitsaysiscapturedwhenanodehasbeendiscovered. It returns the node’ ENR. •NodeInserted -Saysthatanodehasbeeninsertedintotheroutingtable.Returns the node inserted ID, and if it replaced another node ID, it also returns it •SessionEstablished - Handshake process has been established with a node from 46 our local routing table - Returns the ENR of the node with whom we have established the session •SocketUpdated -Multiaddressofanodehasbeenupdated-Returnsthenew address in the form of the following struct : pub struct SocketAddrV4 { ip: Ipv4Addr, port: u16, } •TALKRequest - Talkrequest received - Returns a TalkRequest struct composed of the following : pub struct TalkRequest { id: RequestId, node_address: NodeAddress, protocol: Vec<u8>, body: Vec<u8>, sender: Option<mpsc::UnboundedSender<HandlerIn>>, } Node Update Next, nodes that are stored under our DHT routing table can have their ENRs updated. These ENRs might include new IP addresses / Ports, so we need to take this into consideration and perform updates accordingly. For those reasons, we can utilize the SocketUpdated event stream from Discv5 as follows: Event::SocketUpdated(addr) => { info!(%addr, "Socket updated"); //Find key in dht and update addr + port let ip =addr.ip().to_string(); let port =addr.port(); let node =Node::new(ip, port); interface.ping(node); //Pinging node to add it to dht }, Code Explanation: This time, the stream provides a SocketAddr struct, so there is no need to derive any ENR. We simply derive the address and port, create a new Node instance and ping it. As usual, if the node responds to the PONG, the DHT Protocol will automatically store the new node information under our routing table. Since we do not receive the ENR under this stream event, we are unable to directly eliminate the old node information from the DHT. However, as discussed in section 2.2.6, the Distributed Hash Table stores the node information in a queue that follows a 47 FIFO setup. This means that the least used addresses will find themselves at the end of the list, and will then be further replaced by this process and therefore eliminated from our buckets automatically. 4.2.5 TALK Request & TALK RESP This is a practical exploration of the TALK request from discv5 to showcase how it can be utilized, but it doesn’t serve much purpose for now for the implemented code. However, for future work, the information being passed can serve for further optimization of our DHT querying system. As previously explained, upon receiving a new connection with a new node, we map its node ID to 0 in our routing table. To maintain this value, we are performing a TALK request to all the known peers periodically, passing to them in the body the number of known peers. The TALK request is then detected with the help of the event stream, and we parse the information received to update the data within our dht using the node ID as the key, which is received from the node address, and the value is received from the body. This information exchange between the nodes can be split in two parts, the talk request, and the talk response. TALK REQ The TALK REQ is sent periodically along with the rest of our loop function. The Code : let protocol ="peer_size".as_bytes(); //Finding all node Ids known to the current disc let ids =discv5.table_entries_id(); for node in ids{ //Finding known enr from the node ids let enr =discv5.find_enr(&node); //If enr is found, perform a talk request with it if let Some(found_enr) =enr { let _=talk(&discv5, &interface, &found_enr, protocol).await; } } As we can see, we first start by defining the protocol over which we will be comunicating, which is a String transformed into bytes before transfer. This can serve as an identifier to the receiving node to know which data to pass / function to execute. We are then using discv5 to call a method called table entries id(), which in turn returns a list of node IDs that are known the local node. We then loop through each of these ids to retrieve the latest known ENR we have in store on the routing table of discv5 associated to that node ID with the function find enr, passing the ID to it. If we do have an ENR for the given node ID, because the field is optional in case we have none, we execute a talk request by performing our talk function. 48 The talk function takes the discv5 server as argument, the dht interface, the ENR that we have just found from the node ID and finally the protocol over which we will be executing the communication with the node. The talk function can be depicted below: pub async fn talk( discv5: &Discv5, interface: &Protocol, enr: &enr::Enr<CombinedKey>, protocol: &[u8], )-> Result<(), String>{ let peer_count =discv5.connected_peers().to_string(); let request_data =format!("{}",peer_count).as_bytes().to_vec(); // Send the talk request match discv5 .talk_req(enr.clone(), protocol.to_vec(), request_data) .await { Ok(talk_response) => { // Process the talk response // Assuming the response is also in the form "node_id|peer_size" if let Ok(response_str) =std::str::from_utf8(&talk_response) { let parts: Vec<&str> = response_str.split('|').collect(); if parts.len() == 2{ let remote_node_id =parts[0]; let remote_peer_size =parts[1]; interface .put(remote_node_id.to_string(),remote_peer_size.to_string()); } } Ok(()) }, Err(e) => Err(e.to_string()), } } Code explanation: we execute the asynchronous talk req method from discv5 passing the receiving node’s ENR, the protocol and the request data. We then match the response to what can be expected, which is whether an empty response or a talk response. When it is a talk response, we are receiving the same information but for the receiving node. Meaning that if node A communicates over a talk request with node B to send how many nodes it can reach, node B will proceed by replying to node A with the same information from its side. After the talk request is completed and a talk response is 49 received, we will end up having both nodes information up to date under our DHT. TALK RESP In turn, the talk response is executed once a talk request is received. In order to achieve this, we are also making use of the event stream from discv5 here. We can read the following code to achieve this second part of the process : Event::TalkRequest(talk_request) => { if talk_request.protocol() == "peer_size".as_bytes() { let request_body =talk_request.body(); let known_peers_remote =match std::str::from_utf8(request_body) { Ok(v) => v, Err(_) => { let _=talk_request.respond(vec![]); return; } }; let node_id =talk_request.node_id().to_string(); info!("talk request received from peer {}",node_id); // Storing known peer size to node ID interface.put(node_id, known_peers_remote.to_string()); let known_peers =discv5.connected_peers(); let self_id =discv5.local_enr().id(); if let Some(enr_id) =self_id { let response =enr_id +"|" + &known_peers.to_string(); let _=talk_request.respond(response.as_bytes().to_vec()); } } } Code Explanation: We receive the talk request by capturing it from the event stream. It has one parameter which has all the information needed, including the ID of the node, and the body containing the information that we will be storing under the routing table, which is the number of peers that the sender node is able to reach. We define the node ID and the known peers number, that we then use to store under our DHT. We then proceed by retrieving the number of connected peers under our local discv5 using the method from the library discv5.local enr().id(), and emit a response to the talk request by passing the node ID and the known peers in a single string separated by ”—”, transformed to bytes. 50 Upon running the code and executing it on both the local computer and the droplet from digital ocean, we can see that the TalkRequest is being correctly received by the nodes: Figure 15: Node A Receiving talk request Figure 16: Node B: Receiving talk request 4.2.6 DHT Interface Integration with Discv5 Alongside our Discv5 server, we also go through the creating of our Distributed Hash Table interface, which will be responsible for the storage and retrieval of data by exposing its functionalities with a REST API that is reachable by anyone that knows the IP address / Port of any node in the system. We integrate our DHT by first retrieving the bootstrap Node to pass it along to the DHT as well, by deriving the node information from the ENR to make sure we have a secure connection as follows: //Using bootstrap.json to get an optional Node //that we pass to our interface to bootstrap let bootstrap_result =get_bootstrap_if_exists(bootstrap_file); //Starting root with local ip address and port + 1 let root =Node::new(utils::get_local_ip().unwrap(), 8001); //DHT interface responsible for adding nodes and data let dht_protocol =Arc::new(Protocol::new( root.ip.clone(), root.port.clone(), bootstrap_result, )); 51 We now have our distributed hash table ready for usage, and it is bootstrapped with a secure and reachable node. In our application, we are relying on the capturing of these events to keep provide node information and updates to the Kademlia DHT as follows: Node Insertion Event::Discovered(enr) => { //Derive ip address and port from enr as well as node ID info!(%enr, "Enr discovered"); //Pinging new discovered node to store it in our dht let enr_info =derive_info(&enr); if let Some(ip) =enr_info.udp4{ let node =Node::new(ip.ip().to_string(), ip.port() + 1); //Storing the new node by pinging it let res =interface.ping(node); //Mapping the nodeId to the current known peer size //for later optimzation let id =derive_id_from_enr(&enr); if let Some(node_id) =id { //Initializing known peers to 0 interface.put(node_id.to_string(), 0.to_string()); info!("ENR mapped to node ID successfully"); } if res{ info!("Node stored under our DHT successfully"); }else{ info!("Failed at receiving PONG response. Node not stored."); } }else{ info!("Could not derive node information") } }, Code explanation: The discovered event stream captures the ENR of the node that has been newly discovered. This variable is passed to the internal function where we will derive the node information from it using the derive info function. The derive info function is as follows: pub fn derive_info(enr: &enr::Enr<CombinedKey>)-> NodeInfo { let node_id =enr.node_id(); let udp_port =enr.udp4_socket(); NodeInfo { node_id: node_id, 52 udp4: udp_port, } } Once our function returns the node ID and udp address containing both the IP address and the port, we create a Node struct using this derived information and ping it. If the ping returns a pong, the node is automatically handled and stored in our DHT local routing table. Furthermore, we initialize a value 0 to mapped to the node ID, representing the number of nodes that this peer has within the routing table of its ENR. This number is a form of initialization, and will later be modified using the TALK request, as presented in section 4.2.5. We can derive the following sequence logic between nodes: Figure 17: Process Flow for Node Discovery followed by Node insertion in DHT 53 4.3 Demonstration Upon exposing our endpoints, we are now able to see if the DHT is truly storing and retrieving data. This process will serve us to prove four key aspects: •Our routes are operational and reachable. •The Discv5 is operating as intended, discovering new nodes, generating ENRs from which we can derive node information that we can then use to communicate with. •We are correctly deriving the node information and storing a new node instance under our DHT. •We are able to store data with the DHT logic across the nodes, and this data can then be retrieved. We can validate all these three points if we are able to make a request to our endpoints to store data, and then retrieving it. For this, we will be using Postman, a platform for testing API endpoints, to make the API calls to a random node and see if the data is stored and then retrieved. Figure 18: Store request Figure 19: Retrieve request As we can see in figure 18, we are making a call to our local machine which is running the first node instance, node A. We are making a call by passing the localhost Ip address 127.0.0.1, followed by the port that is used to create our web application. We are passing the data needed for the Store POST request, which is the key and the value, that are used to store the data in our DHT. Upon querying the key we just stored in figure 19, we successfully receive the value printed out. 54 We can wrap all of these functionalities within a sequence diagram to have a clearer understanding of how the process works: Figure 20: Sequence diagram of our application 55 messages that were sent over an unsecure connection have been dropped by a time out. The node with the new ip address responds to the WHOAREYOU by sending an authentication packet, and a new session is established. The recipient node discovers that the ENR is invalid, and sends a one time session Ping message, to which a PONG message will be responded to, therefore determining the most recent Ethereum Node record maintained, and storing it under the routing table. The node then propagates its new ENR to the rest of the other nodes by going through this same process. 5.5 ENR Update Much like the IP update, we have an initial session establishment between the nodes as we can see in figure 29. We now have our secure connections ready for messaging between the peers. Figure 29: Session Establishment Between Nodes Figure 30: Updating ENR & Emitting a PING Request Node A proceeds and updates its local UDP socket contained in the key value pairs of its local ENR as we can see in 30. Figure 31: Socket Updated Event 62 The node responds to ping requests from other peers with a new ENR, which results in a SocketUpdated Event, to the other peers that are connected to it. We can trace those messages to the rest of the nodes and see that the messages have been successfully received. The neighbouring peers have now the new ENR of the initial node. The test results in a success as well. The ENR update process went smoothly. 5.6 Concurrent Requests in Discv5 Having explored the foundational aspects of node interaction within the discv5 protocol, including dynamic IP address adaptation and ENR updates, we now turn our attention towards understanding how discv5 manages concurrent operations. This aspect is critical, as discv5 is designed to operate in highly dynamic and distributed network environments where multiple operations occur simultaneously. 5.6.1 General Concurrent Requests In this test case, we want to see how a node typically reacts to several concurrent WHOAREYOU requests. We have 2 nodes, A and B. Node A will be executing parallel FIND NODE requests to node B. Node B will be set with a short term session timeout in order to have an expired session (a few seconds), while receiving the requests. We can then demonstrate how discv5 handles the queue of incoming messages while making sure that they are only communicated over a secure connection. Node B is serving as the bootstrap node . Therefore, node A has the ENR of node B. It proceeds by sending a random packet to node B. This packet is received by node B, which will go through the handshake process of establishing a secure connection with node A as depicted in 32. Figure 32: Session Establishment Between Node A & B 63 We can see that in the figure 33 that as soon as the new ENR has been added to our routing table and that there is a secure session between both nodes, the node B looks back at the received queries in order to start responding to them over this newly established secure connection. Figure 33: 1st Response over Secure Connection Since we set discv5 to have a short session timeout, we can see the following happening in figure 34: 1. Multiple parallel requests are being sent over from node A to node B. 2. Session times out. We can read that the messages are being received without a session. 3. Node B is sending multiple WHOAREYOU requests as a response to the FINDNODE parallel requests 4. It detects that a WHOAREYOU packet has already been sent, therefore aborts from the second packet. 5. we proceed with the Authentication. Once the Authentication packet is received and the node is added again to our routing table, Node B now sends the response to the FINDNODE requests. The process runs logically in a sequential manner that mitigates any redundancy in the WHOAREYOU packets being sent, establishing a new connection again with the same node and ensuring the all requests are treated over a secure connection. The process ends with a success yet again. 64 Figure 34: Session Timeout - Managing Requests 5.6.2 Concurrent Requests before establishing Secure Session In this test case, we will be sending TALK requests in parallel from one node to another, before there even is a secure connection between the node. Figure 35: 2 TALK Requests Prior to Handshake We can see in this figure 35 that 2 TALK requests have been sent as soon as the discv5 services started across the 2 nodes. 65 Figure 36: Handshake Process Establishment The sending node notices that there is no secure connection established with the recipient node, and proceeds by sending a random packet to establish this secure connection with it. We can see a traced message by the sender peer saying that the request has been queued. We can see that one message has been placed under ”active requests”, and the rest of the messages (2 in total) are placed into ”pending requests” locally. It so proceeds and establishes the handshake process with the recipient node, exchanging authentication data as usual, and the node is now connected and officially added to the routing table in a designated k-bucket. We now delve into the last step of this test: Figure 37: TALK Responses As we can see, the sender node has the previous events received stored and fetches them back in order to re-send them over this newly established secure connection with the node 2, 1 by 1, by sending the queued TALK requests to the designated node. The messages are then received by the node in question, and the test results in a successful management of concurrent requests with no session establishment, by caching the requests, establishing a secure connection with the node and then going back to the requests in order to finally receive a response to them. 66 5.7 Eclipse Attack Having explored the dynamic behavior of discv5 in handling IP and ENR updates as well as managing concurrency in distributed environments, we now shift our focus towards the security implications of these network dynamics. Eclipse attacks represent a critical vulnerability in peer-to-peer networks as seen in section 2.3.5, where malicious nodes manipulate network connections to isolate a victim from honest peers. This type of attack is particularly concerning because it directly undermines the trust and reliability of network routing mechanisms. By monopolizing the victim’s routing table with attacker-controlled entries, an adversary can e↵ectively sever the victim’s connections to the rest of the network, leading to misinformation or denial of access. In the beginning, we see a list of multiple nodes, with on the one hand the victim node, and on the other hand an attacker node along with several malicious nodes that will come in hand-in-hand in the attack process with the attacker. These attackers will try to fill the routing table of the victim node in order to isolate it from the rest of the network. We finally have an honest node, which we will be using to check once the attack has been made, whether or not it is still able to reach our victim node. We can see the following list being printed out: Figure 38: Eclipse Attack Scenario Nodes List We then can attest that all the nodes are filling the routing table of the victim node by connecting to it as follows: Figure 39: Attacking Nodes Information & Session Establishment 67 As we can witness, all the attackers are being inserted into our routing table. The attacker node has successfully established a secure connection with the victim node upon making a handshake process. Since both are now connected, the victim node was vulnerable to the attacker’s incoming connection and has discovered the other accomplice nodes in the attack by detecting them with the Discovered Event from discv5 and then inserting them as a consequence. Finally, we can print out the routing table of the victim node: Figure 40: List of k-buckets from our Victim Node We can read in figure 40 the following information: •index - Index of our k-bucket from the routing table •num entries - Number of node entries in the given k-bucket •num connected - Number of currently connected nodes •num disconnected - Number of disconnected nodes In figure 40, we have the last buckets from our routing table. We are specifically interested in the last one. Upon receiving the state of the k-buckets, we witnessed that all of them have 0 entries, except of the last k-bucket of index 256, which has 8 entries, all nodes being currently connected. This is due to the initial settings the nodes have been created with, which is a limit to the number of nodes per buckets, set to 8. We can conclude that the attackers have successfully filled that last 8-bucket, but have dramatically failed at taking over the entire routing table with this parameter being set. The attack is therefore unsuccessful, and the victim node has not been isolated from the network. The honest node can still connect to the victim node since there are still empty 8-buckets in its routing table. However, if we were to remove the parameter, the victim node emits ”Table full” error and the test case results in failure, since the victim’s routing table is full of the ”incoming” attacker node ids. 5.8 Results Evaluation Upon reviewing and analyzing each test case, we have covered potential scenarios that can occur when networking with discv5, and the results have been really well handled. We started by covering basic scenarios of communication between nodes when faced with changes of IP address with an invalid ENR, to then covering the potential concurrent request cases. Finally, we assessed how discv5 reacts to a major security attack that can be made to distributed systems, the eclipse attack. 68 Upon running the first test cases, we notice that nodes make sure that other peers always have the latest ENR. Nodes detect that an ENR is invalid when there is a di↵erent IP address as well, and make sure that they request a valid ENR, that the peer would generate. This ENR will then get propagated to ensure that it always has the latest valid one. Furthermore, sending requests to a node with no established session are not handled directly. They are cached within a queue of messages that are pending, and only treated once a handshake process is made between the nodes. Finally, the networking system responds successfully to the eclipse attack, where a malicious node tries to fill the routing table of another victim node. It fails in doing so by adding a parameter to set a limit of peers per bucket within the routing table, which would result in only one bucket being filled by malicious nodes, and the victim node is still able to handle upcoming requests from other honest peers. 6 Outline 6.1 Conclusion In this thesis, we embarked on a practical exploration of the discv5 protocol, an essential component in the architecture of distributed systems. This investigation began with a theoretical overview of distributed systems, providing a foundational understanding necessary for delving into more complex concepts. We further examined the intricacies of discv5 and Distributed Hash Tables (DHT), focusing on their critical role in enabling peers within a network to discover and connect with one another efficiently. Through the implementation of a system utilizing both DHT and discv5, this thesis demonstrated the operational capabilities of these technologies in creating and maintaining a structured pure p2p network. Notably, the use of discv5’s routing tables and Remote Procedure Calls facilitated e↵ective node discovery on the networking layer in a decentralized manner, underscoring the protocol’s utility in real-world applications with a focus on having no central entity. Matching this technology with the distribution of data across the system of nodes with Kademlia Distributed Hash Tables goes hand-in-hand in order to fulfill those needs. Our empirical analysis included a series of tests designed to challenge the protocol under various conditions, such as IP address changes without ENR updates, ENR updates themselves, and handling concurrent requests both before and after the establishment of a secure session. Additionally, the simulation of an eclipse attack, which was mitigated by the protocol’s inherent limitations on node numbers per bucket, further highlighted the resilience of discv5 under potential security threats. The findings from these tests confirmed that discv5 is not only theoretically sound but also robust in practical applications, capable of handling dynamic network changes and security challenges efficiently. However, several functionalities are promised theoretically, but are yet to be implemented practically. These would be interesting for an establishment of sub-protocols within the networking between peers. 69 6.2 Future Work While this thesis has made significant contributions to understanding and applying the discv5 protocol in distributed systems, several promising areas remain unexplored, presenting opportunities for future research. One of the most notable areas involves the practical implementation of the topic advertisement mechanism, a theoretical aspect of discv5 that has yet to be fully realized in practical scenarios. Some of these can vary into the following: •Implementation of Topic Advertisement Mechanisms •Clustering Network Based on Topics •Scalability and Performance Optimization with topic-based clustering •Security Implications of Topic-Based Clustering •Integration with Other Protocols By pursuing these areas of future work, researchers and developers can continue to refine and expand the capabilities of discv5, potentially transforming it into a more versatile tool for managing distributed networks. This not only supports the growth and efficiency of existing systems but also opens up new possibilities for innovative and decentralized social media platforms. 70 References [1] Big o notation. URL: https://en.wikipedia.org/wiki/Big_O_notation. [2] Block cipher mode of operation. URL: https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation. [3] Discv5 testing plans github repository. URL:https://github.com/ackintosh/discv5-testground. [4] Distributed hash tables wikipedia definition. URL:https://en.wikipedia.org/wiki/Distributed_hash_table. [5] Ethereum bootstrap nodes. URL:https://github.com/ethereum/go-ethereum/blob/master/params/ bootnodes.go#L23. [6] Overlay network. URL:https://en.wikipedia.org/wiki/Overlay_network. [7] Rust crate official documentation. URL:https://docs.rs/discv5/latest/discv5/#:~:text=Discovery%20v5% 20is%20a%20protocol,optionally%20IP%20address%20and%20port. [8] Serde - framework official homepage. URL: https://serde.rs. [9] Struct discv5::discv5. URL: https://docs.rs/discv5/latest/discv5/struct.Discv5.html. [10] Testground github repository. URL:https://github.com/search?q=discv5&type=repositories. [11] Tokio - framework official homepage. URL: https://tokio.rs. [12] What is a cryptographic nonce? URL: https://www.okta.com/identity-101/nonce/#:~:text=Nonce%20in% 20cryptography%20means%20œnumber,it%20is%20only%20used%20once. [13] What is the osi model. URL: https://www.imperva.com/learn/application-security/osi-model/. [14] D. Acharya. Resource Discovery in Distributed Systems, 2022. URL:https://www.researchgate.net/publication/363431109_Resource_ Discovery_in_Distributed_Systems. 71