Malware Analysis on Android

Malware Analysis on Android Análisis de Malware en Android Final Degree Project in Computer Science Degree Faculty of Computer Science & Engineering Universidad Complutense de Madrid Course 2020-2021 Authors Daniel Puente Arribas José Ignacio Daguerre Garrido Ramón Costales de Ledesma Advisors: Marcos Sánchez-Élez Martín and Inmaculada Pardines Lence 2 Abstract In the XXI century, the world has witnessed the creation, development and proliferation of mobile devices until the massive usage apparent nowadays. The portability, instantaneity and ease of use that these devices offer has encouraged the great majority of the population to have one of them at arm’s length. Thus, these devices have become a coveted target for malicious developers. This is the reason why the security of mobile devices has become a vital topic that must be addressed, since a suitable solution has yet to be found. From this necessity arises the present work, in which we elaborate the beginning of a response that serves as a starting point to promote further development that achieves the desired objective. With Android being the most representative Operating System among mobile devices, we are going to study the analysis of malware on Android and develop a static and dynamic antivirus based on signatures, permissions and logs, since they will prove useful when trying to detect malicious applications. Keywords (10 max) Android, Malware, Vulnerability, Antivirus, Hash, Log, Permission, Static Analysis, Dynamic Analysis, Cloud. 3 4 Resumen En el siglo XXI se ha podido apreciar la aparición, desarrollo y proliferación de los dispositivos móviles hasta llegar a la masificación que tiene lugar en la actualidad. La portabilidad, instantaneidad y facilidad de uso que ofrecen ha hecho que la mayoría de la población tenga uno siempre al alcance de su mano. Es por ello que se han convertido en un objetivo codiciado por los desarrolladores de programas maliciosos. Así pues, la seguridad de estos dispositivos se ha convertido en un punto clave que debe ser abordado, ya que hasta la fecha no se ha encontrado una solución apropiada. De esta necesidad surge el presente trabajo, en el que elaboramos el comienzo de una respuesta que sirve como punto de partida para fomentar un posterior desarrollo que alcance el objetivo deseado. Siendo Android el sistema operativo más representativo entre los dispositivos móviles, vamos a hacer un estudio del análisis del malware en Android y a desarrollar un antivirus estático y dinámico basado en firmas, permisos y logs, pues estas evidencias serán de gran ayuda en la labor de detección de aplicaciones maliciosas. Palabras clave (máx 10) Android, Malware, Vulnerabilidad, Antivirus, Hash, Log, Permiso, Análisis estático, Análisis dinámico, Cloud. 5 6 Acknowledgements We would like to express our most sincere gratitude to both our project advisors, Marcos Sánchez-Élez Martín and Inmaculada Pardines Lence, for their excellent guidance throughout the production of this project. 7 Index 1. Introduction 11 1.1 Motivations 11 1.2 State of the Art 12 1.3 Objectives 14 1.4 Work planning 14 1.5 Document Organization 21 2. Android 22 2.1 Android Operating System 22 2.1.1 Introduction 22 2.1.2 Applications 24 2.1.3 Components 27 2.1.4 Intents 28 2.2 Android’s Vulnerabilities 28 2.2.1 General Vulnerabilities 28 2.2.2 Rooting 31 2.3 Android’s Security Systems 33 2.3.1 Linux Security 33 2.3.2 Application Sandbox 33 2.3.3 SELinux 34 2.3.4 System’s Security 34 2.3.5 User’s Security 35 2.3.6 Apps Security 35 2.4 Android’s Malware 35 2.4.1 Malware trends 35 2.4.2 Malware types, aims and characteristics 38 2.4.3 Transmission methods, detection and prevention 42 3. Workspace 46 3.1 Tools and samples 46 3.1.1 Tools 46 3.1.1 Samples 48 3.2 Decisions 49 4. Analysis Methods 51 4.1 Cryptographic Signature Analysis 51 4.2 Heuristic Log Analysis 53 4.2.1 Android logs 53 4.2.2 Log handling 55 4.3 Permission Analysis 56 5. Analysis Methods Implementation 59 5.1 Cryptographic Signature Analysis Implementation 59 8 5.2 Heuristic Log Analysis Implementation 65 5.2.1 Application 65 5.2.2 Server 79 5.2.3 Problems overcome 84 5.3 Permission Analysis Implementation 85 5.3.1 Dataset of permissions 85 5.3.2 Classification of permissions 89 5.3.3 Permissions analysis and scoring 90 5.3.4 Results 96 6. Android Malware Analyzer App 101 6.1 Fragment Menu 101 6.2 Home Fragment 103 6.3 Apps Information Fragment 104 6.4 Signature Analyzer Fragment 109 6.5 Permission Analyzer Fragment 111 6.6 Log Analyzer Fragment 113 6.7 Previous Results Fragment 118 6.8 Server Settings Fragment 119 6.9 About Us Fragment 122 7. Experimental results 123 7.1 Applications Analyzed 123 7.2 Permission Analysis 132 7.3 Cryptographic Signature Analysis 138 7.4 Heuristic Log Analysis 144 8. Individual Work 152 8.1 Daniel Puente Arribas 152 8.2 José Ignacio Daguerre Garrido 153 8.3 Ramón Costales de Ledesma 154 9. Conclusions & Future Work 155 9.1 Overview 155 9.2 Applied Knowledge 155 9.3 Improvements 156 9.3.1 Signature analyzer 156 9.3.2 Log analyzer 157 9.3.3 Permission analyzer 157 9.4 Future Work 157 9.4.1 Signature analyzer 157 9.4.2 Log analyzer 157 9.4.3 Permission analyzer 158 Bibliography 159 9 ii. Upgrade the app to list the hashes of all the applications. Timestamp: 21 October - 4 November. Developer: Daniel Puente. iii. Upgrade the app to list the permissions of all the applications. Timestamp: 21 October - 4 November. Developer: Daniel Puente. iv. Develop the app details activity functionality and view. Timestamp: 4 - 30 November. Developer: Daniel Puente. v. Upgrade the app to access the logs. Timestamp: 25 November - 16 December. Developers: Daniel Puente and Ramón Costales. vi. Create a mockup local database for the app. Timestamp: 25 November - 16 December. Developer: José Ignacio Daguerre. vii. Upgrade the app to connect to the server. Timestamp: 20 January - 8 February. Developers: All the team. viii. Connect the app to the server. Timestamp: 20 January - 8 February. Developers: All the team. ix. Print the result obtained from the server. Timestamp: 20 January - 8 February. Developers: All the team. x. Divide the app functionality into fragments. Timestamp: 21 - 25 February. Developer: Ramón Costales. xi. Fix navigability. Timestamp: 18 April. Developer: Ramón Costales. Figure 4: App planning 4. Server (see Figure 5): i. Create the server. Timestamp: 20 January - 8 February. Developers: All the team. ii. Create a Spark Streaming script that processes the logs. Timestamp: 20 January - 8 February. Developers: All the team. iii. Develop the multi connection. Timestamp: 21 March - 2 April. Developer: Ramón Costales. iv. Update the Spark Streaming script. Timestamp: 25 - 26 April. Developer: Ramón Costales. v. Process, parse and filter the logs. Timestamp: 26 April - 4 May. Developer: Ramón Costales. 16 vi. Send back the results to the app. Timestamp: 4 - 8 May. Developer: Ramón Costales. vii. Fix bugs. Timestamp: 8 - 11 May. Developer: Ramón Costales. Figure 5: Server planning 5. App [server settings fragment] (see Figure 6): i. List all the connections. Timestamp: 25 - 26 February. Developer: Ramón Costales. ii. Develop the add functionality. Timestamp: 26 - 28 February. Developer: Ramón Costales. iii. Develop the delete functionality. Timestamp: 28 February - 2 March. Developer: Ramón Costales. iv. Save and load the connections. Timestamp: 2 - 5 March. Developer: Ramón Costales. v. Parse the IP and port numbers. Timestamp: 19 - 20 April. Developer: Ramón Costales. Figure 6: Server settings planning 6. App [apps information fragment] (see Figure 7): i. Convert the apps list activity to a new fragment. Timestamp: 25 - 28 February. Developer: Daniel Puente. ii. Convert the app details activity to a new fragment. Timestamp: 25 - 28 February. Developer: Daniel Puente. 17 iii. Develop the permissions view organized by categories. Timestamp: 16 - 23 March. Developer: Daniel Puente. Figure 7: Apps information planning 7. App [permissions analyzer fragment] (see Figure 8): i. Permissions research. Timestamp: 1 - 7 March. Developer: Daniel Puente. ii. Elaborate permissions and domains dataset. Timestamp: 8 - 15 March. Developer: Daniel Puente. iii. Design and code permissions grading algorithms. Timestamp: 24 March - 1 April. Developer: Daniel Puente iv. Develop permission analyzer logic and functionality. Timestamp: 5 - 18 April. Developer: Daniel Puente. v. Develop permissions analyzer view. Timestamp: 19 - 30 April. Developer: Daniel Puente. vi. Finish, test, debug and correct errors. Timestamp: 1 - 12 May. Developer: Daniel Puente. Figure 8: Permission analyzer planning 8. App [signature analyzer fragment] (see Figure 9): i. Upgrade and upload the first database version. Timestamp: 25 November - 20 February. Developer: José Ignacio Daguerre. ii. Development of the Signature Analyzer fragment. Timestamp: 20 - 27 February. Developer: José Ignacio Daguerre. iii. Creation of an activity with the “Check Hash” option. Timestamp: 28 February1 March. Developer: José Ignacio Daguerre. 18 iv. Connection and interaction with the database through the application. Timestamp: 1 - 2 March. Developer: José Ignacio Daguerre. v. Display all the apps in a checkbox list. Timestamp: 3 - 17 March. Developer: José Ignacio Daguerre. vi. Implement a search box in Signature Analyzer Fragment. Timestamp: 1 - 19 April. Developer: José Ignacio Daguerre. vii. Finish Signature Analysis fragment. Timestamp: 22 April - 1 May. Developer: José Ignacio Daguerre. viii. Upgrade Signature Analysis fragment with an updatable database. Timestamp: 3 - 6 May. Developer: José Ignacio Daguerre. ix. Upload the final version of the database. Timestamp: 11 May - 12 May. Developer: José Ignacio Daguerre. Figure 9: Signature analyzer planning 9. App [log analyzer fragment] (see Figure 10): i. Display all the apps in a checkbox list. Timestamp: 5 - 10 March. Developer: Ramón Costales. ii. Program the connection to the server. Timestamp: 10 - 21 March. Developer: Ramón Costales. iii. Clean the code. Timestamp: 2 - 8 April. Developer: Ramón Costales. iv. Develop the fragment that shows the log analysis result. Timestamp: 22 - 25 April. Developer: Ramón Costales. v. Implement expandable elements while showing the result. Timestamp: 11 - 12 May. Developer: Ramón Costales. vi. Check the permissions of the apps when showing the result. Timestamp: 12 - 16 May. Developer: Ramón Costales. 19 Figure 10: Log analyzer planning 10. App [previous results fragment] (see Figure 11): i. Create the PrevResults database. Timestamp: 8 - 12 April. Developer: Ramón Costales. ii. List all the previous results. Timestamp: 12 - 14 April. Developer: Ramón Costales. iii. Develop a fragment that shows the result of an analysis. Timestamp: 14 - 17 April. Developer: Ramón Costales. Figure 11: Previous results planning 11. App [about us fragment] (see Figure 12): i. Create the fragment. Timestamp: 14 - 17 April. Developer: Ramón Costales. Figure 12: About us planning 12. App [home fragment] (see Figure 13): i. Develop the fragment. Timestamp: 18 April. Developer: Ramón Costales. ii. Display elements whose hash is yet to be analyzed. Timestamp: 20 - 22 April. Developer: Ramón Costales. 20 Figure 13: Home planning 1.5 Document Organization This document is structured in 9 chapters. Chapter 1 introduces the context of the project. Chapter 2 describes the Android operating system and reviews its vulnerabilities and the security systems developed for this OS; this chapter ends with a malware overview. Chapter 3names all the tools and samples used in the development of this project and argues the decisions we took in each step of its production. Chapter 4 explains the theory behind the analysis methods we developed for the application and Chapter 5 shows their implementation. Chapter 6 covers the user’s interaction with the application and its layouts. Chapter 7 shows the results obtained from analyzing several applications using the application developed throughout this project. Chapter 8 lists all the individual contributions of each member of the team. Finally, Chapter 9 reviews the conclusions drawn from this project and brainstorms future work and improvements that we didn’t have the time to accomplish. 21 2. Android This chapter introduces the Android Operating System [1], describes the key characteristics of applications and enumerates all types of components and intents. Also, Android’s vulnerabilities are outlined, the rooting method is explained and Android’s different security measures are listed. Lastly, an analysis of Android’s malware types, characteristics and trends is performed. 2.1 Android Operating System 2.1.1 Introduction Android is an Operating System that consists of a stack of open source software based on the Linux Kernel and created specifically for mobile devices. Its architecture is conformed of six layers [21]: -The Linux Kernel: It is the base layer of the Android Operating System, required for carrying out essential functionalities such as process management, memory, network stack, controller model and key security features. It is not the standard kernel, but a specific fork that includes additional elements, such as Binder for inter-process communication, specific drivers, etc. -Hardware Abstraction Layer (HAL): allows the Java API Framework layer to make use of standard interfaces that permit the leverage of device hardware capabilities. It consists of a set of library modules, one for each hardware component, such as the camera. -Android Runtime (ART): Each app runs in its own process and with its own instance of the Android Runtime. It is designed to be able to run multiple virtual machines on low memory devices through DEX files. Before Android 5.0, the Dalvik virtual machine was used. The main difference between Dalvik and ART is that the latter compiles the bytecode files during the installation of the application, so its key objective is to compile the sources in DEX code. It has the Core Library (a particular implementation of the Java API), basic commands accessible through an ADB shell [45], native system daemons and services, and the Init process. -Native C/C++ Libraries: They allow applications to interact at a low level with the kernel. All Android libraries are Open Source. Examples: Bionic (C standard library on Android), WebKit (web page rendering and JavaScript interpreter), SQLite (database), OpenSSL (SSL Sockets), etc. -Java API Framework: Every part of the Operating System is accessible through an API written in Java, which is used for developing applications with the objective of reusing components. It is not exactly a library, since there is an inter-process communication with Binder for limiting accesses for security reasons. Examples: notification manager, packet manager, window manager etc. 22 -System Apps: These are the preinstalled apps on the system that offer functionality for both the user and other apps that may require them. These layers are detailed in Figure 14. [21] Figure 14: The Android software stack 23 2.1.2 Applications APK files Android apps are packaged and distributed in APK (Application Package) files that are based on Java JAR packages. These packages have the following structure [22]: -AndroidManifest.xml – It is the application's configuration file. Several attributes are defined in it, such as the unique identifier of the application, it’s components («activities», «receivers», «content providers», etc.) or the permissions it requires. -classes.dex – Contains the application’s compiled code. -resources.arsc – It is the file containing precompiled resources. -META-INF – It is the directory that stores information corresponding to the digital signature of the application; it contains the following files: -MANIFEST.MF – It contains a complete list of the APK files along with their respective SHA-1 hash. -CERT.SF – It contains the SHA-1 hash of every 3 lines that appear in the MANIFEST.MF. -CERT.RSA – stores the signature of the CERT.SF file and the certificate used to sign the files. -res – It is the directory that stores the resources (images, text files, XML files, etc.) used by the application. -lib – It is the directory that contains the compiled code for different architectures: armeabi, armeabi-v7a, x86 or mips. -assets – It contains non-processed resources. Figure 15 displays the aforementioned contents of APK files. [23] Figure 15: APK file structure 24 As explained in chapters 4 and 5, APK files are of great importance in our project, as we make use of them in all the analysis methods we carry out: - In the Cryptographic Signature Analysis, we use the entire APK file to compute the hash signature of the installed applications. - In the Permission Analysis, we read the permissions that applications require from their respective AndroidManifest.xml file. - In the Heuristic Log Analysis, as in Permission Analysis, we read the permissions that the analyzed applications require. Permissions Of all the files that make up an APK file, the most relevant for our project is the AndroidManifest.xml [24], since it contains the permissions that the application requires. This information is vital for two of the three analysis methods that we perform: the permission analysis and the log analysis. Android restricts access to specific data and actions in order to protect user privacy. In case an application needs to access any of those restricted components, it has to request the specific permission needed for accessing them. These permissions are categorized in two different ways: categorized by type and by group. There are several types of permissions, divided by the scope of restricted data or actions that the application may perform once the permission is granted [25]: -Install-time permissions: They give the app limited access to restricted data, and they allow it to perform restricted actions that minimally affect the system and other apps. The system automatically grants the permissions when the user installs the app. -Runtime permissions: Also known as dangerous permissions, they give the app additional access to restricted data, and they allow it to perform restricted actions that more substantially affect the system and other apps. Therefore, they need to be requested before they can access the restricted data or perform restricted actions. -Signature permissions: If the app declares a signature permission that another app has defined, and if the two apps are signed by the same certificate, then the system grants the permission to the first app at install time. -Normal permissions: They allow access to data and actions that extend beyond the app's Sandbox. However, the data and actions present very little risk to the user's privacy, and the operation of other apps. -Special permissions: They correspond to particular app operations. Only the platform and OEMs (Original Equipment Manufacturers) can define special permissions. The permissions are divided into groups by their functionality [26]. For example, android.permission-group.LOCATION groups the permissions that grant access to the device location, like the permission android.permission.ACCESS_FINE_LOCATION or the permission android.permission.ACCESS_COARSE_LOCATION. 25 For devices without an unlockable bootloader, root access can be achieved by exploiting a kernel or system vulnerability. A privilege escalation exploit, typically packaged in one-click rooting applications, allows an application to run a root shell to install the "su" binary or modify system settings. Another way is via a privileged ADB (Android Debug Bridge) [45]. The system property "ro.secure" of "default.prop" determines the UID (User ID) of the process under which an ADB shell is executed. When the value is 1, the daemon process "adbd", which initially runs as root, changes its UID before creating the ADB shell without root privileges. Otherwise, users can have a shell that can run any program as root. Root Vulnerability Android's permission system forces access controls on security-related resources such as sensors, sensitive data and important communication modules. But if the phone is rooted, this permission system can be avoided. On a rooted phone, processes can run with root privilege and it is possible to access any resource without permission. Many people root the device to uninstall stock apps, flash third-party ROMs, use applications that require root permission, back up the phone... The problem is that when dealing with a rooted mobile phone, malware can access sensitive databases (SMS, Contacts...) and hardware interfaces (camera, microphone...) without having the corresponding permissions beforehand. In these cases, the permission system is not relevant, because it is avoided. Whereas there are applications that offer one-click-root (by clicking a button they are able to root the phone), equally there exist applications that offer one-click-unroot (it removes the "su" binary). Removing root from the phone takes away root permissions from potential malware, so the permission system becomes relevant again, denying malware access to system resources. However, during the time window in which the device is rooted, if the malware has modified the packages.xml file (containing a list of permissions and packages) or apks with root privileges, it may have escalated its permissions, causing permission escalation after root removal to be a backdoor for the malware to abuse resources [46]. They could also delete the certificate restriction for sharing UIDs with another app; if this is done with a privileged app, then privileged permissions of that app or access to its data can be obtained. Moreover, the code could also have been modified to remove the permission access control. Therefore, this type of malware offers a higher level of impact, as they are able to persist after root removal, as well as having a very high detection evasion rate. 32 2.3 Android’s Security Systems 2.3.1 Linux Security At the operating system level, the Android platform uses Linux kernel security such as secure inter-process communication (IPC) to enable secure communications between applications running in different processes [47]. This ensures that even native code is restricted by the application Sandbox. Thus, the system is designed to prevent a malicious application from damaging other apps, the Android system, or the device. The Linux kernel provides Android with several key secure features, including [47]: - A user-based permissions model. - Process isolation. - Extensible mechanism for secure IPC. - The capability to remove unnecessary and potentially insecure parts of the kernel. A fundamental goal of kernel security is to isolate the resources of one user from those of another user, thus [47]: - Prevents one user from reading another user's files. - Ensures that one user does not exhaust the memory of another. - Ensures that one user does not drain another user's CPU resources. - Ensures that one user does not drain another user's devices (telephony, GPS, Bluetooth...). 2.3.2 Application Sandbox The security of Android applications is enforced by the application Sandbox [84], which isolates applications from each other and protects apps and the system from malicious apps. It achieves this by assigning a unique user ID to each app and running it in its own process. The kernel enforces security between apps and the system at the process level through standard Linux facilities such as user and group IDs that are assigned to different apps. By default, apps cannot interact with each other and have limited access to the OS. As the application Sandbox is located in the kernel, this security model extends to both system applications and native code. All software above the kernel, such as operating system libraries, application frameworks, application runtime (ART), and all applications, are running inside an application Sandbox. Generally, to evade the application Sandbox on a properly configured device, kernel security must be compromised. Nevertheless, the individual protections that force the application Sandbox are not invulnerable, so protection into depth is important to prevent a single vulnerability from compromising the operating system or other apps. With each Android version, protections have been added to protect the application Sandbox, such as in 9.0, 33 which forced all non-privileged apps to run in individual SELinux Sandboxes, providing mandatory per-app access control, to improve the separation of apps, prevent overwriting of secure defaults, and prevent apps from making their data accessible to everyone. It is not a good idea to make data accessible to everyone, as this can be an information leak and a popular target for malware. From Android version 9 onwards this is not allowed. Therefore, for file sharing it is used by content providers or MediaStore class for those media files that should be accessible to everyone. 2.3.3 SELinux Android uses Security-Enhanced Linux (kernel security module) [48] to apply access control policies and set mandatory access controls on processes. It applies Mandatory Access Control (MAC) instead of Discrete Access Control (DAC). This implies that instead of the owner of a resource controlling the access permissions attached to that resource, any access is queried to a central authority. This ensures that the software runs only at the lowest privilege level, mitigating the effects of potential attacks. 2.3.4 System’s Security The system partition contains the Android kernel, as well as system libraries, the application runtime (ART), the application framework and applications [47]. The partition is read-only. When the device is booted in safe mode, third-party applications can be launched manually by the device owner, but are not launched by default. File system permissions ensure that a user cannot alter or read another user's files, unless the developer explicitly shares files with other applications [47]. In Android, each application runs as its own user. Verified boot ensures the integrity of the device's software, starting from a hardware root of trust till the system partition [87]. During boot, each stage cryptographically verifies the integrity and authenticity of the next stage before executing it. This makes privilege escalation non-persistent, because it detects file system modifications and compromised devices are not allowed to boot. Android provides a set of cryptographic APIs for use by applications [47]. This includes implementations of standard and commonly used cryptographic primitives, such as AES, RSA, DSA and SHA. Additionally, these APIs can be used by high-level protocols, such as SSH and HTTPS. It also has a KeyChain class that allows applications to use system credential storage for private keys and certificate chains. By default, only the kernel and a small subset of core applications can be run with root permissions. Android does not prevent a user or application with root permissions from modifying the operating system, kernel, or any other application. In general, root has full access to all applications and their data. Users who change permissions on an Android device to allow root access to applications increase their exposure to malicious applications and potential application crashes. 34 2.3.5 User’s Security Android supports full file system encryption, so all user data can be encrypted in the kernel [47]. It also allows full disk encryption, so that a single key (protected by the device password) protects the entire user data partition; at boot time the user must provide credentials before any part of the disk becomes accessible. It also supports file-based encryption, allowing different files to be encrypted with different keys that can be unlocked independently. Encrypting data with a key stored in the device does not protect application data from users with root permissions. Applications can add a layer of data protection by using encryption with a key stored outside the device, such as on a server, or a user password. This provides temporary protection while the key is not present, but at some point the key must be given to the application, making it accessible to users with root permissions. A more robust approach to protect data from possible access by users with root permissions is the use of hardware solutions. Manufacturers can implement hardware solutions that limit access to specific content. Android also allows for pre-access verification of the device through a password given by the owner. Not only does it prevent access, but it also protects the cryptographic keys for file system encryption. In the case the device is lost or stolen, the encryption of the entire file system uses the device's password to protect the encryption key, so that modifying the bootloader or operating system is not enough to gain access to the user's data. 2.3.6 Apps Security Applications can only access a limited set of resources managed by the OS [49]. Nevertheless, applications usually need access to a different set of resources outside the Sandbox such as the camera or Bluetooth. This requires the use of protected APIs, which are intended to be used by applications through permissions. Since permissions are managed by the OS, in order to use these APIs, applications must define the permissions in the manifest and then the device owner either accepts or rejects them during installation. In case an application tries to access a protected API not declared in the manifest, a security exception is raised and returned to the application, denying its access to the requested resource. 2.4 Android’s Malware 2.4.1 Malware trends Malware development for mobile devices has increased considerably in the last few years. In 2019, there were already more than 27 million malware programs in the Android mobile 35 sector [50]. A growth of 690,000 new malware programs was observed, resulting in an increasing number of botnets targeting Android systems [50]. Most infections are due to malicious apps obtained from third parties, which has increased by around 85% per year since 2011 [50]. Figure 23 below shows the growth of malware samples on Android from 2012 to 2018. [40] Figure 23: New Android malware samples per year In 2020, threats on Android devices are divided into four different categories [51]: Malware, which accounts for approximately three quarters of the total; Adware, which represents 15.4% of the total; Riskware and PUA (Potentially Unwanted Applications) being almost negligible at 6% and 4% respectively. As shown in Figure 24, in 2020 malicious activity increased by 30% in March, which coincided with COVID-19 crisis [52]. As workplace work has been forced to move to home, much of the workload has shifted to home, which is often less protected than a company's network. 36 [53] Figure 24: The Android threat activity in Q2 compared to Q1 2020 Figure 25 illustrates the top mobile threats detected by Kaspersky in both 2019 and 2020. It shows that the use of Adware has doubled in a single year. [54] Figure 25: Distribution of new mobile threats by type in 2019 and 2020, Kaspersky 37 Out of the top ten malware families detected, five of them make intrusive use of ads, with Android/Hiddad topping the list [53]. It is worth mentioning that Trojan droppers represent a third of the top ten families detected in the second quarter of 2020, as shown in Figure 26. [53] Figure 26: Top ten detected families in Q2 2020 2.4.2 Malware types, aims and characteristics Common mechanisms: -Persistence: Usually, malware samples seek to persist on the device. One of the most common mechanisms to achieve persistence is through component hiding. One possible technique to achieve this objective is to disable the activity component registered in the Launcher by the application at the moment of its installation, as a result of which the application's icon disappears. To further enable code execution, malware should implement a receiver component to log system events and a service component for background execution. This allows the malware to run in background at system events, such as start-up or WiFi activation, even though the icon is not visible. Examples of this can be seen in samples of spyware, RATs, clickers or ransomware, as they can deploy their full functionality from background execution. -Denial of service: Malware seeks to block access to screens from which the application could be removed. It does this by using services that, while running in the background, detect when the application is trying to be uninstalled and overlay a component that prevents it from doing so. To achieve this, they usually implement the GET_TASKS permission to get the application that is running in the foreground, as well as the BIND_DEVICE_ADMIN permission to register as Device Administrator. This section lists the main categories of malware that can be found on the Android operating system. Each category describes its objective and strategies for identification by the analyst [55]. 38 Adware This is the most common type of malware on Android devices. When faced with applications with ads, it is controversial to classify them as malware, as it is difficult to establish a limit at which the use of ads starts to be abusive or simply another monetization system. The main feature of this malware is the inclusion of API keys in the AndroidManifest.xml file to obtain the functionality of various ads services, such as AdMob, Baidu, Adwhirl or Ad-X. These API keys contain the identifiers that the ads services use to identify which app is displaying the ads, thus enabling monetary reward. Another characteristic used by more aggressive samples comes from the inclusion of permissions such as: -SYSTEM_ALERT_WINDOW: It overlays the current window with another one of your choice. -GET_TASKS: Allows you to see what other application is running. If adware detects that a browser is being used, it can redirect the user to an ad page. Phishing The aim of this type of malware is to steal sensitive user information (usually username and password) by deception, pretending to be a legitimate application that hides malware. Detecting phishing is relatively easy if it tries to pass itself off as a legitimate application. In such a case, by comparing the digital certificates, we can check whether they are applications programmed by the same developer or whether we are dealing with a case of impersonation, causing it to fall into the category of phishing. Malware of this type requires permissions that allow it to send stolen information, such as access to the Internet, SMS, etc. This allows us to identify this malware, especially if it should not require these permissions given the functionality it promises (a social networking application should not need access to SMS). Also, as they are copies of other applications, it is common for there to be discrepancies with the original or functional errors. Spyware This category covers all types of applications which seek to steal information from a device, such as phone number, email account, contacts, location, installed applications, calls, messages, microphone access, device ID, operating system, MAC address, etc. In its code, if there is no obfuscation, there probably are strings related to information they are looking for, such as email, location, model, phone, SMS, etc. During execution, data is frequently either sent to a server, posted on a forum or sent by SMS, which involves network traffic. Data may be sent plain or encrypted, making it difficult to identify. 39 The permissions required by the spyware depend on the information that needs to be extracted, for example: -ACCESS_WIFI_STATE: It searches for network information from the device. -READ_CONTACTS: Access contacts. -ACCESS_COARSE_LOCATION oACCESS_FINE_LOCATION: Para acceder a la localización. -READ_SMS o el RECEIVE_SMS: Access messages. -PROCESS_OUTGOING_CALLS y el READ_PHONE_STATE: Phone calls. RAT RAT stands for Remote Access Tool or, if it is hidden inside another application, Remote Access Trojan. The aim of this type of malware is to gain remote control of a device. These actions can be: accessing web pages, installing applications, sending SMS, sending user information, changing device configurations, etc. As control is remote, the application must communicate with a C&C (Command & Control) server. Commonly, these servers give the malware developer the opportunity to distribute commands to specific devices. It is for this reason that the ultimate goal of this type of malware is the creation of botnets that allow them to launch distributed attacks or black hat SEO (Search Engine Optimization) techniques (they are used to improve the positioning of a website in the search engine results list). This malware normally requires as many permissions as possible, allowing for a wider range of actions. Keyloggers This type of malware collects keystrokes that have been pressed by the user and sends them to an external server. Some controversy also arises with applications with this functionality, as there are keyboard applications that collect keystrokes and statistics to improve their services, raising a dilemma as to whether they should be considered malware or not. Such applications usually have the following permissions: -BIND_INPUT_METHOD: Must be required by an InputMethodService, to ensure that only the system can bind to it. -ACCESS_NETWORK_STATE: Allows applications to access information about networks. -INTERNET 40 Tapjacking Malware of this type is designed to trick the user into pressing on the screen, performing a different function than the one the user thinks she/he is performing. The two most typical implementation techniques are based on Toast and WindowManager. Toast is a system for displaying text messages in pop-up format. Clicks on it are non-functional, so they affect whatever is underneath the pop-up. These messages can be designed using XML, so they can be made to look similar to a dialogue with buttons. This would allow the malware developer to design a pop-up which guides the user's taps to where she/he wants them to go. It is common for such applications to require the GET_TASKS permission to know which application is open and thus which application the user's taps on the Toast are working on. Clickers The purpose of this kind of malware is to load web pages and click on links to improve the ranking of that page (black hat SEO), in ads with the aim of creating a large number of hits that generate a financial benefit for the developer, or redirect a victim to download other malware. It is common for many of the click-accounting systems on websites to be JavaScript code. Clickers must therefore have the ability to load HTML code and interpret JavaScript. Some declare the SYSTEM_ALERT_WINDOW permission. Ransomware Its aim is to inhibit access to device resources, typically to demand a financial payment. On computers it is usually implemented by encrypting files; Nonetheless, on Android this method is less common due to the application Sandbox, which limits the resources that each application can access, even if a device is rooted , the ransomware could escalate privileges and gain access to all files. Therefore, the most common in Android are activity blockers, which require permissions to identify the application that is in the foreground and overlap with it, causing the user to be unable to use their device. These applications usually require the following permissions: -RECEIVE_BOOT_COMPLETED: Launch ransomware as soon as the device boots up. -USER_PRESENT oSCREEN_ON: Detect if the user is interacting with the device. -WRITE_SETTINGS: Modify settings on the device. - BIND_DEVICE_ADMIN: Allows SystemUI to request third party controls. 41 Android Virtual Machine An Android emulator was required in order to test the analyzer with real malware. The machine has been configured with the virtualization environment VMware, and the version of the operating system is Android 8. PackageManager and PackageInfo In order to handle the metadata information that the installed applications contains, the analyzer uses the classes PackageManager and PackageInfo. The PackageManager is a class for retrieving various kinds of information related to the application packages that are currently installed on the device [78]. The PackageInfo is a class that contains overall information about the contents of a package. This corresponds to all the information collected from AndroidManifest.xml [77]. Github Desktop Github Desktop is an application that enables users to interact with GitHub using a GUI instead of the command line or a web browser [64]. We used this tool to manage the versions of the application and to seamlessly merge each contribution of all the participants of the team, allowing a parallel development. 3.1.1 Samples In addition to the tools necessary to develop the work presented here, we have had to look for samples of malicious applications or permissions misinformation to test the tool. Malicious Apps Hashes Despite efforts to acquire a database containing the digital signatures (hash function) encrypted in different encryption algorithms (SHA, MD5, etc.) of all existing malware, we could only find a list of MD5 hashes of malware samples [76]. Nowadays most common encryption algorithms for digital signatures of applications are SHA1, SHA2, and MD5. Therefore, we tried to obtain a dataset which contains one of these algorithms. It is worth mentioning that the most secure option is the use of SHA256 algorithm or higher as it causes less collisions, so we have therefore adapted the Cryptographic Signature Analysis so that it remains functional when using a different hashing algorithm. 48 Permissions Dataset The permissions dataset is mostly used by the permission analysis but it is also used in other aspects of the application. This sample has been made from various sources, specifically from the permissions API reference page of the Android Developers Official Site [79] and from the Android Permissions site [80] so as to get a dataset as complete as possible. What is stored in this dataset are the permission constants, the level of danger, a description of the permissions and the group they belong to. Domains Dataset The Domains dataset is also used by the permission analysis. This sample has been made from the Android Permissions site [42] where all the permissions are assigned to a specific group. In this dataset it is stored the permission groups, an alias of the domain and a description of the group. 3.2 Decisions In this section we present a summary of the design decisions that we have made throughout the development process of this work and that affect the final form of the developed tool. - We decided to implement a log analysis for Android because we wanted to have a dynamic analysis to add some functionality over the two other methods of analysis. Also, since we did not find much information about it, we wanted to try and create something relatively new. - We opt to implement a permission analyzer because it can give the user plenty of information about what the application is trying to achieve. Also, by giving a score we believe we can show visually and effectively if it is actually a benevolent application. Finally, we think that it can raise the user’s awareness regarding permission granting and encourage them to check the permissions of the applications being installed on their device. - We chose to implement a signature analyzer since if there is a collision found with its signature, it is almost certain that it is malware, implying that this analysis method gives an almost absolute certainty to the user. - We decided to implement a Server Settings Fragment because every time the EC2 instance is launched, a new IP is set for the instance. If we had not developed this fragment, each time we launched the instance we would have needed to write the new IP in the application code, build the APK and install it on our mobile phones. Also, this fragment allows the user to create their own server and only have to worry about adding the IP without changing any code. - We chose to process the logs on the cloud because doing it locally would have implied that the application would probably lag or crash due to the amount of processing power needed. It also implies that the battery usage is reduced, since less power is needed. Finally, this allows the user to set up their own server for processing the logs. 49 - We also concluded that we would only analyze the applications installed by the user and not the stock applications since these applications are developed by relevant companies which are globally trusted. Also, since the user cannot uninstall them, it would only mean that the analyses would take more time and they would not gain anything. - We decided to implement the upgrading and adjustment of the application code in case of a database replacement, because in case of being able to acquire a data set with encryption algorithms better than MD5 or even containing several types of algorithms applied for a single application. - We decided to develop our app for at least Android 8.x (Oreo) versions, as they moved from an Install-time permissions policy to a Runtime permissions policy. In addition, we make sure we are up to date with version 11, which can be considered the most up to date version, as version 12 is still in testing. Finally, we decided to work with this version because, in the permission analysis, the app retrieves the category of the installed apps. This action can only be performed with a 26 API level which corresponds to Android 8. 50 4. Analysis Methods This chapter introduces the theory required for understanding the three analysis methods we have performed in the application, which are the Cryptographic Signature Analysis, the Heuristic Log Analysis and the Permission Analysis. 4.1 Cryptographic Signature Analysis This section focuses on the concepts of cryptographic signatures and their subsequent analysis to detect malware. In the literature on this subject, fingerprints calculated with a hashing algorithm are often referred to as signatures. There are currently other methods of malware detection, but the use of signatures or hash functions by comparing with the results of previously detected and analyzed malware is still the most functional technique for antivirus or security systems. Google Play Store requires that each APK must be signed with two digital certificates: an App signing key (used to sign APKs that are installed on a user's device) and an Upload key (used to sign the app bundle or APK before you upload it for app signing with Google Play). As part of Android security, the signing key never changes during the lifetime of an application, so it not only ensures that Android applications are trustworthy, but also verifies that the application has been provided by a trusted source [65]. If a third party manages to take an App signing key without the knowledge or permission of an app developer, it could sign and distribute the app that maliciously replaces the authentic application or corrupts it. Moreover, it could also sign and distribute apps under your identity that attack other apps or the system itself, or corrupt or steal user data. The certificate fingerprint is a short and unique representation of a certificate that is often requested by API providers alongside the package name to register an app to use their service. The MD5, SHA-1 and SHA-256 fingerprints of the upload and app signing certificates can be found on the app signing page of the Play Console. When you are trying to publish an application you must have previously signed it by yourself providing the SHA-1 of your signing certificate or you upload it to the Play Console, and Play App Signing takes care of the rest. Google Play Store checks that the package name and certificate match with the application and if they do not match it is not offered to users but if it is an update of an existing application in the store it will consider it as a new application and will not offer it to users as an update [66]. The analysis of cryptographic signatures is based on database queries, which store the information obtained from previously reported or analyzed malicious files or applications. This information contains the summary functions of the malicious files which are used to uniquely and unambiguously identify each file hosted in the database. Cryptographic signatures are a mathematical algorithm (hash function) that maps a data set, regardless of its size, to a bit-array of a fixed size. They are essential for malware detection 51 since in case of even the slightest modification of the data or code, the bit-array changes extensively. They are also deterministic so that a malicious file always generates the same bit array when applying the same hash function and if the hash function chosen has a weak collision, it would be impossible to find one malicious file and another valid file containing the same hash. There are different types of algorithms used for hash function generation, but the most common are SHA2 (256, 384 or 512 bits), SHA1 (160 bits) and MD5 (128 bits). The notorious difference between the previously mentioned algorithms focuses on the length of the generated hash string, the longer the length of the string the lower the probability of a collision. We can see in Figure 28 an analysis of an Android application performed by the online tool VirusTotal [81], which makes use of a database management system that stores signatures. It focuses on performing file queries remotely using a hash function (SHA256) in order to check if the files are malicious. [81] Figure 28: VirusTotal - Analysis of an Android apk 52 4.2 Heuristic Log Analysis Log files are computer-generated text files that are automatically produced whenever a specific event takes place in a specific environment, such as an operating system, application, server, etc. They contain information about usage, activities and operations. This information is useful for troubleshooting and debugging the environment, since they keep a record of everything that has happened in a textual format. They typically have the LOG file extension. Each operating system has different methods of starting or stopping logs recording, since both the environment and the specific events that trigger them are different. Therefore, each OS is uniquely configured to generate log files in response to specific events. In the case of Linux, it divides log files into four categories: Application logs, Event logs, Service logs and System logs [67]. 4.2.1 Android logs Android Logging System consists of different circular buffers, which provide logging for different parts of the system. These log buffers are [68]: -radio: This buffer contains radio/telephony related messages. -events: This buffer stores binary system event messages. -main: This is the default log buffer, which does not contain system and crash log messages (it contains the applications logs). This is the only buffer available to apps. -system: This buffer contains the system logs. -crash: This buffer stores logs related to crashes. -kernel: This buffer stores kernel related logs. -security: This is the security log buffer. -stats: This buffer corresponds to statistics logs. Each message in the log consists of a tag indicating the part of the system or application that the message came from, a timestamp, the message log level and the log message itself. The log level is a character that encodes the priority of the log entry (it is Android’s terminology for severity level). Here we list all the possible values it can take, ordered from lowest to highest priority [68]: -V: Verbose (lowest priority) -D: Debug -I: Info -W: Warning -E: Error -F: Fatal -S: Silent (highest priority, on which nothing is ever printed) The Log class (android.util.Log [69]) is an API that allows users to create log entries based on their log level. It contains several public methods for logging in each priority. For 53 example, for Verbose priority the user can use the method Log.v(), for Warning priority the user can use Log.w(), etc. Typically, these methods take two arguments: the log’s tag (for example, it could be the name of the activity that creates the log) and its message. The API then creates the log entry with the passed values and adds the timestamp, the identifier of the issuing process and thread and other information. On the other hand, the Logcat command-line tool is used for reading logs. The user can run logcat through an adb shell using the following syntax: [adb] logcat [<option>] ... [<filter-spec>] … This command has a wide variety of options; these are the most relevant [70]: --b <buffer>: Specifies the log buffer that is going to be read. --c: Clears the entire buffer. --d: Dump the log contents. --v <format>: Sets the output format for log messages. The default is threadtime format. There are several output formats that modify the output so that they display certain metadata fields. The following list corresponds to the supported output formats [70]: -brief: Display priority, tag, and PID of the issuing process. -long: Display all metadata fields. -process: Display PID only. -raw: Display the raw log message with no other metadata fields. -tag: Display the priority and tag only. -thread: A legacy format that shows priority, PID, and TID of the thread issuing the message. -threadtime (default): Display the date, invocation time, priority, tag, PID, and TID of the thread issuing the message. -time: Display the date, invocation time, priority, tag, and PID of the process issuing the message. Now we show some examples of the most relevant formats: - The default output format has the following structure: Date Time PID TID Priority Tag: Message 05-16 20:04:58.151 6992 7560 i camera : open camera: 1, package name: com.whatsapp - The brief output format has the following structure: Priority/Tag( PID): Message I/ActivityManager( 585): Starting activity: Intent { action=android.intent.action...} - The long output format has the following structure: 54 [ Date Time PID: TID Priority/Tag ] Message [ 05-28 18:30:53.542 3716: 3733 I/com.whatsapp ] Background young concurrent copying GC freed 25395(1669KB) AllocSpace objects, 0(0B) LOS objects, 24% free, 6753KB/8917KB, paused 262us total 112.240ms 4.2.2 Log handling Log files record a large amount of information that conveys everything that is happening in the system. This makes log files an important element to consider if we want to analyze what applications are running on the system and try to figure out what actions they are carrying out. Therefore, log analysis is a scrutiny method widely used in the industry of malware detection. The most typical use cases for log analysis are [71]: -Compliance with security policies, audits or regulations. - System troubleshooting. -Forensics. - Security incident response. - Understanding online user behaviour. -Performance improvement. Depending on the use case, the behaviour of log analysis differs according to the context of the log files. After all, both the data and objective behind a network log analysis are not the same as a system log analysis. Hence, log analysis must interpret messages within the context of the application or system. Nonetheless, they usually have some procedures in common [71]: -Normalization: Converting log messages from different sources into a uniform format. -Pattern recognition: Selecting incoming log messages and comparing them with a previously established dataset to filter or handle the logs in different ways. -Classification and tagging: Ordering and classifying log messages into different categories based on specific keywords, dates, etc for later usage. -Correlation analysis: Collecting messages from different systems and finding all the messages belonging to one single event. -Artificial Ignorance: Discarding log entries which are known to be uninteresting. The normalization and classification procedures ensure an ease of use while handling the log messages. Secondly, the pattern recognition and correlation analysis procedures grant the analyst the information necessary to draw useful conclusions from the log entries. Lastly, the artificial ignorance procedure ensures the certainty of the results as well as a better performance. Log analysis is a type of dynamic analysis, since it examines the behaviour of a system, network or applications while they are in execution. This implies it is a time and resources consuming process, since huge amounts of information are generated each second and 55 every log must be checked. Nevertheless, its greatest advantage is that it allows the administrator to discover how the analyzed element is interacting with the system, which helps discover malfunction or damages. There are tools centred in Android log analysis to monitor system use. As an example, SolarWinds Loggly [72] has several functionalities that allow the user to perform an analysis of the logs of his device: it aggregates all of the Android logs on the cloud so that the user can monitor and analyze them by means of search queries and simplified charts and dashboards. Another example is AndroidLogViewer [73], which displays the logs of the user’s Android device and allows him to search in them using regular expressions, filter them by tag, PID, priority, etc and more. 4.3 Permission Analysis Android app permissions are considered to be a filter that helps to preserve user privacy by protecting access to restricted information, such as the user's system status and contact information, and to restricted actions, such as connecting to a linked device or recording audio. They lie in the AndroidManifest.xml file [24] and there are different classes depending on their purpose and restriction scope that they grant. This section covers the importance of Android permissions from a malware analysis approach and provides a detailed explanation of how they are assessed and classified. Although the previous analysis and all the information declared in the AndroidManifest file must be considered as relevant, an Android permissions assessment is undoubtedly one of the sections to which more attention should be paid as a starting point when analyzing malware on Android. All system functionality that the application wants to access have to be declared within the AndroidManifest.xml file under the following tags structure: <manifest> <uses-permission /> <permission /> <permission-group /> … </manifest> Since in this work the permission analysis is performed on the <uses-permission /> tag, next, we review what its objective is. <uses-permission> tag This tag indicates what permissions an application requires, referring to hardware and software components that are on the device and that the application can make use of. From a malware analysis perspective, the key is to find some kind of unusual behaviours and other indicators. Therefore, in a study of declared permissions it is important to bare in mind the following tasks [74] as a guide: 56 - Identify those applications that request a large amount of permissions. These kinds of applications generally demand additional permissions without actually requiring them, which is a sign of unusual behaviour and might be a hint of a malware entry. - Identify the functions that the application intends to perform through the declared permissions. - Identify the permissions that it does not make sense to declare according to the supposed nature of the application. For example, an application whose supposed functionality is to allow the user to change the wallpaper desktop background, but which, through its permissions, requests sending of SMS messages. - Identify the permissions that the application does not declare, but it would be expected to declare according to the supposed nature of the application. For example, a photography application that does not require access to the camera. - Identify the permissions that the application declares and, according to its classification, look for those which are considered to be invasive and potentially dangerous. For example, permissions that delete packages, mount/unmount filesystems, read logs, etc. On the other hand, if the application installation process is analyzed, some peculiarities should be observed depending on the type of installation that is carried out: - In case of installing through the Google Play Store, the permissions are shown to the user grouped by categories to request their consent. At this point it is important to note a peculiarity that occurs when updating applications that are already installed on the device, since there may be threats that make use of it: ○ If a permission from a category that has not been previously approved is added, a confirmation dialog is shown to the user in order to approve the new group of permissions. ○ If a permission from a category that has been previously approved is added, no confirmation is requested from the user. So there might be the possibility that an application that initially requested a certain category, after an update, incorporates a new permission that belongs to the same category without asking the user for any confirmation. For example, an app that initially requests the Messaging category because it uses the android.permission.READ_SMS permission, after an update, it incorporates the android.permission.WRITE_SMS permission without asking the user for approval. - In case of installing an APK through alternative markets such as Amazon AppStore or Aptoide (in any of the cases it is necessary to have allowed the installation from 57 -InstalledAppsAdapter: It is a view adapter which controls how the Recyclerview shows the view. It also maps all application’s information from the xml file to functions in the adapter. There is implemented a listener which wraps the entire item and if the checkbox is clicked, it is set to the opposite of what it was, so that with the use of notifyDataSetChanged() method, the list of apps is refreshed with its respective checkbox from RecyclerView. -SignatureAnalyzerFragment: It is the main class where all button’s functionalities are defined with their own listener and mapped to XML components based on the graphical view. Unfortunately, MD5 has been cryptographically broken and considered insecure. For this reason, it is always recommended to store cryptographic signatures using a different hashing algorithm, so we decided to implement a method (Figure 35) in which it examines the first line of a database added and it checks which hashing algorithm is used by its character length. This makes our app capable of updating the database management in case there is a hash database substitution. Figure 35: Method to recognise which algorithm is used in the database Problems overcome The first version of the database presented a problem which was the repetition of the tuples when initialising our database, so to solve it we proceeded to modify hash, d_grp and idperm columns to type UNIQUE. Regarding the development of the cryptographic signature analyzer, several modifications were made until we reached the final version. After managing to display the list of applications installed on the Android device along with a checkbox, we decided to implement the SELECT ALL option to make it straightforward for the user to make a full selection of apps. In addition, it was decided to add a Searchbox as it was difficult to find a specific app among all the installed ones. Being aware of the fact that the Malware List table was not secure enough in terms of the hashing algorithm used, we structured the cryptographic signature analyser so that it can be functional with any database containing a different hashing algorithm. 64 5.2 Heuristic Log Analysis Implementation In the discussions carried out by the members of the team for the design of the malware detection tool, we came to the conclusion that a heuristic analysis of the log files was necessary. The reasons behind our decision to implement this analysis are the following: - To alert the user of all the applications that are currently running on the device, since some of them may be running in the background without the user’s knowledge. - To let the user know what target elements (camera, SMS, storage, etc) each running application is trying to access. - To let the user establish specific keywords that will be monitored. The result shows all the applications that shared a log entry with that specific keyword. - To check if the applications that accessed certain target elements had the permissions necessary to do so. Of all the procedures commonly used in log analysis, we use the following: -Classification and tagging: We classify the logs by package name and keyword. -Correlation analysis: We group all the logs of each application and correlate all the logs of a specific application by the target elements accessed. -Artificial Ignorance: We only use those logs that contain something related to applications, since there are plenty of system logs that are not interesting for malware detection purposes. We have developed this analysis dividing its functionality into two parts: -Application: It connects the Android mobile device to the server, extracts its log entries, sends them to the server along with the package names of all the user apps installed on the device and some filters set by the user. After the analysis is stopped, it retrieves the result from the server and stores that result in the Previous Results database located in the Android device. Finally, it displays the conclusion visually, after processing the result. -Server: Whenever an incoming connection from any mobile device arrives, it processes the logs received based on the filters established by the user and, after storing all the logs, it summarizes the entries of interest. Once the stop signal is received, it retrieves the results obtained and sends them back to the application installed in the mobile device through the connection. 5.2.1 Application Setup Reading logs is not something that a regular application should do. For this reason, the permission needed for accessing them, called android.permission.READ_LOGS, is a forbidden permission, which means that it cannot be granted by the user like normal permissions. One way of granting this type of permissions to applications is by rooting the device, which is extremely risky. Instead, the user can use ADB (Android Debug Bridge) [45] to open a shell that communicates with the device. 65 First of all, the user must be able to access the developer options of the Android device: - This is achieved by clicking 7 times the build number, kernel version or other value inside the information about the phone (depends on the device). - Inside the developer options the user must allow the USB debugging. - The user needs to have installed ADB through platform-tools [82] on the computer to which the phone is going to be plugged. - After plugging the phone to the computer and accepting to connect to said computer, the user must enter the directory of platform-tools through a terminal (CMD in Windows). Using the command adb devices we can check if the device is being detected. If it is, the user must execute the command adb shell pm grant com.example.androidmalwareanalyzerandroid.permission.READ_LOGS, which grants AndroidMalwareAnalyzer the READ_LOGS permission. Classes developed As Figure 36 shows, three directories were developed to contain the functionality of the Log Analysis: -serverSettings: Handles the IP and Port numbers used to connect to the server. -logAnalyzer: All the functionality of the analysis is contained inside this directory, except saving the result obtained from the server and displaying it. -prevResults: Contains the PrevResultsDB, which is the database used to store the results of both the signature and log analyses. It also contains the functionality needed for displaying said results. Figure 36: Organization of the Server Settings, Log Analyzer and Previous Results classes 66 Next, we explain each class that composes the Server Settings: -ServerSettingsFragment: This Fragment is used to handle the addresses (IP:Port) that the Log Analyzer will use to connect to the user. All the already stored addresses are shown through a ListView.These addresses can be selected by the user, simply by pressing them. The address selected by the user is the one that is going to be used for the connection. This Fragment also displays one button for adding addresses and one for deleting the selected address. The addresses are stored inside SharedPreferences. When the user adds a new address, both IP and Port numbers are parsed. The parsing method is shown in Figure 37. Figure 37: ServerSettingsFragment - Address parsing Next, we explain each class that composes the Log Analyzer: -LogAnalyzerFragment: This Fragment is the entrypoint of the log analysis. It is used for displaying two buttons. Both buttons redirect to the LogAnalyzerFragmentApplications, passing a boolean parameter, which tells the class if the applications to be listed have to be user applications (installed by the user) or system applications (stock-apps, system services). The user will have to choose between one of them. -LogAnalyzerFragmentApplications: This Fragment is the frontend of the analyzer. It first creates a RecyclerView and sets its Adapter with a new class, called 67 LogAppsAdapter. Then, it also initializes the LogAnalyzerConnect. In this Fragment, the user can type some keywords that will be monitored while analyzing the logs. When the analysis is started, the LogAnalyzerConnect is executed. When the analysis is stopped, it saves in the PrevResultsDB the result obtained and calls the class ShowLogResult to display the result. This class also has an alert message system that, based on an integer, displays a Snackbar with information regarding the status of the analysis (Figure 38). Figure 38: LogAnalyzerFragmentApplications - status -LogAppsAdapter: This class extends RecyclerView.Adapter. It is responsible for controlling the checkbox list of applications. It first retrieves all the applications installed on the device, both user and system apps (Figure 39). The package name, application name and icon of all the applications are stored in two ArrayLists of PackageInfoStruct classes, which we developed to store typical values of applications (Figure 40). One of the arrays contains the user apps and the other the system apps. Depending on whether the user chose to list user or system applications, the corresponding array is used in the checkbox list. For each application, this Adapter shows its icon and application name and a checkbox; if no application name is found, the package name is printed. 68 Figure 39: LogAppsAdapter - get installed apps Figure 40: LogAppsAdapter - PackageInfoStruct -LogAnalyzerConnect: This class extends AsyncTask [86]. It is responsible for extracting the logs, sending them to the server and retrieving the result from the server. It first loads the selected address that was stored in SharedPreferences in the ServerSettingsFragment. When this AsyncTask is executed, the AsyncTask method doInBackground() is called. This method first connects to the server using the loaded address (Figure 41). Once connected to the server, the logs are sent through the connection (Figure 42). When the user presses the button to stop the analysis, the result is retrieved through the connection (Figure 43). Finally, when doInBackground() finishes,the AsyncTask method onPostExecute() sends the result back to LogAnalyzerFragmentApplications. Figure 41: LogAnalyzerConnect - New connection 69 Figure 42: LogAnalyzerConnect - Extract and send logs Figure 43: LogAnalyzerConnect - Retrieve result - LogResult: This class is responsible for converting the result from text into an array of LogInteractions classes (Figure 44), which we developed to store information needed to display the result. It also has a function that reduces the size of the result, as the data retrieved from the server may have some application results split over more than one line, allowing it to be reduced to a single line. This way, the database saves some space. Figure 44: LogResult - Attributes 70 Next, we explain each class that composes the Previous Results: -PrevResultsDB: This class acts as a database for the results obtained from the log and signature analysis methods. This database contains a table called prevResults, which is formed by the columns shown in Table 4. The datetime of the completion of the analysis is stored to know when the analysis was performed. The analysis_type row differentiates between the signature and log methods. The apps_analysed value is used to know what elements were analyzed and the analysis_result stores the result obtained from the analysis. The two main methods of this database are insertToDB() (inserts a new result into the table) and readAllFromDB() (retrieves all the results performed). PrevResults Column name Type Schema _id INTEGER PRIMARY KEY AUTOINCREMENT date TEXT Date and time when the analysis was finished analysis_type TEXT Describes the analysis type (log or signature) apps_analysed TEXT List of the keywords and package names of the applications analyzed analysis_result TEXT Result of the analysis Table 4: Previous Result Table structure -PrevResultsFragment: this Fragment is the entrypoint of the Previous Results. It first gets all the results stored in the database through the method readAllFromDB() and displays them in a RecyclerVIew whose Adapter is an instance of PrevResultsAdapter. If the user selects one of the elements of the list, she/he will be redirected to ShowResult if it is a signature analysis result or ShowLogResult if it is a log analysis result. If no analysis has been performed, the message "No analysis performed yet" is displayed. -PrevResultsAdapter: This class extends RecyclerView.Adapter. For each element, it displays the date when the analysis was performed and the analysis type (signature or log). -ShowResult: This Fragment displays the result obtained from a signature analysis. Initially, this class calls its method getInstalledApps() (see Figure 45). First of all, this method checks if no applications have been analyzed. If true, then the list of apps analyzed will display “None”. If at least one application has been analyzed, it gets the application name, package name and application iconn of the analyzed 71 applications. This process is similar to the one used in LogAppsAdapter. These applications are then displayed using the PrevResultsRecyclerView class, with the ElementsAdapter. Afterwards, a TextView displays the result. Figure 45: ShowResult - getInstalledApps -PrevResultsRecyclerView: This class extends RecyclerView. It was developed to establish a dynamic size for the list, since the list had to have the necessary height to wrap its contents, but also maximum height, so that it did not take up too much space. As seen in Figure 46, this was accomplished by overriding the onMeasure() method and setting a height of 750 at most. Figure 46: PrevResultsRecyclerView - onMeasure -ElementsAdapter: This class extends RecyclerView.Adapter. It displays all the applications or elements analyzed (the keywords specified in LogAnalyzerFragmentApplications). 72 -ShowLogResult: This Fragment is used to show the results of a log analysis. It firstly creates an instance of LogResult and calls its method getList(), which converts the result from text to an array of LogInteractions. Afterwards, the method getInstalledApps() of ShowLogResult is called (see Figures 47 and 48). The functionality of this method is quite similar to the also named getInstalledApps() method of ShowResult. First, if no elements were analyzed, an ArrayList is created with only one element that will be displayed as “Everything”, since not choosing any element to analyze implies analyzing all the user applications. If at least one element was analyzed, the ArrayList is set with the application name, package name and application icon of the analyzed apps. Then, it traverses all the elements and sub elements present in the result, adding the application name, package name and application icon of the applications and the name of keywords and target elements to two ArrayLists of LogInteractions. The elements of the result can either be applications (which implies that their sub elements are the target elements they have accessed) or keywords set by the user (which implies that their sub elements are the applications that have accessed that keyword). Having the three ArrayLists, first the elements analyzed (apps and keywords) are displayed through a PrevResultsRecyclerView, with the ElementsAdapter. Then, a list of all the applications that were running during the analysis is shown in a RecyclerView, with the LogElementsAdapter for the elements (applications) and LogSubElementsAdapter for the sub elements (target elements) since the applications that have accessed at least one target element can be expanded. Next, a list of the keywords specified by the user and that were found during the analysis is shown in a RecyclerView, with the LogElementsAdapter for the elements (keywords) and LogSubElementsAdapter for the sub elements (applications) since the keywords that have been accessed by one application can be expanded. Finally, the setPermissionsList() method checks if the applications that have accessed a target element had the permissions necessary to do so (see Figure 49) (we decided it would be better to check whether the apps have a permission within a specific permission group instead of comparing all possible permissions, because the permissions needed are well summarized by their permission group; the only exception is NFC, since it is a very specific permission inside the permission group Network, which is extremely broad). This is achieved by verifying all the permissions granted to those applications (see Figure 50) and contrasting if one of those permissions allows the app to access the specific target element. For this purpose a list with the target elements and their corresponding necessary permissions () has been created to check if the applications have them granted; the values are displayed in Table 5. The result of checking the permissions is shown in a RecyclerView, with the LogElementsAdapter for the elements (apps / keywords) and LogSubElementsAdapter for the sub elements (target elements / apps) since the applications that have accessed at least one target element can be expanded. A symbol at the right of each element shows if the necessary permission is granted (green check) or not (red cross). 73 sudo curl -O \\ http://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz sudo tar xvf ./spark-2.2.0-bin-hadoop2.7.tgz sudo mkdir /usr/local/spark sudo cp -r spark-2.2.0-bin-hadoop2.7/* /usr/local/spark Add /usr/local/spark/bin to the PATH: export PATH="$PATH:/usr/local/spark/bin" Include the internal hostname and IP to /etc/hosts. For example: 127.0.0.1 localhost 172.30.4.210 ip-172-30-4-210 Finally, download the two necessary scripts that we have developed for the server: -server.c: Manages incoming connections, parses incoming logs, sends them to the analyzer.py script and, when the user has stopped the analysis, it gives the result back to the user. -analyzer.py: Spark Streaming script that stores, splits and reduces the logs. To execute the scripts, open two terminals. In the first one execute: gcc -Wall -g server.c -o server ./server In the other one execute: spark-submit analyzer.py Next, the functionality of both scripts (server.c and analyzer.py) is going to be explained in detail. Server.c The server.c script performs three functions described below: 1. Connection management First, the script waits until it can connect with analyzer.py via TCP. The socket established for this connection is identified by the address localhost and the port 9999. Once the connection with the script has been established, the server begins to accept client connections. We have programmed this script to always be in execution, allowing all clients to connect to the server whenever they need. For this reason, the server has been developed as a concurrent server with an accept-and-fork pattern. We can see this pattern in Figure 52, but after the method accept() returns, the server forks. 80 [85] Figure 52: Elementary TCP Socket When the connection has been established, the function receiveData() is called. 2. Log management The logs are managed inside the method receiveData().This method first receives the keywords and applications that are used as filters and the package name of all the user applications of the client’s device. Then, the process loops until the client sends a specific signal that instructs the server that the user has finished sending logs. For ease of use, we decided to lower case all the logs received. Now, the server checks if the logs contain any of the package names from among all the user applications on the client device, which were received at the start of this method. If one is found, we consider that log to be of interest, since we only want to monitor the behaviour of the user applications. Since we want to show the user what applications are running, we send the logs related to the target elements to be monitored to the analyzer.py script, concatenating at the beginning of the log the IP of the device of the client and the package name of the application found within the log. 81 We decided that there are several target elements that require special attention in terms of possible malicious behavior. These target elements are: - location - gps - camera - microphone - sound - recorder - telephony - bluetooth - wifi - network - messaging - mms - sms - sdcard - storage - contacts - nfc - mail - account Now, three checks are performed: -The keywords and applications filters are empty: we check if there are any target elements in the log. If there is one found, we send it to analyzer.py, concatenating the client’s device IP, the package name of the application found inside the log and that target element. -The keywords filter is not empty: we check if there are any keywords of the keywords filter in the log. If there is one found, we send it to analyzer.py, concatenating the client’s device IP, the keyword and the package name of the application found inside the log. -The applications filter is not empty: we check if the application is in the specified applications filter. If it is, we check if there are any target elements in the log. If there is one found, we send it to analyzer.py, concatenating the client’s device IP, the package name of the application found inside the log and that target element. 3. Getting the result Since spark streaming stores everything in several partitions, we created the function getResults(), which opens every file inside every subdirectory found in the Result directory. Then, we send to the client all the results that contain his IP and delete those results. Analyzer.py 82 We have programmed this script to always be in execution, allowing all clients to connect to the server whenever they need. Figure 53 shows the code of this script. Figure 53: analyser.py First, the script waits until it can connect with the server.c via TCP. The socket established for this connection is identified by the address localhost and the port 9999. Once the connection with the script has been established, the server begins to accept client connections. It is a Spark Streaming script, which implies that it is constantly getting information. First, it saves the logs received in the AndroidLogs directory. Then, it parses those logs and reduces them with the following format ((IP, package name, target element), number of grouped logs). Finally, it saves those reductions in the Result directory. Figure 54 shows a snippet of the script in execution. Figure 54: analyzer.py running 83 5.2.3 Problems overcome -The IP and Port numbers of the server were initially hard-coded. The problem of this approach is that the IP of the EC2 instance changes each time the instance is launched, which implies that the application had to be modified, compiled and installed whenever the cloud instance was launched. For this reason, we decided to implement a TextView where the user could write the IP and Port. We then realized that every time the application was launched, the user had to input the IP and Port, which was a very exhausting task. Finally, we opted to develop the ServerSettings class for storing the IP and Ports. With this class, the user has to input them only once. - At first, all types of applications were printed (user and system apps). This made it too difficult for the user to choose a specific application, so we decided to divide them into two different fragments. We also decided to keep the system applications because it can be very interesting when performing a log analysis to analyze by Camera, Bluetooth, etc. - In Android, an application cannot connect to a socket in the UI Thread (main thread of execution for the application). This meant that the connection to the server had to be executed in another thread. We decided to achieve this by using the AsyncTask class for its simplicity in programming and the fact that its computation runs in a background thread and its result is posted to the UI thread. - When sending data to the server, we realized that recv() call did not always read all the data sent. More specifically, if the size of the data was too big, only a fraction of that data was received. We at first thought that it could be due to the size of the buffer where the data was being stored, but we later realized that we shouldn't expect to receive the data in the same number of read calls as there were write calls. For that reason, we used termination characters, to specify the end of the data being sent. We used ‘#’ for the user app package names, ‘\n’ for the logs and ‘Q’ for signaling the end of the communication. - At first we performed the log filtering in the application. A problem arose, which was that some devices could not handle such a large amount of processing. For that reason, we decided to move the log filtering to the server. This also allowed us to store all the logs in the server before doing any filtering. - One of the biggest challenges we faced was how to extract useful information from the logs. The main problem with logs is that they share a shared format, but the message itself does not have a common form, so each message is different. We first thought that it would be interesting to know which applications created a log entry with a level above Warning, because that could reveal applications with bugs in the code or applications that access specific protected items. In the end, we decided that the best way was to check for specific patterns inside those messages, and that’s how we came up with the idea of the target elements. 84 - We also faced some issues returning the result from LogAnalyzerConnect to LogAnalyzerFragmentApplications, as it runs on a different thread. We managed this by creating a delegate function in the UI thread and passing it to LogAnalyzerConnect. This function acts as an asynchronous response that is called within AsyncTask’s onPostExecute() method, which is called when all processing has already been done. 5.3 Permission Analysis Implementation Another aspect that must be considered in the development of a malware analysis tool is the study of the relevance of the permissions requested by each application. Permissions play a very important role when analyzing malware as they help support user privacy by protecting access to restricted data and restricted actions from malicious purposes [55]. This section goes through the motivations behind performing this assessment and explains all the details and steps followed during the implementation of the permission analysis. The permissions determine what is allowed to be done by an app. In order to perform a more comprehensive and elaborate malware study, the team has concluded that a permission analysis would make a solid supporting feature to our Android Malware Analyzer App. To carry out an analysis of this style, the main thing is to remain neutral and always rely on objective facts to achieve the most accurate result possible in the permission assessment. The fact of fulfilling these conditions determines the rigor and precision of the analysis. The basis and what we are trying to evaluate in this analysis are the permissions requested by the application, which can be found inside the AndroidManifest.xml file [24] under the <uses-permission /> tag [88]. The following sections describe how these permissions have been handled and analyzed. 5.3.1 Dataset of permissions Before starting with the analysis it is essential to create a dataset that collects all the existing permissions and information about them. The aim of having this list is to know the installed apps permissions and to classify them according to the domain that they belong to and their level of danger. For this work, the team considers that the best implementation of the permission dataset is to have a database formed by two tables, one with the permissions information and another with the groups information (see Figure 55). 85 Figure 55: Permissions Dataset Relational Model The table Permissions (Table 6) contains all the relevant information regarding Android permissions. As it concerns the identification of this table, the Permissions ID field is the constant value of the permission which is stored in the <uses-permissions/> tag of the AndroidManidfest.xml. The aim of the score field is to grade the permissions according to how exposed the information is as well as the scope of restricted actions you can perform when the system grants you that permission. The values taken by the scores range from 0 to 6 and they are explained in the 5.3.2 section. Table 6: Fragment of the table Permissions The table Domains (Table 7) contains all the relevant information regarding the different groups which the permissions belong to. This table just stores the identification of the group and an alias and a description to facilitate the understanding of the domain to the user and show him at a high level what functionalities of the device are used by the app. 86 Table 7: Fragment of the table Domains In order to get a permissions dataset as complete as possible, the team relies on the permissions API reference page [79] from the Android for Developers official site. This page provides developers with a full list of permissions recognized by Android. In Figure 56 all the information used in the permissions dataset is highlighted. 87 [79] Figure 56: Android Developers, API Reference Page 1 - Permission ID, 2 - Permission Score, 3 - Permission Description, 4 - Permission Alias, 5 - API level (not used) As additional information, the team has also used the dataset of permissions form the source androidpermissions.com [80] which completes the list provided by Android Developers. Furthermore, this site relates the permissions with their domain and gives an explanation. In Figure 57 all the information used in the domains dataset is highlighted. 88 [80] Figure 57: androidpermissions.com 1 - Domain ID, 2 - Domain Alias, 3 - Domain Description, 4 - Permission ID, 5 - Permission Description 5.3.2 Classification of permissions Even though Android already classifies its permissions according to the scope of restricted data and actions that an app can access and perform when the system grants that permission [9], the team decides to adopt a more detailed way of labelling the permissions. In Table 8 it can be seen this adaptation. The column Score is the value stored in the database that allocates the permission scope and the column Permission Class is the permission classification according to Android. Table 8: Explanatory table of the different levels assigned to the permissions 89 permissions is 10 times the average number of permissions according to the category. This way, for an application of category Games (ideal = 5), depending on the number of permissions, the Quantitative Score is higher or lower: 5. Sort the list of apps The next and final step of the permission analyzer is to sort into a list the installed apps according to their Final Score. This score is computed as the lowest value between the Quantitative and Qualitative scores. 5.3.4 Results Once the analysis of permissions is done, the results are shown to the user. These results consist on: - The average of the Final Scores (Figure 61). Figure 61: Permission Analysis Results view 96 - List of the installed applications that request forbidden permissions (Figure 62 and Figure 63). Figure 62: Apps that request special permissions [91] Figure 63: Aptoide requests “Install Packages, which is a special permission - List of installed applications that request dangerous permissions (Figure 64 and 65). Figure 64: Apps that request dangerous permissions 97 Figure 65: Adobe Scan requests several dangerous permissions - List of installed applications that request deprecated permissions (Figure 66 and 67). Figure 66: Apps that request deprecated permissions Figure 67: Instagram requests two deprecated permissions: “Uninstall_Shortcut” (see Figure 68) and “Use_Fingerprint” (see Figure 69) 98 [98] Figure 68: Uninstall Shortcut is a deprecated permission [92] Figure 69: Use Fingerprint is a deprecated permission - List of installed applications that request unknown permissions (Figure 70 and 71). Figure 70: Apps that request unknown permissions Figure 71: Instagram requests an unknown permission: “Billing” 99 - List of installed applications sorted by Final Score (Figure 72). Figure 72: Apps sorted by Final Score 100 6. Android Malware Analyzer App This chapter shows the structure of the application developed as a result of the work carried out in this project. This chapter provides a detailed explanation of each layout of the application and the interaction of the user with it. Through this chapter we will refer to the application developed as AMA, (Android Malware Analyzer). We have also designed a logo as shown in Figure 73. Figure 73: Application Logo 6.1 Fragment Menu The AMA application has been structured through the use of Fragments. Android Fragments represent a reusable portion of the app's UI that defines and manages its own layout and are attached to an Activity. As shown in Figure 74 and 75, AMA consists of 8 main fragments, accessible by a fragment menu, which can be revealed by clicking on the top left three bar icon. The main fragments are: -Home: Entry point of the application. Shows some starting information of the analysis performed. -Apps Information: Lists all the applications installed on the device and when one of them is selected displays information about it. -Signature Analyzer: Fragment in charge of carrying out the signature analysis. -Permission Analyzer: Fragment in charge of carrying out the permission analysis. -Log Analyzer: Fragment in charge of carrying out the log analysis. -Previous Results: Displays the results of previously performed analysis. -Server Settings: Handles the IPs used to establish the connection to the server. -About Us: Shows some information about the project and the three analyses. 101 Figure 74: Android project organisation Figure 75: Fragment menu 102 6.2 Home Fragment The Home Fragment, shown in Figure 76, is the visual entry point of the application. It accesses the PrevResults database to inform the user about the last analysis performed, displaying the analysis type as well as the date and time. If no analysis is performed, the text “None” is displayed. Below this information, the application shows to the user all the apps whose hash has not been analyzed yet. We have implemented this functionality because we have thought it would be highly recommended to encourage the user to analyze all the applications installed on his device. If all the apps had been analyzed, the text “None” is displayed. Figure 76: Home Fragment 103 6.3 Apps Information Fragment The Apps Information Fragment (Figure 77) shows the user a list of the installed applications. By clicking on a specific app, the user gets the different information that the package provides, the application scores, the requested permissions and a list of the domains that the app has access to. Figure 77: App Information Fragment The application scores are the quantity and quality results of the assessment performed by the permission analyzer. The overall score of the application is the lowest value of both scores, Adobe Scan would have an overall score of 3.82 whereas Among Us would have an 8.75 (Figure 78). As seen previously this is based on the permissions requested by the applications. Other relevant information that the package provides is the package name, the path and the categorization of the app. Next, the fragment shows all the domains of the app, that is, all the permission groups that the application requests. They are displayed in the form of an expandable list that, when clicked, the permissions that belong to that group appear. It also displays to the user a brief description of that domain (see Figure 79). 104 Figure 78: App Details Fragment (Scores and Package Information) Figure 79: Apps Details Fragment (Domains that Adobe Scan has access to) 105 Finally, at the bottom, there is a list that contains the installed applications sorted by final score (Figure 90). If the user decides to click in one of the apps, the application is redirected to the Permission List Fragment (Figure 80). Figure 88: Permissions Analyzer Fragment Figure 89: Permissions Analyzer Fragment Figure 90: Top and Bottom of the applications list sorted by scores 112 6.6 Log Analyzer Fragment Firstly, the Log Analyzer Fragment lets the user decide if the apps she/he wants to analyze are user applications (applications installed by him, such as Whatsapp, Instagram, AndroidMalwareAnalyzer, etc) or functionalities (pre-installed apps, such as Camera, Gmail, etc) as shown in Figure 91. Figure 91: Log Analyzer Fragment Depending on the decision the user has made, the list of applications that are displayed changes accordingly, as shown in Figure 92. If the connection to the server has been correctly set and selected in the Server Settings Fragment (see section 6.8), an alert message as the one in Figure 92 is displayed, telling the user to select the apps to monitor. 113 Figure 92: Log Analyzer Fragment - Applications However, if no address has been correctly set up in the Server Settings Fragment, the alert shows an error message telling the user to add a connection in the Server Settings section (Figure 93). Figure 93: Log Analyzer Fragment - No connections 114 As Figure 94 displays, after the previous step the user can select the applications that wants to analyze. If none is selected, all of them are analyzed, which will consume more time compared to selecting some applications. The user can also input some keywords separated by commas, which are used later on during the analysis (see section 5.2.2 for more information). When the user wants to start the analysis, she/he needs to press the Start Analysis button. During this time, the connection is established and the application begins to send logs to the server (Figure 94). Figure 94: Log Analyzer Fragment - Analysis Started The user is able to see if something goes wrong through the alert messages that appear on the screen (Figure 95). Since the device is now being analyzed, the user can interact with the application she/he wants to analyze, so that AMA gets its logs. As it is a beta version, we recommend accessing the AndroidMalwareAnalyzer once in a while, because it can sometimes get stuck if it is kept in the background for too long. 115 Figure 95: Log Analyzer Fragment - Connection Error After some time, the user can press the Stop Analysis button to stop the connection and retrieve the results. As shown in Figure 96, the result first introduces the analyzed elements. Then it shows the apps analyzed, which can be expanded to show the target elements they accessed. Some of them may not be expandable if no target element was recognized. The number displayed indicates how many times that element appeared in a log entry. After that, the keywords analyzed are shown, which can also be expanded to display which application accessed them. Lastly, the user can see if the applications have the permissions needed to access those target elements. 116 Figure 96: Log Analyzer Fragment - Show Result 117 6.7 Previous Results Fragment This fragment displays all the log and signature analyses that have been made previously. This is a simple way to keep track of what is happening on your mobile phone with the applications installed. In addition, analysis over time of this data could indicate unwanted operation of installed apps; for example, an app accessing an item too much even though the user is not aware of having that app in use (at least in the foreground). However, we have decided that the permission analysis should not be saved, since it would take up a lot of space due to the amount of information and because it is more interesting to calculate it each time. In section 5.2.1 the contents of the PrevResults table are shown (Table 4). As shown in Figure 97, each element of the list of analyses shows the type of analysis and the day and time. They are ordered by both the date and time, in descending order. Figure 97: Previous Results Fragment If one of the results is selected, the specific data is displayed. Figure 98 displays the results of the signature analysis and log analysis that have been selected. 118 Figure 98: Previous Result Selected 6.8 Server Settings Fragment In this fragment communication with the server is configured. The user can set the IP and Port numbers that are going to be used in the Log Analyzer for connecting to the server. They are stored using SharedPreferences. The user can select each of the saved connections, which is the one used when connecting to the server. As shown in Figure 99, there are two buttons: the Add button and the Delete button. If the Add button is pressed, a popup appears, asking the user to input the IP and Port number in the format IP:Port (Figure 100). If the user presses the Cancel button, the operation is discarded. 119 Figure 99: Server Settings Fragment Figure 100: Server Settings Fragment - Add IP:Port If the user presses the Add button, the user’s input is parsed. In case something is wrong with the IP or Port numbers (i.e. if it is not a number, one of the bytes of the IP is inferior to 0 or superior to 255, etc) the connection is not added and an alert message tells the user what went wrong, as shown in Figure 101. If the Delete button is pressed, the selected address is deleted. Once it has been removed, an alert message notifies the user of the deletion (Figure 102). 120 Figure 101: Server Settings Fragment - Wrong Input Alert Figure 102: Server Settings Fragment - Delete address 121 Figure 108: Permission manager - TikTok -Figure 109 displays the information obtained from the Apps Information Fragment of AndroidMalwareAnalyzer. It can be seen that the category in which this application falls into is Social & Communication, which is correct for the app's functionalities. The package name of the application is com.zhiliaoapp.musically, which seems to be quite suspicious since the current name is TikTok. However, it might be legitimate because Musically is the former name of the app. 128 Figure 109: Apps Information - TikTok -Diccionario de la Lengua Española (DLE) - DLE is the official application that the Real Academia Española (RAE) and the Asociación de Academias de la Lengua Española (ASALE) made available to consult the Spanish Dictionary. - Its main functionality is the search of the meaning of specific words in the dictionary, being able to apply different filters. It also has some links that allow the user to obtain information about the app and the entities behind its development. -Figure 110 shows the permissions required by the app and whether they have been granted or denied. In this case, it has not required any. 129 Figure 110: Permission manager - DLE -Figure 111 displays the information obtained from the Apps Information Fragment of AndroidMalwareAnalyzer. It can be seen that the category in which this application falls into is Productivity, which is correct for the app's functionalities. Also, the package name of the application is es.rae.dle, which seems to be legitimate. 130 Figure 111: Apps Information - DLE - Nasip Kismet degilmis - It is a supposed TV remote control that actually does not work as a remote control, but as a Clicker. -Figure 112 displays the information obtained from the Apps Information Fragment of AndroidMalwareAnalyzer. It can be seen that the category in which this application falls into is Undefined, which is already suspicious. Also, the package name of the application is com.ndsonkentucki.kuma, which also seems suspicious. 131 Figure 112: Apps Information - Nasip Kismet degilmis - SmartcardService - It is a malware that obtains the phone number, MAC address, device usage and records SMS and voice conversations. 7.2 Permission Analysis In this section, the permission analyzer will examine the apps mentioned in the previous section. Based on the result, the permissions requested will be discussed to see if their use is legitimate or if it contradicts its alleged functionality. -Instagram -Figure 113 shows the overall score obtained by analyzing the app with the permission analyzer. As seen, it scored a 4.75 out of 10, which is a low score. This is due, as we will later see, to its huge number of permissions and the high ratio of dangerous permissions. Figure 113: Overall Score - Instagram - Accessing the result of the analysis, shown in Figure 114, it can be seen that the application requests several types of permissions. Specifically, this app requests 15 dangerous permissions, 2 deprecated, 14 normal and 11 132 unknown, which gives it a rating of 5.8 out of 10 in quality. In terms of quantity, it requests 42 permissions in total, hence it has a 4.75. Of all of them, the most notable permissions are those related to the Camera, Accounts, Contacts, Location, Microphone, Phone, Storage and Billing, which make sense within all the functionalities that the app offers. All in all, it can be appreciated that this application requests a great number of permissions, which could be dangerous in case the application were malicious. In this case, this application is the legitimate one, and those permissions are needed to perform several of the functionalities of the app. Figure 114: Permission Analyzer - Instagram -WhatsApp -Figure 115 shows the overall score obtained by analyzing the app with the permission analyzer. As seen, it scored a 2.5 out of 10, which is a very low score. This is due, as we will later see, to its large number of permissions and the poor balance of permission types according to the security levels. Figure 115: Overall Score - Whatsapp 133 - Accessing the result of the analysis, shown in Figure 116, it can be seen that the application requests several types of permissions. Specifically, this app requests 22 dangerous permissions, 3 deprecated, 1 signature, 20 normal and 14 unknown, which gives it a rating of 5.73 out of 10 in quality. In terms of quantity, it requests 60 permissions in total, hence it has a 2.5. Of all of them, the most notable permissions are those related to the Camera, Accounts, Contacts, Location, Microphone, Phone, SMS, Read Call Log, NFC, Biometric, Storage and Billing, which make sense within all the functionalities that the app offers. All in all, it can be appreciated that this application requests a great number of permissions, which could be dangerous in case the application were malicious. In this case, this application is the legitimate one, and those permissions are needed to perform several of the functionalities of the app. Figure 116: Permission Analyzer - WhatsApp -TikTok -Figure 117 shows the overall score obtained by analyzing the app with the permission analyzer. As seen, it scored a 2.25 out of 10, which is a very low score. This is due, as we will later see, to its huge number of permissions and the high ratio of unknown and dangerous permissions. 134 Figure 117: Overall Score - TikTok - Accessing the result of the analysis, shown in Figure 118, it can be seen that the application requests several types of permissions. Specifically, this app requests 9 dangerous permissions, 2 deprecated, 12 normal and 39 unknown, which gives it a rating of 5.08 out of 10 in quality. In terms of quantity, it requests 62 permissions in total, hence it has a 2.25. Of all of them, the most notable permissions are those related to the Camera, Audio, Contacts, Microphone, Phone, Storage and Billing, which make sense within all the functionalities that the app offers. What is quite rare is the great amount of unknown permissions that apparently request to read/write the system settings. Apart from that, it can be appreciated that this application requests a great number of permissions, which could be dangerous in case the application were malicious. In this case, this application is the legitimate one, and those permissions are needed to perform several of the functionalities of the app. Figure 118: Permission Analyzer - TikTok -Diccionario de la Lengua Española (DLE) -Figure 119 shows the overall score obtained by analyzing the app with the permission analyzer. As seen, it scored a 9.6 out of 10, which is a very high 135 score. This is due, as we will later see, to its impeccable quality and quantity of permissions. Figure 119: Overall Score - DLE - Accessing the result of the analysis, shown in Figure 120, it can be seen that the application only requests normal permissions, which gives it a rating of 10 out of 10 in quality. In terms of quantity, it only requests two permissions, hence it has a 9.6. Within the functionalities of the application, it makes sense that it uses the Internet and Foreground Service permissions. All in all, it can be appreciated that this application is highly reliable, as it is far from what could be considered a dangerous or suspicious application. Figure 120: Permission Analyzer - DLE -Nasip Kismet degilmis -Figure 121 shows the overall score obtained by analyzing the app with the permission analyzer. As seen, it scored a 8.8 out of 10, which is a very high score. This is due, as we will later see, to the good quality and quantity of permissions. 136 Figure 121: Overall Score - Nasip Kismet degilmis - Accessing the result of the analysis, shown in Figure 122, it can be seen that the application only requests normal and signature permissions, which gives it a rating of 10 out of 10 in quality. In terms of quantity, it only requests six permissions, hence it has a 8.8. Within the functionalities of the application, it should request different permissions, which is suspicious. All in all, it can be appreciated that this application does not request dangerous permissions, but it is suspicious that it does not request the permissions it should need to keep up with its functionality. Figure 122: Permission Analyzer - DLE -SmartcardService -Figure 123 shows the overall score obtained by analyzing the app with the permission analyzer. As seen, it scored a 4.13 out of 10, which is a very low score. This is due, as we will later see, to the huge number of dangerous permissions it requests. Figure 123: Overall Score - SmartcardService 137 Figure 136: Signature Analyzer - SmartcardService 7.4 Heuristic Log Analysis In this section, the log analyzer will analyze the apps mentioned in section 7.1. The result will then be discussed. -Instagram - While the log analyzer was collecting the logs, we accessed Instagram and interacted a bit with the main page, looking at recent posts and some stories. We then tried to take a photo, for which it asked for permission to access the camera which we decided not to grant. After that, we accessed Directs and tried to send an audio to a contact, for which it asked for permission to access the microphone which we decided not to grant. Finally, we saw some of the posts and Reels that appeared in the Explore tab and accessed a specific account after searching for it in the search bar. - After we were done, we stopped the analysis and the results were obtained (Figure 137). It can be seen that during the execution of the analysis, four applications were running: Instagram, AndroidMalwareAnalyzer, WhatsApp and Stickers. WhatsApp probably had some service running in the background, so it is not surprising to see that it generated some logs. As for the Stickers app, perhaps it would be interesting for the user to analyze it separately since it should not be running, but that is not the scope of this 144 analysis. Four target elements have been detected during the execution of Instagram: Camera, Messaging, Microphone and Internet. Since there are a low number of logs related to each target element, the user should not worry, as they are probably false positives or, having been asked to grant permissions to the camera and microphone, it may have generated some logs related to them. For this reason, finding that the app has no access to the camera and microphone should not be alarming. Figure 137: Log Analyzer - DLE - Reading the logs that were stored on the server, we saw that Camera was triggered when the camera activity was loaded: “06-15 17:07:55.469 1375 8410 v windowmanager: adding window{a67672 u0 khcd.4zp.reel_composer_camera} to window{620175e u0 com.instagram.android/com.instagram.mainactivity.mainact ivity}”. Regarding the Microphone, the logs were generated when we were prompted to grant the permission: “06-15 17:08:03.273 31338 31338 v grantpermissionsactivity: logged buttons presented and clicked permissiongroupname=android.permission-group.microphone uid=10278 package=com.instagram.android presentedbuttons=25 clickedbutton=8” 145 - From this analysis we can conclude that this application performs the promised functionalities and does not attempt to do anything outside of its supposed behaviour. -WhatsApp - While the log analyzer was collecting the logs, we accessed a WhatsApp conversation to take a photo and send it. We also activated the microphone and sent a document stored in the device. Finally, we made a video call with another contact. - After we were done, we stopped the analysis and the results were obtained (Figure 138). It can be seen that during the execution of the analysis, three applications were running: WhatsApp, AndroidMalwareAnalyzer and Instagram. Instagram probably had some service running in the background, so it is not surprising to see that it generated some logs. Five target elements have been detected during the execution of Instagram: Camera, Microphone, Messaging, SDcard and Internet. As the number of logs regarding the microphone and camera, their numbers are high enough to be sure that those have been accessed. Figure 138: Log Analyzer - WhatsApp - Reading the logs that were stored on the server, we saw that the Camera logs were generated when the camera was opened and when the camera activity 146 was called. As an example, we show this log: “06-15 19:33:12.549 14324 15103 i camera : open camera: 1, package name: com.whatsapp”. Regarding the SDcard, the logs were generated when we searched the locally stored documents to send one of them: “06-15 19:33:12.444 14324 14324 w com.whatsapp: type=1400 audit(0.0:10590): avc: granted { getattr } for pid=14324 name="/" dev="sdcardfs" ino=11048 scontext=u:r:untrusted_app:s0:c147,c256,c512,c768 tcontext=u:object_r:sdcardfs:s0 tclass=filesystem”. - From this analysis we can conclude that this application performs the promised functionalities and does not attempt to do anything outside of its supposed behaviour. -TikTok - While the log analyzer was collecting the logs, we accessed TikTok and interacted a bit with the main page, looking at recent posts. We then recorded a video, for which it asked for permission to access the camera which we decided to grant. - After we were done, we stopped the analysis and the results were obtained (Figure 139). It can be seen that during the execution of the analysis, just five applications were running: TikTok, AndroidMalwareAnalyzer, Zity, Wible and Instagram. Regarding Instagram, it probably had some service running in the background, so it is not surprising to see that it generated some logs. As for Zity and Wible apps, perhaps it would be interesting for the user to analyze it separately since it should not be running, but that is not the scope of this analysis. Four target elements have been detected during the execution of TikTok: Camera, Messaging, Location and SDCard. As it concerns the Camera, it makes sense to find logs because during the analysis it has been used. Regarding the rest of the logs, since there are a low number of logs related to each target element, the user should not worry as they are probably false positives. For this reason, finding that the app has no access to the location should not be alarming. 147 Figure 139: Log Analyzer - TikTok - Reading the logs that were stored on the server, we saw that Camera was triggered when the camera was opened: “06-15 19:14:15.738 31375 32062 i cameramanager: open camera: 1, package name: com.zhiliaoapp.musically”. - As regards the SDcard, the logs were: “06-15 19:14:57.411 31375 31375 w frescoioboundex: type=1400 audit(0.0:7330975): avc: granted { read open } for pid=31375 path="/storage/emulated/0/android/data/com.zhiliaoapp.mu sically/cache/picture/fresco_cache/v2.ols100.1/35/eq_mss q_qyzoxfxqyavkx5b5wjw.cnt" dev="sdcardfs" ino=254257 scontext=u:r:untrusted_app:s0:c99,c257,c512,c768 tcontext=u:object_r:sdcardfs:s0 tclass=file” - As to the Location, the logs found were “06-15 19:14:12.993 2089 2467 i pg_ash : unp_gps:com.zhiliaoapp.musically uid:10355 result:true”. - Finally, regarding to the SDCard, the logs found were “06-15 19:14:12.991 2009 5099 d assistantservice-1030200: handlemessage app switch 148 frompackage:com.huawei.android.launcher, topackage:com.zhiliaoapp.musically”. - From this analysis we can conclude that this application performs the promised functionalities and does not attempt to do anything outside of its supposed behaviour. -Diccionario de la Lengua Española (DLE) - While the log analyzer was collecting the logs, we accessed the DLE application and did a couple of searches for the meanings of different words. We also accessed two of the links in the application, which redirect to the default browser to display information from the RAE and the app. - After we were done, we stopped the analysis and the results were obtained (Figure 140). It can be seen that during the execution of the analysis, three applications were running: DLE, AndroidMalwareAnalyzer and Firefox Focus. Firefox Focus appears because the DLE application links redirected the user to the default browser, which in the case of the device on which we tested the analysis, was Firefox Focus. Two target elements have been detected during the execution of the DLE application: Messaging and Storage. Since there are a low number of logs related to each target element, the user should not worry, as they are probably false positives. For this reason, finding that the app has no access to the storage should not be alarming. 149 Figure 140: Log Analyzer - DLE - Reading the logs that were stored on the server, we realized that Messaging was triggered because when switching from the DLE application to Firefox Focus and back, several logs like “06-13 17:40:41.598 1975 4130 d assistantservice-1030200: handlemessage app switch frompackage:com.huawei.android.launcher, topackage:es.rae.dle” were generated, which is detected as Messaging by a false positive. Storage has been detected because two logs were created in this form “06-13 17:40:31.424 13742 13760 i hwapicachemangerex: apicache path=/storage/emulated/0 state=mounted key=es.rae.dle#10122#”. - From this analysis we can conclude that this application performs the promised functionalities and does not attempt to do anything outside of its supposed behaviour. -Nasip Kismet degilmis and SmartcardService - Due to their malicious nature, it is necessary to perform an analysis of these two applications in a secure environment such as a virtual machine. When trying to perform the analysis, we realized that the applications crashed each time they were opened, probably because they were programmed to work in 150 an older Android version. For this reason, we were unable to perform the log analysis for these applications. 151 8. Individual Work This chapter summarises the contributions of each participant in this project, listing everything learned in the process of carrying it out. The work has been carried out jointly and fairly, dividing the work into each of the points that make it up. These are the cryptographic signatures analysis, the analysis of logs and the analysis of permissions. 8.1 Daniel Puente Arribas My work has had more weight in the analysis of permissions. The beginning of my work consisted of a global study of the metadata contained within the PackageInfo object. In this way, I learned about its uses and characteristics to understand how to obtain the data and say which fields were the most useful and relevant to develop our analysis. Next, I began the development of the Apps Information fragment and my first approach to the logical and visual development of the application consisted in showing the list of all the installed applications, as well as the relevant information of each one. Once this was done, I continued with the development of the fragment that shows the details and information for each application. During this stage, I started to work with the requested permissions and I saw the need to create a view dedicated to the permissions in order to classify them according to the scope of restricted data that the apps can access, and the scope of restricted actions that apps can perform when the system grants them permission. This way, before getting into the permissions analysis, I did a deep research about Android permissions and learned how to declare permissions, where to declare them, how to access them and how to classify them according to the domain which they belong to and according to the level of access they request to the device. Subsequently, I built a dataset composed of permissions and permission groups out of various sources with the objective of using it later in permissions analysis. The interesting thing about this dataset is that it turned out to be very complete and very useful when it comes to correctly classifying the permissions. After all the information had been collected, the dataset information was added as tables to the application's SQLite database. Once I obtained the necessary knowledge about Android permissions and having created a dataset that stores them, I began to develop the Permission Analyzer fragment. To do this, I came up with two algorithms that assign two different ratings in order to evaluate applications based on the permissions they request. Actually, these two ratings provide two very interesting views on applications; a quantitative view and another deeper and qualitative view. After I have finished designing the logical and functional part of the permissions analyzer, I implemented it in the application and I continued developing the GUI of the analyzer. 152 Regarding this document, I wrote the permission analyzer sections of chapters 4, 5, 6 and 9. We also divided chapter 3, section 1.4 and the abstract equally. I also wrote my corresponding section of chapter 8 and sections 1.1 and 1.3. Lastly, I was in charge of correctly citing the bibliography. To conclude, my main contributions in this project focus on the analysis of applications metadata, the research of Android permissions and in relation to the extraction of information from the apps and the analysis of permissions, elaborating the functional and logic design as well as their user interface design and its corresponding implementation. 8.2 José Ignacio Daguerre Garrido I created the local database with its two tables of malware signatures and existing Android permissions, which is essential to perform the static analysis. I was in charge of creating an EC2 instance of AWS to be able to monitor the logs sent from an Android device in order to eliminate the cost of doing it locally. I was also in charge of establishing a connection between this instance and a TCP server, which connected remotely with the application developed in Android Studio. On the other hand, I have participated in the implementation of the Cryptographic Signature Analysis fragments, being one of the three fundamental pillars that make up the whole application. Regarding this document, I wrote the signature analyzer sections of chapters 4, 5, 6 and 9. We also divided chapter 3, section 1.4 and the abstract equally. I also wrote my corresponding section of chapter 8 and section 1.2. Alongside Ramón, we wrote chapter 2. I have been able to understand not only the internal structure of Android applications but also how Linux is composed and the security services and structure that Android provides. Despite the lack of knowledge of the Android Studio tool due to its complex structure and use, thanks to this project I have been able to deal with this tool and clearly understand its usability, as well as discovering the benefits and implementations that can be applied to the development of Android applications. As Android Studio is programmed in Java, I have been able to strengthen my Java programming skills and become more fluent. I have also learned how to make connections between AWS instances and mobile devices through a TCP server using sockets. I have found Spark Streaming really exciting, which is responsible for processing data in real time, in our case the logs of an Android device. It is going to be helpful to implement applications that offer greater efficiency and performance. It is undoubtedly the case that we live in an age when people are unaware of the security and structure of their mobile device and the risks involved. What I have learnt is that being knowledgeable about how application permissions work gives you a certain basic understanding of how to deal with malware. 153