▼ Attribution of the malware to the developers writing the malware is an important factor in cybercrime investigative work. Clustering together not only malware of the same family, but also inter-family malware relations together provides more information about the authors and aid further malware analysis work. In this report, previous work which concluded attribution on compiled binaries can be done by a programmer their style is questioned. Given insight on this matter, this report explores new clustering techniques for both static and dynamically derived features from malware binaries. Both methods are complementary as they provide very different types of data. In the static analysis, the data for the similarity comparison is derived from disassembled binaries, while in dynamic analysis the choice was made to record system calls executed by the malware during ecution. We use a finer granularity than when comparing the data of the complete binaries with each other, such that instead of differences, fine similarities among malware families can be found. Evaluation of clusters is a difficult subject, because of its unsupervised nature and data quality related causes. However, upon manual inspection of the generated clusters, the newly developed clustering methods confirm previously discovered similarities but also find new connections among malware families. Advisors/Committee Members: Verwer, S.E. (mentor), Lagendijk, R.L. (graduation committee), Finavaro Aniche, M. (graduation committee), Delft University of Technology (degree granting institution).

▼ 

The internet traffic is constantly rising nowadays due to the significant increase of the devices connected to the Internet. As a consequence, many cyber risks have arisen. Cybercriminals are trying to exploit the vulnerabilities of these devices to cause damage and gain profit. Monitoring the network traffic and detecting such threats has become essential in order to keep safe systems that are connected to the Internet. The powerful properties of state machines and the sequential nature of the network traffic data, makes them an interesting and promising solution for the implementation of an intrusion detection system. The goal of this thesis is to implement a new state-merging heuristic which will speedup the state machine building procedure without a significant loss on the quality of the model, and use it to detect malicious host on network traffic data. The new state-merging heuristic is utilizing the Locality-sensitive Hashing concept to store the future traces of each state and simplify the consistency check for the merge of two states. The network traffic data used are in the NetFlow format, and they are encoded and converted into traces in order to build the state machine model and measure its performance. The state machine built is modeling a malicious behavior and used to classify other hosts. We show that the models built can effectively detect the malicious hosts, with its performance being comparable to the one of a state-of-the-art model. At the same time, the time needed to build the model is much less when compared to the time needed by other state-merging heuristics.

Computer Science | Cyber Security

Advisors/Committee Members: Verwer, S.E. (mentor), Lagendijk, R.L. (graduation committee), Finavaro Aniche, M. (graduation committee), Delft University of Technology (degree granting institution).

▼ 

Since the introduction of the Web, online platforms have become a place to share opinions across various domains (e.g., social media platforms, discussion fora or webshops). Consequently, many researchers have seen a need to classify, summarise or categorise these large sets of unstructured user-generated content. A field related to this task is also known as opinion mining in which various applications have focused on sentiment analysis techniques to classify opinionated documents based on sentiment. More recent, researchers have focused on stance classification to classify opinionated documents based on stance in controversial debates. However, in the case of such controversial debates it would be equally interesting to know the underlying reasons behind a stance in order to truly understand a discussion. We can call these underlying reasons as perspectives. Few have focused on distilling such perspectives from text and in this research we aim to explore the use of an unsupervised model - called joint topic models - to perform the task of perspective discovery. We define perspective discovery on a controversial debate as the process of automatically finding and extracting a structured overview of perspectives from unstructured text. The aim is to quantify how well existing joint topic models can extract human understandable perspectives between and within stances for more fine-grained opinion mining on textual debates. To perform this evaluation we propose an evaluation setup with an extensive user study. This setup focuses on the topic model’s clustering ability of perspectives as well as the human understandability of the topic model’s output. Based on the results we may derive that topic models can discover some of the perspectives from text. Moreover, the results suggest that users are not influenced by their pre-existing stance when interpreting the output of topic models.

Computer Science

Advisors/Committee Members: Tintarev, N. (mentor), M.%22%29&pagesize-30">Houben, G.J.P.M. (graduation committee), Finavaro Aniche, M. (graduation committee), Draws, T.A. (mentor), Delft University of Technology (degree granting institution).

▼ 

Code quality of software products often degrades while they grow. Counteracting the degradation of code quality or improving it requires immense effort. Tools that reduce this effort are a hot topic in software engineering research. Software Modularization in particular aims to aid in the process of improving the quality of code structure, by finding flaws in code structure and suggesting improvements. Much research has been done in this field, however, most of it is applied on small to medium scale codebases. In addition, the quality of solutions implied by this research is often not properly validated. This thesis aims to apply an existing approach to an enterprise-level codebase, namely that of Adyen, and validating the results with developers experienced with the code. We achieve this by taking a graph-based approach, applying the NSGA-II algorithm, and introducing a new metric called the Estimated Build Cost of module Cache Breaks. We evaluate the approach in two ways. First, we performed a controlled experiment exploring the feasibility of the approach on larger scale codebases. For this experiment, we apply the approach to the Adyen codebase. The results show that the approach is scalable and shows a significant improvement of code quality in terms of the metrics used. We then performed a user study where we explore the feasibility of these results in practice. For this study, solutions generated during the experiment are split up and filtered to form groups with less than 10 changes, which are reviewed by developers that have changed that code recently or are a senior with experience in that specific area of the code. The results show that the algorithm is successful in identifying flaws in the codebase. However, the improvements it suggests are less precise and require future work.

Computer Science

Advisors/Committee Members: Finavaro Aniche, M. (mentor), van Deursen, A. (graduation committee), Yorke-Smith, N. (graduation committee), Delft University of Technology (degree granting institution).

▼ Privacy-preserving data aggregation protocols have been researched widely, but usually cannot guarantee correctness of the aggregate if users are malicious. These protocols can be extended with zero-knowledge proofs and commitments to work in the malicious model, but this incurs a significant computational cost on the end users, making adoption of such protocols less likely. We propose a privacy-preserving data aggregation protocol for calculating the sum of user inputs. Our protocol gives the aggregator confidence that all inputs are within a desired range. Instead of zero-knowledge proofs, our protocol relies on an asynchronous probabilistic hypergraph-based detection algorithm with which the aggregator can quickly pinpoint malicious users. Our protocol is robust to user dropouts and is non-interactive apart from the registration phase. We describe several optional extensions to our protocol for temporal aggregation, dynamic user joins and leaves, and differential privacy. We analyse our protocol in terms of security, privacy, and detection rate. Finally, we compare the runtime complexity of our protocol with a selection of related protocols. Advisors/Committee Members: Erkin, Z. (mentor), Picek, S. (graduation committee), Finavaro Aniche, M. (graduation committee), Delft University of Technology (degree granting institution).

▼ Active state machine learning algorithms are a class of algorithms that allow us to infer state machines representing certain systems. These algorithms interact with a system and build a hypothesis of what the state machine describing that system looks like according to the behavior they observed. Once the algorithm arrives at a hypothesis, it sends it to a so-called equivalence oracle to be checked. The equivalence oracle returns a counterexample trace describing different behavior between the hypothesis and the real system if one exists. Traditionally, this task is done using a technique called the W-method, which provides certain guarantees about finding a counterexample if one exists, but this is rather slow due to the thorough search it needs to do. In real systems, there are sometimes patterns to the counterexamples to be found during the learning process, which can be exploited to speed up the process of finding new counterexamples. Instead of performing an exhaustive search each time, we first use the already observed patterns to generate new traces to try before falling back to traditional techniques in case no counterexample was found in the first step. This allows us to find counterexamples more quickly than usual, and because we can always fall back on the techniques that do provide guarantees we do not have to sacrifice correctness. In the same amount of time, our method provides an up to 26x increase in the number of states found compared to the W-method, while remaining fully black box and without degrading worst-case performance. We also show that, when not working in a fully black-box scenario, fuzzing can be a successful complementary technique to apply next to active learning. However, we also show that relying on fuzzing alone does not always fully capture the behavior of a system, and when correctness is critical such as in the case of model checking, it needs to be supplemented with additional techniques to result in fully correct models. Finally, we use our home-grown state machine learning library to participate in the 2020 RERS challenge, and apply the techniques we studied so far to attain a perfect score on the sequential LTL track. Advisors/Committee Members: Verwer, S.E. (mentor), Lagendijk, R.L. (graduation committee), Finavaro Aniche, M. (graduation committee), Delft University of Technology (degree granting institution).

▼ 

Software maintenance is an essential and time-consuming task during the software development cycle. Readability of test code is a crucial element for performing programming tasks, such as testing, bug fixing and maintaining code. Hence poorly written tests are difficult to maintain and lose their value to developers. In order to overcome this problem, we need to understand how programmers read the test code. Therefore we conducted an empirical study to analyze the various reading patterns in novices and professionals using a sophisticated eye tracking device. Our results show that (i) all programmers first comprehended the production code and then switched between test and production codes, (ii) novices had higher fixations reading test code and assert statements, (iii) professionals revisited the test code more than novices, (iv) professionals had significantly lesser test code coverage than novices, and (v) there is a significant difference in reading test code between novice and professionals.

Electrical Engineer | Embedded Systems

Advisors/Committee Members: Finavaro Aniche, M. (mentor), Larios Vargas, E. (mentor), Zaidman, A.E. (graduation committee), Scharenborg, O.E. (graduation committee), Delft University of Technology (degree granting institution).

▼ 

Refactorings tackle the challenge of architectural degradation of object-oriented software projects by improving its internal structure without changing the behavior. Refactorings improve software quality and maintainability if applied correctly. However, identifying refactoring opportunities is a challenging problem for developers and researchers alike. In a recent work, machine learning algorithms have shown great potential to solve this problem. This thesis used RefactoringMiner to detect refactorings in open-source Java projects and computed code metrics by static analysis. We defined the refactoring opportunity detection problem as a binary classification problem and deployed machine learning algorithms to solve it. The models classify between a specific refactoring type and a stable class using the metrics as features. Multiple machine learning experiments were designed based on the results of an empirical study of the refactorings. For this work, we created the largest data set of refactorings in Java source code to date, including 92800 open-source projects from GitHub with a total of 33.67 million refactoring samples. The data analysis revealed that Class- and Package-Level refactorings occur most frequently in early development stages of a class, Method- and Variable-Level refactorings are applied uniformly during the development of a class. The machine learning models achieve high performance ranging from 80% to 89% total average accuracy for different configurations of the refactoring opportunity prediction problem on unseen projects. Selecting a high Stable Commit Threshold (K) improves the recall of the models significantly, but also strongly reduces the generalizability of the models. The Random Forest (RF) classifier shows great potential for the refactoring opportunity detection, it can adapt to various configurations of the problem, identifies a large variety of relevant metrics in the data and is able to distinguish different refactoring types. This work shows that for solving the refactoring opportunity detection problem a large variety of metrics is required, as a small set of metrics cannot represent the complexity of the problem.

http://doi.org/10.5281/zenodo.4267824 Appendix: Data Analysis and Machine Learning Experiments ShowEdit http://doi.org/10.5281/zenodo.4267711 Appendix: Refactoring Data Set ShowEdit https://github.com/refactoring-ai/Data-Collection Repository link Refactoring Mining Tool ShowEdit https://github.com/refactoring-ai/Machine-Learning Repository link Machine Learning Pipeline

Computer Science

Advisors/Committee Members: Finavaro Aniche, M. (mentor), van Deursen, A. (graduation committee), Erkin, Z. (graduation committee), Delft University of Technology (degree granting institution).

▼ 

Monitoring software behaviour is being done in various ways. Log messages are being output by almost any kind of running software system. Therefore, learning how software behaves from doing analysis over log data can lead to new insights about the system. However, the number of log messages in a computer system grow fast, and analysing the log data by hand is a time-consuming job. The objective of this study is to propose and implement a scalable architecture for doing real-time log analysis. Log data is structured so that analysis can take place, and the solution is horizontally scalable in every module so that the approach can scale with an ever-growing software solution. The focus of the study is on scalability, and ease-of-use of the implementation of the proposed approach. The proposed solution can scale horizontally and the test set up showed that reporting features for anomalies remained instantaneous when processing 1.2 million log lines per minute. The usability of the proposed approach is tested in a case study at Weave, where bugs were found by running the proposed solution in a controlled environment.

Computer Science

Advisors/Committee Members: Finavaro Aniche, M. (mentor), van Deursen, A. (graduation committee), Katsifodimos, A. (graduation committee), Delft University of Technology (degree granting institution).

▼ 

As organizations start to adopt machine learning in critical business scenarios, the development processes change and the reliability of the applications becomes more important. To investigate these changes and improve the reliability of those applications, we conducted two studies in this thesis. The first study aims to understand the evolution of the processes by which machine learning applications are developed and how state-of-the-art lifecycle models fit the current needs of the fintech industry. Therefore, we conducted a case study with seventeen machine learning practitioners at the fintech company ING. The results indicate that the existing lifecycle models CRISP-DM and TDSP largely reflect the current development processes of machine learning applications, but there are crucial steps missing, including a feasibility study, documentation, model evaluation, and model monitoring. Our second study aims to reduce bugs and improve the code quality of machine learning applications. We developed a static code analysis tool consisting of six checkers to find probable bugs and enforcing best practices, specifically in Python code used for processing large amounts of data and modeling in the machine learning lifecycle. The evaluation of the tool using 1000 collected notebooks from Kaggle shows that static code analysis can detect and thus help prevent probable bugs in data science code. Our work shows that the real challenges of applying machine learning go much beyond sophisticated learning algorithms  – more focus is needed on the entire lifecycle.

Computer Science

Advisors/Committee Members: van Deursen, A. (mentor), Finavaro Aniche, M. (graduation committee), Liem, C.C.S. (graduation committee), Miranda da Cruz, L. (graduation committee), Delft University of Technology (degree granting institution).

▼ 

Code duplication is a form of technical debt frequently observed in software systems. Its existence negatively affects the maintainability of a system in numerous ways. In order to tackle the issues that come with it, various automated clone detection techniques have been proposed throughout the years. However, the vast majority of them operate using the entire codebase as input, resulting in redundant calculations and undesirable delays when this process is repeated for every new revision of a project. On the other hand, newer incremental techniques address this by storing intermediate information that can be reused across revisions. However, all these approaches are language-specific, utilizing language parsers to generate more sophisticated source code representations, in an attempt to detect more complex types of clones. As a result, less popular languages, for which finding or building a parser is challenging, are unfortunately not supported. In this study we propose LIICD, a language-agnostic incremental clone detector, capable of detecting exact-match clones. We assess its performance and compare it with a state-of-the-art commercial-grade detector, found within the Software Improvement Group (SIG). Furthermore, we use a similarity estimation technique called Locality Sensitive Hashing (LSH) in an attempt to extend and improve the original approach. Our experiments result in some interesting findings. Firstly, the proposed incremental detector is very efficient and able to scale well for larger codebases. Additionally, it provides a significant improvement compared to a non-incremental commercial-grade detector. Lastly, our LSH-based extension proves to have difficulties matching our original approach's performance. However, future suggestions indicate how the potential of the technique can be further investigated.

Computer Science

Advisors/Committee Members: van Deursen, A. (mentor), Finavaro Aniche, M. (graduation committee), Poulsen, C.B. (graduation committee), di Biase, M. (mentor), Delft University of Technology (degree granting institution).

▼ 

Web APIs can have constraints on parameters, such that not all parameters are either always required or always optional. Sometimes the presence or value of one parameter could cause another parameter to be required. Additionally, parameters could have restrictions on what kinds of values are valid. We refer to these as inter-parameter and single-parameter constraints respectively. Having a clear overview of the constraints can help API consumers to integrate without the need for additional support and with fewer integration faults. We developed two approaches for identifying parameter constraints in complex web APIs. One approach uses online documentation to infer inter-parameter constraints, the other depends on static code analysis to extract inter- and single-parameter constraints from the control flow of the API’s source code. In our case study at several APIs at Adyen, the documentation- and code-based approach can identify 21% and 53% percent of the constraints respectively. When the constraints identified by both approaches are combined, 66% of the inter-parameter constraints can be identified. Code analysis is able to identify 78% of the single-parameter constraints.

Computer Science and Engineering

Advisors/Committee Members: Finavaro Aniche, M. (mentor), van Deursen, A. (graduation committee), Poulsen, C.B. (graduation committee), Akimov, A. (graduation committee), Delft University of Technology (degree granting institution).

▼ 

Mistakes in binary conditions are a source of error in many software systems. They happen when developers use < or > instead of <= or >=. These boundary mistakes are hard to find for developers and pose a manual labor-intensive work. While researches have been proposing solutions to identify errors in boundary conditions, the problem remains a challenge. In this thesis, we propose deep learning models to learn mistakes in boundary conditions and train our model on approximately 1.6M examples with faults in different boundary conditions. We achieve an accuracy of 85.06%, a precision of 85.23% and a recall of 84.82% on a controlled dataset. Additionally, we perform tests on 41 real-world boundary condition bugs found from GitHub and try to find bugs from the Java project of Adyen. However, the false-positive rate of the model remains an issue. We hope that this work paves the way for future developments in using deep learning models for defect prediction.

Computer Science

Advisors/Committee Members: Finavaro Aniche, M. (mentor), Paridon, Onno van (graduation committee), Hauff, C. (graduation committee), van Deursen, A. (graduation committee), Delft University of Technology (degree granting institution).

▼ 

Static Analysis is of indispensable value for the robustness of software systems and the efficiency of developers. Moreover, many modern-day software systems are composed of interacting subsystems written in different programming languages. However, in most cases no static validation of these interactions is applied. In this thesis, we identify the Cross-Language Static Semantics Problem, which is defined as "How to provide a formal and executable specification of the static semantics of interactions between parts of a software system written in different languages?" We investigate current solutions to this problem, and propose criteria to which an all-encompassing solution to this problem must adhere. After that, we present a design pattern for the Statix meta-DSL for static semantics specification that allows to model loosely coupled, composable type system specifications. This pattern entails that the semantic concepts of a particular domain are encoded in an interface specification library, which is integrated in the type system of concrete languages. This allows controlled but automated composition of type systems. We show that, under some well-formedness criteria, the system provides correct results. A runtime, executing composed specifications, is implemented using PIE pipelines for partial incrementality, and integrated in the command-line interface and Eclipse IDE platforms, using the Spoofax 3 Framework. This allows using multi-language analysis in concrete projects. The design pattern, and the accompanying runtime are validated using two case studies. These case studies show that the approach is effective, even in a case where there is an impedance mismatch in the data models of the involved languages.

Computer Science

Advisors/Committee Members: Visser, E. (mentor), Poulsen, C.B. (graduation committee), Finavaro Aniche, M. (graduation committee), Mosses, P.D. (graduation committee), Cockx, J.G.H. (graduation committee), Delft University of Technology (degree granting institution).

▼ 

Learning to program is not a easy task, as has become evident from the abundance of research papers concerning the subject. One of the learning barriers of learning a new programming language is understanding their error message, as coding errors have to be resolved before the programmer can run the code or add new functionality to the program. If the error message does not give the (kind of) information the programmer needs or uses terminology the programmer does not understand, it becomes a lot harder to fix the error it is reporting. This thesis aimed to enhance the understanding about why the error messages of the Python programming language fail to be understood by novice programmers. An experiment was performed where novice programmers constructed how they thought the error messages should look like when encountering errors often made by novice programmers. This thesis also introduces the Error Message Component Framework, a framework which can be used to determine the different components of information an error message contains and the structure of these components. This framework was used to compare the original error message to the custom error messages created by the participants of the experiment, to investigate if the original message matched the novice programmer's information expectations. This thesis concludes that even though some errors are often made by novice programmers, their resulting error messages may not targeted towards this group based on the terms used in these messages, the component of information these messages contains and the lack of precision of the information.

Software Technology

Advisors/Committee Members: Hermans, Felienne (mentor), van Deursen, A. (graduation committee), M.%22%29&pagesize-30">Specht, M.M. (graduation committee), Finavaro Aniche, M. (graduation committee), Delft University of Technology (degree granting institution).