Web robot detection using supervised learning algorithms
| dc.contributor.advisor | He, Hongmei | |
| dc.contributor.advisor | Starr, Andrew | |
| dc.contributor.author | Chen, Hanlin | |
| dc.date.accessioned | 2023-09-28T10:45:48Z | |
| dc.date.available | 2023-09-28T10:45:48Z | |
| dc.date.issued | 2020-06 | |
| dc.description.abstract | Web robots or Web crawlers have become the main source of Web traffic. Although some bots perform well, such as search engines, other bots can perform DDoS attacks, posing a huge threat to websites. The project aims to develop an offline system that can effectively detect malicious web robots, which is not only conducive to network traffic cleaning, but also conducive to improving the network security of IoT systems and services. A comprehensive literature review for the years 2010-2019 was conducted to identify the research gap. The key contributions of the research are: 1) it provided a systematic methodology to address the web robot detection problem based on the log file from industrial company; 2) it provided an approach of feature engineering, thus overcoming the challenge of curse of dimensionality; 3) It made a big progress in the accuracy of off-line web robot detection through a holistic study on the three types of machine learning techniques based on real data from industry. Three algorithms based on Keras sequential model, random forest, and SVM, were developed with python to detect web robots from human visitors on the TensorFlow 2.0 platform. Experimental results suggested that random forest obtained the best performance in accuracy and training time...[cont.] | en_UK |
| dc.description.coursename | Manufacturing | en_UK |
| dc.identifier.uri | https://dspace.lib.cranfield.ac.uk/handle/1826/20293 | |
| dc.language.iso | en | en_UK |
| dc.publisher | Cranfield University | en_UK |
| dc.publisher.department | SATM | en_UK |
| dc.rights | © Cranfield University, 2020. All rights reserved. No part of this publication may be reproduced without the written permission of the copyright holder. | en_UK |
| dc.subject | Web robot | en_UK |
| dc.subject | Web crawler | en_UK |
| dc.subject | Random forest | en_UK |
| dc.subject | Sequential model | en_UK |
| dc.subject | SVM | en_UK |
| dc.subject | Feature importance | en_UK |
| dc.subject | TensorFlow 2.0 | en_UK |
| dc.title | Web robot detection using supervised learning algorithms | en_UK |
| dc.type | Thesis or dissertation | en_UK |
| dc.type.qualificationlevel | Masters | en_UK |
| dc.type.qualificationname | MRes | en_UK |