Browsing by Author "Ince, Bilkan"

Now showing 1 - 4 of 4
  • Thumbnail Image
    ItemOpen Access
    Adversarial proximal policy optimisation for robust reinforcement learning
    (AIAA, 2024-01-04) Ince, Bilkan; Shin, Hyo-Sang; Tsourdos, Antonios
    Robust reinforcement learning (RL) aims to develop algorithms that can effectively handle uncertainties and disturbances in the environment. Model-free methods play a crucial role in addressing these challenges by directly learning optimal policies without relying on a pre-existing model of the environment. This abstract provides an overview of model-free methods in robust RL, highlighting their key features, advantages, and recent advancements. Firstly, we discuss the fundamental concepts of RL and its challenges in uncertain environments. We then delve into model-free methods, which operate by interacting with the environment and collecting data to learn an optimal policy. These methods typically utilize value-based or policy-based approaches to estimate the optimal action-value function or the policy directly, respectively. To enhance robustness, model-free methods often incorporate techniques such as exploration-exploitation strategies, experience replay, and reward shaping. Exploration-exploitation strategies facilitate the exploration of uncertain regions of the environment, enabling the discovery of more robust policies. Experience replay helps improve sample efficiency by reusing past experiences, allowing the agent to learn from a diverse set of situations. Reward shaping techniques provide additional guidance to the RL agent, enabling it to focus on relevant features of the environment and mitigate potential uncertainties. In this paper, a robust reinforcement learning methodology is adapted utilising a novel Adversarial Proximal Policy Optimisation (A-PPO) method integrating an Adaptive KL penalty PPO. Comparison is made with DQN, DDQN and a conventional PPO algorithm.
  • Thumbnail Image
    ItemOpen Access
    Detect and avoid considerations for safe sUAS operations in urban environments
    (IEEE, 2021-11-15) Celdran Martinez, Victor; Ince, Bilkan; Kumar Selvam, Praveen; Petrunin, Ivan; Seo, Min-Guk; Anastassacos, Edward; Royall, Paul G.; Cole, Adrian; Tsourdos, Antonios; Knorr, Sebastian
    Operations involving small Unmanned Aerial Systems (sUAS) in urban environments are occurring ever more frequently as recognized applications gain acceptance, and new use cases emerge, such as urban air mobility, medical deliveries, and support of emergency services. Higher demands in these operations and the requirement to access urban airspace present new challenges in sUAS operational safety. The presence of Detect and Avoid (DAA) capability of sUAS is one of the major requirements to its safe operation in urban environments according to the current legislation, such as the CAP 722 in the United Kingdom (UK). The platform or its operator proves a full awareness of all potential obstacles within the mission, maintains a safe distance from other airspace users, and, ultimately, performs Collision Avoidance (CA) maneuvers to avoid imminent impacts. Different missions for the defined scenarios are designed and performed within the simulation model in Software Tool Kit (STK) software environment, covering a wide range of practical cases. The acquired data supports assessment of feasibility and requirements to real-time processing. Analysis of the findings and simulation results leads to a holistic approach to implementation of sUAS operations in urban environments, focusing on extracting critical DAA capability for safe mission completion. The proposed approach forms a valuable asset for safe operations validation, enabling better evaluation of risk mitigation for sUAS urban operations and safety-focused design of the sensor payload and algorithms.
  • Thumbnail Image
    ItemOpen Access
    Optimization of a robust reinforcement learning policy
    (AIAA, 2023-01-19) Ince, Bilkan; Shin, Hyosang; Tsourdos, Antonios
    A major challenge for the integration of unmanned air vehicle (UAV) in the current civil applications is the sense-and-avoid (SAA) capability and the consequent possibility of mid-air collision avoidance. Although UAS have been shown to be efficient under different and varied conditions, their safety, reliability, and compliance with aviation regulations remain to be proven. In autonomous collision avoidance, UAS sense hazards with the sensors equipped on them and make decisions on manoeuvres autonomously for collision avoidance at the minimum safe time before impact. Thus, it is required for each individual UAS to have capabilities to recognize urgent threats and undertake the evasive manoeuvres immediately. Most of the current sense and avoid algorithms are composed of separated obstacle detection and tracking algorithm and decision-making algorithm on avoidance manoeuvre. Implementing artificial intelligence (AI), reinforcement learning (RL) algorithm combines both sense and avoid functions through state and action space. An autonomous agent learns to perform complex tasks by maximizing reward signals while interacting with its environment. It may be infeasible to test a policy in all contexts since it is difficult to ensure it works as broadly as intended. In these cases, it is important to trade-off between performance and robustness while learning a policy. This work develops an optimization method for a robust reinforcement learning policy for a nonlinear small unmanned air systems (sUAS), in AirSim using a model-free architecture. Using an on-line trained reinforcement learning agent, the difference of an optimized robust reinforcement learning (RRL) policy together with a conventional RL and RRL algorithm will be reproduced.
  • Thumbnail Image
    ItemOpen Access
    Sense and avoid considerations for safe sUAS operations in urban environments
    (IEEE, 2024-05-07) Ince, Bilkan; Martinez, Victor Celdran; Selvam, Praveen Kumar; Petrunin, Ivan; Seo, Minguk; Tsourdos, Antonios
    Operations involving small Unmanned Aerial Systems (sUAS) in urban environments are occurring ever more frequently as recognized applications gain acceptance, and new use cases emerge, such as urban air mobility, medical deliveries, and support of emergency services. The presence of Detect and Avoid (DAA) capability of sUAS is one of the major requirements for its safe operation in urban environments. The platform or its operator proves a full awareness of all potential obstacles within the mission, maintains a safe distance from other airspace users, and, ultimately, performs Collision Avoidance (CA) maneuvers to avoid imminent impacts. Communication and navigation defined scenarios are designed and performed within the simulation model in Systems Tool Kit (STK) software environment, covering several practical cases. The acquired data supports the assessment of feasibility and requirements for real-time processing. Utilizing Unreal Engine and MATLAB analysis of the findings and simulation results leads to a holistic approach to implementation of sUAS operations in urban environments, focusing on extracting critical DAA capability for safe mission completion. The proposed approach forms a valuable asset for safe operations validation, enabling better evaluation of risk mitigation for sUAS urban operations and safety-focused design of the sensor payload and algorithms.