Mirosław Tomera

Path planning for a mobile agent concerns the problem of searching for a collision-free and optimal path between the initial and target positions. The space in which the agent moves contains a number of obstacles modeled by ordered grids, each representing the obstacle location in the space of the agent movement. The final boundary of an obstacles is formed by its actual boundary plus a minimum safe distance, taking into account the size of the agent, which allows to treating the obstacles as points in the environment. In the article, reinforcement learning algorithms were used to determine the path, including numerous design methods: dynamic programming (policy iteration algorithm and value iteration algorithm), Monte Carlo method (Monte Carlo control algorithm), temporal-difference (TD) learning (Q-learning algorithm and Sarsa algorithm), eligibility traces (Q()) algorithm and Sarsa() algorithm), planning and learning (Dyna-Q algorithm), and gradient methods (Q-learning algorithm with Adam optimizer and Sarsa algorithm with Adam optimizer). The reinforcement learning algorithms operate on the principle of determining the agent's policy, which seeks the minimum distance between the initial and target positions of the mobile agent, while avoiding obstacles. These learning procedures differ between, dynamic programming, which requires a good knowledge of the environment model to determine the agent's policy, and other methods which do not require this knowledge. The aim of the reported work was to examine the above named algorithms in terms of their effectiveness and speed of finding the optimal solution. Based on the results of the simulation studies, the most effective methods turned out to be that using gradient methods for optimization, i.e.: Q-learning with Adam optimizer.

The focus of this work is the determination of the optimal path for a mobile agent to take in an environment with static obstacles, using reinforcement learning (RL). The paper explains the work examining different RL algorithms, such as Q-learning and Sarsa in the classic version and enhanced with the Adam gradient optimiser. The work investigates the impact of the Adam gradient optimiser on the rate and stability of finding the optimal solution. The analysis includes a comparison of the learning rate, the number of steps in a single episode and the stability of the learning process. The results reveal that the considered Q-learning and Sarsa algorithms supplemented with the Adam optimiser achieve a higher performance, characterised by a faster determination of the optimal transition path, than the same algorithms without the Adam gradient optimiser. The results could be particularly useful in practical applications for routing transitions in fields like mobile robotics. Praca koncentruje się na wyznaczaniu optymalnej trasy przejścia agentaruchomego w środowisku z przeszkodami statycznymi, przy wykorzystaniu uczenia przezwzmacnianie. W pracy przeanalizowano algorytmy uczenia przez wzmacnianie takie jak:Q-learning i Sarsa w wersji klasycznej oraz rozszerzonej o optymalizator gradientowy Adam.W pracy zbadano wpływ optymalizatora Adam na szybkość i stabilność znajdowaniaoptymalnego rozwiązania. Analiza obejmowała porównanie szybkości uczenia, liczbykroków w pojedynczym odcinku oraz stabilności procesu uczenia. Wyniki wykazały, żerozważane algorytmy Q-learning i Sarsa uzupełnione o optymalizator Adam osiągająwyższą skuteczność działania, charakteryzującą się szybciej wyznaczaną optymalną trasąprzejścia, aniżeli algorytmy te bez optymalizatora Adam. Uzyskane wyniki mogą byćszczególnie przydatne w praktycznych zastosowaniach służących do wyznaczanie trasyprzejścia, w takich dziedzinie jak robotyka mobilna.

The article presents the synthesis of a multi-layer group control system for a marine autonomous surface vessel with the use of modern control theory methods. First, an evolutionary programming algorithm for determining the optimal route path was presented. Then the algorithms—dynamic programming with neural state constraints, ant colony, and neuro-phase safe control algorithms—were presented. LMI and predictive line-of-sight methods were used for optimal control. The direct control layer is implemented in multi-operations on the principle of switching. The results of the computer simulation of the algorithms were used to assess the quality control.

The main goal of the research is to design an efficient controller for a dynamic positioning system for autonomous surface ships using the backstepping technique for the case of full-state feedback in the presence of unknown external disturbances. The obtained control commands are distributed to each actuator of the overactuated vessel via unconstrained control allocation. The numerical hydrodynamic model of CyberShip I and the model of environmental disturbances are applied to simulate the operation of the ship control system using the time domain analysis. Simulation studies are presented to illustrate the effectiveness of the proposed controller and its robustness to external disturbances.

The paper presents the design of a hybrid controller used to control the movement of a ship in different operating modes, thereby improving the performance of basic maneuvers. This task requires integrating several operating modes, such as maneuvering the ship at low speeds, steering the ship at different speeds in the course or along the trajectory, and stopping the ship on the route. These modes are executed by five component controllers switched on and off by the supervisor depending on the type of operation performed. The desired route, containing the coordinates of waypoints and tasks performed along consecutive segments of the reference trajectory, is obtained by the supervisory system from the system operator. The former supports switching between component controllers and provides them with new set-points after each change in the reference trajectory segment, thereby ensuring stable operation of the entire hybrid switching controller. The study also presents designs of all controller components, which are done using a complex mathematical model of the selected ship, after its simplification depending on the type of controller. The developed control system was tested on the training ship Blue Lady and used to train captains at the Ship Handling Research and Training Center near Iława in Poland. The conducted research involved an automatic movement of the ship from one port to another. The performed transit route required the ship to leave the port, pass the water area, and berth at the port of destination. The study revealed good quality of the designed hybrid controller.