Intelligent Management using Collaborative Reinforcement Multi-agent System

In large infrastructures such as transportation, health and energy systems, collaborative monitoring system is needed, where there are special needs for intelligent distributed networking systems with learning schemes. Agent Reinforcement Learning (RL) for autonomous network management, in general, is one of the challengeable methods in a dynamic complex cluttered environment over a network. The goal for autonomous network management using RL is self-management to manage optimized agent work-flow without minimal human dependency by learning process . The system is needed by the development of computational multi-agents learning process in large distributed networking nodes, where the agents have limited and incomplete knowledge, and they only access local information in distributed networking nodes. Reinforcement Learning can become an effective technique to transfer and share information among agents, as it does not require a priori knowledge of the agent behavior or environment to accomplish its tasks [Megherbi]. Such a knowledge is usually acquired and learned automatically and autonomously by trial and error. Reinforcement Learning is Machine Learning techniques that will be adapted to the various networking environments for automatic networks[I-D.jiang-nmlrg-network-machine-learning]. Thus, this document provides motivation, learning technique, and use case for network machine learning.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in .

Reinforcement Learning is a system capable of autonomous acquirement and incorporation of knowledge. It can continuously self-improve learning process with experience and attempts to maximize cumulative reward to manage an optimized learning knowledge by multi-agents-based monitoring systems[Teiralbar]. The maximized reward can be increasingly optimizing of learning speed for agent autonomous learning process.

Reinforcement learning is an emerging technology in terms of monitoring and managing network system to achieve fair resource allocation for nodes within the wire or wireless mesh setting. Monitoring parameters of the network and adjusts based on the network dynamics can demonstrate to improve fairness in wireless environment Infrastructures and Resources [Nasim]. The fundamental goal for Reinforcement Learning is self-management, which is comprised of a couple of properties such as self-healing (adaptive function in the environment and heal problems automatically) and self-optimizing (function for automatically determine ways to optimize their behavior against a set of well-defined goals) .

There are many different networking management issues such as connectivity, traffic management, fast internet without latency and etc. We expect that ml-based mechanism such as reinforcement learning will provide solutions of networking issues with multiple cases against human operating capacities even if it is a challengeable area due to a multitude of reasons such as large state space search, complexity in giving reward, difficulty in agent action selection, and difficulty in sharing and merging learned information among the agents in a distributed memory node to be transferred over a communication network [Minsuk].

Autonomous vehicle is capable of self-management for automotive driving without human supervision depending on optimized trust region policy by reinforcement learning that enables learning of more complex and special network management environment. Such a vehicle provides a comfortable user experience safely and reliably on interactive communication network [April] [Markus].

The adaptive multi-agent system, which is combined with complexities from interacting game player, has developed in a field of reinforcement learning. In the early game theory, the interdisciplinary work was only focused on competitive games, but Reinforcement Learning has developed into a general framework for analyzing strategic interaction and has been attracted field as diverse as psychology, economics and biology [Ann]. AlphaGo is also one of the game theories using reinforcement learning, developed by Google DeepMind. Even though it began as a small learning computational program with some simple actions, it has now trained on a policy and value networks of thirty million actions, states and rewards for optimal management using learning process.

Wireless sensor network (WSN) consists of a large number of sensors and sink nodes for monitoring systems to manage event parameters such as temperature, humidity, air conditioning, etc. Reinforcement learning in WSNs has been applied in a wide range of schemes such as cooperative communication, routing and rate control. The sensors and sink nodes are able to observe and carry out optimal actions on their respective operating environment for network and application performance enhancements [Kok-Lim].

Reinforcement Learning is used to enhance multicast routing protocol in wireless ad hoc networks, where each node has different capability. Routers in the multicast routing protocol are determined to discover optimal route with a predicted reward, and then the routers create the optimal path with multicast transmissions to reduce the overhead in Reinforcement Learning[Kok-Lim].

Agent reinforcement Learning is ml-based unsupervised algorithms based on an agent learning process. Reinforcement Learning is normally used with a reward from centralized node (the global environment), and capable of autonomous acquirement and incorporation of knowledge. It is continuously self-improving and becoming more efficient as the learning process from an agent experience to optimize management performance for autonomous learning process.[Sutton][Madera]

Distance and Frequency algorithm uses the state occurrence frequency in addition to the distance to goal. It avoids deadlocks and lets the agent escape the Dead, and it was derived to enhance agent optimal learning speed. Distance-and-Frequency is based on more levels of agent visibility to enhance learning algorithm by an additional way that uses the state occurrence frequency.[Al-Dayaa]

Autonomous multi-agent learning process for network management environment is related to transfer optimized knowledge between agents on a given local node or distributed memory nodes over a communication network.

This is a technique how agents can share information for optimal learning process. The quality of agent decision making often depends on the willingness of agents to share a given learning information collected by agent learning process. Sharing Information means that an agent would share and communicate the knowledge learned and acquired with or to other agents using reinforcement learning. Agents normally have limited resources and incomplete knowledge during learning exploration. For that reason, the agents should take actions and transfer the states to the global environment under reinforcement learning, then it would share the information with other agents, where all agents explore to reach their destination via a distributed reinforcement reward-based learning method on the existing local distributed memory nodes. MPI (Message Passing Interface) is used for communication way. Even if the agents do not share the capabilities and resources to monitor an entire given large terrain environment, they are able to share the needed information to manage collaborative learning process for optimized management in distributed networking nodes.[Chowdappa][Minsuk]

A new technical method for agent sub-goal selection in distributed nodes is introduced to reduce the agent initial random exploration with a given selected sub-goal. [TBD]

The architecture using Reinforcement Learning describes a collaborative multi-agent-based system in distributed environments as shown in figure 1, where the architecture is combined with a hybrid architecture making use of both a master and slave architecture and a peer-to-peer. The centralized node(global environment), assigns each slave computing node a portion of the distributed terrain and an initial number of agents.

+-------------+ +-----------------+ | |<......>| node 1 |<.......>| terrain 1 | | | +-----------------+ | Global env. | | (node 0) | +-----------------+ | |<......>| node 2 |<.......>| terrain 2 | +-------------+ +-----------------+ Reinforcement Learning actions involve interacting with a given environment, so the environment provides an agent learning process with the elements as followings: Agent actions, states and cumulative rewards One or more obstacles, and goals Initially, random exploration in a given node Next, optimal explorations under reinforcement learning Additionally, agent actions with states toward its goal as below: Agent continuously actions to avoid an obstacle based on its policy and move to one or more available positions until it reaches its goal(s) After an agent reaches its destination, it can use the information collected by initial random learning process to next learning process for optimal management Agent learning process is optimized in the following phase and exploratory learning trials In shown as Figure2, we illustrate the fundamental architecture for relationship of an action, state and reward, and each agent explores to reach its destination(s) under reinforcement learning. The agent does an action that leads to a reward from achieving an optimal path toward its goal. Our works will be extended depending on the architecture.

+---------------------+ ....state and reward........+ Global Environment +|<............ . +---------------------+ . +------+------+ . | Multi-agent | . +------+------+ +---------------+ . ............action.........>+ Destiantion +................... +---------------+

In this section, we deal with case of a collaborative distributed multi-agent, where each agent has same or different individual destination in a distributed environment. Since sharing information scheme among the agents is problematic one, we need to expand on the work described by solving the challenging cases. Basically, the main proposed algorithm is presented by distributed multi-agent reinforcement learning as below: Proposed Algorithm (1) Let Ni denote the number of node (i= 1, 2, 3 ...) (2) Let Aj denote the number of agent (3) Let Dk denote the number of destination (4) Place initial number of agents Aj, in random position (Xm, Yn) (5) Every Aj in Ni -----> (a) Do initial exploration (random) to corresponding Dk -----> (b) Do exploration (using RL) for Tx denote the number of trial Random Trial (1) Let Si denote the the current state (2) Relinquish Si so that the other agent can occupy the position (3) Assign the agent new position (4) Update the current state Si -> Si+1 Optimal Trial (1) Let Si denote the the current state (2) Let ACj denote an action (3) Let DRm denote discount reward (4) Choose ACj <- Policy(Si, ACj) (5) Move an available posiion (6) Update learning process in the global environment (7) Update the current state Si < Si+1 Multi-agent reinforcement learning in distributed nodes can improve the overall system performance to transfer or share information from one node to another node in following cases; expanded complexity in RL technique with various experimental factors and conditions, analyzing multi-agent sharing information for agent learning process.

Sub-goal selection is a scheme of a distributed multi-agent RL technique based on selected intermediary agent sub-goal(s) with the aim of reducing the initial random trial. The scheme is to improve the multi-agent system performance with asynchronously triggered exploratory phase(s) with selected agent sub-goal(s) for autonomous network management. [TBD]

There are no IANA considerations related to this document.

[TBD]

David Meyerm, who chief scientist and VP in Brocade, has provided significant comment and feedback for the draft.