Supplementary Material for Pheromones to Policies: RL for Biological Swarms

Supplementary Material for Pheromones to Policies: RL for Biological Swarms 1 Cross-learning equivalence derivation 1.1 Case 1: Site i is Chosen Assume site iis chosen at time t. Then τi(t+ 1) = ρ τi(t)+Q, τk(t+ 1) = ρ τk(t) (∀k=i).(1) Only τi(t) receives an increment of Q, while all other pheromone levels τk(t) for k=iare solely subject to evaporation by the factor ρ. Therefore, when normalizing to compute Pi(t+1), the additional pheromone Qcontributes only to the chosen site’s term, resulting in QAiin the denominator. Hence the probability of choosing site iat time t+ 1 becomes: Pi(t+ 1) = ρ τi(t)+QAi ρPM j=1 τj(t)Aj+Q Ai .(2) We rewrite τi(t) in terms of Pi(t) by noting τi(t) = Pi(t) AihM X j=1 τj(t)Aji.(3) Substitute into (2): Pi(t+ 1) = ρ τi(t)Ai+Q Ai ρPM j=1 τj(t)Aj+Q Ai =ρPi(t) AiPM j=1 τj(t)AjAi+Q Ai ρPM j=1 τj(t)Aj+Q Ai =ρ Pi(t)PM j=1 τj(t)Aj+Q Ai ρPM j=1 τj(t)Aj+Q Ai . (4) Define r=Q Ai ρPM j=1 τj(t)Aj+Q Ai .(5) Rewriting to highlight the increment in Pi(t) yields Pi(t+ 1) = Pi(t)+r[ 1 −Pi(t)].(6) Thus, if site iis chosen, Pi(t) is incremented by a factor proportional to [1 −Pi(t)]. 1 1.2 Case 2: Site i is Not Chosen If site iis not chosen at time t, then there is no new pheromone added to τi(t), so τi(t+ 1) = ρ τi(t).(7) Meanwhile, exactly one other site (say k=i) is chosen and thus gets reinforced. Hence, τk(t+ 1) = ρ τk(t)+Q, and for all others j={i, k}: τj(t+ 1) = ρ τj(t). (8) We can now convert these τi(t+ 1) values to updated probabilities by normalizing over the sum of all pheromone×attractiveness terms: Pi(t+ 1) = ρ τi(t)Ai M X j=1ρ τj(t)+∆jAj ,(9) where ∆j=Qif jwas chosen, and 0 otherwise. Since exactly one site kis chosen, M X j=1ρ τj(t)+∆jAj=ρ M X j=1 τj(t)Aj+Q Ak.(10) Hence Pi(t+ 1) = ρ τi(t)Ai ρPM j=1 τj(t)Aj+Q Ak .(11) Expressing τi(t) in terms of Pi(t): Pi(t) = τi(t)Ai PM j=1 τj(t)Aj =⇒τi(t) = Pi(t) AihM X j=1 τj(t)Aji. (12) Substitute back to get Pi(t+ 1) = ρPi(t) AiPM j=1 τj(t)AjAi ρPM j=1 τj(t)Aj+Q Ak =Pi(t)ρPM j=1 τj(t)Aj ρPM j=1 τj(t)Aj+Q Ak . (13) Factor out Pi(t) and rewrite the fraction: Pi(t+ 1) = Pi(t)1−Q Ak ρPM j=1 τj(t)Aj+Q Ak.(14) Define r=Q Ak ρPM j=1 τj(t)Aj+Q Ak .(15) 2 Then the update law becomes Pi(t+ 1) = Pi(t)−r Pi(t),(16) showing that if site iis not chosen, its probability Pi(t) decreases by a fraction rproportional to Pi(t) itself. 3