Look for invariants
Observation 1: Let white stones represent $$$+1$$$, and black stones represent $$$-1$$$. The difference between the values of two people's stones is constant. If the starting value of stones is $$$a_i$$$, then the ending value will be $$$a_i + C$$$, for some constant $$$C$$$.
Observation 2: Define a pair of stones to be a black stone and a white stone. Note that these pairs can be freely moved. BW,0,0 B,B,B 0,0,BW
Observation 3: If there are $$$N-2$$$ pairs, and one extra stone, they can be eliminated BW, BW, BW, B, 0 W, W, W, 0, W 0, 0, 0, B, 0
Observation 4: If there are less than $$$N-2$$$ pairs, they cannot be removed without changing $$$C$$$. Proof: Every operation changes $$$C$$$ by $$$1$$$. Thus, to not change $$$C$$$, you must perform an even number of operations. Since the total number of stones changes by $$$N-2$$$ each operation, after an even number of operations, thus changing the total number of stones by a multiple of $$$2(N-2)$$$.
Thus, for a given $$$C$$$, the minimum number of pairs can be computed. It can be shown, that the value of $$$C$$$ that minimizes the stones is the negative of the median of $$$a_i$$$, plus or minus 1. Thus, you can simply test those values.
