Because for one heap multiple of 4 is always losing. All other remainders are winning. When you do mod it gets the smallest number of sticks and doesn't change the remainder. Now it goes back to any old nim sum where you can remove any number of sticks. Check out 11.5.1 of guide to cp.

You should look at the last part of chapter Algorithm Design Topics specifically 8.3.3. Then use that idea with divide and conquer optimization. This is the only way to do it, I think. You can't just use Convex Hull Trick for everything. The recurrence clearly shows that it is a divide and conquer DP optimization problem.

When you do %4 it isn't changing anything the outcome is still the same. The only difference is that now it becomes just like nim game 1 because everything is below 4 and you can remove 1, 2, or 3. This is essentially nim game 1 where you can remove anything from any pile. Now you can do nim sum just like in nim game 1.