I was thinking of a way to solve this problem in O(1) and I am not getting the right answer..

My logic was that..

the answer is (a+b)ab/2 and the equation is x+my=mc(I changed b to c for convinience..)

so upon applying the am>=gm inequality in the following way..

((a+b)+a+(2*m-1)b)/3>=((2*m-1)^1/3)*answer

so answer would be floor(LHS/(2*m-1)^1/3) and LHS is 2*m*c as (a,b) lies on the line..

but I was getting a wrong answer..

Sorry if I missed something obvious and any help would be really appreciated..