sambit1993's blog

How to calculate memory usage of a program?
By sambit1993, 11 years ago, In English

I am implementing a online judge in python. The judge is to run on a UNIX machine. To enforce memory limits I am using ulimit and to run a program I am using subprocess.Popen() . I want to calculate the exact memory usage. How can I do so?

  • Vote: I like it
  • +3
  • Vote: I do not like it

»
11 years ago, # |
  Vote: I like it +1 Vote: I do not like it

Use resource.getrusage():

http://docs.python.org/3/library/resource.html#resource.getrusage

The maximum used memory is ru_maxrss.

→ Reply
  • »
    »
    11 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    I used ~~~~~ p = subprocess.Popen(cmd, shell=True,stdout=subprocess.PIPE, stderr=subprocess.PIPE) print getrusage(resource.RUSAGE_CHILDREN).ru_maxrss ~~~~~

    but the output is always zero am I doing something wrong. cmd is the command to be executed.

    → Reply
    • »
      »
      »
      11 years ago, # ^ |
        Vote: I like it 0 Vote: I do not like it

      You need to wait for the child first: http://pubs.opengroup.org/onlinepubs/9699919799/functions/getrusage.html

      → Reply
      • »
        »
        »
        »
        11 years ago, # ^ |
        ← Rev. 4   Vote: I like it 0 Vote: I do not like it

        Okay.. Now I am using


        ~~~~~
p=subprocess.Popen("./"+cmd+"<in.txt>out.txt",shell=True,stdout = subprocess.PIPE, stderr = subprocess.PIPE)
p.wait()
print getrusage(resource.RUSAGE_CHILDREN).ru_maxrss

        ~~~~~

        But still I am getting about the same value(~3430) for any program that I am executing. But the programs: https://ideone.com/KK1rak and http://ideone.com/3HfV9f take different memory according to ideone.

        → Reply
        • »
          »
          »
          »
          »
          11 years ago, # ^ |
            Vote: I like it 0 Vote: I do not like it

          That is because you measure maxrss of the shell process and not of your program. You have to use shell=False.

          Also, it seems that resource usage is cumulative, so if you want to run multiple programs and obtain individual data about them, you'd have to fork a Python process for each time.

          → Reply
          • »
            »
            »
            »
            »
            »
            11 years ago, # ^ |
            ← Rev. 2   Vote: I like it 0 Vote: I do not like it

            I changed to this.

            with open('in.txt','r') as infile, open('out.txt', 'w') as outfile:

            `p=subprocess.check_call("./"+cmd,shell=False,stdin=infile,stdout = outfile, stderr = subprocess.PIPE)`

            Res= getrusage(resource.RUSAGE_CHILDREN)

            print Res.ru_maxrss

            Still I am getting a constant value(~5060).

            → Reply
            • »
              »
              »
              »
              »
              »
              »
              11 years ago, # ^ |
              ← Rev. 2   Vote: I like it 0 Vote: I do not like it

              Now it seems you forgot the wait... This works for me:

              import subprocess
import resource

p = subprocess.Popen("./a.out")
p.wait()
print(resource.getrusage(resource.RUSAGE_CHILDREN).ru_maxrss)

              EDIT: I did not notice you changed Popen to check_call, it will wait for the process indeed.

              → Reply
»
11 years ago, # |
  Vote: I like it +8 Vote: I do not like it

Maybe you could take a look at this opensource online judge (written in Python) and see how they do it :)

→ Reply