bayashi_cl's blog

Expand your Python library into a single source file.
By bayashi_cl, history, 5 years ago, In English

Hello. This is my first blog post.

I have created a tool that combines Python packages that exist in separate files into a single source. This tool allows you to import and use libraries for competitive programming in the same way as common packages (like numpy).Similar tools are webpack and oj-bundle.

By the way, in the code generated by this tool, the definitions of functions and classes in the library are expressed as "strings". I think that this may violate the rules against obfuscation (Can-do's and Can't-do's 4). Can the code generated by this tool be used in the contest? I would like to hear your opinions.

Thank you very much.

  • Vote: I like it
  • +2
  • Vote: I do not like it

Comments Comments (5)
| Write comment?
»
5 years ago, hide # |
 
Vote: I like it 0 Vote: I do not like it

Auto comment: topic has been updated by bayashi_cl (previous revision, new revision, compare).

→ Reply
»
4 years ago, hide # |
← Rev. 2  
Vote: I like it +3 Vote: I do not like it

I tried out this, it literally pulls in the entire library. My 10 lines code got expanded to 400K lines. Am I using it correctly ?

This is my code test.py

from sympy import symbols, expand
x = symbols('x')
p1 = 1+2*x+3*x**2
p2 = 3*x**3+2*x**4+x**5
print(p1*p2)
r = expand(p1*p2)
print(r)

And I ran.

python -m expander test.py -o test_expanded.py -m sympy

It generated close to 400K lines of code. Surely it must be more intelligent than this, it should only pull in the used function and its dependencies recursively and nothing more. If I have to write code in a way like changing the coding style from import sympy to looking like from sympy import symbols, expand I am okay to make that adjustment.

→ Reply
  • »
    »
    4 years ago, hide # ^ |
     
    Vote: I like it 0 Vote: I do not like it

    Thank you for using my tool.

    I am sorry, but it is not possible to expand and submit a large module like sympy with this tool.

    Currently, this tool recursively expands all "dependent" modules, such as the imported modules and their parent modules, and modules imported by that module... and so on. (In the case of sympy, this is directly due to the fact that sympy/__init__.py imports many submodules.) This is because it is very difficult to identify which parts of the source code are really needed; even when Python imports modules, it should not be "loading only the necessary parts".

    I would like to see this tool used for smaller packages like PyRival or your own competitive programming libraries. (And I have submitted code expanded with this tool several times and never received a warning.)

    → Reply
    • »
      »
      »
      4 years ago, hide # ^ |
       
      Vote: I like it 0 Vote: I do not like it

      I copy manually from sympy all the time, it takes only few hundred lines most of the times. My own libraries aren't going to be too simple either. What is the point of using library if you are having to write each library of just one or two files. I will try out pyrival and let you know. If it can't extract out code from complex packages then it woundn't be of much use yet, but with some improvement hopefully it will. If recursively pulling out library is going to be so difficult, I would suggest we can design alternative expander to extract code based on code coverage. Only the hot path of the code may be extracted and other paths may not(after running through a set of test data). There are lots of opensource code coverage tools to get started.

      Anyways thanks for the quick reply. I wasn't able to get in touch with you on github.

      → Reply