How to parse a domain-specific language on top of Python? What architecture with IPython?

Question

I want to construct a domain-specific language as a superset of Python. Cryptic commands like

f7:10y=x^2

designed to minimize typing shall be parsed into plain Python

for k in range(7,10):
    f[k].set_y( expr='x^2' )

before being executed. Probably, the command-line interface shall be IPython.

What would be an appropriate architecture: Shall I implement the cryptic-to-plain-Python translation in the IPython command-line shell or in its kernel daemon? Are there helpful libraries / tutorials / examples?

Or more generically: Are there examples how to add complex syntactic sugar to Python?

Answer 1

A solution that will work everywhere is not really an option here. What you are trying to achieve is modifying the core of the Python parser.

The options are:

1. Use a preparser. Within IPython there are hooks available for this: http://ipython.org/ipython-doc/dev/config/inputtransforms.html

   Here's an example for your use case:

   import re
   
   range_search = re.compile('''
       (?P<variable>.+)
       (?P<start>\d+):(?P<stop>\d+)
       (?P<k>[^=]+)=(?P<v>.+)
   ''', re.VERBOSE)
   
   range_replace = '''for k in range(\g<start>, \g<stop>):
       \g<variable>[k].set_\g<k>(expr='\g<v>')'''
   
   print range_search.sub(range_replace, 'f7:10y=x^2')
   
   @StatelessInputTransformer.wrap
   def inline_loop(line):
       return range_search.sub(range_replace, line)
   

2. Recompile Python with your modifications

3. Use a language which is more flexible in this regard like ruby

Answer 2

With PyPy this is fairly easy to do. You can just add some things to the grammar file and you're done.

Here's an example: https://bitbucket.org/adonohue/units/src/bb1b20dd739f73fe495723d24cd266b67549f5c9/unitPython/patches?at=default

It allows the units library to do things like these:

print(2cm / 0.5 s)
-> 4.0 cm / s