Tutorial

This section describes how to set up a fully-functional project using Speedy Antlr. You can find the completed example here.

In this example, a we will put together a fictional Python module called spam that implements an Antlr parser for a grammar called MyGrammar.

antlr4-cpp-runtime

src/spam/parser/cpp_src/antlr4-cpp-runtime

This directory contains a copy of Antlr’s C++ runtime source. This is required for our extension to be built against.

Future releases of the runtime can be downloaded from Antlr’s download page

generate_parsers.sh

src/spam/parser/generate_parsers.sh

This script is what the developer uses to re-generate Antlr targets, as well as the Speedy Antlr accelerator files.

As usual, generate targets from your grammar file using Antlr. We want to generate both Python and C++ targets since both will be used together.

antlr4="java -Xmx500M -cp /usr/local/lib/antlr-4.8-complete.jar org.antlr.v4.Tool"

# Generate C++ target with visitor
$antlr4 -Dlanguage=Cpp -visitor -no-listener -o cpp_src MyGrammar.g4

# Generate Python target
$antlr4 -Dlanguage=Python3 -no-visitor -no-listener -o . MyGrammar.g4

After the targets are generated, invoke speedy-antlr-tool via Python to generate the accelerator files.

# Run speedy-antlr-tool to generate parse accelerator
python3 <<EOF
from speedy_antlr_tool import generate

generate(
    py_parser_path="MyGrammarParser.py",
    cpp_output_dir="cpp_src",
)
EOF

You’ll notice this last step generates the following files:

  • sa_mygrammar.py

  • cpp_src/sa_mygrammar_cpp_parser.cpp

  • cpp_src/sa_mygrammar_translator.cpp/.h

  • cpp_src/speedy_antlr.cpp/.h

sa_mygrammar.py

src/spam/parser/sa_mygrammar.py

This module provides the entry-point for the C++ based parser, as well as a pure Python fall-back implementation. When calling the parse() function, the fall-back implementation is automatically used if the C++ version failed to install.

setup.py

setup.py

This example setup script shows how to gracefully omit the C++ accelerator if it fails to build. Recall from earlier, if the extension is not avialable, the parse() wrapper function will automatically choose the Python equivalent.

LICENSE-3RD-PARTY

LICENSE-3RD-PARTY

Since you’ll be bundling the Antlr C++ runtime in your package’s distribution (source and binary), be a good steward of open-source software and include a copy of Antlr’s BSD license.

.travis.yml

.travis.yml

If you’ve attempted to install this example by now, you’ve probably noticed that it takes a looong time. This is because all the C++ files (antlr has many) are getting compiled.

If you plan to publish your package to PyPi, it is good practice to also publish binary distributions. This eliminates the need for the end-user to install a compiler and build everything from source.

Since you probably dont have access to every variant of Windows/Linux/macOS, this is typically done using a continuous integration service like Travis-CI. This YAML file tells Travis-CI how to run your project’s tests, and how to deploy to PyPi. I’m also using cibuildwheel to automate building all the different distribution variants.