Fitting Java and Python with JPY

Posted by in Java

There are many libraries in Java (more than 176,649 unique artifacts indexed just on Maven Central), but sometimes you can not find what you are looking for, except for a Python equivalent. In a previous project, I had to deal with custom MaxMind databases. Maxmind provides a Java library with a database reader, but does not provides a database writer. After some researches, I found one official lib in Perl, and an other (unofficial) in Python. Since, I didn’t have the time to redevelop the equivalent in Java (this was just a POC…), I’ve decided to use a bridge on one of these two libraries. I didn’t find any easy and production-ready solution to call some Perl in Java (exposing the library as a remote service, create a native process from Java and play with the shell,…) so I tried the unofficial project in Python because some bridges can fit Python and Java together.

In this post I will explain how to use a Python project or library in your Java projects by using JPY. JPY is a bi-directional Python-Java bridge which you can use to embed or call Python code in Java programs or vice versa. JPY is a Python module written in C and compiled into a shared library used by the Python JPY and by the JPY Java library as a native library.

JPY is not deployed on the central Maven repositories, so you have to clone it and compile it on your Linux/Mac or Windows. After the setup, you should see these files into your JPY project:

tree ~/git/jpy
├── build
│   ├── bdist.linux-x86_64
│   ├── lib.linux-x86_64-2.7
│   │   ├── jdl.so
│   │   ├── jpyconfig.properties
│   │   ├── jpyconfig.py
│   │   ├── jpyconfig.pyc
│   │   ├── jpy.so
│   │   ├── jpyutil.py
│   │   └── jpyutil.pyc
├── dist
│   └── jpy-0.9_SNAPSHOT-cp27-cp27mu-linux_x86_64.whl
├── target
│   └── jpy-0.9-SNAPSHOT.jar
...

Now you can use the freshly compiled JAR in your project. Create a new project with a layout like this:

├── pom.xml
├── requirements.txt
├── src
│   ├── main
│   │   ├── assembly
│   │   │   ├── __init__.py
│   │   │   └── zip.xml
│   │   ├── java
│   │   │   └── fr
│   │   │       └── layer4
│   │   │           └── jpy
│   │   │               ├── Main.java
│   │   │               ├── MyPlugin.java
│   │   │               └── ZipUtils.java
│   │   ├── python
│   │   │   └── my_plugin.py
│   │   └── resources
│   │       └── jpyconfig.properties

Some explanations:
– the Main.java class is the Java class calling our Python scripts.
– my_plugin.py is a module inside the project.
– requirements.txt contains all the Python dependencies. It will contain only “requests==2.11.1” for our example.
– we use the exec-maven-plugin to fetch the dependencies in the requirements.txt file via a “pip install”.
– all the dependencies are zipped with a blank file “init.py” in the root, into an archive named deps-bundle.zip.
– this archive is embed in the final jar.
– a file jpyconfig.properties must be provided in the classpath and defined the mandatory properties for JPY. After having built the JPY project, you can find this file in the build/lib.linux-x86_64-2.7 directory. Mine is:

# Created by 'jpyutil.py' tool on 2017-02-13 15:37:50.987325
# This file is read by the jpy Java API (org.jpy.PyLib class) in order to find shared libraries
jpy.jpyLib = /home/devil/git/jpy/build/lib.linux-x86_64-2.7/jpy.so
jpy.jdlLib = /home/devil/git/jpy/build/lib.linux-x86_64-2.7/jdl.so
jpy.pythonLib = /usr/lib/x86_64-linux-gnu/libpython2.7.so
jpy.pythonPrefix = /usr
jpy.pythonExecutable = /usr/bin/python

NOTE
Since the JPY libs depend on the architecture (Linux, Windows, x86…) of the target host, this file should be deployed on every hosts on which you plan to install your application and never be included in the final archive of your project.

my_plugin.py is really simple, it defines two methods, one using the Python requests module:

import requests

class MyPlugin:
    def process(self, arg):
        return arg.split();
    def curl(self, url):
        return requests.get(url);

In MyPlugin.java, we can find the same methods:

public interface MyPlugin {
    String[] process(String arg);
    PyObject curl(String url);
}

Now here is the magic:

package fr.layer4.jpy;

import org.jpy.PyInputMode;
import org.jpy.PyLib;
import org.jpy.PyModule;
import org.jpy.PyObject;
import org.springframework.util.FileSystemUtils;
import org.springframework.util.ResourceUtils;
import org.springframework.util.StringUtils;

import java.io.FileInputStream;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Properties;

public class Main {

    public static void main(String... args) throws Exception {

        PyLib.Diag.setFlags(PyLib.Diag.F_OFF);

        // Prepare required system properties like 'jpy.jpyLib' and others
        Properties properties = new Properties();
        properties.load(new FileInputStream(ResourceUtils.getFile("classpath:jpyconfig.properties")));
        properties.forEach((k, v) -> System.setProperty((String) k, (String) v));

        if (!PyLib.isPythonRunning()) {
            List<String> extraPaths = Arrays.asList(
                    "classpath:deps-bundle.zip", // REQUIRED: contains all the dependencies
                    "classpath:my_plugin.py" // OPTIONAL: my custom scripts
            );
            List<String> cleanedExtraPaths = new ArrayList<>(extraPaths.size());

            Path tempDirectory = Files.createTempDirectory("lib-");
            Runtime.getRuntime().addShutdownHook(new Thread(() -> FileSystemUtils.deleteRecursively(tempDirectory.toFile())));
            cleanedExtraPaths.add(tempDirectory.toString());

            extraPaths.forEach(lib -> {
                if (lib.startsWith("classpath:")) {
                    try {
                        String finalLib = lib.replace("classpath:", "");
                        Path target = Paths.get(tempDirectory.toString(), finalLib);
                        try (InputStream stream = Main.class.getClassLoader().getResourceAsStream(finalLib)) {
                            Files.copy(stream, target, StandardCopyOption.REPLACE_EXISTING);
                        }
                        if (finalLib.endsWith(".zip")) {
                            ZipUtils.extract(target.toFile(), tempDirectory.toFile());
                        }
                    } catch (Exception e) {
                        e.printStackTrace();
                    }
                } else {
                    cleanedExtraPaths.add(lib);
                }
            });

            PyLib.startPython(cleanedExtraPaths.toArray(new String[]{}));
        }

        try {
            // Exec a python script
            PyObject.executeCode("print 'this is from python'", PyInputMode.SCRIPT);

            // Proxify the call to a python class
            PyModule plugInModule = PyModule.importModule("my_plugin");
            PyObject plugInObj = plugInModule.call("MyPlugin");
            MyPlugin plugIn = plugInObj.createProxy(MyPlugin.class);

            String[] results = plugIn.process("AAA bbb c ddd");
            System.err.println(StringUtils.arrayToCommaDelimitedString(results));

            PyObject response = plugIn.curl("https://api.github.com/users/treydone");
            System.err.println(response);
            System.err.println(response.getAttribute("status_code").getIntValue()); // r.status_code
            System.err.println(response.getAttribute("text").getStringValue()); // r.text
            System.err.println(response.call("json").call("get", "login").getStringValue()); // r.json['login']
            System.err.println(response.getAttribute("headers").call("get", "content-type").getStringValue()); // r.headers['content-type']
        } finally {
            PyLib.Diag.setFlags(PyLib.Diag.F_OFF);
            PyLib.stopPython();
        }
    }
}

Many lines of code, I agree. All this configuration, just for this:

PyModule plugInModule = PyModule.importModule("my_plugin");
PyObject plugInObj = plugInModule.call("MyPlugin");
MyPlugin plugIn = plugInObj.createProxy(MyPlugin.class);

With JPY you can “implement” Java interfaces using Python by proxifying a Java interface with Python modules or classes. If you call methods of the resulting Java object, jpy will delegate the calls to the matching Python module functions or class methods.

Package the whole project as a fat-jar, run it, and you should see these outputs in the console:

> java -cp target/jpy-test-1.0-SNAPSHOT.jar:/home/devil/git/jpy/build/lib.linux-x86_64-2.7 fr.layer4.jpy.Main

this is from python
AAA,bbb,c,ddd
200
{"login":"Treydone","id":1038029,"avatar_url":"https://avatars.githubusercontent.com/u/1038029?v=3","gravatar_id":"","url":"https://api.github.com/users/Treydone","html_url":"https://github.com/Treydone","followers_url":"https://api.github.com/users/Treydone/followers","following_url":"https://api.github.com/users/Treydone/following{/other_user}","gists_url":"https://api.github.com/users/Treydone/gists{/gist_id}","starred_url":"https://api.github.com/users/Treydone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Treydone/subscriptions","organizations_url":"https://api.github.com/users/Treydone/orgs","repos_url":"https://api.github.com/users/Treydone/repos","events_url":"https://api.github.com/users/Treydone/events{/privacy}","received_events_url":"https://api.github.com/users/Treydone/received_events","type":"User","site_admin":false,"name":"Vincent Devillers","company":"Layer4","blog":"https://blog.layer4.fr","location":"Paris","email":null,"hireable":true,"bio":null,"public_repos":26,"public_gists":9,"followers":23,"following":22,"created_at":"2011-09-09T08:13:08Z","updated_at":"2017-02-01T08:40:52Z"}
Treydone
application/json; charset=utf-8

Great!

Here is the pom.xml used for this project:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>fr.layer4</groupId>
    <artifactId>jpy-test</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>org.jpy</groupId>
            <artifactId>jpy</artifactId>
            <version>0.9-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-core</artifactId>
            <version>4.3.3.RELEASE</version>
        </dependency>
    </dependencies>

    <build>
        <resources>
            <resource>
                <directory>src/main/python</directory>
                <includes>
                    <include>*.py</include>
                </includes>
            </resource>
            <resource>
                <directory>src/main/resources</directory>
                <includes>
                    <include>*</include>
                </includes>
            </resource>
            <resource>
                <directory>${project.build.directory}</directory>
                <includes>
                    <include>deps-bundle.zip</include>
                </includes>
            </resource>
        </resources>

        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>1.3.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>exec</goal>
                        </goals>
                        <phase>initialize</phase>
                        <configuration>
                            <workingDirectory>${project.build.directory}/dist</workingDirectory>
                            <executable>pip</executable>
                            <arguments>
                                <argument>install</argument>
                                <argument>-r</argument>
                                <argument>${basedir}/requirements.txt</argument>
                                <argument>--target=${project.build.directory}/dist</argument>
                            </arguments>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.5.4</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>assembly</goal>
                        </goals>
                        <phase>initialize</phase>
                        <configuration>
                            <descriptors>
                                <descriptor>src/main/assembly/zip.xml</descriptor>
                            </descriptors>
                            <finalName>deps</finalName>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.0.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <transformers>
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>fr.layer4.jpy.Main</mainClass>
                                </transformer>
                            </transformers>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>jpyconfig.properties</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

All the sources can be found on Github

Sources:
https://github.com/bcdev/jpy
http://jpy.readthedocs.io/en/latest/intro.html#current-limitations
http://docs.python-requests.org/en/master/

Credits:
“Snake on Grey Wood” by Pixabay is licensed under CC0 1.0 / Resized

Published the