Compile external library for Python

Owned by Mina Hiruma (比留間 美菜） (Unlicensed)
Last updated: Aug 22, 2024 by Tachibana Yuta (立花 裕太)
11 min read

 

Introduction

Please follow the steps below if you are using SDK V1.61 or earlier version.
This modification will be incorporated in the next SDK version update.

  • Please delete the following part in “external/build/sample/docker/Dockerfile”.
    6th line: ENV http_proxy=http://proxy.mei.co.jp:8080/ https_proxy=http://proxy.mei.co.jp:8080/

  • Please modify the following parts in “external/build/sample/docker/docker_volume/module_compile.sh”

    4th line:
    　Before：NUMPY_ARCHIVES=v1.19.4.tar.gz
    　After：NUMPY_ARCHIVES=numpy-1.19.4.tar.gz
    8th line:
    　Before: OPENCV_ARCHIES=opencv-3.4.7.zip
    　After: OPENCV_ARCHIVES=opencv-3.4.7.zip

 

Please follow the steps below if you are using SDK V1.70 or earlier version.
This modification will be incorporated in the next SDK version update.

  • Please delete the following part in “external/build/sample/docker/Dockerfile”.

    4th line:
    　Before：RUN python -m pip install Cython
    　After：RUN python -m pip install Cython==0.29.36

 

Also, this tutorial refers to the SDK installation directory as ${SDK_DIR}.

Building external libraries is not supported. Please build the external library by referring to the information provided in this chapter. Please note that we cannot provide support even if you contact us.

 

Table of contents

 

Overview

Python module needs to be compiled when using external library with Python version AdamApp.

Here, we navigate how to build external library for Python from the environment building to module compilation.

External library used in this tutorial

  • NumPy 1.14.9

  • OpenCV 3.4.7

  • sqlite 3.40.0

  • Python 3.7.9

  • pycurl 7.45.2

  • OpenSSL 1.1.1

 

Establish Docker environment

Docker environment is built over the Ubuntu to which AdamAppSDK is installed.

OS used in this tutorial for installation

  • SDK v1.71 and earlier : Ubuntu 18.04.2 LTS (bionic)

  • SDK v1.80 or later : Ubuntu 22.04.3 LTS (Jammy Jellyfish)

AdamAppSDK development environment must be built in advance. (This tutorial does not include this step)

If you have already created an AdamAppSDK development environment using SDK ver.1.80 or later, there are some duplicate steps. Please omit unnecessary parts as appropriate. Also, please note that the Docker image created using .devcontainer/Dockerfile when building the AdamAppSDK development environment and the Docker image explained in this chapter are different Docker images.

In addition, as the installation require the use of apt-get command and such, prepare the Internet environment that allows you to run “apt-get”, and setup management setting such as root privilege for installation.

Reference: Docker official document

For VirtualBox environments, Docker compilation may fail on host-guest shared directories. If it fails, try working in a directory on Ubuntu.

 

Install the package

Install the package required to establish Docker environment.

$ sudo apt-get update $ sudo apt-get install \ apt-transport-https \ ca-certificates \ curl \ gnupg-agent \ software-properties-common \ qemu-user-static

 

Add repository(When using the apt-key command)

Add apt repository for Ubuntu provided by Docker to your system.

Add Docker official PGP key to your system.

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - OK

Make sure the fingerprint looks like the following after adding PGP key.

$ sudo apt-key fingerprint 0EBFCD88 pub rsa4096 2017-02-22 [SCEA] 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88 uid [ unknown ] Docker Release (CE deb) docker@docker.com sub rsa4096 2017-02-22 [S]

Then, add Docker official apt repository to your system.

Check if you successfully added apt repository to “/etc/apt/sources.list”.

 

Add repository(When not using the apt-key command)

Add apt repository for Ubuntu provided by Docker to your system.

Create a keyrings directory in the /etc/apt directory in advance.

Add Docker official PGP key to your system.

Then, add Docker official apt repository to your system.

 

Install Docker execution file

Check if the system is up-to-date and then install Docker using the following apt-get command.

 

(Optional) Proxy setting of Docker

You may need proxy setting for the Docker if you need a proxy server to connect to the Internet.

Reference: Docker official document

 

Change the setting for Docker command

Change the setting for Docker command in order to make it available to general users.

Log out and log in again to apply the setting change.

 

Confirm if Docker runs successfully

Check the installed Docker can be run in your system using the following command.

 

Create Docker image

Create a Docker image used to compile Python module

 

Edit Dockerfile

You need to create a file called Dockerfile in which the program to create an image is written when creating a Docker image.

We provide you with a Dockerfile template to create necessary image for compiling NumPy and OpenCV module with ease.

external/build/sample/docker/Dockerfile

 

Build an image from a Dockerfile

Docker image is built within the directory where Dockerfile is placed.

Check the directory to see if a repository “compile/ubuntu-rel” appears when you finish building the image.

 

How to install qemu

qemu must be installed in the development environment in order to create container version additional apps. Describe how to install qemu. If you have completed creating the Docker image, there is no need to install qemu, so please proceed to the next chapter.

 

install qemu

 

The third line also has a means to do the following:

The official document (Japanese) as of June 2023 guides here.

Docker Buildx | Docker ドキュメント (matsuand.github.io)

 

Preparing docker buildx

  • Add and reflect environment variables to use experimental features and BuildKit.

  • Open configuration file.

  • Add the following to the end

    Save file.

  • Check if buildx is available.

    Output example

     

  • Check the currently buildable architectures.

    Output example

    If you see linux/arm64 in this output, success.

 

Compile Python module(Numpy, OpenCV)

Steps

Follow the steps below to compile Python module for AdamApp

  1. Place source file in “${SDK_DIR}/external/build/sample/docker/docker_volume” directory

  2. Write compilation program in “${SDK_DIR}/external/build/sample/docker/docker_volume/module_compile.sh”.

  3. Run docker and proceed compilation.

  4. The compiled module will be placed in “${SDK_DIR}/external/build/sample/docker/docker_volume/install_path” directory.

Here we take you through NumPy and OpenCV compilation as a tutorial.

 

Download source code

Download Numpy and OpenCV source code from the URLs below.

This tutorial uses Numpy 1.14.9, OpenCV 3.4.7.
Select the version to download according to the functions you use.

Numpy 1.14.9:

https://github.com/numpy/numpy/archive/v1.19.4.tar.gz

OpenCV 3.4.7:

https://github.com/opencv/opencv/archive/3.4.7.zip

Place the downloaded files to “${SDK_DIR}/external/build/sample/docker/docker_volume” directory.

 

Update compilation program

Write compilation program in the file: “${SDK_DIR}/external/build/sample/docker/docker_volume/module_compile.sh”.

”module_compile.sh” is provided as a sample script to compile Numpy and OpenCV.

Here we proceed without changing “module_compile.sh”.

 

Compile

Compile NumPy module and OpenCV module using docker.

Run Docker in “${SDK_DIR}/external/build/sample/docker” directory.

The compiled module is created in “external/build/sample/docker/docker_volume/install_path”.

 

Deployment in AdamApp

Deploy the modules to use them with AdamApp.

Here we take “${SDK_DIR}/src/adamapp-py/jpeg_app” as an example.

 

Deploy Numpy

NumPy module is installed in the following directory.

${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/site-packages/numpy-1.19.4-py3.7-linux-aarch64.egg/numpy

Copy and paste to AdamApp source directory without changing the directory structure.

${SDK_DIR}/src/adamapp-py/jpeg_app/python/site-packages

Then, the directory structure will be as follows.

${SDK_DIR}/src/adamapp-py/jpeg_app/python/site-packages/numpy

 

Deploy OpenCV

OpenCV module is installed in the following directory.

${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/dist-packages/cv2

Copy and paste to AdamApp source directory without changing the directory structure.

${SDK_DIR}/src/adamapp-py/jpeg_app/python/site-packages

Then, the directory structure will be as follows.

${SDK_DIR}/src/adamapp-py/jpeg_app/python/site-packages/cv2

Deploy both Open CV library and Python module as Open CV requires them both.

OpenCV library is installed in the following directory.

${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib

Copy “libopencv_world.so.3.4.7” and paste it to the following directory.

例）${SDK_DIR}/src/adamapp-py/jpeg_app/external/lib/aarch64-linux-gnu/

Then create a symbolic link.

 

Delete files no longer needed

AdamApp does not use (_pycache_), Python cash file.

We recommend that you delete the files to reduce the folder size.

In addition, the information needed for debugging library is included in the OpenCV library you created.

We recommend that you delete this file to reduce the folder size as you do not need this to run the application.

 

Build the sample application

Build the sample application "jpeg_app" and install it on the camera (eg, it can be installed from the green frame in the image below).

Please perform the build method according to each SDK version.

Open the app screen (red frame button in the image below), and if the jpeg image taken by the camera is displayed, it is successful.

 

Compile Python module(SQLite)

Numpy and OpenCV were compiled. As the next tutorial, we will compile SQLite.

 

Download source code

Download sqlite and Python source code from the URLs below.

This tutorial uses sqlite 3.40.0, Python 3.7.9.
Select the version to download according to the functions you use.

sqlite 3.40.0

https://www.sqlite.org/2022/sqlite-autoconf-3400000.tar.gz

Python 3.7.9

https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tgz

Place the downloaded files to “${SDK_DIR}/external/build/sample/docker/docker_volume” directory.

 

Update compilation instructions file

Put the compilation instructions in “${SDK_DIR}/external/build/sample/docker/docker_volume/module_compile.sh” file.

Please copy the content below.

 

Compile

Compile sqlite module and Python module using docker.

Run Docker in “${SDK_DIR}/external/build/sample/docker” directory.

The compiled module is created in “external/build/sample/docker/docker_volume/install_path”.

 

Deployment in AdamApp

Deploy the modules to use them with AdamApp.

Here we take “${SDK_DIR}/src/adamapp-py/sqlite_app” as an example. 

 

Deploy SQLite

SQLite module is installed in the following directory.

${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib

libsqlite3.so
libsqlite3.so.0
libsqlite3.so.0.8.6
libsqlite3.la
libsqlite3.a

Copy the above 5 files to the AdamApp source directory.

${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu

Then, the directory structure will be as follows.

${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu/libsqlite3.so
${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu/libsqlite3.so.0
${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu/libsqlite3.so.0.8.6
${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu/libsqlite3.la
${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu/libsqlite3.a

 

Deploy sqlite for Python

sqlite for Python module is installed in the following directory.

${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/sqlite3

Copy and paste to AdamApp source directory without changing the directory structure.

${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages

Then, the directory structure will be as follows.

${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages/sqlite3

sqlite for Python requires not only modules but also libraries, so place them together.

The sqlite for Python library is installed in the following directory.

${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/lib-dynload

Copy “_sqlite3.cpython-37m-aarch64-linux-gnu.so” to the following directory.

${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages/sqlite3

Then, the directory structure will be as follows.

${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages/sqlite3/_sqlite3.cpython-37m-aarch64-linux-gnu.so

Next, modify dbapi2.py so that the sqlite for Python library can be loaded when importing.

${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages/sqlite3/dbapi2.py

Modify as follows.

Before) from _sqlite3 import *
↓
After) from sqlite3._sqlite3 import *

 

Build the sample application

Before building the sample app, format the SD card using the screen below. (Please confirm in advance whether the SD card can be formatted. All data on the SD card will be erased.)
Once formatting is complete, the SD card as ext (ext4) and press the Set button.

 

Build the sample application "sqlite_app" and install it on the camera (eg, it can be installed from the green frame in the image below).

Please perform the build method according to each SDK version.

Open the app screen (red frame button in the image below),

and if the following screen is displayed, it is successful.

 

Compile Python module(Pycurl)

SQLite was compiled. As the next tutorial, we will compile Pycurl.

Download source code

Download PyCurl and OpenSSL source code from the URLs below.

This tutorial uses pycurl 7.45.2, OpenSSL 1.1.1.
Select the version to download according to the functions you use.

pycurl 7.45.2

https://files.pythonhosted.org/packages/a8/af/24d3acfa76b867dbd8f1166853c18eefc890fc5da03a48672b38ea77ddae/pycurl-7.45.2.tar.gz

OpenSSL 1.1.1

https://github.com/openssl/openssl/archive/OpenSSL_1_1_1i.tar.gz

Place the downloaded files to “${SDK_DIR}/external/build/sample/docker/docker_volume” directory.

 

Update compilation instructions file

Put the compilation instructions in “${SDK_DIR}/external/build/sample/docker/docker_volume/module_compile.sh” file.

Please copy the content below.

 

Compile

Compile pycurl module and OpenSSL module using docker.

Run Docker in “${SDK_DIR}/external/build/sample/docker” directory.

The compiled module is created in “external/build/sample/docker/docker_volume/install_path”.

 

Deployment in AdamApp

Deploy the modules to use them with AdamApp.

Here we take “${SDK_DIR}/src/adamapp-py/hogehoge_app” as an example.

 

Deploy PyCurl

PyCurl module is installed in the following directory.

${SDK_DIR}/external/build/sample/docker/docker_volume/pycurl-7.45.2/build/lib.linux-aarch64-3.7

pycurl.cpython-37m-aarch64-linux-gnu.so

Copy the above file to the AdamApp source directory.

${SDK_DIR}/src/adamapp-py/hogehoge_app/python/site-packages

Then, the directory structure will be as follows.

${SDK_DIR}/src/adamapp-py/hogehoge_app/python/site-packages/pycurl.cpython-37m-aarch64-linux-gnu.so

 

Modify the source code to load PyCurl

Open “${SDK_DIR}/src/adamapp-py/hogehoge_app/python/pymain.py” and add as follows.

import os;

Add the following one line below.

import pycurl;

 

Build the sample application

Build the sample application "hogehoge_app" and install it on the camera (eg, it can be installed from the green frame in the image below).

Please perform the build method according to each SDK version.

 

Compile a Python module (Numba)

This article describes the steps to build Numba, which is used to speed up Python.

 

Download source code

Several modules are required to run Numba. Please download each source code from the URL below. Also, please select a version of each module that is compatible with Python 3.7.9.

Numba 0.56.4:
https://github.com/numba/numba/archive/refs/tags/0.56.4.zip

Numpy 1.19.4:
https://github.com/numpy/numpy/archive/v1.19.4.tar.gz

LLVM 11.1.0:
https://github.com/llvm/llvm-project/archive/refs/tags/llvmorg-11.1.0.tar.gz

llvmlite 0.39.1:
https://github.com/numba/llvmlite/archive/refs/tags/v0.39.1.tar.gz

importlib_metadata 6.7.0:
https://github.com/python/importlib_metadata/archive/refs/tags/v6.7.0.zip

typing_extensions 4.7.1:
https://github.com/python/typing_extensions/archive/refs/tags/4.7.1.zip

zipp 3.15.0:
https://github.com/jaraco/zipp/archive/refs/tags/v3.15.0.zip

Place the downloaded file in the following directory. ${SDK_DIR}/external/build/sample/docker/docker_volume

 

Update the compilation instructions file

The compilation procedure is described in the file below.
${SDK_DIR}/external/build/sample/docker/docker_volume/module_compile.sh

Please copy the content below.

 

compile

Compile each module using Docker.
Run Docker in the directory below.
${SDK_DIR}/external/build/sample/docker

Compiled modules are placed in the following directories.
${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/site-packages

 

Place in AdamApp

Place the created module so that it can be used by AdamApp.
The tutorial uses the following as an example.
${SDK_DIR}/src/adamapp-py/yuv_yolov5_app

 

Place the module

Copy the compiled module to the AdamApp source directory, keeping the directory structure as is.

$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/site-packages ${SDK_DIR}/src/adamapp-py/yuv_yolov5_app/python/

update pymain.py

Change the source code to be compatible with numba. Replace the attached file "pymain.py" with the file below.
\src\adamapp-py\yuv_yolov5_app\python\pymain.py

Build the sample app

Build the sample app "yuv_yolov5_app" and install it on the camera. Please perform the build method according to each SDK version. When you open the app screen, if there is a frame around the object (such as a person) on the screen, it is a success.

Please also refer to the operation explanation of the sample application.

 

If it fails, a message will be displayed on the application screen indicating that the application has failed to start. From now on, please use the message content and debugging tools to solve the problem and create an application.