Introduction

Please follow the steps below if you are using SDK V1.61 or earlier version.
This modification will be incorporated in the next SDK version update.

Please delete the following part in “external/build/sample/docker/Dockerfile”.
6th line: ENV http_proxy=http://proxy.mei.co.jp:8080/ https_proxy=http://proxy.mei.co.jp:8080/
Please modify the following parts in “external/build/sample/docker/docker_volume/module_compile.sh”
4th line:
　Before：NUMPY_ARCHIVES=v1.19.4.tar.gz
　After：NUMPY_ARCHIVES=numpy-1.19.4.tar.gz
8th line:
　Before: OPENCV_ARCHIES=opencv-3.4.7.zip
　After: OPENCV_ARCHIVES=opencv-3.4.7.zip

Please follow the steps below if you are using SDK V1.70 or earlier version.
This modification will be incorporated in the next SDK version update.

Please delete the following part in “external/build/sample/docker/Dockerfile”.
4th line:
　Before：RUN python -m pip install Cython
　After：RUN python -m pip install Cython==0.29.36

Also, this tutorial refers to the SDK installation directory as ${SDK_DIR}.

Overview

Python module needs to be compiled when using external library with Python version AdamApp.

Here, we navigate how to build external library for Python from the environment building to module compilation.

External library used in this tutorial

NumPy 1.14.9
OpenCV 3.4.7
sqlite 3.40.0
Python 3.7.9
pycurl 7.45.2
OpenSSL 1.1.1

Establish Docker environment

Docker environment is built over the Ubuntu to which AdamAppSDK is installed.

OS used in this tutorial for installation

Ubuntu 18.04.2 LTS (bionic)

AdamAppSDK development environment must be built in advance. (This tutorial does not include this step)

In addition, as the installation require the use of apt-get command and such, prepare the Internet environment that allows you to run “apt-get”, and setup management setting such as root privilege for installation.

Reference: Docker official document

For VirtualBox environments, Docker compilation may fail on host-guest shared directories. If it fails, try working in a directory on Ubuntu.

Install the package

Install the package required to establish Docker environment.

$ sudo apt-get update
$ sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common \
    qemu-user-static

Add repository

Add apt repository for Ubuntu provided by Docker to your system.

Add Docker official PGP key to your system

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
OK

Use -x option if you need a proxy server to connect to the Internet.

（Change URL and port number for the proxy server according to the proxy server you are using.)

$ curl -x http://proxy.example.com:8080/ -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Make sure the fingerprint looks like the following after adding PGP key.

$ sudo apt-key fingerprint 0EBFCD88
pub   rsa4096 2017-02-22 [SCEA]
      9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
uid           [ unknown ] Docker Release (CE deb) docker@docker.com
sub   rsa4096 2017-02-22 [S]

Then, add Docker official apt repository to your system.

$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

Add environment variable “http_proxy” and “https_proxy” as indicated below if you need a proxy server to connect to the Internet.

(Change URL and port number for the proxy server according to the proxy server you are using.)

$ sudo http_proxy=http://proxy.example.com:8080 https_proxy=https://proxy.example.com:8080 add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

Check if you successfully added apt repository to “/etc/apt/sources.list”.

$ cat /etc/apt/sources.list
(Omitted)
deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable
# deb-src [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable

Install Docker execution file

Check if the system is up-to-date and then install Docker using the following apt-get command.

$ sudo apt-get update
$ sudo apt-get install docker-ce docker-ce-cli containerd.io

(Optional) Proxy setting of Docker

You may need proxy setting for the Docker if you need a proxy server to connect to the Internet.

Reference: Docker official document

Change the setting for Docker command

Change the setting for Docker command in order to make it available to general users.

$ sudo usermod -a -G docker $(whoami)

Log out and log in again to apply the setting change.

Confirm if Docker runs successfully

Check the installed Docker can be run in your system using the following command.

$ docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Create Docker image

Create a Docker image used to compile Python module

Edit Dockerfile

You need to create a file called Dockerfile in which the program to create an image is written when creating a Docker image.

We provide you with a Dockerfile template to create necessary image for compiling NumPy and OpenCV module with ease.

external/build/sample/docker/Dockerfile

FROM arm64v8/ubuntu:bionic

ADD qemu-aarch64-static /usr/bin
ADD exec_entry.sh /bin

RUN apt-get update \
 && apt-get install -y --no-install-recommends \
    build-essential gcc-8 g++-8 gfortran-8 python3.7 python3-distutils python3-setuptools python3.7-dev python3-pip cmake unzip \
 && apt-get -y clean && rm -rf /var/lib/apt/lists/*
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 800 --slave /usr/bin/g++ g++ /usr/bin/g++-8 \
 && update-alternatives --install /usr/bin/gfortran gfortran /usr/bin/gfortran-8 800 \
 && update-alternatives --install /usr/bin/python python /usr/bin/python3.7 800
RUN python -m pip install Cython
RUN chmod +x /bin/exec_entry.sh

ENTRYPOINT ["/bin/exec_entry.sh"]

※Do not change 1st, 3rd, 4th, 14th, and 16th line above when editing Dockerfile.

Add the following lines before

“RUN apt-get update \”

if you need a proxy server to connect to the Internet.

（Change URL and port number for the proxy server according to the proxy server you are using.)

ENV http_proxy=http://proxy.example.com:8080/ https_proxy=http://proxy.example.com:8080/

Build an image from a Dockerfile

Docker image is built within the directory where Dockerfile is placed.

$ cd ${SDK_DIR}/external/build/sample/docker/
$ docker build -t compile/ubuntu-rel:0.1 .

If it fails with an error, please do the following:
How to install qemu - Technology Partner FAQ (En) - Confluence (atlassian.net)

Check the directory to see if a repository “compile/ubuntu-rel” appears when you finish building the image.

$ docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
compile/ubuntu-rel   0.1                 ca4409cf01fd        45 minutes ago      559MB

Make sure you see the following two files in the directory where Dockerfile is placed.

exec_entry.sh
qemu-aarch64-static

Do not delete these files as you will need them when creating Python module.

Compile Python module(Numpy, OpenCV)

Steps

Follow the steps below to compile Python module for AdamApp

Place source file in “${SDK_DIR}/external/build/sample/docker/docker_volume” directory
Write compilation program in “${SDK_DIR}/external/build/sample/docker/docker_volume/module_compile.sh”.
Run docker and proceed compilation.
The compiled module will be placed in “${SDK_DIR}/external/build/sample/docker/docker_volume/install_path” directory.

Here we take you through NumPy and OpenCV compilation as a tutorial.

The compilation fails if you work within shared folder over VirtualBox environment.
Create and move the folders to Ubuntu before you start.

Download source code

Download Numpy and OpenCV source code from the URLs below.

This tutorial uses Numpy 1.14.9, OpenCV 3.4.7.
Select the version to download according to the functions you use.

Numpy 1.14.9:
https://github.com/numpy/numpy/archive/v1.19.4.tar.gz
OpenCV 3.4.7:
https://github.com/opencv/opencv/archive/3.4.7.zip

Place the downloaded files to “${SDK_DIR}/external/build/sample/docker/docker_volume” directory.

Update compilation program

Write compilation program in the file: “${SDK_DIR}/external/build/sample/docker/docker_volume/module_compile.sh”.

Be aware the following points when updating the compilation program.

”${SDK_DIR}/external/build/sample/docker/docker_volume” will be mounted as “/home/docker” after you run docker.
Source files and compile directory are also placed in ”/home/docker”.
Install compiled modules to “/home/docker/install_path”.
You need NumPy module to compile OpenCV.

”module_compile.sh” is provided as a sample script to compile Numpy and OpenCV.

Here we proceed without changing “module_compile.sh”.

Compile

Compile NumPy module and OpenCV module using docker.

Run Docker in “${SDK_DIR}/external/build/sample/docker” directory.

$ docker run --rm -it --name aarch64-ubuntu -v `pwd`/docker_volume:/home/docker compile/ubuntu-rel:0.1

The compiled module is created in “external/build/sample/docker/docker_volume/install_path”.

Be patient while compiling OpenCV, it may take a while.
If a compilation error occurs with SDK V1.70 or earlier, please confirm that the external/build/sample/docker/Dockerfile described in the "Introduction" section has been corrected. If it is not corrected, start over from creating the Docker image.

Deployment in AdamApp

Deploy the modules to use them with AdamApp.

Here we take “${SDK_DIR}/src/adamapp-py/jpeg_app” as an example.

Deploy Numpy

NumPy module is installed in the following directory.

${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/site-packages/numpy-1.19.4-py3.7-linux-aarch64.egg/numpy

Copy and paste to AdamApp source directory without changing the directory structure.

${SDK_DIR}/src/adamapp-py/jpeg_app/python/site-packages

$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/site-packages/numpy-1.19.4-py3.7-linux-aarch64.egg/numpy src/adamapp-py/jpeg_app/python/site-packages

Then, the directory structure will be as follows.

${SDK_DIR}/src/adamapp-py/jpeg_app/python/site-packages/numpy

Deploy OpenCV

OpenCV module is installed in the following directory.

${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/dist-packages/cv2

Copy and paste to AdamApp source directory without changing the directory structure.

${SDK_DIR}/src/adamapp-py/jpeg_app/python/site-packages

$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/dist-packages/cv2 src/adamapp-py/jpeg_app/python/site-packages

Then, the directory structure will be as follows.

${SDK_DIR}/src/adamapp-py/jpeg_app/python/site-packages/cv2

Deploy both Open CV library and Python module as Open CV requires them both.

OpenCV library is installed in the following directory.

${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib

Copy “libopencv_world.so.3.4.7” and paste it to the following directory.

Please create manually after the external folder.

例）${SDK_DIR}/src/adamapp-py/jpeg_app/external/lib/aarch64-linux-gnu/

$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/libopencv_world.so.3.4.7 src/adamapp-py/jpeg_app/external/external/lib/aarch64-linux-gnu

Then create a symbolic link.

$ cd ${SDK_DIR}/src/adamapp-py/jpeg_app/external/lib/aarch64-linux-gnu/
$ ln -s libopencv_world.so.3.4.7 libopencv_world.so.3.4
$ ln -s libopencv_world.so.3.4 libopencv_world.so

Delete files no longer needed

AdamApp does not use (_pycache_), Python cash file.

We recommend that you delete the files to reduce the folder size.

$ cd ${SDK_DIR}/src/adamapp-py/yuv_yolo_app/python/site-packages/numpy/
$ find . -name *pycache* -print | xargs rm -rf

In addition, the information needed for debugging library is included in the OpenCV library you created.

We recommend that you delete this file to reduce the folder size as you do not need this to run the application.

$ cd ${SDK_DIR}/src/adamapp-py/yuv_yolo_app/external/lib/aarch64-linux-gnu/
$ aarch64-linux-gnu-strip -strip-debug libopencv_world.so.3.4.7

Build the sample application

Build the sample application "jpeg_app" and install it on the camera (eg, it can be installed from the green frame in the image below).

See here for how to build.

Open the app screen (red frame button in the image below), and if the jpeg image taken by the camera is displayed, it is successful.

Compile Python module(SQLite)

Numpy and OpenCV were compiled. As the next tutorial, we will compile SQLite.

Download source code

Download sqlite and Python source code from the URLs below.

This tutorial uses sqlite 3.40.0, Python 3.7.9.
Select the version to download according to the functions you use.

sqlite 3.40.0
https://www.sqlite.org/2022/sqlite-autoconf-3400000.tar.gz
Python 3.7.9
https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tgz

Place the downloaded files to “${SDK_DIR}/external/build/sample/docker/docker_volume” directory.

Update compilation instructions file

Put the compilation instructions in “${SDK_DIR}/external/build/sample/docker/docker_volume/module_compile.sh” file.

Please copy the content below.

#!/bin/sh -x

HOME_PATH=/home/docker

LIB_INSTALL_DIR=${HOME_PATH}/install_path

SQLITE_ARCHIVES=sqlite-autoconf-3400000.tar.gz
SQLITE_DIR=sqlite-autoconf-3400000
SQLITE_BUILD_DIR=sqlite-autoconf-3400000/build

PYTHONPATH=$PYTHONPATH:${NUMPY_SITE_PACKAGES}
export PYTHONPATH

PYTHON_ARCHIVES=Python-3.7.9.tgz
PYTHON_DIR=Python-3.7.9

## for sqlite
cd ${HOME_PATH}
tar zxf ${SQLITE_ARCHIVES}
mkdir -p ${SQLITE_BUILD_DIR}
cd ${SQLITE_DIR}
./configure --prefix=${LIB_INSTALL_DIR}
make
make install

## for python
cd ${HOME_PATH}
tar zxf ${PYTHON_ARCHIVES}
cd ${PYTHON_DIR}
./configure --prefix=${LIB_INSTALL_DIR}
make
make install

Compile

Compile sqlite module and Python module using docker.

Run Docker in “${SDK_DIR}/external/build/sample/docker” directory.

$ docker run --rm -it --name aarch64-ubuntu -v `pwd`/docker_volume:/home/docker compile/ubuntu-rel:0.1

The compiled module is created in “external/build/sample/docker/docker_volume/install_path”.

Deployment in AdamApp

Deploy the modules to use them with AdamApp.

Here we take “${SDK_DIR}/src/adamapp-py/sqlite_app” as an example.

An SD card is required for sqlite_app to work. Only cameras that can insert an SD card can be checked. Also, depending on the firmware of the camera, it may not work unless the SD card is formatted with ext4.

Deploy SQLite

SQLite module is installed in the following directory.

${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib

libsqlite3.so
libsqlite3.so.0
libsqlite3.so.0.8.6
libsqlite3.la
libsqlite3.a

Copy the above 5 files to the AdamApp source directory.

${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu

$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/libsqlite3.so ${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu
$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/libsqlite3.so.0 ${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu
$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/libsqlite3.so.0.8.6 ${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu
$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/libsqlite3.la ${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu
$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/libsqlite3.a ${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu

Then, the directory structure will be as follows.

${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu/libsqlite3.so
${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu/libsqlite3.so.0
${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu/libsqlite3.so.0.8.6
${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu/libsqlite3.la
${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu/libsqlite3.a

Deploy sqlite for Python

sqlite for Python module is installed in the following directory.

${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/sqlite3

Copy and paste to AdamApp source directory without changing the directory structure.

${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages

$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/sqlite3 ${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages

Then, the directory structure will be as follows.

${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages/sqlite3

sqlite for Python requires not only modules but also libraries, so place them together.

The sqlite for Python library is installed in the following directory.

${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/lib-dynload

Copy “_sqlite3.cpython-37m-aarch64-linux-gnu.so” to the following directory.

${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages/sqlite3

$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/lib-dynload/_sqlite3.cpython-37m-aarch64-linux-gnu.so ${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages/sqlite3

Then, the directory structure will be as follows.

${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages/sqlite3/_sqlite3.cpython-37m-aarch64-linux-gnu.so

Next, modify dbapi2.py so that the sqlite for Python library can be loaded when importing.

${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages/sqlite3/dbapi2.py

Modify as follows.

Before) from _sqlite3 import *
↓
After) from sqlite3._sqlite3 import *

Build the sample application

Build the sample application "sqlite_app" and install it on the camera (eg, it can be installed from the green frame in the image below).

See here for how to build.

Open the app screen (red frame button in the image below),

and if the following screen is displayed, it is successful.

Compile Python module(Pycurl)

SQLite was compiled. As the next tutorial, we will compile Pycurl.

Download source code

Download PyCurl and OpenSSL source code from the URLs below.

This tutorial uses pycurl 7.45.2, OpenSSL 1.1.1.
Select the version to download according to the functions you use.

pycurl 7.45.2
https://files.pythonhosted.org/packages/a8/af/24d3acfa76b867dbd8f1166853c18eefc890fc5da03a48672b38ea77ddae/pycurl-7.45.2.tar.gz
OpenSSL 1.1.1
https://github.com/openssl/openssl/archive/OpenSSL_1_1_1i.tar.gz

Place the downloaded files to “${SDK_DIR}/external/build/sample/docker/docker_volume” directory.

Update compilation instructions file

Put the compilation instructions in “${SDK_DIR}/external/build/sample/docker/docker_volume/module_compile.sh” file.

Please copy the content below.

#!/bin/sh -x

HOME_PATH=/home/docker

OPENSSL_ARCHIVES=openssl-OpenSSL_1_1_1i.tar.gz
OPENSSL_DIR=openssl-OpenSSL_1_1_1i

PYCURL_ARCHIVES=pycurl-7.45.2.tar.gz
PYCURL_DIR=pycurl-7.45.2

apt update
apt install libcurl4-openssl-dev libssl-dev

cd ${HOME_PATH}
tar zxf ${OPENSSL_ARCHIVES}
cd ${OPENSSL_DIR}
./Configure linux-aarch64
make

export PYCURL_SSL_LIBRARY=openssl
export CPPFLAGS=-I${HOME_PATH}/${OPENSSL_DIR}/include/openssl
export LDFLAGS=-L${HOME_PATH}/${OPENSSL_DIR}

## for pycurl
cd ${HOME_PATH}
tar zxf ${PYCURL_ARCHIVES}
cd ${PYCURL_DIR}
python3.7 setup.py --with-openssl --openssl-dir=${HOME_PATH}/${OPENSSL_DIR} build

Compile

Compile pycurl module and OpenSSL module using docker.

Run Docker in “${SDK_DIR}/external/build/sample/docker” directory.

$ docker run --rm -it --name aarch64-ubuntu -v `pwd`/docker_volume:/home/docker compile/ubuntu-rel:0.1

The compiled module is created in “external/build/sample/docker/docker_volume/install_path”.

Deployment in AdamApp

Deploy the modules to use them with AdamApp.

Here we take “${SDK_DIR}/src/adamapp-py/additional_info_sample_app” as an example.

Deploy PyCurl

PyCurl module is installed in the following directory.

${SDK_DIR}/external/build/sample/docker/docker_volume/pycurl-7.45.2/build/lib.linux-aarch64-3.7

pycurl.cpython-37m-aarch64-linux-gnu.so

Copy the above file to the AdamApp source directory.

${SDK_DIR}/src/adamapp-py/additional_info_sample_app/python/site-packages

$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/pycurl-7.45.2/build/lib.linux-aarch64-3.7/pycurl.cpython-37m-aarch64-linux-gnu.so ${SDK_DIR}/src/adamapp-py/additional_info_sample_app/python/site-packages

Then, the directory structure will be as follows.

${SDK_DIR}/src/adamapp-py/additional_info_sample_app/python/site-packages/pycurl.cpython-37m-aarch64-linux-gnu.so

Modify the source code to load PyCurl

Open “${SDK_DIR}/src/adamapp-py/additional_info_sample_app/python/pymain.py” and add as follows.

import os;
Add the following one line below.
import pycurl;

Build the sample application

Build the sample application "additional_info_sample_app" and install it on the camera (eg, it can be installed from the green frame in the image below).

See here for how to build.

Open the app screen (red frame button in the image below), and if the string is displayed like "body", it is successful.

If it fails, a message will be displayed on the application screen indicating that the application has failed to start. From now on, please use the message content and debugging tools to solve the problem and create an application.

Compile external library for Python

Introduction

Table of contents

Overview

External library used in this tutorial

Establish Docker environment

OS used in this tutorial for installation

Install the package

Add repository

Install Docker execution file

(Optional) Proxy setting of Docker

Change the setting for Docker command

Confirm if Docker runs successfully

Create Docker image

Edit Dockerfile

Build an image from a Dockerfile

Compile Python module(Numpy, OpenCV)

Steps

Download source code

Update compilation program

Compile

Deployment in AdamApp

Deploy Numpy

Deploy OpenCV

Delete files no longer needed

Build the sample application

Compile Python module(SQLite)

Download source code

Update compilation instructions file

Compile

Deployment in AdamApp

Deploy SQLite

Deploy sqlite for Python

Build the sample application

Compile Python module(Pycurl)

Download source code

Update compilation instructions file

Compile

Deployment in AdamApp

Deploy PyCurl

Modify the source code to load PyCurl

Build the sample application