Introduction
Please follow the steps below if you are using SDK V1.61 or earlier version.
This modification will be incorporated in the next SDK version update.
Please delete the following part in “external/build/sample/docker/Dockerfile”.
6th line: ENV http_proxy=http://proxy.mei.co.jp:8080/ https_proxy=http://proxy.mei.co.jp:8080/Please modify the following parts in “external/build/sample/docker/docker_volume/module_compile.sh”
4th line:
Before:NUMPY_ARCHIVES=v1.19.4.tar.gz
After:NUMPY_ARCHIVES=numpy-1.19.4.tar.gz
8th line:
Before: OPENCV_ARCHIES=opencv-3.4.7.zip
After: OPENCV_ARCHIVES=opencv-3.4.7.zip
Please follow the steps below if you are using SDK V1.70 or earlier version.
This modification will be incorporated in the next SDK version update.
Please delete the following part in “external/build/sample/docker/Dockerfile”.
4th line:
Before:RUN python -m pip install Cython
After:RUN python -m pip install Cython==0.29.36
Also, this tutorial refers to the SDK installation directory as ${SDK_DIR}.
Building external libraries is not supported. Please build the external library by referring to the information provided in this chapter. Please note that we cannot provide support even if you contact us.
Table of contents
Overview
Python module needs to be compiled when using external library with Python version AdamApp.
Here, we navigate how to build external library for Python from the environment building to module compilation.
External library used in this tutorial
NumPy 1.14.9
OpenCV 3.4.7
sqlite 3.40.0
Python 3.7.9
pycurl 7.45.2
OpenSSL 1.1.1
Establish Docker environment
Docker environment is built over the Ubuntu to which AdamAppSDK is installed.
OS used in this tutorial for installation
Ubuntu 18.04.2 LTS (bionic)
AdamAppSDK development environment must be built in advance. (This tutorial does not include this step)
In addition, as the installation require the use of apt-get command and such, prepare the Internet environment that allows you to run “apt-get”, and setup management setting such as root privilege for installation.
Reference: Docker official document
For VirtualBox environments, Docker compilation may fail on host-guest shared directories. If it fails, try working in a directory on Ubuntu.
Install the package
Install the package required to establish Docker environment.
$ sudo apt-get update $ sudo apt-get install \ apt-transport-https \ ca-certificates \ curl \ gnupg-agent \ software-properties-common \ qemu-user-static
Add repository(When using the apt-key command)
Add apt repository for Ubuntu provided by Docker to your system.
Add Docker official PGP key to your system.
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - OK
The apt-key command is scheduled to be deprecated. The abolishment date has not yet been determined, but please consider alternative methods on your own.
Use -x option if you need a proxy server to connect to the Internet.
(Change URL and port number for the proxy server according to the proxy server you are using.)
$ curl -x http://proxy.example.com:8080/ -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
Make sure the fingerprint looks like the following after adding PGP key.
$ sudo apt-key fingerprint 0EBFCD88 pub rsa4096 2017-02-22 [SCEA] 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88 uid [ unknown ] Docker Release (CE deb) docker@docker.com sub rsa4096 2017-02-22 [S]
Then, add Docker official apt repository to your system.
$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
Add environment variable “http_proxy” and “https_proxy” as indicated below if you need a proxy server to connect to the Internet.
(Change URL and port number for the proxy server according to the proxy server you are using.)
$ sudo http_proxy=http://proxy.example.com:8080 https_proxy=https://proxy.example.com:8080 add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
Check if you successfully added apt repository to “/etc/apt/sources.list”.
$ cat /etc/apt/sources.list (Omitted) deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable # deb-src [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable
Add repository(When not using the apt-key command)
The apt-key command is scheduled to be deprecated, so an example of an alternative method is provided. Please consider alternative methods on your own. Also, if you have already created a Docker image in an environment using the apt-key command, it may not work properly unless you create it again from the Docker container. please note.
Add apt repository for Ubuntu provided by Docker to your system.
Create a keyrings directory in the /etc/apt directory in advance.
$ mkdir /etc/apt/keyrings
Add Docker official PGP key to your system.
$ sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker-key.asc
Then, add Docker official apt repository to your system.
$ echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker-key.asc] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
Install Docker execution file
Check if the system is up-to-date and then install Docker using the following apt-get command.
$ sudo apt-get update $ sudo apt-get install docker-ce docker-ce-cli containerd.io
(Optional) Proxy setting of Docker
You may need proxy setting for the Docker if you need a proxy server to connect to the Internet.
Reference: Docker official document
Change the setting for Docker command
Change the setting for Docker command in order to make it available to general users.
$ sudo usermod -a -G docker $(whoami)
Log out and log in again to apply the setting change.
Confirm if Docker runs successfully
Check the installed Docker can be run in your system using the following command.
$ docker run hello-world Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64) 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker ID: https://hub.docker.com/ For more examples and ideas, visit: https://docs.docker.com/get-started/
Create Docker image
Create a Docker image used to compile Python module
Edit Dockerfile
You need to create a file called Dockerfile in which the program to create an image is written when creating a Docker image.
We provide you with a Dockerfile template to create necessary image for compiling NumPy and OpenCV module with ease.
external/build/sample/docker/Dockerfile
FROM arm64v8/ubuntu:bionic ADD qemu-aarch64-static /usr/bin ADD exec_entry.sh /bin RUN apt-get update \ && apt-get install -y --no-install-recommends \ build-essential gcc-8 g++-8 gfortran-8 python3.7 python3-distutils python3-setuptools python3.7-dev python3-pip cmake unzip \ && apt-get -y clean && rm -rf /var/lib/apt/lists/* RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 800 --slave /usr/bin/g++ g++ /usr/bin/g++-8 \ && update-alternatives --install /usr/bin/gfortran gfortran /usr/bin/gfortran-8 800 \ && update-alternatives --install /usr/bin/python python /usr/bin/python3.7 800 RUN python -m pip install Cython==0.29.36 RUN chmod +x /bin/exec_entry.sh ENTRYPOINT ["/bin/exec_entry.sh"]
※Do not change 1st, 3rd, 4th, 14th, and 16th line above when editing Dockerfile.
Add the following lines before
“RUN apt-get update \”
if you need a proxy server to connect to the Internet.
(Change URL and port number for the proxy server according to the proxy server you are using.)
ENV http_proxy=http://proxy.example.com:8080/ https_proxy=http://proxy.example.com:8080/
Build an image from a Dockerfile
Docker image is built within the directory where Dockerfile is placed.
$ cd ${SDK_DIR}/external/build/sample/docker/ $ docker build -t compile/ubuntu-rel:0.1 .
If it fails with an error, please do the following:
How to install qemu - Technology Partner FAQ (En) - Confluence (atlassian.net)
Check the directory to see if a repository “compile/ubuntu-rel” appears when you finish building the image.
$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE compile/ubuntu-rel 0.1 ca4409cf01fd 45 minutes ago 559MB
Make sure you see the following two files in the directory where Dockerfile is placed.
exec_entry.sh
qemu-aarch64-static
Do not delete these files as you will need them when creating Python module.
Compile Python module(Numpy, OpenCV)
Steps
Follow the steps below to compile Python module for AdamApp
Place source file in “${SDK_DIR}/external/build/sample/docker/docker_volume” directory
Write compilation program in “${SDK_DIR}/external/build/sample/docker/docker_volume/module_compile.sh”.
Run docker and proceed compilation.
The compiled module will be placed in “${SDK_DIR}/external/build/sample/docker/docker_volume/install_path” directory.
Here we take you through NumPy and OpenCV compilation as a tutorial.
The compilation fails if you work within shared folder over VirtualBox environment.
Create and move the folders to Ubuntu before you start.
Download source code
Download Numpy and OpenCV source code from the URLs below.
This tutorial uses Numpy 1.14.9, OpenCV 3.4.7.
Select the version to download according to the functions you use.
Numpy 1.14.9:
https://github.com/numpy/numpy/archive/v1.19.4.tar.gz
OpenCV 3.4.7:
Place the downloaded files to “${SDK_DIR}/external/build/sample/docker/docker_volume” directory.
Update compilation program
Write compilation program in the file: “${SDK_DIR}/external/build/sample/docker/docker_volume/module_compile.sh”.
Be aware the following points when updating the compilation program.
”${SDK_DIR}/external/build/sample/docker/docker_volume” will be mounted as “/home/docker” after you run docker.
Source files and compile directory are also placed in ”/home/docker”.
Install compiled modules to “/home/docker/install_path”.
You need NumPy module to compile OpenCV.
”module_compile.sh” is provided as a sample script to compile Numpy and OpenCV.
Here we proceed without changing “module_compile.sh”.
Compile
Compile NumPy module and OpenCV module using docker.
Run Docker in “${SDK_DIR}/external/build/sample/docker” directory.
$ docker run --rm -it --name aarch64-ubuntu -v `pwd`/docker_volume:/home/docker compile/ubuntu-rel:0.1
The compiled module is created in “external/build/sample/docker/docker_volume/install_path”.
Be patient while compiling OpenCV, it may take a while.
If a compilation error occurs with SDK V1.70 or earlier, please confirm that the external/build/sample/docker/Dockerfile described in the "Introduction" section has been corrected. If it is not corrected, start over from creating the Docker image.
Deployment in AdamApp
Deploy the modules to use them with AdamApp.
Here we take “${SDK_DIR}/src/adamapp-py/jpeg_app” as an example.
Deploy Numpy
NumPy module is installed in the following directory.
${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/site-packages/numpy-1.19.4-py3.7-linux-aarch64.egg/numpy
Copy and paste to AdamApp source directory without changing the directory structure.
${SDK_DIR}/src/adamapp-py/jpeg_app/python/site-packages
$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/site-packages/numpy-1.19.4-py3.7-linux-aarch64.egg/numpy src/adamapp-py/jpeg_app/python/site-packages
Then, the directory structure will be as follows.
${SDK_DIR}/src/adamapp-py/jpeg_app/python/site-packages/numpy
Deploy OpenCV
OpenCV module is installed in the following directory.
${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/dist-packages/cv2
Copy and paste to AdamApp source directory without changing the directory structure.
${SDK_DIR}/src/adamapp-py/jpeg_app/python/site-packages
$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/dist-packages/cv2 src/adamapp-py/jpeg_app/python/site-packages
Then, the directory structure will be as follows.
${SDK_DIR}/src/adamapp-py/jpeg_app/python/site-packages/cv2
Deploy both Open CV library and Python module as Open CV requires them both.
OpenCV library is installed in the following directory.
${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib
Copy “libopencv_world.so.3.4.7” and paste it to the following directory.
Please create manually after the external folder.
例)${SDK_DIR}/src/adamapp-py/jpeg_app/external/lib/aarch64-linux-gnu/
$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/libopencv_world.so.3.4.7 src/adamapp-py/jpeg_app/external/external/lib/aarch64-linux-gnu
Then create a symbolic link.
$ cd ${SDK_DIR}/src/adamapp-py/jpeg_app/external/lib/aarch64-linux-gnu/ $ ln -s libopencv_world.so.3.4.7 libopencv_world.so.3.4 $ ln -s libopencv_world.so.3.4 libopencv_world.so
Delete files no longer needed
AdamApp does not use (_pycache_), Python cash file.
We recommend that you delete the files to reduce the folder size.
$ cd ${SDK_DIR}/src/adamapp-py/yuv_yolo_app/python/site-packages/numpy/ $ find . -name *pycache* -print | xargs rm -rf
In addition, the information needed for debugging library is included in the OpenCV library you created.
We recommend that you delete this file to reduce the folder size as you do not need this to run the application.
$ cd ${SDK_DIR}/src/adamapp-py/yuv_yolo_app/external/lib/aarch64-linux-gnu/ $ aarch64-linux-gnu-strip -strip-debug libopencv_world.so.3.4.7
Build the sample application
Build the sample application "jpeg_app" and install it on the camera (eg, it can be installed from the green frame in the image below).
See here for how to build.
Open the app screen (red frame button in the image below), and if the jpeg image taken by the camera is displayed, it is successful.
Compile Python module(SQLite)
Numpy and OpenCV were compiled. As the next tutorial, we will compile SQLite.
Download source code
Download sqlite and Python source code from the URLs below.
This tutorial uses sqlite 3.40.0, Python 3.7.9.
Select the version to download according to the functions you use.
sqlite 3.40.0
https://www.sqlite.org/2022/sqlite-autoconf-3400000.tar.gz
Python 3.7.9
Place the downloaded files to “${SDK_DIR}/external/build/sample/docker/docker_volume” directory.
Update compilation instructions file
Put the compilation instructions in “${SDK_DIR}/external/build/sample/docker/docker_volume/module_compile.sh” file.
Please copy the content below.
#!/bin/sh -x HOME_PATH=/home/docker LIB_INSTALL_DIR=${HOME_PATH}/install_path SQLITE_ARCHIVES=sqlite-autoconf-3400000.tar.gz SQLITE_DIR=sqlite-autoconf-3400000 SQLITE_BUILD_DIR=sqlite-autoconf-3400000/build PYTHONPATH=$PYTHONPATH:${NUMPY_SITE_PACKAGES} export PYTHONPATH PYTHON_ARCHIVES=Python-3.7.9.tgz PYTHON_DIR=Python-3.7.9 ## for sqlite cd ${HOME_PATH} tar zxf ${SQLITE_ARCHIVES} mkdir -p ${SQLITE_BUILD_DIR} cd ${SQLITE_DIR} ./configure --prefix=${LIB_INSTALL_DIR} make make install ## for python cd ${HOME_PATH} tar zxf ${PYTHON_ARCHIVES} cd ${PYTHON_DIR} ./configure --prefix=${LIB_INSTALL_DIR} make make install
Compile
Compile sqlite module and Python module using docker.
Run Docker in “${SDK_DIR}/external/build/sample/docker” directory.
$ docker run --rm -it --name aarch64-ubuntu -v `pwd`/docker_volume:/home/docker compile/ubuntu-rel:0.1
The compiled module is created in “external/build/sample/docker/docker_volume/install_path”.
Deployment in AdamApp
Deploy the modules to use them with AdamApp.
Here we take “${SDK_DIR}/src/adamapp-py/sqlite_app” as an example.
An SD card is required for sqlite_app to work. Only cameras that can insert an SD card can be checked. Also, depending on the firmware of the camera, it may not work unless the SD card is formatted with ext4.
Deploy SQLite
SQLite module is installed in the following directory.
${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib
libsqlite3.so
libsqlite3.so.0
libsqlite3.so.0.8.6
libsqlite3.la
libsqlite3.a
Copy the above 5 files to the AdamApp source directory.
${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu
$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/libsqlite3.so ${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu $ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/libsqlite3.so.0 ${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu $ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/libsqlite3.so.0.8.6 ${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu $ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/libsqlite3.la ${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu $ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/libsqlite3.a ${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu
Then, the directory structure will be as follows.
${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu/libsqlite3.so
${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu/libsqlite3.so.0
${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu/libsqlite3.so.0.8.6
${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu/libsqlite3.la
${SDK_DIR}/src/adamapp-py/sqlite_app/external/lib/aarch64-linux-gnu/libsqlite3.a
Deploy sqlite for Python
sqlite for Python module is installed in the following directory.
${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/sqlite3
Copy and paste to AdamApp source directory without changing the directory structure.
${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages
$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/sqlite3 ${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages
Then, the directory structure will be as follows.
${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages/sqlite3
sqlite for Python requires not only modules but also libraries, so place them together.
The sqlite for Python library is installed in the following directory.
${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/lib-dynload
Copy “_sqlite3.cpython-37m-aarch64-linux-gnu.so” to the following directory.
${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages/sqlite3
$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/install_path/lib/python3.7/lib-dynload/_sqlite3.cpython-37m-aarch64-linux-gnu.so ${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages/sqlite3
Then, the directory structure will be as follows.
${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages/sqlite3/_sqlite3.cpython-37m-aarch64-linux-gnu.so
Next, modify dbapi2.py so that the sqlite for Python library can be loaded when importing.
${SDK_DIR}/src/adamapp-py/sqlite_app/python/site-packages/sqlite3/dbapi2.py
Modify as follows.
Before) from _sqlite3 import *
↓
After) from sqlite3._sqlite3 import *
Build the sample application
Build the sample application "sqlite_app" and install it on the camera (eg, it can be installed from the green frame in the image below).
See here for how to build.
Open the app screen (red frame button in the image below),
and if the following screen is displayed, it is successful.
Compile Python module(Pycurl)
SQLite was compiled. As the next tutorial, we will compile Pycurl.
Download source code
Download PyCurl and OpenSSL source code from the URLs below.
This tutorial uses pycurl 7.45.2, OpenSSL 1.1.1.
Select the version to download according to the functions you use.
pycurl 7.45.2
OpenSSL 1.1.1
https://github.com/openssl/openssl/archive/OpenSSL_1_1_1i.tar.gz
Place the downloaded files to “${SDK_DIR}/external/build/sample/docker/docker_volume” directory.
Update compilation instructions file
Put the compilation instructions in “${SDK_DIR}/external/build/sample/docker/docker_volume/module_compile.sh” file.
Please copy the content below.
#!/bin/sh -x HOME_PATH=/home/docker OPENSSL_ARCHIVES=openssl-OpenSSL_1_1_1i.tar.gz OPENSSL_DIR=openssl-OpenSSL_1_1_1i PYCURL_ARCHIVES=pycurl-7.45.2.tar.gz PYCURL_DIR=pycurl-7.45.2 apt update apt install libcurl4-openssl-dev libssl-dev cd ${HOME_PATH} tar zxf ${OPENSSL_ARCHIVES} cd ${OPENSSL_DIR} ./Configure linux-aarch64 make export PYCURL_SSL_LIBRARY=openssl export CPPFLAGS=-I${HOME_PATH}/${OPENSSL_DIR}/include/openssl export LDFLAGS=-L${HOME_PATH}/${OPENSSL_DIR} ## for pycurl cd ${HOME_PATH} tar zxf ${PYCURL_ARCHIVES} cd ${PYCURL_DIR} python3.7 setup.py --with-openssl --openssl-dir=${HOME_PATH}/${OPENSSL_DIR} build
Compile
Compile pycurl module and OpenSSL module using docker.
Run Docker in “${SDK_DIR}/external/build/sample/docker” directory.
$ docker run --rm -it --name aarch64-ubuntu -v `pwd`/docker_volume:/home/docker compile/ubuntu-rel:0.1
The compiled module is created in “external/build/sample/docker/docker_volume/install_path”.
Deployment in AdamApp
Deploy the modules to use them with AdamApp.
Here we take “${SDK_DIR}/src/adamapp-py/additional_info_sample_app” as an example.
Deploy PyCurl
PyCurl module is installed in the following directory.
${SDK_DIR}/external/build/sample/docker/docker_volume/pycurl-7.45.2/build/lib.linux-aarch64-3.7
pycurl.cpython-37m-aarch64-linux-gnu.so
Copy the above file to the AdamApp source directory.
${SDK_DIR}/src/adamapp-py/additional_info_sample_app/python/site-packages
$ cp -R ${SDK_DIR}/external/build/sample/docker/docker_volume/pycurl-7.45.2/build/lib.linux-aarch64-3.7/pycurl.cpython-37m-aarch64-linux-gnu.so ${SDK_DIR}/src/adamapp-py/additional_info_sample_app/python/site-packages
Then, the directory structure will be as follows.
${SDK_DIR}/src/adamapp-py/additional_info_sample_app/python/site-packages/pycurl.cpython-37m-aarch64-linux-gnu.so
Modify the source code to load PyCurl
Open “${SDK_DIR}/src/adamapp-py/additional_info_sample_app/python/pymain.py” and add as follows.
import os;
Add the following one line below.
import pycurl;
Build the sample application
Build the sample application "additional_info_sample_app" and install it on the camera (eg, it can be installed from the green frame in the image below).
See here for how to build.
Open the app screen (red frame button in the image below), and if the string is displayed like "body", it is successful.
If it fails, a message will be displayed on the application screen indicating that the application has failed to start. From now on, please use the message content and debugging tools to solve the problem and create an application.