AI model convert tool: Evaluate AI model
- Rie Sakito (崎戸 梨恵) (Unlicensed)
- Yoshitaka Fukushima (福島 佳孝)
- Masakazu Fujii (藤井 雅和)
Table of contents
Preparation
Please refer here for the environment construction procedure.
Conversion
Please refer to the following page for model conversion.
AI model convert tool: Tensorflow
AI model convert: ONNX(PyTorch)
Evaluate AI model
Please evaluate in one of the following ways.
Running the converted model on the i-PRO camera
Store the converted AI model file in the /data_CV2X/cnn/ directory (for CV2X cameras) or /data_CV5X/cnn/ directory (for CV5X cameras) in the application folder being developed.
Also, implement the application according to the I/F described in the API documentation. Implementation examples are also included in various sample applications.
Inference time
Measure the time before and after the inference execution API (Adam_AI_RunNet) call
Precision
Evaluate the accuracy using the data obtained by the API (ADAM_CV_GetCpuData) that acquires the data of the output layer of the model.Using sample app for evaluation
The CV tool includes a sample app that allows you to obtain inference time and output layer data for the converted model. This page explains how to use it.Using simulator in CV tool
It is possible to perform inference for the converted model. Please refer here on how to use it.
Sample app for evaluation
Preparing for the evaluation
Use the Chrome extension's ADAM OPERATION UI for evaluation.
For details on how to install, refer here.
Also, make sure you have an i-PRO network camera that the app can install on.
DnnSdApp
Install the app
First, copy the DnnSdApp package (DnnSdApp_V0_5_ambaCV2X5X.ext) in the container to the host PC.
[Work Directory]Any directory
$ cd [Work Directory]
$ sudo docker run -it --rm -v $(pwd):/work [image name] /bin/bash
$ cp /home/cvtool/app/DnnSdApp_V0_5_ambaCV2X5X.ext /workLaunch a browser and access the detailed setting
Set [Basic] - [SD memory card] - [Operation Mode] - [SD memory card] and [Ext. software mode] to "On".
If you do not want to use the SD card, please select "Not use".
When uploading a model, use the method of transferring it on a TFTP server.
Move to the Ext. software and install DnnSdApp.
Change settings to match your rating model
Configure various settings with ADAM OPERATION UI.
layernamein: Input layer name
layernameout: Output layer name (separated by comma for multiple settings)
DnnSdApp may not work properly, when “/” is contained in layernamein or layernameout.
NETNAME: Model name
TftpServerIP：TFTP server address where models are stored
*Set if SD card is not used
ChannelNum：Channels of model
ImgHeight：Height of input image
ImgWidth：Width of input image
PixelFormat：Pixel format of model
Prepare evaluation images
Compress the images to be used for evaluation (dnn.tar.gz) .
Follow the folder structure below, either jpeg or mp4 only can be used.
tar cvzf dnn.tar.gz dnnFolder configuration | Remarks | ||
|---|---|---|---|
dnn/ | test_jpeg/ | yyy1.jpg | jpeg placement directory, file names are arbitrary File extension: ".jpg", ".jpeg", ".JPG", ".JPEG"
|
yyy2.jpg | |||
: | |||
test_mp4/
| zzz1.mp4 | mp4 placement directory, file name to be deployed is arbitrary File ectension: ".mp4" | |
zzz2.mp4 | |||
: | |||
Upload images to DnnSdApp
Open the app screen and upload the image data.
Upload the model file to the app
Upload the model file.
Run the app
After placing the model and image, click the "Start" button to start execution.
Download the results
Once the run is complete, you can get the result file from the Download button.
SsdSdApp
The operation is similar to DnnSdApp.
Please replace the folder name with "SSD" ⇒ "DNN".