AI model convert tool: Evaluate AI model

Owned by Rie Sakito (崎戸 梨恵) (Unlicensed)
Last updated: Jun 19, 2024 by Masakazu Fujii (藤井 雅和)
3 min read

Table of contents

 

Preparation

Please refer here for the environment construction procedure.

 

Conversion

Please refer to the following page for model conversion.

　AI model convert tool: Caffe

　AI model convert tool: Tensorflow

　AI model convert: ONNX(PyTorch)

 

Evaluate AI model

Please evaluate in one of the following ways.

  1. Running the converted model on the i-PRO camera
    Inference time
    Measure the time before and after the inference execution API (Adam_AI_RunNet()) call
    Precision
    Evaluate the accuracy using the data obtained by the API (Adam_AI_GetOutput()) that acquires the data of the output layer of the model.

  2. Using sample app
    The CV tool includes a sample app that allows you to obtain inference time and output layer data for the converted model. This page explains how to use it.

  3. Using simulator in CV tool
    It is possible to perform inference for the converted model. Please refer here on how to use it.

 

Sample app for evaluation

Preparing for the evaluation

Use the Chrome extension's ADAM OPERATION UI for evaluation.

For details on how to install, refer here.

Also, make sure you have an i-PRO network camera that the app can install on.

Product Number: Installation conditions for applications - FAQ - Development Partner Portal (En) (i-pro.com)

 

 

DnnSdApp

Install the app

First, copy the DnnSdApp package (DnnSdApp_V0_5_ambaCV2X5X.ext) in the container to the host PC.

[Work Directory]Any directory

$ cd [Work Directory] $ sudo docker run -it --rm -v $(pwd):/work [image name] /bin/bash $ cp /home/cvtool/app/DnnSdApp_V0_5_ambaCV2X5X.ext /work

Launch a browser and access the detailed setting

Set [Basic] - [SD memory card] - [Operation Mode] - [SD memory card] and [Ext. software mode] to "On".

If you do not want to use the SD card, please select "Not use".
When uploading a model, use the method of transferring it on a TFTP server.

Move to the Ext. software and install DnnSdApp.

Change settings to match your rating model

Configure various settings with ADAM OPERATION UI.

layernamein: Input layer name

layernameout: Output layer name (separated by comma for multiple settings)

DnnSdApp may not work properly, when “/” is contained in layernamein or layernameout.

NETNAME: Model name

TftpServerIP：TFTP server address where models are stored
　　　　　　　*Set if SD card is not used

ChannelNum：Channels of model

ImgHeight：Height of input image

ImgWidth：Width of input image

PixelFormat：Pixel format of model

Prepare evaluation images

Compress the images to be used for evaluation (dnn.tar.gz) .

Follow the folder structure below, either jpeg or mp4 only can be used.

tar cvzf dnn.tar.gz dnn

Folder configuration

Remarks

Folder configuration

Remarks

dnn/

test_jpeg/

yyy1.jpg

jpeg placement directory, file names are arbitrary

File extension: ".jpg", ".jpeg", ".JPG", ".JPEG"

 

yyy2.jpg

:

test_mp4/

　

　

zzz1.mp4

mp4 placement directory, file name to be deployed is arbitrary

File ectension: ".mp4"

zzz2.mp4

:

Upload images to DnnSdApp

Open the app screen and upload the image data.

Upload the model file to the app

Upload the model file.

Run the app

After placing the model and image, click the "Start" button to start execution.

Download the results

Once the run is complete, you can get the result file from the Download button.

SsdSdApp

The operation is similar to DnnSdApp.

Please replace the folder name with "SSD" ⇒ "DNN".