This tutorial [5] requires creation of extensible platform setup as described in TE0950, Vitis 2023.2.1, Extensible Platform [1].
This tutorial [5] requires installation of AMD DPUCV2DX8G for Vitis 2023.2.1 as described in TE0950, Vitis 2023.2.1, Integration of DPUCV2DX8G [3].
The target system is TE0950-02 (with Versal xcve2302 ES1 device) or TE0950-03 (with Versal xcve2302 device).
The TE0950 extensible bring up script are using Linux make utility.
The board configuration files are identical with [6], [7] for the Versal device.
The Artix device is using the bring up scripts and board configuration files [6], [8].
Description of the AMD DPUCV2DX8G can be find in AMD product guide PG425 [9].
Download package TE0950_2023_2_1_dpucv2dx8g_AI_3_5.zip accompanying this tutorial from:
Download the Vitis-AI 3.5 repository.
In browser, open page:
https://github.com/Xilinx/Vitis-AI/tree/master
Click on green Code button and download Vitis-AI-3.master.zip file.
Unzip to
~/work/Vitis-AI-3.5
The directory contains the Vitis-AI 3.5 framework.
Starting point for exploration of these Vitis AI 3.5 examples is this Xilinx www page.
https://xilinx.github.io/Vitis-AI/3.0/html/index.html
In Ubuntu 20.04, install docker
sudo apt install docker.io
Test the docker installation:
sudo docker run hello-world
Install precompiled docker images
sudo docker pull xilinx/vitis-ai-tensorflow-cpu:latest
sudo docker pull xilinx/vitis-ai-tensorflow2-cpu:latest
sudo docker pull xilinx/vitis-ai-pytorch-cpu:latest
Vitis AI 3.5 models can be downloaded with python tool downloader.py
cd ~/work/Vitis-AI-3.5/model_zoo
Example:
Select pytorch model framework
input: pt
Select model (vehicle_type_resnet18_pt-vek280-r3.5.0)
input num: 1
chose model (all)
Input num: 0
devel@ubuntu:~/work/Vitis-AI-3.5/model_zoo$ python3 downloader.py
Tip:
you need to input framework and model name, use space divide such as tf vgg16
tf:tensorflow1.x tf2:tensorflow2.x cf:caffe dk:darknet pt:pytorch all: list all model
input:pt
chose model
0 : all
1 : pt_vehicle-type-classification_3.5
2 : pt_inceptionv3_0.3_3.5
3 : pt_squeezenet_3.5
4 : pt_face-mask-detection_3.5
5 : pt_OFA-yolo_3.5
6 : pt_OFA-yolo_0.5_3.5
7 : pt_bert-tiny_3.5
8 : pt_resnet50_0.3_3.5
9 : pt_HRNet_3.5
10 : pt_yolox-nano_3.5
11 : pt_pointpillars_3.5
12 : pt_OFA-resnet50_0.88_3.5
13 : pt_resnet50_0.7_3.5
14 : pt_inceptionv3_3.5
15 : pt_OFA-resnet50_0.74_3.5
16 : pt_fadnet_0.65_3.5
17 : pt_fadnetv2_3.5
18 : pt_OFA-resnet50_0.45_3.5
19 : pt_vehicle-make-classification_3.5
20 : pt_bert-large_3.5
21 : pt_vehicle-color-classification_3.5
22 : pt_OFA-resnet50_0.60_3.5
23 : pt_xilinxSR_3.5
24 : pt_resnet50_0.6_3.5
25 : pt_OFA-depthwise-resnet50_3.5
26 : pt_bert-base_3.5
27 : pt_resnet50_0.4_3.5
28 : pt_3D-UNET_3.5
29 : pt_SESR-S_3.5
30 : pt_inceptionv3_0.5_3.5
31 : pt_psmnet_3.5
32 : pt_fadnet_3.5
33 : pt_inceptionv3_0.4_3.5
34 : pt_OFA-rcan_3.5
35 : pt_resnet50_0.5_3.5
36 : pt_OFA-resnet50_3.5
37 : pt_fadnetv2_0.51_3.5
38 : pt_OFA-yolo_0.3_3.5
39 : pt_resnet50_3.5
40 : pt_movenet_3.5
41 : pt_inceptionv3_0.6_3.5
input num: 1
chose model type
0: all
1 : GPU
2 : vek280
3 : v70
input num:0
pt_vehicle-type-classification_3.5.zip
100.0%|100%
vehicle_type_resnet18_pt-vek280-r3.5.0.tar.gz
100.0%|100%
vehicle_type_resnet18_pt-v70-DPUCV2DX8G-r3.5.0.tar.gz
100.0%|100%
done
devel@ubuntu:~/work/Vitis-AI-3.5/model_zoo$
Delete model packages for v70-DPUCV2DX8G:
./vehicle_type_resnet18_pt-v70-DPUCV2DX8G-r3.5.0.tar.gz
Models for AMD DPUCVDX8H configuration for the TE0950 board have to be compiled from data present in zip archive:
./pt_vehicle-type-classification_3.5.zip
unzip this archive to directory
./pt_vehicle-type-classification_3.5
An arch.json is produced by Vivado in the process of compilation of the AMD DPUCV2DX8G.
In case of TE0821 system with ID=3, [4] the arch.json file describing the AMD DPUCV2DX8G has been created in:
/home/devel/work/dpucv2dx8g-trd/vitis_prj/hw/binary_container_1/ link/vivado/vpl/prj/prj.runs/prj/prj.gen/sources_1/bd/vu_sys/ip/ vu_sys_DPUCV2DX8G_1_0/arch.json
Copy the arch.json file for the AMD DPUCV2DX8G configuration for the TE0950 to
~/work/Vitis-AI-3.0/model_zoo/arch.json
Copy fie compile_model.sh from the repository associated to this application note to the directory
~/work/Vitis-AI-3.5/model_zoo/compile_model.sh
Change mode to executable for file compile_model.sh
Chmod +x ~/work/Vitis-AI-3.5/model_zoo/compile_model.sh
Change directory to
~/work/Vitis-AI-3.5
Start pytorch docker
sudo ./docker_run.sh xilinx/vitis-ai-pytorch-cpu:latest
In pytorch docker WorkFlow, change directory to model_zoo
cd model_zoo
Start pytorch compiler ftramework
conda activate vitis-ai-pytorch

Parameters for model compilation script compile_model.sh :
Pytorch model is indicated by pt
Model name can be found in the readme file in classification directory:
/home/devel/work/Vitis-AI-3.5/examples/vai_library/samples/vehicleclassification/readme
Part of vehicleclassification/readme file content
Valid model name:
vehicle_make_resnet18_pt
vehicle_type_resnet18_pt
Directory is name of the unzipped directory associated to the model
Quantized model is name of file *.xmodel located in directory associated to the model in subdirectory quantized
/work/Vitis-AI-3.5/model_zoo/pt_vehicle-type-classification_3.5/ quantized/ResNet_0_int.xmodel
The file name *.xmodel can be different for different models but the location is in the subdirectory quantized and the extension is .xmodel
Command for compilation of this pytorch model example:
./compile_model.sh pt vehicle_type_resnet18_pt pt_vehicle-type-classification_3.5 ResNet_0_int.xmodel
Compilation listing:
vitis-ai-user@ubuntu:/workspace/model_zoo$ conda activate vitis-ai-pytorch (vitis-ai-pytorch) vitis-ai-user@ubuntu:/workspace/model_zoo$ ./compile_model.sh pt vehicle_type_resnet18_pt pt_vehicle-type-classification_3.5 ResNet_0_int.xmodel ************************************************** * VITIS_AI Compilation - Xilinx Inc. ************************************************** [UNILOG][INFO] Compile mode: dpu [UNILOG][INFO] Debug mode: null [UNILOG][INFO] Target architecture: DPUCV2DX8G_ISA1_C20B1 [UNILOG][INFO] Graph name: ResNet_0, with op num: 171 [UNILOG][INFO] Begin to compile... [UNILOG][INFO] Total device subgraph number 3, DPU subgraph number 1 [UNILOG][INFO] Compile done. [UNILOG][INFO] The meta json is saved to "/workspace/model_zoo/./compiled_output_pt/vehicle_type_resnet18_pt/meta.json" [UNILOG][INFO] The compiled xmodel is saved to "/workspace/model_zoo/./compiled_output_pt/vehicle_type_resnet18_pt/vehicle_type_resnet18_pt.xmodel" [UNILOG][INFO] The compiled xmodel's md5sum is 6a4a4c3c1bbc88da22c9415b7f87c7de, and has been saved to "/workspace/model_zoo/./compiled_output_pt/vehicle_type_resnet18_pt/md5sum.txt" (vitis-ai-pytorch) vitis-ai-user@ubuntu:/workspace/model_zoo$
Classification Model: pt_vehicle-type-classification_3.5 is compiled for AMD DPUCV2DX8G. It contains these 3 files:
~/work/Vitis-AI-3.5/model_zoo/compiled_output_pt/ vehicle_type_resnet18_pt/md5sum.txt ~/work/Vitis-AI-3.5/model_zoo/compiled_output_pt/ vehicle_type_resnet18_pt/meta.json ~/work/Vitis-AI-3.5/model_zoo/compiled_output_pt/ vehicle_type_resnet18_pt/vehicle_type_resnet18_pt.xmodel
One additional file is needed in the compiled model. This file is present in archive with precompiled model for the AMD DPUCV2DX8G.
Open archive
vehicle_type_resnet18_pt-vek280-r3.5.0.tar.gz
copy file
/vehicle_type_resnet18_pt/vehicle_type_resnet18_pt.prototxt
to file
./compiled_output_pt/vehicle_type_resnet18_pt/vehicle_type_resnet18_pt.prototxt
Perform similar compilation steps for all 10 models selected for the demonstration in this application note:
Vehicleclassification of type of vehicle Model (already compiled as an commented example):
1 : pt_vehicle-type-classification_3.5
Vehicleclassification of make of vehicle Model
19 : pt_vehicle-make-classification_3.5
Find related model names in sample application readme file:
~/work/Vitis-AI-3.5/examples/vai_library/samples/ vehicleclassification/readme
Classification of color of vehicle Model:
21 : pt_vehicle-color-classification_3.5
Classification Model:
39 : pt_resnet50_3.5 8 : pt_resnet50_0.3_3.5 27 : pt_resnet50_0.4_3.5 35 : pt_resnet50_0.5_3.5 24 : pt_resnet50_0.6_3.5 13 : pt_resnet50_0.7_3.5
Find related model names in sample application readme file:
~/work/Vitis-AI-3.5/examples/vai_library/samples/classification/readme
Face detection Model:
4: pt_face-mask-detection_3.5
Find the expected model names in sample application readme file:
~/work/Vitis-AI-3.5/examples/vai_library/samples/yolov4/readme
Results of compilations will be in folder
./compiled_output_pt/*
It will be used to target Vitis AI 3.5 demos on TE0950 with AMD DPUCV2DX8G.
Copy this archive into directory indicating configuration for the AMD DPUCV2DX8G like:
~/work/DPUCV2DX8G/compiled_output_pt.zip
Delete content of the directory
~/work/Vitis-AI-3.5/model_zoo/compiled_output_pt/*
Demos from the Vitis AI 3.5 library can be compiled in Vitis 2023.2.1 for Petalinux 2023.2 run-time on TE0950 board.
SW examples from the Vitis AI 3.5 package to be moved to TE0950 board are:
~/work/Vitis-AI-3.5/examples/vai_library/samples/vehicleclassification ~/work/Vitis-AI-3.5/examples/vai_library/samples/classification ~/work/Vitis-AI-3.5/examples/vai_library/samples/yolov4
Compress them to:
~/work/vehicleclassification.zip ~/work/classification.zip ~/work/yolov4.zip
Archives to be moved to TE0950 are:
~/work/DPUCV2DX8G/compiled_output_pt.zip ~/work/vehicleclassification.zip ~/work/classification.zip ~/work/yolov4.zip
Vitis AI 3.5 TensorFlow models can be downloaded with python tool downloader.py
cd ~/work/Vitis-AI-3.5/model_zoo
List TensorFlow models
devel@ubuntu:~/work/Vitis-AI-3.5/model_zoo$ python3 downloader.py Tip:you need to input framework and model name, use space divide such as tf vgg16 tf:tensorflow1.x tf2:tensorflow2.x cf:caffe dk:darknet pt:pytorch all: list all model input:tf chose model 0 : all 1 : tf_RefineDet-Medical_0.85_3.5 2 : tf_RefineDet-Medical_0.88_3.5 3 : tf_efficientnet-edgetpu-S_3.5 4 : tf_resnetv2-152_3.5 5 : tf_mlperf_resnet50v1.5_3.5 6 : tf_inceptionv3_0.4_3.5 7 : tf_inceptionv3_0.2_3.5 8 : tf_rcan_0.98_3.5 9 : tf_mobilenetv2-1.4_3.5 10 : tf_inceptionv4_0.4_3.5 11 : tf_resnetv1-152_0.51_3.5 12 : tf_inceptionv4_0.2_3.5 13 : tf_resnetv1-101_3.5 14 : tf_superpoint_3.5 15 : tf_mobilenetv2-1.0_3.5 16 : tf_efficientdet-d2_3.5 17 : tf_vgg16_0.5_3.5 18 : tf_inceptionv1_0.16_3.5 19 : tf_mobilenetv1-1.0_0.12_3.5 20 : tf_yolov4-416_3.5 21 : tf_resnetv1-101_0.57_3.5 22 : tf_resnetv1-101_0.35_3.5 23 : tf_vgg16_3.5 24 : tf_vgg19_0.24_3.5 25 : tf_bert-base_3.5 26 : tf_vgg19_0.39_3. 27 : tf_vgg16_0.43_3.5 28 : tf_resnetv1-50_0.65_3.5 29 : tf_efficientnet-edgetpu-L_3.5 30 : tf_RefineDet-Medical_0.75_3.5 31 : tf_resnetv2-101_3.5 32 : tf_mobilenetv1-1.0_3.5 33 : tf_mobilenetv1-0.25_3.5 34 : tf_inceptionv3_3.5 35 : tf_HFNet_3.5 36 : tf_resnetv2-50_3.5 37 : tf_RefineDet-Medical_3.5 38 : tf_resnetv1-152_0.6_3.5 39 : tf_resnetv1-152_3.5 40 : tf_inceptionv1_0.09_3.5 41 : tf_mobilenetv1-1.0_0.11_3.5 42 : tf_ssdmobilenetv1_3.5 43 : tf_ViT_3.5 44 : tf_RefineDet-Medical_0.5_3.5 45 : tf_ssdmobilenetv2_3.5 46 : tf_resnetv1-50_0.38_3.5 47 : tf_mlperf_ssdresnet34_3.5 48 : tf_efficientnet-edgetpu-M_3.5 49 : tf_inceptionv4_3.5 50 : tf_yolov4-512_3.5 51 : tf_yolov3_3.5 52 : tf_resnetv1-50_3.5 53 : tf_vgg19_3.5 54 : tf_inceptionv1_3.5 input num:
Process of compilation models is similar to pytorch model. Change directory to:
~/work/Vitis-AI-3.5
Start tensorflow docker
sudo ./docker_run.sh xilinx/vitis-ai-tensorflow-cpu:latest
In tensorflow docker WorkFlow, change directory to model_zoo
cd model_zoo
Start tensorflow compiler ftramework
conda activate vitis-ai-tensorflow
There are only 3 parameters for model compilation script compile model.sh :
The tensorflow model is indicated by tf
Model name can be found in the readme file in the sample application.
Directory is name of the unzipped directory associated to the model.
Vitis AI 3.5 TensorFlow2 models can be downloaded with python tool downloader.py
cd ~/work/Vitis-AI-3.5/model_zoo
List TensorFlow 2 models
devel@ubuntu:~/work/Vitis-AI-3.5/model_zoo$ python3 downloader.py Tip: you need to input framework and model name, use space divide such as tf vgg16 tf:tensorflow1.x tf2:tensorflow2.x cf:caffe dk:darknet pt:pytorch all: list all model input:tf2 chose model 0 : all 1 : tf2_2D-UNET_3.5 2 : tf2_resnet50_3.5 3 : tf2_efficientnet-b0_3.5 4 : tf2_efficientnet-lite_3.5 5 : tf2_inceptionv3_3.5 6 : tf2_mobilenetv3_3.5 7 : tf2_mobilenetv1_3.5 8 : tf2_yolov3_3.5 input num:
Process of compilation models is similar to pytorch model. Change directory to:
~/work/Vitis-AI-3.5
Start tensorflow2 docker
sudo ./docker_run.sh xilinx/vitis-ai-tensorflow2-cpu:latest
In tensorflow2 docker WorkFlow, change directory to model_zoo
cd model_zoo
Start vitis-ai-tensorflow2 compiler framework
conda activate vitis-ai-tensorflow2
There are only 3 parameters for model compilation script compile model.sh :
Tensorflow2 model is indicated by tf2
Model name can be found in the readme file in the sample application.
Directory is name of the unzipped directory associated to the model.
Sample application projects are part of the Vitis AI 3.5 package. The download and compilation process will be demonstrated only on a small subset of sample applications for the TE0950 board.
Use WinSCP utility for copy of Vitis AI 3.5 examples:
From PC
~/work/vehicleclassification.zip ~/work/classification.zip ~/work/yolov4.zip
To te0950 board
~/vehicleclassification.zip ~/classification.zip ~/yolov4.zip
Use WinSCP for copy of compiled Vitis 3.5 model archives from PC:
~/work/DPUCV2DX8G/compiled_output_pt.zip
To te0950 board
~/DPUCV2DX8G/compiled_output_pt.zip
On TE0950 board, unzip Vitis AI 3.5 sample applications from
~/vehicleclassification.zip ~/classification.zip ~/yolov4.zip
To directories:
~/vehicleclassification ~/classification ~/yolov4
Copy directory:
~/DPUCV2DX8G/compiled_output_pt/face_mask_detection_pt
To directory:
~/yolov4/face_mask_detection_pt
Copy directories:
~/DPUCV2DX8G/compiled_output_pt/vehicle_make_resnet18_pt ~/DPUCV2DX8G/compiled_output_pt/vehicle_type_resnet18_pt
To directories:
~/vehicleclassification/vehicle_make_resnet18_pt ~/vehicleclassification/vehicle_type_resnet18_pt
Copy directories:
~/DPUCV2DX8G/compiled_output_pt/chen_color_resnet18_pt ~/DPUCV2DX8G/compiled_output_pt/resnet50_pt ~/DPUCV2DX8G/compiled_output_pt/resnet50_pruned_0_3_pt ~/DPUCV2DX8G/compiled_output_pt/resnet50_pruned_0_4_pt ~/DPUCV2DX8G/compiled_output_pt/resnet50_pruned_0_5_pt ~/DPUCV2DX8G/compiled_output_pt/resnet50_pruned_0_6_pt ~/DPUCV2DX8G/compiled_output_pt/resnet50_pruned_0_7_pt
To directories:
~/classification/chen_color_resnet18_pt ~/classification/resnet50_pt ~/classification/resnet50_pruned_0_3_pt ~/classification/resnet50_pruned_0_4_pt ~/classification/resnet50_pruned_0_5_pt ~/classification/resnet50_pruned_0_6_pt ~/classification/resnet50_pruned_0_7_pt
On target board, compile all sample applications:
sh-5.1# cd ~/yolov4 sh-5.1# chmod +x build.sh sh-5.1# ./build.sh sh-5.1# cd ~/vehicleclassification sh-5.1# chmod +x build.sh sh-5.1# ./build.sh sh-5.1# cd ~/classification chmod +x build.sh sh-5.1# ./build.sh
Test Vitis AI 3.5 applications.
Commands used for execution of applications are described in sample application readme files:
~/yolov4/readme ~/vehicleclassification/readme ~/classification/readme
In TE0950 terminal use export command:
On module-based systems supporting modules TE0821 [24] or modules TE0820 [25], use export command:
sh-5.1# export XLNX_VART_FIRMWARE=/run/media/mmcblk1p1/dpu.xclbin
Change directory to
~/yolov4
Test face mask detection with input from file sample_face_mask.jpg
sh-5.1# ./test_jpeg_yolov4 face_mask_detection_pt sample_face_mask.jpg
Output: face mask detection coordinates on terminal.
Test performance of face mask detection with input from files listed in test_performance_face_mask.list file.
sh-5.1# ./test_performance_yolov4 face_mask_detection_pt test_performance_face_mask.list -s 60 -t 3
Output: data about performance (in FPS) of face mask detection on terminal.
-s 60 defines length of benchmark in seconds.
-t 3 defines use of 3 SW threads (with one AMD DPU)
Test face mask detection with USB www camera input
sh-5.1# ./test_video_yolov4 face_mask_detection_pt 0 -t 1

Face mask detection Vitis AI 3.5 application with camera input, AMD DPUCV2DX8G.
Output: Video output with face mask coordinates in video and performance (in FPS) displayed as text in video.
0 defines USB www camera device number
-t 1 defines use of 1 SW thread
The SD card created finally in [3] contains combination of test_free_run IPs and DPUCV2DX8G unit. While the face_mask detection application is running, go to second TE0950 terminal, and run the test_free_run application by command:
sh-5.1# cd /run/media/mmcblk1p1/ sh-5.1# ./test_free_run dpu.xclbin
Both applications can run in parallel.
The face_mask detection Vitis AI 3.5 application with USB camera input can be stopped by pointing mouse to the X11 video output window and typing Esc key on the keyboard.
Vitis AI 3.5 sample test applications located in the directory ~/vehicleclassification have analogical format for calling and parameters.
Two different AI models can be used to classify make or type of the vehicle.
Vitis AI 3.5 sample test applications located in the directory ~/classification have analogic format.
Several different AI models can be used to:
Mesured performance indicators are summarized for TE0950 with AMD DPUCV2DX8G:
Vitis AI 3.5 examples | Performance with input from camera e2e [FPS] | Power with input from camera e2e [W] | Performance with input from file e2e [FPS] | Power with input from file e2e [W] | TeraOps with input from file e2e [Tops] |
Face detection Model: pt_face-mask-detection_3.5 | 20.0 | 13.0 | 210 | 13.1 | 0.14 |
Vehicle make Model: pt_vehicle-make-classification_3.5 | 20.0 | 13.0 | 730 | 15.0 | 2.65 |
Vehicle type Model: pt_vehicle-type-classification_3.5 | 20.0 | 13.0 | 730 | 15.0 | 2.65 |
Vehicle color Model: pt_vehicle-color-classification_3.5 | 20.0 | 13.0 | 730 | 15.0 | 2.65 |
General classification Model: pt_resnet50_3.5 | 20.0 | 13.1 | 272 | 14.8 | 2.23 |
General classification Model: pt_resnet50_0.3_3.5 | 20.0 | 13.1 | 295 | 14.6 | 1.71 |
General classification Model: pt_resnet50_0.4_3.5 | 20.0 | 13.1 | 317 | 14.5 | 1.55 |
General classification Model: pt_resnet50_0.5_3.5 | 20.0 | 13.0 | 333 | 14.3 | 1.36 |
General classification Model: pt_resnet50_0.6_3.5 | 20.0 | 13.0 | 365 | 14.2 | 1.20 |
General classification Model: pt_resnet50_0.7_3.5 | 20.0 | 13.0 | 428 | 14.1 | 1.07 |
Measurement conditions:
Not available