GitHub - onnx/onnx-tensorrt: ONNX-TensorRT: TensorRT backend for ONNX
source link: https://github.com/onnx/onnx-tensorrt
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
README.md
TensorRT backend for ONNX
Parses ONNX models for execution with TensorRT.
See also the TensorRT documentation.
Supported TensorRT Versions
Development on the Master branch is for the latest version of TensorRT 6.0 with full-dimensions and dynamic shape support.
For version 6.0 without full-dimensions support, clone and build from the 6.0 branch
For version 5.1, clone and build from the 5.1 branch
For versions < 5.1, clone and build from the 5.0 branch
Full Dimensions + Dynamic Shapes
Building INetwork objects in full dimensions mode with dynamic shape support requires calling the following API:
C++
const auto explicitBatch = 1U << static_cast<uint32_t>(nvinfer1::NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);
builder->createNetworkV2(explicitBatch)
Python
import tensorrt
explicit_batch = 1 << (int)(tensorrt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
builder.create_network(explicit_batch)
For examples of usage of these APIs see:
Supported Operators
Current supported ONNX operators are found in the operator support matrix.
Installation
Dependencies
Building
For building on master, we recommend following the instructions on the master branch of TensorRT as there are new dependencies that were introduced to support these new features.
To build on older branches refer to their respective READMEs.
Executable usage
ONNX models can be converted to serialized TensorRT engines using the onnx2trt executable:
onnx2trt my_model.onnx -o my_engine.trt
ONNX models can also be converted to human-readable text:
onnx2trt my_model.onnx -t my_model.onnx.txt
See more usage information by running:
onnx2trt -h
Python modules
Python bindings for the ONNX-TensorRT parser are packaged in the shipped .whl files. Install them with
pip install <tensorrt_install_dir>/python/tensorrt-6.0.1.5-cp27-none-linux_x86_64.whl
TensorRT 6.0 supports ONNX release 1.5.0. Install it with:
pip install onnx==1.5.0
ONNX Python backend usage
The TensorRT backend for ONNX can be used in Python as follows:
import onnx import onnx_tensorrt.backend as backend import numpy as np model = onnx.load("/path/to/model.onnx") engine = backend.prepare(model, device='CUDA:1') input_data = np.random.random(size=(32, 3, 224, 224)).astype(np.float32) output_data = engine.run(input_data)[0] print(output_data) print(output_data.shape)
C++ library usage
The model parser library, libnvonnxparser.so, has its C++ API declared in this header:
NvOnnxParser.h
Important typedefs required for parsing ONNX models are declared in this header:
NvOnnxParserTypedefs.h
Docker image
Tar-Based TensorRT
Build the onnx_tensorrt Docker image using tar-based TensorRT by running:
git clone --recurse-submodules https://github.com/onnx/onnx-tensorrt.git
cd onnx-tensorrt
cp /path/to/TensorRT-6.0.*.tar.gz .
docker build -f docker/onnx-tensorrt-tar.Dockerfile --tag=onnx-tensorrt:6.0.6 .
Deb-Based TensorRT
Build the onnx_tensorrt Docker image using deb-based TensorRT by running:
git clone --recurse-submodules https://github.com/onnx/onnx-tensorrt.git
cd onnx-tensorrt
cp /path/to/nv-tensorrt-repo-ubuntu1x04-cudax.x-trt6.x.x.x-ga-yyyymmdd_1-1_amd64.deb .
docker build -f docker/onnx-tensorrt-deb.Dockerfile --tag=onnx-tensorrt:6.0.6 .
Tests
After installation (or inside the Docker container), ONNX backend tests can be run as follows:
Real model tests only:
python onnx_backend_test.py OnnxBackendRealModelTest
All tests:
python onnx_backend_test.py
You can use -v flag to make output more verbose.
Pre-trained models
Pre-trained models in ONNX format can be found at the ONNX Model Zoo
Recommend
-
92
GitHub is where people build software. More than 28 million people use GitHub to discover, fork, and contribute to over 80 million projects.
-
53
ONNX (Open Neural Network eXchange) is an open format for the sharing of neural network and other machine learned models between various machine learning and deep learning frameworks. As the open big data serving engine,...
-
488
README.md TensorRT Pose Estimation This project features multi-instance pose estimation accelerated by NVIDIA TensorRT. It is ideal for appl...
-
42
How to Convert Your Keras Model to ONNX Intuition I love Keras for its simplicity. With about 10 minutes, I can build a deep learning model with its sequential or functional API with elegant code. However, Keras always load...
-
16
Operationalizing ML models with ONNX, C# .... and Pokemon!On .NET Live - Operationalizing ML models with ONNX, C# .... and Pokemon! - YouTube
-
251
README.md This repository represents Ultralytics open-source research into future object detection methods, and incorporates lessons learned and best practices evolved...
-
11
作者丨立交桥跳水冠军 来源丨https://zhuanlan.zhihu.com/p/272767300 编辑丨GiantPandaCV 之前几个月参与了Open MMlab的...
-
16
0x0. 背景 最近看了一些ONNX的资料，一个最大的感受就是这些资料太凌乱了。大多数都是在介绍ONNX模型转换中碰到的坑点以及解决办法。很少有文章可以系统的介绍ONNX的背景，分析ONNX格式，ONNX简化方法等。所以，综合了相当多资料之后我准备写一篇...
-
199
Continuing from Introducing OnnxSharp and ‘dotnet onnx’, in this post I will look at using OnnxSharp to set dynamic batch s...
-
3
Llama 2 Powered By ONNX This is an optimized version of the Llama 2 model, available from Meta under the Llama Community License Agreement found on this repository. Microsoft permits you to use, modify, redistribu...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK