Using C++ library From Rust - DepthAI RGB Camera Example

There has been a long time since the my last post, and many things happend. I have been persuaded by Alex to join AdVentura Works SA to a very exiting adventure, touching one of my passions in embedded software, robotics, AI/CV/ML. I have left Zest Labs GmbH and joined the new startup, where we are making the warehouse operations more safe and efficient. We have lots of opportunities in the area of tele-operations, AI assited operations, tasks and operation optimisations.

Our new project requires top performance on the edge, and we cannot afford a single ms lost in compute. As long time fan of GoLang - I have been challenged to squize even more performance. Our first POCs were written in Go and it performed really well for consumer grade hardware, but when we touch the edge, there is much more to ask. Go is excellent language, producing blazingly fast binaries, but there is still a garbage collector, and much more importantly - memory copy when using C bindings. In our area, there are a lot of modules and libraries written in C and C++. We have to be able to wrap those functionallities and squize the best performance out of them.

All this drove me to Rust - well - not so surprizing. Rust is very close to the hardware, often creates faster binaries than the once written in low level languages like C and C++, and at the same time it is safe and pleasurable to write in.

Our goal is to be able to use existing libraries written in C and C++ in our new applications, without compromizing on performance and safety.

All this made me to experiment, so that I can showcase the interface between already written libraries and our software. Calling external (static or dynamic libraries) is not a trivial problem. C is quite streamlined - in Rust you can use extern "C" and be quite sure that the things will work. As for C++ - Rust compiler cannot have a good picture what to interface. Majory of the basic aproaches are to wrap C++ code in C libraries and then to call them from Rust. As you can understand - this is not trivial and requires lots of time and effort. So let’s see what are the options:

• bindgen - Bindgen parses header files and generates Rust bindings. Works well for C but for C++ is not perfect
• cbindgen - Works in the other direction - parses Rust code and generates C or C++ headers
• cxx crate - Works both ways - can generate c++ bindings and vise versa.
• cpp crate - Can embed C++ code in your rust code

Kudos to Tobias Hunger for his post/presentation.

So lets get our hands durty and write some Rust to call C++ ready library. I have chosen something that will work for me in my projects. We have bought a OpenCV AI Kit: OAK—D to utilise in our project. So the task is to be able to integrate the existing DepthAI Core Library, written in C++, in our Rust core project.

I have taken the short path at the beginning, so that I have less complexity, and just to make the showcase, how to make the things done in the future, so I have chosen to write the existing example app from depthai-core in rust - rgb_encoding.cpp.

The ultimate goal will be to have the same functionallity as in the C++ example, but written in Rust.

For the onces, who don’t want to read more of my rumbles - just check the repository that I have published Using DepthAI Core from Rust

For the rest - lets explore the code

First we will define a c++ header and implementation, where we will define a class that instantiates the DepthAI Device, and creates a pipeline. Further we will use the same class to fetch the H264/5 frames and write them to a file. Here I have chosen the aproach to control the thread and fetching the frames from Rust - as the call is synchonios - I have the luxory to handle it in Rust. Some of you will criticize me for using a call back function, instead of diretctly fetching the frame data. This has been done intentionally, so that in the future I can change the code to push the frames in a WebRTC pipeline asynchroniously.

include/depthai_wrapper.h

pragma once
#include "rust/cxx.h"
#include <memory>

namespace dev
{
  namespace pnkv
  {

    struct DepthAISource;

    class DepthAIClient
    {
    public:
      DepthAIClient();
      ::std::int32_t next_frame(DepthAISource &src) const;  // [1]

    private:
      class impl;
      std::shared_ptr<impl> impl;
      void push_frame(DepthAISource &src) const;
    };

    std::unique_ptr<DepthAIClient> new_depthai_client();     // [2]

  }
}

• [1] next_frame function will be called from Rust code to obtain the next Video frame from the camera, which will trigger call back to post_frame, which will be Rust bound function
• [2] Method to use for constructing the C++ object - called from Rust code.

src/deothai_wrapper.cc

#include "include/depthai_wrapper.h"
#include "depthai-rust/src/main.rs.h"                        // [3]
#include <algorithm>
#include <functional>
#include <set>
#include <string>
#include <atomic>
#include <thread>
#include <chrono>
#include <memory>
#include <iostream>
#include <unordered_map>
#include "depthai/depthai.hpp"

namespace dev
{
  namespace pnkv
  {

    // Implement simple RawData wrapper for Rust
    class DepthAIClient::impl
    {
      friend DepthAIClient;

    private:
      std::shared_ptr<dai::DataOutputQueue> outputQueue;
      std::shared_ptr<dai::Device> device;
    };

    DepthAIClient::DepthAIClient() : 
      impl(new class DepthAIClient::impl)                    // [4]
    {
      // Create pipeline
      auto pipeline = std::make_shared<dai::Pipeline>();

      // Define sources and outputs
      auto camRgb = pipeline->create<dai::node::ColorCamera>();
      auto videoEnc = pipeline->create<dai::node::VideoEncoder>();
      auto xout = pipeline->create<dai::node::XLinkOut>();

      xout->setStreamName("camRgb");

      // Properties
      camRgb->setBoardSocket(dai::CameraBoardSocket::RGB);
      camRgb->setResolution(dai::ColorCameraProperties::SensorResolution::THE_720_P);
      videoEnc->setDefaultProfilePreset(30, dai::VideoEncoderProperties::Profile::H264_BASELINE);

      // Linking
      camRgb->video.link(videoEnc->input);
      videoEnc->bitstream.link(xout->input);

      auto deviceInfoVec = dai::Device::getAllAvailableDevices();
      const auto usbSpeed = dai::UsbSpeed::SUPER;
      auto openVinoVersion = dai::OpenVINO::Version::VERSION_2021_4;

      std::map<std::string, std::shared_ptr<dai::DataOutputQueue>> qRgbMap;
      std::vector<std::shared_ptr<dai::Device>> devices;

      // Get the first device
      for (auto &deviceInfo : deviceInfoVec)
      {
        auto device = std::make_shared<dai::Device>(openVinoVersion, deviceInfo, usbSpeed);
        device->startPipeline(*pipeline);
        impl->device = device;
        auto outputQueue = impl->device->getOutputQueue("camRgb", 30, true);
        impl->outputQueue = outputQueue;
        break;
      }
    }

    // Start fetching encoded frames and pushing them to DepthAISource
    ::std::int32_t 
      DepthAIClient::next_frame(DepthAISource &src) const     // [5]
    {
      push_frame(src);
      return 0;
    }

    void DepthAIClient::push_frame(DepthAISource &src) const
    {
      // Output queue will be used to get the encoded data from the output defined above
      auto h265Packet = impl->outputQueue->get<dai::ImgFrame>();
      ::rust::Slice<const ::std::uint8_t> data = 
        ::rust::Slice<const ::std::uint8_t>(h265Packet->getData().data(), 
              h265Packet->getData().size());
      post_frame(src, data, h265Packet->getData().size());    // [6]
      return;
    }

    std::unique_ptr<DepthAIClient> new_depthai_client()
    {
      return std::make_unique<DepthAIClient>();
    }

  }
}

• [3] This header will be generated by cxx
• [4] Constuct the object and obtain the instance to Dai::device and Dai::outputQueue
• [5] Method called from rust to fetch the next frame
• [6] post_frame is method defined in Rust, that will receive the call when the data is ready from the Camera queue

Now comes the time to use the cxx.

src/main.rs

#[cxx::bridge(namespace = "dev::pnkv")]                         // [7]
mod ffi {
    // Rust types and signatures exposed to C++.
    extern "Rust" {
        type DepthAISource;                                     // [8]

        fn post_frame(src: &mut DepthAISource, 
            data: &[u8], size: u32);                            // [9]
    }

    // C++ types and signatures exposed to Rust.
    unsafe extern "C++" {
        include!("include/depthai_wrapper.h");                  // [10]

        type DepthAIClient;                                     // [11]

        fn new_depthai_client() -> UniquePtr<DepthAIClient>;    // [12]
        fn next_frame(&self, src: &mut DepthAISource) -> i32;   // [13]
    }
}

pub struct DepthAISource {
    file: std::fs::File,
}

impl DepthAISource {
    pub fn new() -> Self {
        Self {
            file: std::fs::File::create("video.h264").unwrap(),
        }
    }
    pub fn close(&mut self) {
        self.file.flush().unwrap();
    }
}

pub fn post_frame(src: &mut DepthAISource, data: &[u8], size: u32) {
    debug("post_frame: {} {}", data.len(), size);
    src.file.write_all(data).unwrap();
}

• [7] - cxx macro to define the bridge
• [8] - definition of rust structure, that will be passed by reference to C++ methods
• [9] - rust external function call back, that is called from c++ code

The rest is syntactic sugar, so that we have the code assistance and verification in rust

• [10] - include the c header
• [11] - define the type that will be implemented in c++
• [12] - the constructor called from rust to instantiate the object
• [13] - function defined in c++ that we will call from rust

How do we build/generate the bridge - at the end cxx needs to generate C++ bindings. This is done using the Rust build scripts

In the project file build.rs we will use cxx_build macro, that needs to be added as development crate.

    cxx_build::bridge("src/main.rs")                             // [13]
        .file("src/depthai_wrapper.cc")                          // [14]
        .includes(&[                                             // [15]
            ".",
            "/usr/include/opencv4",
            "./deps/depthai-core/build/install/include",
            "./deps/depthai-core/build/install/include/depthai-shared/3rdparty",
            "./deps/depthai-core/build/install/lib/cmake/depthai/dependencies/include",
        ])
        .flag_if_supported("-std=c++14")
        .compile("depthai-rust");

    println!("cargo:rerun-if-changed=src/main.rs");              // [16]
    println!("cargo:rerun-if-changed=src/depthai.cc");
    println!("cargo:rerun-if-changed=include/depthai.h");
    println!(
        "cargo:rustc-link-search={}",
        "./deps/depthai-core/build/install/lib"
    );                                                           // [17]
    println!("cargo:rustc-link-lib=dylib=depthai-core");         // [18]
    println!("cargo:rustc-link-lib=dylib=depthai-opencv");       // [19]

• [13] - define the main source to compile with bridge definition
• [14] - add C++ compilation source
• [15] - add the includes needed for the compilation
• [16] - instruct rustc to rebuild the sources in case files change
• [17] - add shared libraries location (this is where depthai-core has built the dynamic libraries)
• [18]/[19] - add shared libraries to be linked with

Note: that before building the project, we have to build the DepthAI core share libraries. (see the README.md and Makefile in the project)

The project can be built using cargo build or you can directly run it using cargo run - in both cases, if you have not installed depthai-core in the standard location - make sure to adjust the Library path - LD_LIBRARY_PATH=$(CURRENT_DIR)/deps/depthai-core/build/install/lib.

The project has been tested only on Linux amd64 system.

~Enjoy