JeVois: Open-Source Quad-Core Smart Machine Vision Camera

JeVois combines a video sensor + quad-core CPU + USB video + serial port, all in a tiny, self-contained package (28 cc or 1.7 cubic inches). Insert a microSD card loaded with the provided open-source machine vision algorithms (including OpenCV 3.2), connect to your desktop, laptop, and/or Arduino, and give your projects the sense of sight immediately.

The smart camera captures video frames and directly processes them on the quad-core processor that is in the camera. Results are streamed over USB to a host computer, and/or over serial port to a micro-controller. The JeVois smart camera can find interesting objects in cluttered scenes using a visual attention algorithm, can recognize some of these things using deep neural networks, can detect and decode QR-codes, barcodes, and ArUco augmented-reality markers, can detect and follow roads, can detect human faces, or can even be used as an eye-tracker. Many of these algorithms operate at 30, 60, or 120 frames/s. The software is open-source so that anyone can implement their own machine vision algorithms.

Included machine vision capabilities:

- Visual attention: find interesting things in cluttered environments
- Gist: recognize different places, e.g., recognize a kitchen vs a bedroom
- Read digits, recognize objects using deep neural networks
- ArUco: detect & decode simple patterns
- Detect & decode Qr-codes and bar codes
- Background subtraction: find moving things
- GPU image processing examples
- NEON-accelerated image processing example
- Build your own 120Hz eye tracker
- Dense SIFT feature extraction
- Quad-core edge detection demo
- Detect and track objects by their color
- Fast optical flow
- Detect and follow roads for autonomous cars
- Detect and recognize objects using SURF
- Segment images into super pixels
- Save video to micro-SD for later analysis

Hardware and Software frameworks:

The JeVois framework operates as follows: video is captured from the camera sensor, processed on the fly through some machine vision algorithm directly on the camera's own processor, and the results are streamed over USB to a host computer and/or over serial to a micro-controller.

To the host computer, the JeVois smart camera is just another USB camera. Different vision algorithms are selected by changing USB camera resolution and frame rate. Users or machines can also interact with the JeVois smart camera, change its settings, or listen for text-based vision outputs over serial link (both hardware serial and serial-over-USB are supported).

Three major modes of operation:
Demo/development mode: the smart camera outputs a demo display over USB that shows the results of its analysis, possibly along with simple results communicated over serial port (e.g., coordinates and content of any Qr-code that has been identified).
Text-only mode: the smart camera provides no USB output, but only text strings, for example, commands for a pan/tilt controller.
Pre-processing mode: the smart camera outputs video that is intended for machine consumption, for example an edge map computed over the video frames captured by the camera sensor, or a set of image crops around the 3 most interesting objects in the scene. This video can then be further processed by the host computer, for example, using a massive deep neural network running on a cluster of high-power GPUs to recognize the three most interesting objects that the smart camera has detected. Text outputs over serial are of course also possible in this mode.

But, really, anything is possible, since the whole JeVois software framework is open-source.

Specs at a glance:

The JeVois smart camera is a complete Linux computer. It can run on its own, without a host PC. Thanks to the cooling fan, it can run under full processing load without overheating and while maintaining a constant 1.34 GHz CPU speed.

Developer friendly

- All software open source (GPL)
- Full Linux operating system, boots in 5 seconds
- Custom kernel modules for camera & USB
- Buildroot framework to easily add software packages and create SD image
- JeVois C++17 video capture, processing & streaming framework
- Switch machine vision modules on the fly by changing output resolution
- Download pre-programmed machine vision modules or create your own
- CMake build system
- Full cross-compiler suite (compile all software on your desktop)
- Compile and run the same software on desktop and on JeVois hardware, at the same time (very useful for development & debugging)
- Operating system and vision software all stored on micro-SD card, hacker-friendly and unbrickable. Smart camera can use microSD to save data.

- Included libraries:
• Neuromorphic algorithms for visual attention & scene understanding
• OpenCV 3.2 machine vision algorithms
• All opencv-contrib modules (object recognition, ArUco, etc)
• ZBar library for barcode & QRcode detection and decoding
• tiny-dnn library for deep convolutional neural networks
• GPU-accelerated image processing using OpenGL ES2.0 shaders
• Support for NEON multimedia processor instructions
• Vlfeat library for visual feature computation
• OF_DIS library for fast motion flow computation
• Eigen3, TBB, OpenMP, etc

Host Computer Requirements

The JeVois smart camera can work as a standalone computer, with no USB video streaming. In such case, one would usually simply stream commands to an Arduino or similar over the serial port. All you need then is to provide power to the JeVois camera's mini-USB connector.
For video streaming over USB:

Linux (including Raspberry Pi): Works out of the box, no drivers needed, all functionality is available. You can switch between different vision processing modes on the fly and at runtime, by selecting different camera resolutions and frame rates on your host computer. For example, [email protected] may run the visual attention algorithm, while [email protected] may run the QRcode detection algorithm. A configuration file on the MicroSD card establishes the mapping between USB video resolution and frame rate, camera sensor resolution and frame rate, and vision algorithm to run.

Windows, Mac OS X: Streaming works but selecting among different available video resolutions has become increasingly difficult with newer versions of Windows and OS X. It seems that these operating systems do not want to let you choose a camera resolution or frame rate. This choice, however, is how one can switch on the fly between different vision algorithms on the JeVois camera. One workaround is to configure your JeVois smart camera with only one vision algorithm (achieved by commenting out entries for other resolutions in a configuration file on the MicroSD card). Then the host computer will have no choice but to use that one.

Android: The camera is detected but streaming video does not work yet. We are working on this and we suspect this has to do with deviations from the USB video class (UVC) standard on the Android side. For example, a USB packet sniffer has revealed that the Android device insists on querying camera controls that were not declared as being supported by the JeVois hardware.

iOS: The JeVois smart camera, and any other USB camera we have tried, are currently reported as not being a supported device. This may change in the future.

To learn more about JeVois do visit their KickStarter page and make a pledge if you are interested.

Source: Kickstarter