Low-Cost Surveillance Robot Using Raspberry Pi

Introduction

The progression in the field of Artificial Intelligence, Big Data Analytics and Sensor Fusion combined with miniaturized computing is not only helping Fortune 500 companies in better operational management but also helping Law Enforcement Agencies in Real-time Monitoring and action, all these technological changes are sprouting seed of change in different ways in our life.

Around the world, multiple government and private organizations like Tesla, Google, Facebook, and Uber are investing in autonomous vehicles capable for plying on busy roads. Tesla's fully self-driven Model S car is available for purchase.

Digital Voice Assistants like Siri, Cortana, and Google Assistant are changing the way we interact with our mobile phone, every organization is investing in Chat Bots to automate multiple manual activities.

Amazon Alexa, Google Home or open-source Mycroft Mark 1 are examples of voice-assistant gadgets that could be used to interact with your internet enabled smart devices at home.

Top notch companies (like Google, Microsoft, Apple, Facebook, Amazon) who have developed cognitive services for serving their customers are bringing those services to their cloud platform in form of APIs for developer’s usage.

Along with above-mentioned proprietary technologies multiple Open Source technologies are available in form renowned OpenCV a Computer Vision SDK, AI platforms [Tensor Flow, Open AI] along with open-source single-board hardware [Raspberry Pi, Arduino, Orange Pi, Banana Pi, Beagle Board] are bringing advanced technologies for consumption of developers.

Considering that open-source and proprietary software along with low-cost hardware components are available for developers and with the aim to create an autonomous robot worked on a project called "Kwik".

Introducing Kwik

"Kwik" is a low-cost robotics solution, it combines some of the above mentioned Open source and proprietary technologies.
Goal for first version of Kwik (i.e. Technology Demonstration 1 [TD1]) was to build an Unmanned Ground Vehicle (UGV) capable enough to be remotely controlled over internet and respond to voice command, object tracking and following could be part of TD2 and ultimately TD 3 with autonomous navigation assisted by GPS and computer vision.

Kwik TD1 is capable of:
1.    Remotely controlled using custom Mobile application.
2.    Share Live camera feed over 4G Network/Wi-Fi to Remote Control application for navigation.
a.    Send live feed to YouTube, Azure Live Stream or other RTMP compatible Media Services
b.    Capture single or multiple snapshots for time-lapse video
c.    Video recording
d.    Support HTTP & RTSP streaming.
3.    Move forward, reverse, left or right, apply the brake, speed up or slow down.
a.    Support differential drive
b.    Automated collision avoidance by sensing obstruction both in forward and reverse movement.
4.    Control camera Pan/Tilt movement.
5.    Remotely control all Kwik capabilities using voice command.





Kwik also has the capability to act as voice control digital assistant using Mycroft deployment.

In next section we are going to look into tools & technologies used to develop Kwik, we will also try to answer some of the relevant questions like What & Why while how to be part of future papers.

Hardware Components


Component
Quantity
Description
Raspberry Pi 3
1
Single board computer capable of translating user command in action, acts as IoT Gateway, State Machine, and Video Streaming device.
16 GB Class 10 SD Card
1
Storage device, Class 10 provide better data transfer rate which is important for system performance.
5V Power Supply
1
Power supply for Raspberry Pi, acts as Power Bank charger also
12V DC Motor 150 RPM
4
Connected to wheels thus supporting  movement
L298 Motor Driver
1
Motor Driver to control DC Motor direction and RPM
Chassis
1
Chassis with dimension to support differential drive
Wheel 4"
4
Wheels with rubber groove for maintaining adequate friction
Resistors
Multiple
Multiple 1K and above resistors.
SG 90 Servo Motor
2
(optional)
Low powered Servo Motor to controlling camera mount pan and tilt movement
Breadboard
1
Basic circuit design and interconnectivity of multiple components
HC SR04 Ultrasonic Sensors
2
(optional)
Distance sensor for front and back obstacle detection
Male & Female Jumper Wire
Multiple
To connect GPIO pins with Breadboard and other components
2 Axis Pan Tilt Camera Mount
1
(optional)
Servo motor mount to support pan and tilt movement
Logitech C 310 Camera
1
For Video Streaming and capturing a snapshot.
List of supported USB camera on Raspberry Pi: http://elinux.org/RPi_USB_Webcams
12V Lithium-Ion Battery
1
Power pack for 4 DC Motors
12 V Battery Charger
1
(optional)
12 V battery charger
10000mah Power Bank
1
Power bank with 5V output to support Raspberry Pi operation.
Bluetooth Speaker
1
(optional)
Play audio
Kinobo USB 2.0 Microphone
1 (optional)
Capture surrounding audio
Android Phone
1
Host of Remote control application

Raspberry Pi 3

Raspberry Pi 3 is a palm-size single board computer that packs the power of 1.2GHz 64bit quad-core Cortex-A53 processor along with 1GB of RAM.  With power usage of just 5V its designed to be network-centric and well capable to support Internet Of Things (IoT) projects, it has built-in support for WIFI, Bluetooth, LAN, 4 USB ports, HDMI & 3.5mm jack for audio output and bunch of GPIO pins for connectivity with multiple sensors. Priced around $35 (INR 2500) makes it most popular single board computer.


While Raspbian (Debian based Linux OS) is officially supported OS there are multiple third-party Operating systems like Microsoft Windows 10 IOT Core, Ubuntu Mate, and RISC OS.

Why Raspberry Pi 3?

Raspberry Pi 3 around the size of wallet combines capability of open source Linux and low-cost hardware, it offers superb performance allowing it to be used for developing a custom solution.

You can use your preferred programming language or framework to develop and deploy an application on Raspberry Pi, backed by opensource communities the platform provide support to not only the below-mentioned programming language but much more.
    1. JAVA
    2. C / C++
    3. Python
    4. Node.JS
    5. C#
   
Some of the use cases where you can develop your solution or even deploy pre-built open source solutions on Raspberry Pi could be:
    • Low-cost desktop for internet access.
    • Home Theater system with Kodi
    • Media server for file storage
    • Web hosting
    • Low-cost robotics
    • Build your own Bot using Alexa
    • Build your home security system
    • Build your own supercomputer by stacking multiple boards.
    • Build your own Car Entertainment System
    • Build home automation system

Considering that Raspberry Pi3 has built-in support for Wi-Fi, Bluetooth, HDMI, multiple USB ports backed by largest open-source community and ever-improving Raspbian OS, optimal performance at a reasonable price are the main reason why to select Raspberry Pi3 over other single board devices.

DC Motor AND Servo Motor

DC Motors are fast rotating motor where a change in Pulse Width Modulation (PWM) could change Rotation Per Minute (RPM), DC motors are ideal for scenarios where higher RPM is required e.g. Fan, Wheel.
Servo Motors are basically DC motors but they are designed to provide higher torque than DC and provide precise control over angular movement i.e.  You can instruct servo to move 45 degrees or 80 degrees depending on requirement.

The servo doesn't require motor driver like L298N as they have the built-in capability but they do require controller like Arduino or Raspberry PI to control PWM. Common use cases are Gears, a robot arm or even aircraft wings.



Why DC Motors & Servo are required?

In case of Kwik TD1 four 12V DC motors are used (i.e. connected in parallel two on each side), L298N Motor diver is used to change PWM and thus RPM.  These 150 RPM motors are responsible for rover movement.

Two SG 90 Servo Motors are used for control Camera mount Pan & Tilt movement, these moves can be controlled remotely. 

L298N Motor Driver

L298N Dual H-Bridge Motor controller is typically used to control motor speed and direction, using this module one can control 2 DC motors or single stepper motor, with maximum voltage of 35V DC you can have larger DC motor as per requirement, 5V output could be used connect other devices like Servo Motor, HC SR04 Ultrasonic Sensors or even Arduino board.


Why L298N Motor Driver Required?

Kwik TD1 as mentioned is capable of moving forward, reverse, left or right, apply the brake, speed up or slow down, differential drive, all these capabilities requires system need to have control over speed and direction of motors which is achieved changing Pulse Width Modulation (PWM) using an L298N motor driver. 

What's Differential Drive?

Differential Drive is a concept of changing speed and direction of each side DC motor and thus achieving change in direction of rover. To make it simpler:
·        If rover has to move towards left, increase in forwarding velocity of left side wheel and change in direction (i.e. reverse) of right side would result in changing direction.
·        If rover has to move towards the right, increase in forwarding velocity of right side wheel and change in direction (i.e. reverse) of left side would result in changing direction.

  

HC SR04 Ultrasonic Sensors

Basically, HC SR04 Ultrasonic Sensors are distance sensors and are used to detect an obstacle within range of 2 - 400 cm, as the name suggests these sensor work by sending ultrasonic sound and capture reflection to detect distance.

Why HC SR04 Ultrasonic Sensors Required?

In Kwik TD1 there are two Ultrasonic Sensors, first at the front and second on back, each gets activated based on the movement i.e. front sensor on forward and vice versa.
As there is lag between rover actual movement and control these are required to put emergency brake in-case of any nearby obstacle and stop collision which may damage rover’s components.

Snapshots - Hardware Integration






Circuit Blueprint


Software Components

 Raspberry Pi Deployment:

Tools & Technology
Description
Raspbian
Raspbian is a Linux Debian-based operating systems for Raspberry Pi, the current version is PIXEL.
Apache Qpid Proton
Qpid is open source message queuing system which implements Advance Messaging Queuing (AMQP) Protocol
Tornado Web Server
Written using Python, Tornado provide capability to communicate over web socket
Python
Programming language of choice
pico2wave
Text to speech library
Azure IOT Hub
Azure IoT Hub Python SDK for device connectivity over cloud
Azure Storage
Kwik Configuration storage
PIGPIO
GPIO library built in C provide connectivity to different sensors, used PIGIO library instead of default Python GPIO library as it provides better performance with Servo Motors
VNC
Remote Desktop Sharing tool
VLC Player
Media player and Video Encoder to support RTSP protocol along with Video recording
( VLC is currently unsupported on PIXEL)
FFMpeg
Media Player and Video Encoder to support RTMP protocol
MJPG Streamer
HTTP Server to support video streaming.
Web Socket Client
Client for Web socket communication
Git
Get the latest version of the application on board.
PyGame
For capturing snapshots
Dataplicity & Ngrok
Creating Cloud to Device Secure tunnel, control Kwik over internet

Python

Python is interpreted, object-oriented, general purpose, high-level programming language, it's widely used across domains be it Web Development, Desktop Application, Network Programming, Big Data Analytics,  Robotics and Artificial Intelligence to name few, thus by learning Python you could start developing an application in a preferred domain.

Python is open source thus it freely usable and distributable, even for commercial use, you can use text editors like  Sublime Text, NotePad ++ to start Python Development, IDE like  Eclipse, VS 2015, PyCharm can also be used at advance level.

Why Python?

Python is an open-source and object-oriented programming language & it's easy to learn & use, powerful, versatile and increase developer productivity.The syntax of the language is designed to be readable.
Python is incredibly popular, powerful dynamic programming languages that's the reason why companies like Google, Dropbox, Spotify, and Netflix use it. 

·         Allows developers to develop & run the code anywhere, including Windows, Mac OS X, UNIX, and Linux.
·         These are multiple Python Version designed for specific context, below are few examples
o    Anaconda: for large-scale data processing, predictive analytics, and scientific computing
o    PyGame: Python game engine
o    SciPy: Data science
o    IronPython: Version targeting the .NET Framework and Mono
o    Django: Web application development
·         Offers automatic memory management.
·         Simple to extend the code by appending new modules that are implemented in other compiled language like C++ or C.


In context of Kwik TD1 Python is the base programming language for development and integration of different components , primary reason for Python selection where its capability to develop application on windows and deploy application on Raspbian, another important reason was availability of Open-Source Artificial Intelligence algorithms, modules, and community on Python that are required for future development of Kwik. 

Apache Qpid Proton

Apache Qpid Proton library is an open source lightweight messaging library for communication between applications, based on AMQP protocol it can act as a message broker, apart from acting as a broker it also has the built-in capability like reliable queuing, topic-based publish-and-subscribe messaging, flexible routing, transactions, and security.
Because of AMPQ protocol capability to provide secure and reliable messaging queue, it's being used across the world, Aadhaar project implements AMPQ which is world largest biometric database with more than 1 billion identities already stored. Google and NASA have been using AMQP within Cloud and Complex Event Processing (CPE) infrastructure. 

Why Qpid Proton Is Required?

Message Queue Telemetry Transport (MQTT) & Advance Message Queuing (AMQP) are two of the most common messaging protocols being used in IoT implementation, two major consideration for selecting AMQP over MQTT was capability of AMQP to provide secure & reliable queue and also compatibility with Azure IoT Hub & Event Hub both of which are AMQP based.
Additional Reading:

In Kwik TD1 Qpid Proton acts as a message broker between State Machine and Tornado/Azure IOT Hub. 

Tornado Web Server

Tornado is open-source Python-based asynchronous high-performance web-server because it’s non-blocking it can handle thousands of connection request thus making it ideal for web-socket implementation. 

Why Tornado Web Server Is required?

Kwik TD1 exposes it's remote control capability  (in LAN) through web-socket implementation using Apache Tornado Web server, Mobile app sends a command to Kwik using exposed Web-socket which in turn transfers the command to Apache Qpid Messaging broker for State machine to capture and take appropriate action.

Ngrok implementation helps in exposing Web Socket connection over the internet and thus supporting remote control, Ngrok in association with Azure IoT Hub provide dual communication layer.  

Azure IoT Hub

Azure IoT Hub provide duplex communication between client and device and vice versa, thus allowing you to securely connects, monitors, and manages IoT devices, AMQP protocol is supported by IoT Hub. Integration with Azure Cortana Analytics suit helps developers in telemetry analytics.

Why Azure IoT Hub Implemented?

Secure Duplex messaging queue is one most important consideration for picking up Azure IoT Hub, the duplex queue not only provide device capability to send telemetry signal to cloud but also provide cloud client to send a command to the device. 

VLC Player, FFMpeg, MJPG Streamer & PyGame

VLC Player, FFMpeg, MJPG Streamer & PyGame are open-source executables with media streaming support, while VLC Player is an all-purpose media player with support to play almost all video/audio codec and also support media streaming.
FFMpeg, MJPG Streamer are good at Audio-Video (AV) streaming, PyGame is good at capturing and playing with images at runtime. 

Why these players are required?

Initial requirement behind Kwik TD 1 was it should be able to stream media to a remote device which could allow better control over it, to complete the requirement you need to implement below capabilities: 

HTTP Streaming :

MJPG Streamer captures images at a rate of almost 5 frames per second (fps) in case of MJPG codec and around 30 fps in case of YUV codec, YUV codec is a uncompressed format and while streaming it consumes more than 90% of CPU.
Logitech C 310 has built-in support for MJPG codec and streaming using MJPG consumes around 5-10% of CPU. 

RTMP Streaming

FFMpeg streamer have built-in support for multiple AV codec, these codec are required for encoding video/audio stream, you can play with  FFMpeg output in a different way, RTMP protocol is one of them, almost all Live streaming sites YouTube, Azure Media Service, Facebook Live and many more supports RTMP protocol , FFMpeg  RTMP output is directed to these sites. 

RTSP Streaming & Recording

VLC Player is an all-purpose media player with support to play almost all video/audio codec and also support media streaming, its capabilities are used to stream RTSP stream over LAN and also record video.

NOTE: Currently VLC player is unsupported on Raspbian PIXEL 

Continuous Snapshot  

MJPG Streamer captures images at a rate of almost 5 frames per second (fps) in case of MJPG codec and around 30 fps in case of raw, these frames could be stored images. 

Snapshot

PyGame an open-source library is used to capture snapshots, this can easily be implemented with MJPG Streamer but PyGame also provide you the capability to play with images e.g. changing contrast, brightness etc.

Kwik TD-1 Layered Architecture



Kwik Android App

A custom Android Mobile App developed to remotely control Kwik on LAN & Internet, currently, Mobile App is responsible for translating user actions (touch and voice command) to Kwik Command, it also acts as a host for Kwik live camera feed thus allowing remote navigation.

Tools and Technology
 Description
Xamarin
Development of Android Application using C# programming language
WebSocket Client
Library for web socket communication
Azure IoT Hub Client
Library support duplex communication with Raspberry Pi
Azure Storage
Library to fetch configuration details from Cloud
Microsoft LUIS Engine
Language understanding system allowing intent identification 
Google Speech To Text API
Library to convert speech to text
Google Text To Speech API
Library to convert text to speech
Stateless
Application State management library

Xamarin

Xamarin is a platform to develop a native cross-platform mobile app, currently owned by Microsoft it's based on Mono platform thus allowing a developer to develop an application for Android & iOS platform using C# programming as language.

It’s a suit designed keeping complete Mobile Development Life Cycle in consideration, you can start writing code in Visual Studio or Xamarin Studio and debug and deploy an application on iOS or Android, Xamarin also provides capability to test your application on multiple devices over the cloud. Backed by Microsoft .NET Framework and Azure SDK it provides rich set of libraries and application that you can consume while developing an application.
·         Allows sharing of same code base across different platform
·         Capability to use Native libraries and features.
·         Xamarin Dev environment isn't only limited to Mobile app but can be used in Game Development, IOT Devices or Wearables

Why Xamarin for App Development

Xamarin allows you to develop a cross-platform mobile app using same code base i.e. you can develop an application for Android and with platform specific changes you can deploy it on iOS, almost 70-80% of code be reused. This was one of the major reason for selection of Xamarin. 

Google Speech to Text

Google Speech To Text (STT) is easy to use library which enables developers to translate Speech to text, supports common language, on each search request audio is recorded and transferred to Google Cloud infrastructure to get processed using advanced machine learning algorithm, the ML algorithm translates it and returns it back to the device.

Additional Reading: https://cloud.google.com/speech/ 

Google Text to Speech

Google Text To Speech (TTS) API enables developers to convert text to audio, designed as screen reader its multi-lingual support is a boon for a developer who wants to develop an accessible application. 

Why Google STT & TTS Used?

Google STT & TTS APIs are core technologies in translating Voice Command to Kwik Command, while Google STT in association with Microsoft LUIS Engine is responsible for command translation Google TTS is responsible for translating Kwik feedback to speech. 

Language Understanding Intelligent Service (LUIS) 

LUIS is part of Microsoft's Cognitive Service and is designed to consume sentences in text format, interpret it and return intent and relevant entities. Exposed as REST API it can support English, French, Italian, Spanish, and Chinese. 

Why LUIS is used even if Google STT is already implemented? 

In Kwik context Google STT is responsible for translating speech to plain text but for extracting intent out of text LUIS engine is used, issue with Speech based algorithms is they return multiple outputs against each sentence (for example if user says "pan camera to right”, STT may return "fan camera to right", "pen camera to write", "pan camera to write", "pan camera to right").

A custom logic is used to extract the best context matched text and send to LUIS, LUIS engine now plays role in properly identifying user intent, once intent is to identify it gets translated in Kwik commands.

Kwik TD 1 Mobile Application Layered Architecture



Message flow between Layers


Challenges and Learning

Kwik as a project is first on many fronts, not only is first IoT project but first venture into hardware and software integration on Raspberry Pi.

Challenges

1.    Integration of Azure IoT Hub platform onto Raspberry Pi and enable duplex communication between Kwik Mobile App and Kwik Raspberry Pi app.
2.    Live video streaming to YouTube, Azure Media Service, and other website is one of the major challenges.
a.    Supported protocol and library understanding, providing support for RTMP protocol which is most common proton for live streaming
b.    Raspberry Pi isn’t well suited for Video streaming using RTMP protocol due to low comping power, handling lag while streaming is still a major challenge, have switched from YUV only supported camera to MJPEG supported camera to lower computation required for video encoding and thus allowing other application to run properly.
c.    Streaming over Mobile network. MJPG Streamer implementation provided the best solution with lowest CPU usage while using a MJPEG supported camera, YUV war stream encoding consumes 80-90% of CPU usage.
3.    Handling power to weight ratio, controlling Kwik movement on a plain is easier than a rough surface.
a.    Forward and reverse movement requires all motors to run in the same direction so motors are capable enough to support movement even on rough surface.
b.    Left & Right movement i.e. differential drive requires motors on each side to run in a different direction and at variable speed, this causes motors to be underpowered to run smoothly on a rough surface. Bigger motor to be used next version.

Learning

1.    Apart from LUIS engine, Azure Event Hub and Azure Storage all other components in both Kwik Raspberry Pi App and Mobile App stack were first to work upon.
2.    Integration of different devices sensors on Raspberry Pi platform.
3.    First ever implementation of Python
4.    Understanding of different Live Streaming protocols and libraries that could be used.
5.    Azure IoT Hub SDK isn’t available on Xamarin Android platform, experimented with multiple libraries and ultimately AMQP Lite library worked well for Cloud-to-Device messaging
6.    Integration of Google Speech-To-Text with LUIS engine & Text-To-Speech integration into Mobile application.
7.    Experimentation with different and implementation of Pico2wave library integration on Raspberry Pi as Text-To-Speech library
8.    Creation of simple circuit using BreadBoard.
9.    Installation of Mycroft and integration of Kinobo USB Microphone for gathering speaker's voice.

Next Steps

1.    Implementation of Differential Drive Kinematics.
2.    Implementation of Computer Vision for object tracking.
3.    Integration of Mycroft with Kwik App thus allowing Kwik to act as voice-based digital assistance.
4.    Support for Hindi along with English for Kwik control.
5.    Provide Google Cardboard based UI for Kwik Navigation.


Comments

Post a Comment