Surveillance with Raspberry Pi NoIR Camera HowTo

There are hundreds of ready-to-use surveillance IP cameras. But the more you look at a specific one the more drawbacks you find. The article will show how to use the RPi camera in a custom surveillance setup. When motion is detected, then spot lights will be turned on and high quality pictures will be taken that are immediately uploaded to google drive.

Rasperry Pi Camera Facts
Dimensions:25 mm x 20 mm x 9 mm
Weight:3 g
Max resolution:5 Megapixel (2592 x 1944)
Sensor type:OmniVision OV5647
Sensor size:3.67 x 2.74 mm
Optical Format:1/4" (Crop factor: 9.4)
Lens:f=3.6 mm, f/2.9
Focal length:4 mm
Video: - 2592×1944 1-15fps, video or stills mode, Full sensor full FOV, default stills capture
- 1920×1080 1-30fps, video mode, 1080p30 cropped
- 1296×972 1-42fps, video mode, 4:3 aspect binned full FOV. Used for stills preview in raspistill.
- 1296×730 1-49fps, video mode, 16:9 aspect , binned, full FOV (width), used for 720p
- 640×480 42.1-60fps, video mode, up to VGAp60 binned
- 640×480 60.1-90fps, video mode, up to VGAp90 binned

(See here for video modes)

Light Conditions - The Key to Good Quality Pictures

In bright sunlight a cheap mobile phone camera can take good pictures because even a small and rather insensitive image sensor gets enough light. Investing in a large and high-quality sensor with high Base ISO and a lens with a large aperture is one way to get better pictures in low light conditions (see ISO explained). But it is a very expensive way. There are two cheaper ways to get better pictures when there is limited sunlight.
First, you can use a camera without an infrared filter as the RPi NoIR Camera. When you check the sunlight spectrum chart on Wikipedia (see here) you will see that there is about as much infrared light than visible light. So you get about twice the illumination if you take pictures without IR filter, in areas where the sun is the light source.
Second, you can use flood light. It is useful to implement the flood lightening with a two step strategy. One LED flood light to detect motion. This needs to have only about 300 lumen (5 Watts LED) and is turned on only at night so that the energy costs are low, but movement can still be detected in the video stream. And as the second step, a strong LED flood light to illuminate the scene. This should have about 4500 lumen (50 Watts LED) or more. This is the light of about 6 - 12 household light bulbs. It is turned on when movement is detected. Even better than one high intensity light, is to use several low intensity lights (e.g. 4 x 1000 lumen) because the light can be directed more granularly. If you do not want the light to be noticeable, then infrared LEDs can be chosen.
These are the two ways that I know and find useful for video recording. Additionally you may use higher quality camera equipment as mentioned at the beginning of the section. Another interesting way is to use software programs that calculate one brighter picture out of several similar pictures (sometimes called hand-held twilight). So you could take pictures or a video with several cheap cameras at once and then let some software combine them to a brighter picture.

What Parts are Needed
  • A Raspberry Pi that is connected to the internet.
  • The Raspberry Pi (NoIR) Camera.
  • A camera housing or at least some double sided tape to fix the camera.
  • A 35 - 700 lumen (0.5 - 10 Watts LED) spot light that can be always on.
  • One or several spot lights with a total of about 3500 - 7000 lumen (50 - 100 Watts LED) that will be on during motion.
  • A relais or remote control AC that will switch LED lights on during motion.
  • Optionally, a wide angle adapter lens that can be attached in front of the existing RPi Camera lens to increase the field of view. Alternatively you can replace the existing lens completely (see here).

Connect and Enable the Camera

See http://www.raspberrypi.org/archives/3890 if your camera is not yet connected.
If the camera is connected to the CSI port and it was enabled in the raspi-config util, then the following command should output a picture raspistill --output test1.jpg.

Update RPi and GPU firmware

Update your existing software with sudo apt-get update && sudo apt-get upgrade -y.
Execute sudo rpi-update to get the newest GPU firmware (see here) and then sudo reboot.

Install UV4L Camera Driver

This Userspace Video4Linux2 (v4l2) driver for the Raspberry Pi CSI Camera Board is needed for motion, MJPG-streamer, SimpleCV, fswebcam and other tools. The following is a summary from linux-projects.org HowTo to get the driver running.

wget http://www.linux-projects.org/listing/uv4l_repo/lrkey.asc && sudo apt-key add ./lrkey.asc  
echo "deb http://www.linux-projects.org/listing/uv4l_repo/raspbian/ wheezy main" | sudo tee -a /etc/apt/sources.list  
sudo apt-get update  
sudo apt-get install uv4l uv4l-raspicam uv4l-raspicam-extras uv4l-server uv4l-uvc uv4l-xscreen uv4l-mjpegstream  
sodu reboot  

After running those commands, the driver will be loaded at boot time because it is linked with /etc/rcS.d/S04uv4l_raspicam and the file /etc/environment will contain the following line LD_PRELOAD=/usr/lib/uv4l/uv4lext/armv6l/libuv4lext.so, which is needed by applications using the driver. After the reboot you can check with ls /dev/video* that the driver created a new video device.
What data is returned when reading from e.g. /dev/video0 is configured in the config file of the driver. It is here: /etc/uv4l/uv4l-raspicam.conf. To make a backup copy of it type sudo cp /etc/uv4l/uv4l-raspicam.conf /etc/uv4l/uv4l-raspicam_backup.conf.
This config file is extremely important. It allows you to set all supported camera properties for images or videos returned when reading from /dev/video0. Well most settings like image size, hue or saturation can also be controlled by the programs using the driver but if not then you can set them in uv4l-raspicam.conf.
To see a list of available configuration options type uv4l --help --driver raspicam --driver-help or see here. There are two interesting commands that show supported settings of the camera device. First v4l2-ctl -d /dev/video0 -l which lists all camera control values and their values like 'brightness', 'sharpness', 'exposuremode', and 'textoverlay' and second v4l2-ctl --list-formats-ext which shows the supported output pixel formats of the Raspberry Pi V4L2 camera driver like 'MJPEG', 'JPEG', 'H264' and 'YU12'(YUV 4:2:0).
After changing the config file, the driver must be restarted with sudo service uv4l_raspicam restart. Sometimes the device cannot be registered as /dev/video0, then you can run sudo pkill uv4l and then restart the driver.

Stream Live Video in Your Browser

If you followed the steps till here, then you can already see your RPi Camera video stream at http://raspberrypi:8080/stream or http://[pi-ip-address]:8080/stream (tested with Google Chrome on Windows 8.1) or in VLC player you can open the network stream at http://[pi-ip-address]:8080/stream/video.mjpeg. The stream is not displayed on Windows Phone 8.1. The stream can only be consumed by one client at the same time. You can even control many settings on the fly from a control panel: http://raspberrypi:8080/panel. Some settings, like the resolution, cannot be changed in the control panel. These must be edited in the driver configuration file and the driver must be restarted afterwards as described in the previous section. If you have DynamicDNS configured and port forwarding configured, then you can access your Camera from the internet (see Free Dynamic DNS Service Subdomain HowTo).
To add password protection for the video stream or to disable it, add the following lines to the /etc/uv4l/uv4l-raspicam.conf core options section and restart the driver.

enable-server = auto                          #'auto', 'required', 'off'  
server-option = --port=8080                   #streaming server port  
server-option = --admin-password=helloworld   #streaming server 'admin' password  
server-option = --user-password=hellow        #streaming server 'user' password  


Taking Pictures and Videos on the Command Line

The command line tools raspistill, raspistillyuv and raspivid are intended to be used with the RPi camera (see here and the UV4L plug-in driver manual here). raspistillyuv seems not to exist in my Raspian Image.
In the next example the shutter speed is set explicitly at 1/400 of a second (2500 micro seconds) so that walking people in the picture are not blurred. Further width and height values are set for a resolution of 1 Megapixel and the jpg quality is set to 75%. The options --nopreview and --timeout 1 will skip the default 5 second preview.

raspistill --nopreview --timeout 1 --shutter 2500 --width 1280 --height 960 --quality 75 --output test2.jpg  

To save as .bmp or .png, simply change the output file name extension. To rotate or flip the image, you can use the --rotation, --hflip and --vflip options. With raspistill you cannot add a text overlay, however with the fswebcam tool this is very easy. You can install it with sudo apt-get install -y fswebcam and take a picture with

fswebcam -v -d /dev/video0 -r 1920x1080 -p mjpeg --title "This picture was taken from directory '`pwd`' at `date`" --no-timestamp --jpeg 100 fswebcam.jpg  

Now let us test how fast pictures can be taken with the following command, which takes as much pictures as possible for 10 seconds

raspistill --nopreview --shutter 2500 --width 1280 --height 960 --quality 75 --timeout 10000 --timelapse 0 --output timelapse_test%04d.jpg  

In 10 seconds, I only get about 17 pictures (7 pictures before the rpi-update). For lower resolutions or with overclocking the RPi's CPU, the amount of pictures does not increase.
Less than two pictures per second is not enough for surveillance because you may miss the frame that allows you to recognize the intruder. So how about using video to generate jpg files? Raspivid cannot output jpg files but avconv can. You can install it with sudo apt-get install -y libav-tools. So the following command will give you 5 jpg files per second for 10 seconds from the /dev/video0 device with pretty good quality (qscale 2) but with lower resolution

avconv -an -r 25 -s 640x480 -f video4linux2 -input_format mjpeg -i /dev/video0 -r 5 -qscale 2 -t 10 -f image2 video_frame_%04d.jpg  

(see avconv -h or avconv Doc for an explanation of the options). Note that you can change the driver settings in /etc/uv4l/uv4l-raspicam.conf to influence the avconv output.
Raspivid is simpler to use as avconv but it can only record .h264 (=MPEG-4 but != mp4) video files. Here are a few examples:
10 second video with 1 Mpx resolution in wide format with 49 frames per second

raspivid --nopreview --shutter 2500 --width 1280 --height 720 --framerate 49 --timeout 10000 --output video_1Mpx_49fps_wide.h264  

Same as above but with low bitrate to reduce file size

raspivid --nopreview --shutter 2500 --width 1280 --height 720 --framerate 49 --timeout 10000 --bitrate 2000000 --output video_1Mpx_49fps_wide_low_bitrate.h264  

Standard format (not wide) but with only 42 frames per second

raspivid --nopreview --shutter 2500 --width 1280 --height 960 --framerate 42 --timeout 10000 --output video_1Mpx_42fps.h264  

Lower resolution but with 90 frames per second

raspivid --nopreview --shutter 2500 --width 640 --height 480 --framerate 90 --timeout 10000 --output video_vga_90fps_slow_motion.h264  

To convert a video to mp4, which is playable in the browser and even on Windows Phone

avconv -i video_1Mpx_49fps_wide_low_bitrate_2000000.h264 -vcodec copy video_1Mpx_49fps_wide_low_bitrate_2000000.mp4  

To convert a series of jpeg files into an mp4 video (very slow)

avconv -qscale 5 -r 5 -b 9600 -i video_frame_%04d.jpg timelapse.mp4  

To extract one jpg file per second (-r 1) from a h264 video with a frame rate of 49 (-r 49).

avconv -r 49 -i video_1Mpx_49fps_wide.h264 -r 1 -f image2 Out%00d.jpg  


Why FFMPEG Needs No Update

ffmpeg is a tool to encode or decode videos and it does a lot more. If you have the package libav-tools installed and type ffmpeg -version on the console, then you will see the text "ffmpeg version 0.8.13" and the text "*** THIS PROGRAM IS DEPRECATED *** This program is only provided for compatibility and will be removed in a future release. Please use avconv instead.".
Well as I learned, Debian, Ubuntu and Raspian do not contain a version of ffmpeg from FFmpeg.org but the ffmpeg tool is supplied by libav.org/avconv. That is why it also says "Copyright (c) 2000-2014 the Libav developers" and why it uses libav libraries. (See here)
So it is not necessary to update ffmpeg to a newer version than 0.8.13 because it is just a wrapper to avconv. But it is possible to install the ffmpeg version from FFmpeg.org but I did not find a package for the RPi for it.

Advantages of Motion Detection with Motion

At first I thought motion detection with a camera was only a nice side effect of having a camera for surveillance. Now I believe that motion detection via video gives better results in many situations even if it is only 2D.
Standard motion detection sensors use passive infrared, ultrasonic or a combination of both. There are also microwave motion sensors that operate based on the Doppler Effect. Advantages of video based motion detection in comparison to passive infrared and ultrasonic are:
Observed area --> With video based motion detection you can detect motion hundreds of meters away, if you use a telephoto lens. Additionally the program 'motion' allows you to configure in which areas of an image it should watch for change (see area_detect and mask_file).
Filter options for noise --> The program 'motion' has a number of options with which you can minimize false motion events. For example threshold is the number of changed pixels after applying noise filters, minimum_motion_frames is the number of picture frames that must contain motion, lightswitch is the amount of light change to ignore, smart_mask_speed is how fast a self-learning mask, that will ignore frequently changing areas like wind blowing grass or trees, is updated, noise_tune uses an algorithm to automatically set and adjust the noise tolerance if the noise_level is not explicitly specified, and despeckle will Erode and/or Dilate the image to reduce noise. (See Motion Guide - Basic Features)

Installing Motion

Surveillance is pretty boring if nothing happens. So the goal is to record only motion and this in the best possible quality.
I will use the program 'motion' to detect change. There is also a fork called motion-mmalcam. It was necessary to use the fork before the UV4L Camera Driver existed. However I did not find additional features in motion-mmalcam that is why I am using motion.
To install 'motion' type sudo apt-get install -y motion. To make a backup copy of the config file type sudo cp /etc/motion/motion.conf /etc/motion/motion_backup.conf, then ensure that user 'pi' is in group motion to be able to read /etc/motion/motion.conf with sudo usermod -a -G motion pi.
You must exit the terminal and login again for the change to take effekt.
Start motion with motion. Using LD_PRELOAD=/usr/lib/uv4l/uv4lext/armv6l/libuv4lext.so motion should not be necessary because /etc/environment already defines the variable. If it says "Error selecting input 0 VIDIOC_S_INPUT: Device or resource busy" then you will likely have the live HTTP stream on http://raspberrypi:8080/stream open. Apparently there can only be one program using /dev/video0.
You will also see the message "Started stream webcam server in port 8081". Unfortunately the documentation is right by saying "You cannot watch the stream with most browsers. Only certain versions of Netscape work" (see MotionDoc - Webcam_Server). This is because the pixel format 'YUV 4:2:0' is not supported by most browsers.
But you can open the stream with VLC. For this, stop motion with Ctrl-C. Then open /etc/motion/motion.conf and change the value "webcam_localhost on" to "webcam_localhost off" (off means to allow non-localhost computers access). You may also want to set the values "control_localhost on" to "control_localhost off" and "control_port 8080" to "control_port 8082". The port of the http interface to view and edit settings must be changed because port 8080 is already used by the http live stream of the UV4L driver. Start motion again. Now you can open the stream in VLC http://[pi-ip-address]:8081/video.mjpeg.
To make motion start at boot time, open /etc/default/motion and change "start_motion_daemon=no" to "start_motion_daemon=yes". Additionally open /etc/init.d/motion and insert
export LD_PRELOAD=/usr/lib/uv4l/uv4lext/armv6l/libuv4lext.so at line 57, underneath the line "chown motion:motion /var/run/motion". Then sudo reboot and check that motion is running by opening the stream with VLC. The camera LED should be permanently on. You can find log messages in the file /var/log/syslog. Every time a motion is detected, motion will save pictures and a video in /tmp/motion. So move the camera or walk in front of it if you want to see pictures in /tmp/motion.
As said before, if one program is using the camera, then no other program can use the camera. So to make the camera available for other programs you can stop motion with sudo service motion stop.

Configuring Motion

Now it is time to optimize the output of the 'motion' tool. The options 'width' and 'height' are very sensitive to CPU consumption. If these are increased too much, then the frame rate will drop. With a resolution of 640x480 the 'motion' program on the RPi is able to produce 3 to 4 pictures per second plus the swf video at almost 100% CPU usage. With the same values but without video motion generates 5 to 7 pictures per second on the RPi. Even though the resolution is disappointing, it has the advantage of saving bandwidth when the images are uploaded to google drive. Now I am wondering what resolution the Banana Pi can achieve with a very similar setup.
Before setting the config values create a new folder where the pictures will be saved with mkdir -p /home/pi/google-drive/camera and set the motion user as owner sudo chown motion:motion /home/pi/google-drive/camera.
To set the config values, open /etc/motion/motion.conf, set target_dir /home/pi/google-drive/camera, width 640, height 480, threshold 6000 (threshold was 1500 so about 2% of 320*240, the new value 6000 is about 2% of 640*480), minimum_motion_frames 2, pre_capture 4, framerate 5, ffmpeg_cap_new off, jpeg_filename %Y-%m-%d/%H-%M-%S--%q.
Now open /etc/uv4l/uv4l-raspicam.conf, set exposure = fixedfps, shutter-speed = 2500.
Now execute sudo service uv4l_raspicam restart and sudo service motion restart.

Synchronization with Google Drive

The direct upload of jpg files from motion to Google Drive is described in a separate article here.

Further reading

Use cases about the V4L2 driver for the Raspberry Pi Camera Module
http://www.linux-projects.org/modules/sections/index.php?op=viewarticle&artid=16

No-IR USB Camera with wide Focus
http://www.delock.de/produkte/F_1068_Kameras_95953/merkmale.html

mjpg-streamer
https://code.google.com/p/mjpg-streamer/

GStreamer
http://gstreamer.freedesktop.org/

Turn your Raspberry Pi into a live HDTV transmitter
http://www.oz9aec.net/index.php/dvb/490-turn-your-raspberry-pi-into-a-live-hdtv-transmitter

Raspberry Pi Streaming video
http://www.mjoldfield.com/atelier/2013/12/raspivid.html