A Video4Linux “proxy” masking a webcam’s background, using TensorFlow and DeepLab. This is just a fun side project distractaiding the current the Covid-19 WFH situation—it does not even have a proper name. If you’re looking for a polished product, there’s Snap Camera, XSplit VCam and others.

That said, I’m happy for pull requests. Please be patient, I suck at responding to anything.

# Add Bazel repository, if necessary.
if ! apt-cache show bazel >/dev/null; then
  curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add -
  echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
fi

# Install build dependencies.
sudo apt install bazel bazel-2.0.0 cmake make libgflags-dev libopencv-dev libgl-dev libegl-dev

# Clone and build repository.
git clone --recursive https://github.com/yath/bgremover
cd bgremover
make -j4

# Install v4l2loopback DKMS module.
sudo apt install v4l2loopback-dkms

Load the v4l2loopback module: sudo modprobe v4l2loopback max_buffers=5 exclusive_caps=1. exclusive_caps prevents the loopback device from being opened for recording for a second time, which is necessary for Chrome. max_buffers=5 is just a reasonable looking amount and can probably be changed.

The make invocation from the previous section should have produced an executable (relative to the project root directory) build/bgr and downloaded a deeplabv3_257_mv_gpu.tflite. The binary expects the .tflite file to be present in the current working directory, so running build/bgr from the project root direct should just work. If you’d like custom backgrounds, place them into $PWD/backgrounds/*.{jpg,png,…}.

bgr defaults to capturing from /dev/video0 and writing to /dev/video2. Use --input_device_number and --output_device_path to change them. --help has a full list of flags.

The masking behaviour can be changed at runtime with several keys (sent to the preview window):

• <space> disables (or re-enables) masking completely.

• s toggles smoothing of the mask.

• b toggles alpha-blending of the mask.

• c and C cycle through the colors set by --color_list. This could be used for emulating a greenscreen and feeding it to OBS, but I haven’t actually tried that.

• i and I cycle through the background images that were available at program startup.

• f freezes the current frame.

• q quits the program.

• DeepLab’s input pixels should be [-1,1], but bgr uses [-0.5,0.5] because it gives better results.

• libgflags-dev shouldn’t be needed for building, but without it gflags and glog are using different namespaces and I haven’t figured out how to summon CMake.

• Much more. Please send a pull request.

Apache 2.0 due to the dependencies used.