Adiel Fernandez
Interaction Designer & Developer
Large-scale Computer Vision System
Disclaimer:
This large scale project was was a collaboration between many different entities including, but not limited to ESI Design, Audio, Video & Controls, and Diversified. My part in the project was solely (and single-handedly) developing the computer vision software and therefore give credit to others for all other project aspects.
Due to non-disclosure agreements the code for this project is in a private repository but I would be happy do discuss any of the non-sensitive project aspects and can provide references upon request.
Overview
As part of a larger renovation of a commercial space, a computer vision system was needed to create interactive visuals across multiple, large LED walls in 6 different corridors. Overall, 14 thermal vision cameras were utilized to get full coverage of the 6 rooms with some rooms requiring multi-camera stitching. All image processing for the 14 cameras was performed on a single machine in real time by a single application. One processed, data regarding people's position, velocity and movement characteristics was sent via OSC to the visuals production team.
Features at a Glance:
- Multi-threading allows simultaneous image processing on 14 different cameras.
- Real time image stitching features to crop and arrage camera feeds within aggregate.
- Camera image warping to correct perspective distortion and "fish-eye" lens effects.
- Image mask drawing tool for masking camera hotspots and project specified "dead zones".
- Custom contrast shifting algorithm to isolate objects from background noise.
- Smart background subtraction allows system to adapt to changing ambient conditions.
- "Blob Joining" algorithm (see below).
- System tracks user movement and sends a still or moving flag for different user interaction states.
Blob Joining Algorithm
Cold ambient temperatures cause people to appear as multiple blobs from warm extremities due to exposed skin while their torsos appear cold from layered clothing. To mitigate this, a blob joining filter was implemented to search through objects and piece them together based on a closeness threshold. After a single blob (A) is found, it searches for nearest neighbors. Once a neighbor (B) is found, it searches again for any new neighbors that might be close to B but not A. This repeats until all blobs are found to be single or joined. This method runs efficiently and smoothly with even hundreds of people within the space.
Screenshots
Six camera aggregate for the largest corridor. Yellow dots are adjusted positions for non-linear mapping to the physical space.
Real-time drawing tool for masking out thermal camera hotspots and project dictated dead zones. Masked pixels in yellow.
UI for single-camera corridor.
Raw feed warped in real-time using control points (in purple) to correct camera warp at boundaries.
Second corridor using multi-camera (4) aggregation in irregular arrangement.
Light blue circles show raw "blob" centers. A light blue line (top right) denotes the joining algorithm has counted them as a single person with a new [adjusted] centroid at the yellow circle.
Blob joining across camera 1 and 2 boundary. Joined blob centroids favor the oldest, most persistent blob.