News & Features » News

UCF brings its own version of artificial intelligence to Orlando’s real-time mass surveillance efforts

by and


Amazon's facial recognition technology isn't the only tool that Orlando is considering in its efforts to monitor the public using street cameras – the University of Central Florida has also developed a mass surveillance system for the city.

UCF researchers have installed an artificial intelligence-powered software that can recognize facial characteristics and body movements to detect assaults, robberies and even explosions in real time.

Orlando was the subject of blistering criticism in May after reports revealed that the police department was piloting Amazon's Rekognition, a facial-recognition technology that plugs into security cameras to identify and track people of interest as they walk down the street. Although currently being tested on a small number of cameras throughout the city with volunteer Orlando police officers, when Rekognition becomes fully operational, it will look for "persons of interest" by tapping into Orlando's network of surveillance cameras and essentially scanning everyone it can see until it finds a match.

But in early 2016, students and professors from UCF's Center for Research in Computer Vision were already using a $1.3 million federal grant to start testing software for the city's surveillance network that would instantly flag suspicious activities.

The Orlando Police Department assigns officers to watch dozens of screens displaying live feeds from the city's estimated network of about 180 security cameras in a second-floor hub called the IRIS room. But the department eventually realized its surveillance capabilities were ineffective, says Raymond Surette, a criminal justice professor at UCF and one of the project's researchers.

"There are simply too many cameras and too few monitors," UCF researchers wrote in a proposal for a grant from the National Institute of Justice. "Thousands of cameras go unwatched and hours of video unviewed."

The IRIS room was also inconsistently staffed with officers assigned to light duty, Surette says.

"The goal is to get the human out of the task of sitting there and watching the screens," Surette says. "Once you get above like 10 or 15 monitors, people just start missing things like crazy. There's this thing called inattentional blindness that kicks in ... psychologically, it's just a boring task."

With surveillance technology, the ultimate goal for police is predicting escalating threats to public safety so that "an assault on the street doesn't turn into a murder on the street," Surette says.

To bolster OPD's surveillance capabilities, UCF researchers proposed a four-function computer vision workstation with analytical software for anomaly detection, face-attribute prediction, body-attribute prediction and action detection to test on cameras they originally planned to install in Orlando's Rosemont neighborhood. First, though, researchers had to develop and test algorithms to train the software using hundreds of video clips containing simple actions like jumping, kicking and punching as well as more intricate activities like playing the violin or sumo wrestling.

Like a human, the software needs to be trained to accurately distinguish the normal from the abnormal. "You need to anticipate that these things will happen, so you show enough examples of it to the system," says Mahdi Kalayeh, a computer vision researcher at UCF whose software is embedded on OPD's workstation.

Action recognition and facial attribute software, Kalayeh says, fall under a field of artificial intelligence called object recognition. Unlike facial recognition, which looks for facial traits to identify a person in real-time within a camera's field of view, action recognition is broader – looking for abnormal movements and gestures to alert police of suspicious events, or "anomalies," like fights, drug deals or robberies.

UCF researchers included anomalies such as abuse, arrest, arson, assault, burglary, explosion, fighting, road accidents, robbery, shooting, shoplifting, stealing and vandalism to train the software. The software could also label actions the surveillance videos recorded, creating a database officers could search by activity.

To get ahead of a potential crime, Kalayeh says, the software is predictive – it's designed to learn, through thousands of examples, what events and conditions precede an anomaly so it can alert law enforcement before future events happen.

Kalayeh wrote the workstation's facial-attribute algorithms, which are tailored to seek more specific characteristics of a person's face as shown in a mugshot or portrait. The software can detect up to 40 different attributes, ranging from large noses, stubble and high cheekbones to whether the person is "attractive," according to a report researchers submitted to the National Institute of Justice in October.

Officers can describe what someone looks like to the software – in essence, an advanced, law enforcement-tuned search engine for faces – and it will display a bank of people who match that description, pulling results from the police department's live surveillance feeds or a predefined bank of images.

"Let's say I couldn't take a photo, but the person was wearing a hoodie, he had black hair, glasses and a mustache," Kalayeh says. An officer could use the program to search those traits within the live video feed of a neighborhood. "And of course it doesn't directly give you the final person, but it narrows down from probably thousands of people to let's say 100 people."

Kalayeh's facial-attribute software looks for the same set of features as facial recognition software, but the two are used for slightly different reasons, he says. Facial-attribute software is more predictive, spotting clues to build a digital picture of a person's mood in order to anticipate their actions. An angry face could lead to hostility, while a happy face wouldn't. Facial recognition software studies the same traits in a surveillance feed to identify a person by name, using an image to match.

"We are not involved in facial-recognition work," Mubarak Shah, the founding director of UCF's CRCV and the study's principal investigator, tells Orlando Weekly. But despite the differences, Kalayeh's facial-attribution algorithms at OPD could be used to supplement Orlando's facial recognition pilot with Amazon.

Similarly, the "body-attribute prediction" recognizes characteristics like "male" and "long hair," and knows if the body it's analyzing is wearing "sunglasses, hat, T-shirt, long sleeve, formal, shorts, jeans, long pants, skirt, face mask, logo [or] stripe," according to the report.

The software was better at recognizing some activities than others – covert drug deals, for example, were hard to distinguish, Surette says, but the more example videos are given to the software, the better it gets at identifying any activity.

"If somebody walks up to their trunk, opens their trunk, and puts in their groceries, that quantitatively is pretty similar to somebody walking up to a trunk, popping it with a crowbar and taking out something," he says. "The videos from these cameras are often obscured or not the best. So you're often dealing with a very limited number of pixels. And pixels translate into data, and the less data you have, the more uncertainty you have."

We welcome readers to submit letters regarding articles and content in Orlando Weekly. Letters should be a minimum of 150 words, refer to content that has appeared on Orlando Weekly, and must include the writer's full name, address, and phone number for verification purposes. No attachments will be considered. Writers of letters selected for publication will be notified via email. Letters may be edited and shortened for space.

Email us at

Support Local Journalism.
Join the Orlando Weekly Press Club

Local journalism is information. Information is power. And we believe everyone deserves access to accurate independent coverage of their community and state. Our readers helped us continue this coverage in 2020, and we are so grateful for the support.

Help us keep this coverage going in 2021. Whether it's a one-time acknowledgement of this article or an ongoing membership pledge, your support goes to local-based reporting from our small but mighty team.

Join the Orlando Weekly Press Club for as little as $5 a month.