Behavior Recognition & Anomaly Detection in Complex Big Data
Pattern of Life (PoL) Overview
Pattern of Life analysis allows for the understanding of complex, large datasets consisting of singular or multiple data types (e.g. video, text, imagery). Often companies, governments and organisations collect data at a rate that exceeds their data analysis throughput. This stored, unused data is known as Dark Data. Some data is collected without a known use or analysis method with the hope future analytics will be able to draw useful conclusions. Dark data may be collected and fused from different sources including sensors and log files. This large amounts of data and information available often leads to data overload. There is a need to automate PoL production.
The output of PoL analysis is most often behavior recognition and anomaly detection. These outputs are used to determine what is currently happening, why is something happening, what actions should be taken, and future predictions. These outputs have applications in nearly every industry, farming, banking, insurance, marketing, genetics, and government intelligence.
There are three scales of population analysis, global, community or local, and individual. Within these three scales two different approaches to PoL analysis may be taken 1. Gather information to predict future actions or events on a known interest and 2. determine the unknown of unknowns. If the question or target of interest is known the difficulties involve developing a query input, sifting through the massive amounts of structured and unstructured data to determine what is relevant, and making connections within the relevant data. When the precise question to be answered is unknown data pattern mining is difficult, since what to ask or what patterns to seek are unknown. Both scenarios presents the challenge of providing a system that can be quickly customized to specific objectives, and contains a mixed data type fusion algorithm capable of discovering patterns to answer unknown questions.
IFT provides PoL analysis on three different scales
Global
Interpret the trends, movements, and international relationships of the global population
Community
Follow the growth, ideals, and relationships within a connected group of a population.
PoL Analysis @ IFT
Custom Frameworks for any Dataset
IFT’s PoL analysis tools allow the user to understand datasets consisting of multiple data types, e.g. video, text, imagery. This provides the customer actionable information behavior patterns and anomaly alerts. Our tools retrieve, analyze, and exploit complex data and dark data.
IFT is able to provide these tools and services because of our expertise in many technologies including lambda architecture, high performance computing, data fusion, game theory, and machine learning. We have designed frameworks for supervised, semi-supervised and unsupervised data analysis. All of these skills allow us to produce efficient tools for complete or incomplete datasets comprised of any combination of data types.
Large, complex datasets need to be resolved and analyzed, to discern abnormal and significant activities from normal patterns of activities.
Creating the Pieces and Putting them Togther
PoL refers to a data analysis technique that can be applied to virtually any temporal data, or data related to an instance in time. POL data analysis can be performed manually, using Artificial Intelligence (AI), or machine learning. Due to the increasing ease and collection of digital data (for example data generated by the Internet of Things (IoT)), and new developments in data mining, AI, and machine learning, POL is becoming more and more associated with computer science. PoL analysis requires historical data, or data that has been collected over a period of time. Patterns from historical data are used to predict future actions or compared to current observations for anomaly detection. IFT’s frameworks detect anomalies in real-time, from many different data sources. Generally, the more historical data present the more accurate PoL analysis. Creating the pieces refers to filtering or pre-processing the historical data to determine relevance. Once the pieces have been created their relationships are established to determine behavior patterns. PoL analysis uses data filtering of historical data to create pieces, and then puts them together to build a picture of happenings in the real world. Although PoL was first made mainstream in the context of military and intelligence agencies, PoL has farther reaching applications. An example includes monitoring credit card spending habits. If a significant changes in spending amounts, locations, or number of purchases is observed customers can quickly be alerted to fraudulent spending. A customer’s spending history is pre-processed to create pieces (individual translations, daily amount spent, monthly number of transactions). The relationships between these pieces (monetary amount, frequency of spending, cites purchases are made) to build a picture of the customers spending habits. Anomalies in these habits are then alerted to the customer for potential fraudulent spending.
CC Image Credits made available under a Attribution-Noncommercial-Share Alike 2.0 license: Network: Simon Cockell, Cranium – GDJ