Analytics & Cognitive Feature in CVI Series: Part-1 CVI Analytics Overview & Data Schema for Analytics

About this blog series

• In this blog series, I would like to focus on the analytics and cognitive features of CVI (IBM IoT Connected Vehicle Insights).
  • The blog content will cover technical perspective of CVI
  • What kind of analytics and cognitive features are supported
  • What kind of technologies are used under CVI analytics and cognitive features
  • What kind of algorithms are used to get insights from big data derived from connected vehicle
  • How CVI analytics manages and utilizes big data
• The blog series are planned as follows:
  • Part-1: CVI Analytics Overview & Data Schema for Analytics (this blog)
  • Part-2: Driving Behavior Analysis
  • Part-3: Trajectory Pattern Analysis / MPP&DP (Most Probable Path & Destination Prediction)
  • Part-4: Traffic Predictor

CVI Architecture Overview

•  Above diagram represents CVI architecture overview including following components:
  
  • VDH (Vehicle Data Hub):
    • A front-end component that collects and manages large volumes of vehicle data from connected vehicles and automotive devices by using a range of protocols and formats
    • See Vehicle Data Hub for more detail
  • Agent System:
    • Real-time analysis: detects, stores, and manages events that are related to vehicles, driving activity, and movement
    • See Agent system for more detail
  • Asset Management
    • Asset management capabilities for managing vehicles, drivers, and the relationships between them.
    • See Registering and managing assets for more detail
  • Context Mapping
    • Geospatial functions, including map matching, road geometry data retrieval, shortest path search for global road networks, and real-time traffic event manipulation
    • Sub component
      • DMM (Dynamic Map Manager)
        • This is core of Context Mapping. It also manages static map information (OSM: Open Street Map, Commercial map) such as topological data (shape of road, intersections) and attributes (speed limit, etc)
      • Context Map
        • It manages dynamic information onto static map such as traffic information, weather information
    • See Context Mapping for more detail
  • Driver Behavior
    • analyzes driving behavior data that is collected via VDH from a connected vehicle or automotive device together with geospatial contextual data
    • See Driver Behavior for more detail
• Our focus on this blog series is Driver Behavior part and Context Map (a part of Context Mapping) which are responsible for big data management, analytics and cognitive features.

Analytics & Cognitive Feature in CVI Overview

• Below is more detailed view focusing on analytics and cognitive feature
• It includes Driver Behavior part and Context Map (a part of Context Mapping) as CVI components (other components are simply described)
• Summary of sub components
  • Driving Behavior Analysis
    • Analyze the behavior of drivers by using the data that is collected from the connected vehicles
    • Technical detail will be described in blog part-1
  • Trajectory Pattern Analysis
    • Identify and analyze the Origin/Destination (O/D) and route patterns from historical driving trips
    • Technical detail will be described in blog part-3
  • MPP&DP (Most Probable Path & Destination Prediction)
    • Predicts the route in the near future and the destination by giving a partial route of current ongoing trip and its context. 
    • Technical detail will be described in blog part-3
  • Probe Historical Data Access
    • It supports to access to historical data on HBase via REST API
    • See Driver Behavior REST API for more detail
  • Traffic Predictor
    • The speed predictor learns and builds models based on historical traffic speed data to predict the speed for a road and the traffic direction.
    • Technical detail will be described in blog part-4
  • Crawler (Traffic / Weather)
    • Periodically collect context data such as real-time traffic flow and weather conditions from external sites and store the data into the context map.
    • See Context Map Crawler for more detail
  • Context Map
    • It manages dynamic information onto static map such as traffic information, weather information

• As big data platform, CVI utilizes Hadoop stack for high scalability of data and computations. For examples,
  • HBase (on HDFS):
    • Scalable historical big data storage collected from connected devices
  • Spark
    • Scalable computation engine such as data crawlers and traffic predictors
  • ZooKeeper
    • Coordination among multiple CVI components and nodes. ZooKeeper is used not only for Hadoop nodes coordination but also for CVI nodes such as VDH, DMM and Agent run on. CVI components are designed to scale horizontally by the coordination process via ZooKeeper.

Data Schema Design with HBase

• HBase is a sparse, distributed, persistent multidimensional sorted map database management system that runs on top of a distributed file system (HDFS).
  • If you're familiar with Java, below represents logical data model of HBase table
    • SortedMap<RowKey, Map<ColumnFamily, SortedMap<Column, SortedMap<Timestamp, Value>>>>
  • See Apache HBase ™ Reference Guide for more detail
  • In CVI, the combination moving object id (e.g. VIN: Vehicle Identification Number) and timestamp is RowKey to distinguish records
• CVI supports extensible data schema to support different types / generation of devices
  • As below diagram shows, it defines 2 parts of HBase schema. One CF=B (ColomnFamily) is for base part for common well-used data such as longitude, latitude, heading, etc. And another CF=O is for other part reserved for project extension.
• As Row Key design shows, this schema is designed best suitable for per moving object (e.g. per VIN) manipulation
  • High performance to scan historical time-series data per moving object
    • (e.g.) scan VIN=Car-A, from 2020-01-01 08:30 - to 2020-01-01 09:00 in order to analyze driving behavior
• CVI analytics and cognitive features focus on per moving object by retrieving historical data from HBase
  • I will focus on each feature in subsequent blog series

• CVI utilizes HBase to support different types / generation of devices
• Below is example of 3 vehicles where each 3 types of vehicles has each own data definition
  • Car-A, family type car, has base columns and other ten additional columns (O1-O10)
  • Bike-B, motorcycle type, has base columns and other five additional columns (O11-O15)
  • Car-C, commercial type car, has base columns and other fifteen additional columns (O1-O10, O16-O20)
• See HBase Mapper - Mapping the VDH car probe extension data to the HBase table column (Restricted content for entitled IBM customers) for more detail to customize your project's specific data schema

Summary

• CVI big data platform is based on Hadoop stack which can support high scalability not only for big data but also for computation.
• The historical data of moving objects are managed on HBase with extensibility design to support each project's unique data schema.
• CVI analytics utilizes historical data on HBase to get insights of each moving object. Detailed will be followed up in subsequent blog series.

Notice

• Information in this blog is based on CVI SaaS V3.1. Information is subject to change in the future.

Revision History

• 2020-09-24: Initial release