Test & Evaluation/Science & Technology Program

C4I & Software Intensive Systems Test (C4T)Test Technology Area (TTA)

Big Data Analytic Technology Needs for T&E

May 11, 2017Mr. Gil TorresPhone: 805-989-9521Email: gilbert.a.torres@navy.milDISTRIBUTION A. Approved for public release: distribution unlimited.

Big Data Data

• Varieties of Big Data data– Structured Data (20%) (Sensor data, Comm, transaction data, man-machine

interface I/O, Schema / RDBMS– Unstructured Data (Imagery, voice, text, chat, email, Docs)– Semi-structured (Self describing -Label/ Value pairs, XML)

• Velocities of Big Data data– Near real Time Situation Awareness– Real time tactical and tactical edge communications– Time Critical interactions– Low latency Distributed Networks

• Volumes of T&E Big Data data– Theater data stream 270 TB of NTM data/year– Future Sensor data Petabytes to Yottabytes

DISTRIBUTION A.

C4TOverview

Innovate T&E: Joint, Early, Often, and Agile

Complex Warfare EnvironmentsBattle increasingly sophisticated adversaries in increasingly complex environments*

Innovative approaches to how we fight, posture our force, and leverage our asymmetric strengths and technological advantages*

Distributed Testing (DT)

• Remove Test Infrastructure Biases• Cross Domain Solutions (CDS) and Multi-Level

Security (MLS)• Assess Big Data Warfighter Systems• Testing Warfighter Systems Employing Agile

Communications• Emulate Contest/Dense Communications

Environments

Modeling & Simulation (M&S)

Test Automation (TA)

• Advance Big Data Collection, Analysis & Visualization

• Improve Test Execution thru Automation/Cloud Computing

• Advance Testing for Next Generation of Handhelds

• Improve Automated Control of Targets

* 2014 Quadrennial Defense Review

• Required Fidelity in Live and Simulated Environments

• V&V Techniques• Aggregation Techniques• Run-time Performance for RT

Applications• Systems, Communications,

Environmental Representations

C4I & Software Intensive Systems Test (C4T)Test Technology Area (TTA)

BIG DATA & CLOUD COMPUTING

C4T TTA Projects Supporting Technology Needsfor Live Virtual & Constructive T&E

DISTRIBUTION A.

Summary of C4T TTA Projects

C4TFAST GRaB (AE)

TNTI

Concept ExplorationTechnology

Characterization & Transition

Engineering, Integration, &

Experimentation

Project Initiation

Technology Demonstration

Technology Assessment

Technology Transition

Concept Formulation(BAA Process)

Transitioned(Tracked)

IPT3N

Development Timeline

Test Automation

Distributed TestingModeling & Simulation LVCT

RAID F-35

WAF BB

RAID PRISM

Domain Project Title Project Acronym

Performer

Test Automation Real-time Automated Insight Engine for Data to Decision RAID F-35 Jet Propulsion Laboratory

Test Automation Real-time Automated Insight Engine for Data to Decision Patuxent River Infrared Signature Measurements

RAID PRISM Jet Propulsion Laboratory

Test Automation Integrated Planning of Tactical, Test Support, & Tactical Engagement Networks

IPT3N Scalable Network Technologies

Modeling & Simulation FAST Gaussian Ray Bundle FAST GRaB NUWC Newport

Modeling & Simulation Weapons Analysis Facility: Broadband Capability WAF BB NUWC

Distributed Testing Technologies for Net-centric Test Interoperability TNTI Leidos Corporation

Distributed Testing Technologies for Interoperability of Live Platforms with Virtual & Constructive Simulation for Distributed Test

LVCT Lockheed Martin Rotary and Mission Systems

FAST GRaB

Simulated Test Environment of Battlefield Robustness

Sensor Platform/Aircraft Movement SimulationUpdate Rate (200 msec)

Terrain Generator100 Meters Per Grid Tile Rocket Sensor Guidance Software

System Under Test (SUT)

Rocket Fly Out Simulation

Update Rate (100 msec)Non-Continuous

Ground Target/VehicleMovement Simulation

Update Rate (1 sec)Non-Continuous

Sensor Update Rate (50 msec); Guidance Update Rate (100 msec); Accuracy ≈ 6m (CEP)

Mature technologies that increase the scope of modeling & simulation across the Department of Defense (DoD) distributed T&E infrastructure.

Modeling & Simulation Technology Use CaseM&S for T&E of Systems in Complex Environments (Representations and V&V)

Use of M&S in T&E will increase Big Datasets

Automated Testing in a Cloud Environment Executing 1000’s of Test Case Scenarios in Parallel

Test Automation Technology Use CaseSoftware Intensive Systems Testing in a Cloud Environment

New virtualization and cloud-based test technologies are needed to automatically create, execute, and control a large number of test cases operating in parallel creating Big datasets.

Software Intensive Systems (SIS) Typically Operate Within a System-of-System (SoS) to Assess Effectiveness

Automated Testing in a Cloud Environment Executing 1000’s of Test Case Scenarios in Parallel

Automated Tools For Testing System

Effectiveness &

PerformanceVirtualized HW

Aspects of System(s) Under Test

Test Harness Technologies • Support automated/agile testing (individual system)• Minimize labor-intensive test procedures • Support of regression, developmental, operational,

and interoperability testing

Virtual Test Harness Virtual Test Harness

Virt

ual T

est H

arne

ssVirtual Test Harness

System Under Test

Software Intensive Systems (SIS) Typically Operate Within a System-of-System (SoS) to Assess Effectiveness

Automated Tools For Testing System

Effectiveness &

PerformanceVirtualized HW

Aspects of System(s) Under Test

Test Harness Technologies • Support automated/agile testing (individual system)• Minimize labor-intensive test procedures • Support of regression, developmental, operational,

and interoperability testing

Virtual Test Harness Virtual Test Harness

Virt

ual T

est H

arne

ssVirtual Test Harness

System Under Test

Automated Testing Creates Big Datasets with Big Data Analytic Needs

Test Automation Technology Use CaseNextGen Handhelds and Widgets

“On-Demand” Test Services for Speed to Fleet for Mobile Applications and Technologies

Big Data Analytics to Improve Speed to Fleet for Mobile Applications & Technologies

DISTRIBUTION A.

Test Automation Domain Technology Use Case T&E Big Data Rapid Analysis

TACTICALTSPI

STRUCTUREDDATA

C2 Operations/Analysts

Improve the test communities ability to analyze and make decisions from large volumes of structured and unstructured test data.

SEARCH:Parallel Search Engine

Technology

TARGET LOCATION

Test Conductor

Advanced Semantic Algorithms

UNSTRUCTUREDDATA

Use Smart Algorithms &Database Design Technologies

to Structure Data

VIDEO CHAT

TEXT VOICE

SCREENS

DISTRIBUTION A.

T&E Big Data Rapid Analysis Technology Needs (1)

• Software algorithms that rapidly process large video datasets captured in multiple file format with multiple focal lengths (e.g. metric zoom lens) and scale factors to correlate into precise angular measurements.

• Algorithms that rapidly process video captured with multiple focal lengths and scale factors into precise angular (e.g. TSPI) measurements;

• Pattern Recognition for T&E Data;• Advanced Multi-Variate Time Series Analytic Techniques;• Integrating Simulation results with collected data to better understand system-

under-test performance;• Advanced Big Data Visualization Techniques Tailored to the T&E Domain; Low

Latency Federated Query on Disparate Big Data Sets;• Artificial Intelligence Techniques applied to T&E data analysis;• Rapid Data Ingest into the big data analysis system;

DISTRIBUTION A.

T&E Big Data Rapid Analysis Technology Needs (2)

• Sample big data representative repositories to evaluate C4T warfighter platforms with reusable and customizable data capture, T&E algorithms, data warehouse and analytics for Big Data warfighting platform analytics’;

• DoD T&E data types fusion, federation & integration technologies;• Near real time analytics for streaming DoD data types; • Hybrid cloud interfaces for efficient Community of Interest (COI) shared Big

Data infrastructure and tools

DISTRIBUTION A.

Distributed Testing Technology Use CaseAssess Big Data Warfighter Systems

Improve ability to create and assess big data warfighter systems under emulated wartime modes of operation

Joint Mission Environment

Distributed Test Infrastructure

Test Control

Distributed Common Ground System (DCGS) -C4I cross Service System employing Data-to-Decisions• Data generators and

simulators for Velocity, Variety, and Volume

• Platform analytics & fusion assessments

• Big Data environment replication

• Effectiveness and Performance of Individual system within SoS

• Automate regression testing

Test Control & Analysis

T&E Instrumentation

Live Warfighter Systems

Virtual and Constructive Systems/Environments

DISTRIBUTION A.

Assess Big Data Warfighter Systems Technology Needs (1)

• Data generators and simulations to stimulate platform inputs with representative velocity, volume, and data type variety requirements including error inputs, and verification and validation of generated datasets.

• Big Data Analytics to evaluate Warfighter Systems; various data types and sources storage, federation, transformations and integration; Platform analytics and responses to inputs; Evaluating platforms Big Data use and share, Effectiveness/performance evaluations

• Analysis of Big Data System fusion algorithms• Analytics to evaluate the correctness of the Big Data platform analytics• Big Data T&E infrastructure monitoring, visualization and control for Big Data

Platform distributed event analytics for real time analysis of streaming input data types including ones that can be located near the source or before storage for batch processing

DISTRIBUTION A.

Assess Big Data Warfighter Systems Technology Needs (2)

• Analytics for Big data birth to deployment artifacts evaluation• Analytics for Big Data platform analysis T&E data history usage• Analytics for Big Data platform deployment monitoring• Automate regression testing for Big Data Platform changes to its big data

capture or analytics• Big Data IT infrastructures: Cloud and noSQL data bases• Analytics to drill down to individual system performance and contribution, and

then determine the impact to the effectiveness of the SoS performance

DISTRIBUTION A.

15

C4T Test Automation ProjectReal-time Automated Insight Engine for Data to Decision (RAID) F-35

Jet Propulsion Laboratory / Pasadena, CA Description: Big data analytical technology developed to automatically analyze, extract, & manage actionable knowledge from terabytes of data per test day.Enables: Automated reuse of knowledge to enable DT quality testing throughout the lifecycle of weapon systems & provide the potential for discovery of “unknown-unknowns” problems.Current Status: Developing RAID knowledge management framework & proof-of-concept insight extraction demo using F-16 data. Continue to try to get F-35 engine data.Transition Partner / Date: Edwards 771st Flight Test Squadron, TRMC Knowledge Management, Edwards 412th Range Squadron / 3QFY18

FY17 Accomplishments: Improved RAID human-like Artificial Intelligence (AI) software to

reuse tester knowledge for data analytics in realistic environment with high degrees of uncertainty.

Integrate reasoner with teleo-reactive prog. Enhanced Associative Symbolic Memory (ASSUME) software to

support probabilistic inferencing. Demonstrated using RAID to accomplish target identification with

Edwards AFB DIADS data. Deliverables (Mo/Yr):Prototype RAID-DIADS software to Edwards AFB. RAID Phase 2 Final Report.

Key Future Events: Enable correlation of test data with knowledge base models and

ontologies. Initial demonstration of RAID capability addressing a

representative F-35 T&E scenario.

DISTRIBUTION A.

16

C4T Test Automation Domain ProjectReal-time Automated Insight Engine for Data to Decision (RAID)

Patuxent River Infrared Signature Measurements (PRISM)

Jet Propulsion Laboratory / Pasadena, CA Description: Develops big data technologies to analyze infrared (IR) videos that utilizes Intelligent Neural Network (NN) based segmentation and 2D/3D correlation used for system identification, tracking and performance assessment.Enables: Real-time automatic segmentation of multiple band IR videos allow rapid data analysis from man-months to hours. Real-time feedback of IR signature quality during test runs. Current Status: Developing neural network based segmentation algorithm & automatic multi-band signature segmentation software. Transition Partner / Date: PAX ATR / 3QFY18

FY17 Accomplishments: Completed segmentation algorithm refinement. Completed applying neural net directly to MW. Completed Initial C++ GUI test program. Mid-Year progress review.

Deliverables (Mo/Yr): Delivered incremental software builds and user’s manual to PAX

ATR PRISM team.

Key Future Events:Complete target feature segmentation and implementation in

complex neural network.Complete testing and optimization Software. Implement real-time automatic segmentation system.

DISTRIBUTION A.

17

C4T Test Automation DomainDecision Engine for Structured & Unstructured Data (DESU)

Morgan State University / Baltimore, MD

Improve Near Real-Time & Post-Test Analysis for T&E Unstructured Big Data

Description: Developing Big Data technologies for rapid analysis of voluminous structured & unstructured heterogeneous data (audio/video/text) in support of data to decisions for automated event pattern discovery.Enables: Assessment of large volumes of test data, real-time data fusion & analysis capability.Current Status: Developing event classifiers & algorithms to correlate within/across data sets.Transition Partner / Date: PAX ATR 4QFY15

FY15 Accomplishments: Completed 100% of Phase 3 exit requirements. Acquired and tested PAX 3-D Shearlet Algorithm.

Deliverables (Mo/Yr): Phase 2 Report & Software.

Key Future Events: None, project successfully completed.

DISTRIBUTION A.

Big Data Analytic Technology Needs for T&EClosing

• Big Data and associated Analytics presents technological challenges for the T&E community game changing innovations are needed NOW

• C4T TTA Broad Area Announcement defines investment topics for Industry and Academia to propose new technology projects– Primary Big Data TopicsT&E Big Data Rapid AnalysisAssess Big Data Warfighter Systems

– Big Data Topics Resulting from New Technology InnovationsM&S for T&E of Systems in Complex EnvironmentsSoftware Intensive Systems Testing in a Cloud EnvironmentT&E for NextGen Handhelds and Widgets

DISTRIBUTION A.

C4TPoints of Contact

• Executing Agent– Mr. Gil Torres– 805-989-9521 – gilbert.a.torres@navy.mil

• Deputy Executing Agent– Ms. Gail Holmes– (401) 832-6972 – gail.holmes@navy.mil

DISTRIBUTION A.