Using Deep Learning with Imagery in ArcGIS - Esri

Vinay Viswambharan

Using Deep Learning with Imagery in

ArcGIS

Session Overview

• AI, Machine Learning & Deep Learning

• Deep Learning Workflow in Pro

• Deep Learning Workflow using arcgis.learn

• Training Models

- ArcGIS Pro

- arcgis.learn

• Types of models and their applications

• Scalable deep learning with Image Server

Neural Networks

TensorFlow

CNTK

Natural Language ProcessingCognitive

Computing

GeoAI

Computer Vision

Dimensionality Reduction

Object Detection

Support Vector Machines

Object Tracking

Keras

PyTorch scikit-learn

fast.ai

Random Forest Machine Learning

Deep Learning

Artificial IntelligenceCaffe

Data Science

Machine Learning

Deep Learning

Artificial Intelligence

• Pixel & Object Based

• Image Segmentation

• Maximum Likelihood

• Random Trees

• Support Vector Machine

• Empirical Bayesian Kriging

• Areal Interpolation

• EBK Regression Prediction

• Ordinary Least Squares Regression and Exploratory Regression

• Geographically Weighted Regression

Classification PredictionClustering

• Spatially Constrained Multivariate Clustering

• Multivariate Clustering

• Density-based Clustering

• Hot Spot Analysis

• Cluster and Outlier Analysis

• Space Time Pattern Mining

Machine Learning in ArcGIS

• Label Objects

• Training Samples Manager

• Export Training Samples

• Detect Objects

• Classify Pixels

• Classify Objects

• Non Maximum Suppression

Data Preparation InferencingTraining

• Train Deep Learning Model

• Object Detection

• Object Classification

• Pixel Classification

• Instance Segmentation

• arcgis.learn module (ArcGIS API for Python)

Deep Learning in ArcGIS

Impervious Surface Classification

Coconut Tree Detection

Building Footprint Extraction

Damaged House Classification

Pixel Classification Object Detection Instance Segmentation Image Classification

End to End Deep Learning – Wide spectrum of deep learning models

Applications of Deep Learning to GIS

Deep Learning with Imagery in ArcGISArcGIS supports end-to-end deep learning workflows

• Tools for:

• Labeling training samples

• Preparing data to train models

• Training Models

• Running Inferencing

• Supports the key imagery deep learning categories

• Supported environments

• ArcGIS Pro

• Map Viewer

• ArcGIS Notebooks/Jupyter Notebook

Part of ArcGIS Image Analyst

Run distributed on ArcGIS Image Server

Train Deep

Learning Model

Detect/Classify

Objects

Prepare data for training

Label

Training Samples

Classify Pixels

Collect Samples Export Training

SamplesTrain

Perform

Inference

Deep Learning Workflow in ArcGIS

Inference

Collect Training Samples / Label data

• Different methods

- Label Objects for Deep Learning – ArcGIS Pro (2.5)

- Training sample manager – ArcGIS Pro

- Feature editing

- ArcGIS Pro

- Map Viewer

- JS Web Apps

• Different data models

- Feature class (local single user)

- Feature services (collaborative experience)

- Classified thematic rasters

Collect

Samples

Export

TrainingSamples

Train Perform

Inference

Export Training Data for Deep Learning

Collect

Samples

Export

TrainingSamples

TrainPerform

Inference

• Exports samples to training images

• Images have associated labels/metadata

• Writes out and ECD

• Used as inputs for model training

• Supports various formats

Train Deep Learning Model

• ArcGIS Pro and ArcGIS API for Python supports training

• ArcGIS Pro “Train Deep Learning Model” tool

• arcgis.learn module in ArcGIS API for Python

• Supported Models:

- Object Detection - SSD, RetinaNet, MaskRCNN

- Object Classification – Feature classifier

- Pixel Classification – UNET, PSPNet

• External Deep Learning Frameworks

- TensorFlow

- CNTK…

Collect

Samples

Export

TrainingSamples

Train Perform

Inference

Perform Inference

• Run on desktop and enterprise

• Parallel processing using enterprise

• Types of inferencing

• Object detection

• Classify objects

• Pixel classification

Collect

Samples

Export

TrainingSamples

Train Perform

Inference

ArcGIS – Deep Learning WorkflowEnd-to-end deep learning workflow

Imagery

Tools to

Generate training data

Training sites

Inference

results

Input Images

Tools & APIs

for Training

models

Inferencing

Tools

Tools to generate training samples

• Image Analyst in ArcGIS Pro

• Image Server on ArcGIS Enterprise

Model Training

- ArcGIS Pro

- Notebooks

Inferencing

• Image Analyst in ArcGIS Pro

• Image Server on ArcGIS Enterprise

DLPK

Deep Learning Package

• Zip with a .dlpk file extension

- Created by Train Deep Learning Model tool and

arcgis.learn (ArcGIS API for Python)

• Contents of the dlpk

- Model definition file (.emd)

- Deep learning model file (framework specific)

- Python Raster Function (.py, optional if using an out-

of-the-box model)

• Can be shared across your organization

Deep Learning Package

End-to-end from raw imagery to structured information products

Deep Learning Workflow in ArcGIS

Image

Management

Labelling Data

PrepTrain

Model

Inferencing AnalysisField

Mobility, Monitoring

ArcGIS being used for each step of the deep learning workflow

Using Deep Learning Tools in

ArcGIS Pro – Well pad detection

Demo

gis

geometry

network

schematics

features

realtime

widgets

mapping

env

geocoding

geo

enrichment

geo

processing

raster

geo

analytics

learn

The arcgis.learn module in

ArcGIS API for Python enables Python developers and data scientists to

easily adopt and apply deep learning in their workflows.

It enables training state-of-the-art deep learning models with a simple,

intuitive API.

Train Models

Before After

• Installing External DL

Frameworks

• Dozens of lines of

Code

• HARD!

• No Installation

(ArcGIS Notebooks)

• 3-5 lines

• EASY

arcgis.learn module

ArcGIS API for Python

Exporting Training Data arcgis.learn.export_training_data

Training DL Modelsarcgis.learn.SingleShotDetector

arcgis.learn.UnetClassifier

arcgis.learn.FeatureClassifier

arcgis.learn.PSPNetClassifier

arcgis.learn.RetinaNet

arcgis.learn.MaskRCNN

Preparing Data (Augmentation)arcgis.learn.prepare_data

Model Managementarcgis.learn.list_models

arcgis.learn.Model

Model.install

Model.uninstall

Model.query_info

Inference APIsarcgis.learn.detect_objects

arcgis.learn.classify_pixels

arcgis.learn.classify_objects

ArcGIS API for PythonNot just “training”!

ArcGIS Pro 2.5 Notebooks

Things you can do today with arcgis.learnObject Detection, Pixel Classification, Feature Classification, Instance Segmentation

Damaged Structures

Roads

Swimming Pools

Building Footprints

Oil Pads

Land Cover

Palm trees Refugee Camps Surface-to-Air missile (SAM) sites

Catfish Brick Kilns

Sinkholes

Training Model using

arcgis.learn

Image ClassificationAssign a label to a given image

Cat

Applications:

- Damaged building classification

- Clean or ‘green’ pools…

Undamaged Damaged

Object ClassificationAssign a label to a given feature

Models (from torchvision):

- Inception

- ResNet

- VGG…

Damage classification using ArcGIS

Pro

Demo

Vinay

Semantic SegmentationAssign a label to each pixel

Cat

Ground

Sky

Turf/Grass

Building

Water

Pixel Classification

Applications:

- Land Cover Classification

- Pervious/Impervious mapping…

Models:

- UNetClassifier

- PSPNetClassifier

Building footprint extraction

Object DetectionFind objects and their location (bounding boxes)

Applications:

- Detect trees, cars, airplanes, …

Models:

- SingleShotDetector

- RetinaNet

Detecting well pads using ArcGIS Pro

Demo

Instance SegmentationFind objects and their precise locations (masks or polygonal features)

Applications:

- Building footprint extraction

Models:

- MaskRCNN

Building footprints

Demo

ArcGIS Enterprise for Scaling Deep Learning

Problem

• Deep learning is an intensive process

• Resource hog

Solution

• Leverage Raster Analytics to scale inferencing

• All desktop inferencing tools are accessible through enterprise

• Clients to invoke distributed inferencing – Map Viewer, ArcGIS Pro, notebooks

• Multi GPU support

• Requires the ArcGIS Image Server license

60,000 buildingsArcGIS Pro: 1 GP100 GPU (16 GB): 4.5 hoursArcGIS Enterprise: 4 nodes RA server with 3 x P40 GPUs (24GB):20 minutes

ArcGIS Enterprise Deep Learning – System Architecture

ArcGIS

Pro

Notebook Server

Webmap

Viewer

ArcGIS Server

ArcGIS Data

Store

ArcGIS

Services GDB

Content Store

Cloud

Hosted Server

Cloud Hosted

Raster Analysis

Cloud Hosted

Image Server

Distributed Raster Data Store

Clo

ud

Ho

ste

d

Cloud GPUs

Blob Storage

Portal

Apps

DesktopAPIs

Image Server Image Server

ArcGIS

Pro

Notebook Server

Webmap

Viewer

ArcGIS Server

ArcGIS Data

Store

ArcGIS

Services GDB

Content Store

Cloud

Hosted Server

Cloud Hosted

Raster Analysis

Cloud Hosted

Image Server


Clo

ud

Ho

ste

d

Cloud GPUs

ND6s

P40 GPU

Blob Storage

Portal

Apps

DesktopAPIs


ArcGIS

Pro

Notebook Server

Webmap

Viewer

ArcGIS Server

ArcGIS Data

Store

ArcGIS

Services GDB

Content Store

Cloud

Hosted Server

Cloud Hosted

Raster Analysis

Cloud Hosted

Image Server


Clo

ud

Ho

ste

d

Cloud GPUs

Blob Storage

Portal

Apps

DesktopAPIs


ND6s

P40 GPUImage Server

Image Server

Image Server

ArcGIS

Pro

Notebook Server

Webmap

Viewer

ArcGIS Server

ArcGIS Data

Store

ArcGIS

Services GDB

Content Store

Cloud

Hosted Server

Cloud Hosted

Raster Analysis

Cloud Hosted

Image Server


Clo

ud

Ho

ste

d

Cloud GPUs

Blob Storage

Portal

Apps

DesktopAPIs


GeoAI Sample Use-CasesIdentifying broken insulators

GeoAI Sample Use-CasesReal-Time detection of cracks (Video)

GeoAI Sample Use-CasesVisualize areas with highest amount of cracks

Real-Time Activity Detection using Deep Learning

with ArcGIS API for Python and Ops Dashboard

GeoAI Sample Use-CasesTracking and anomaly detection

Predict ETA from Downtown San Francisco to different areas under 25 Minutes

GeoAI Sample Use-CasesETA Prediction – Deep learning with network analyst

Recap

• End-to-end Deep learning workflows in ArcGIS

• Geoprocessing tools

- Prepping your model training in ArcGIS Pro and Enterprise

- Inferencing in ArcGIS Pro and Enterprise

• Support 4 key categories of types of deep learning

• Supported in all key clients

• arcgis.learn module for the developer and data scientist

• Multiple model types (SSD, UNET, PSPNet, RetinaNet, MaskRCNN…)

• Scales using enterprise

Print Your Certificate of Attendance

Print Stations Located in 150 Concourse Lobby

Tuesday12:30 pm – 6:30 pm

Expo

Hall B

5:15 pm – 6:30 pm

Expo Social

Hall B

Wednesday10:45 am – 5:15 pm

Expo

Hall B

6:30 pm – 9:30 pm

Networking Reception

Smithsonian National Museum

of Natural History

Download the Esri

Events app and find your event

Select the session

you attended

Scroll down to

“Survey”

Log in to access the

survey

Complete the survey

and select “Submit”

Please Share Your Feedback in the App

Parking lot classification

Using Deep Learning with Imagery in ArcGIS - Esri

Documents

Transcript of Using Deep Learning with Imagery in ArcGIS - Esri