Online Arabic Handwritten Character Recognition using Online-Offline Feature Extraction and Back-

Propagation Neural Network

Amal Ramzi Department of Software Engineering

Faculty of Engineering & IT , Taiz University Taiz, Yemen

amalramzi88@yahoo.com

Ammar Zahary Faculty of computing & IT

University of Science & Technology Sana'a, Yemen

a.zahary@ust.edu

Abstract— The main theme of this paper is performing online handwriting recognition for Arabic character using back propagation neural network and it experiments the performance of it using online features of characters as input to the BPNN in comparison with combining online and offline character features as the input . That's done through the following stages : online data acquisition, online & offline preprocessing, online & offline feature extraction (directional & geometric features) , classification using back propagation neural network to classify the character to one of 15 character classes and finally, delayed strokes handling using logic programming to recognize the character according to the character class and its delayed strokes accounts and positions .

Keywords— online handwriting, Arabic recognition, classification , back propagation, hybrid feature extraction .

I. INTRODUCTION One of the most promising methods of interacting with

small portable computing devices, such as personal digital assistants, is the use of handwriting. In order to make this communication method more natural, we proposed to visually observe the writing process on ordinary paper and to automatically recover the pen trajectory from numerical tablet sequences[1].

Handwriting recognition has been one of the most fascinating and challenging research areas in the field of pattern recognition [2]. It contributes immensely to the advancement of an automation process and can improve the interface between human and machine in numerous applications [3]. The technology is not yet mature enough for the Arabic font written script compared with Latin-based ones [4], so it's still an open and active field. Much work has been done on the recognition of Latin characters but limited work has been done on recognizing Arabic characters. Moreover, most Arabic handwriting recognition in previous works focused on recognizing offline script and little take the online cases, it's very important for the Arabic language to keep pace with time and cope with the new technologies of automation and artificial intelligence .

Handwriting recognition of Arabic script is a difficult problem since it is naturally both cursive and unconstrained. The analysis of Arabic script is further complicated in

comparison to Latin script due to obligatory dots, additional strokes or diacritics that are placed above or below most letters, which called 'delayed strokes'(see Fig. 1), and the Arabic characters change its shapes and forms when it is placed isolated or at initial, middle or at the end of a word (see Fig. 2).

There are two distinct handwriting recognition domains; online and offline, which are differentiated by the nature of their input signals. In offline system, static representation of a digitized document is used in applications such as cheque, form, mail or document processing. On the other hand, online handwriting recognition (OHR) systems rely on information acquired during the production of the handwriting. They require specific equipment that allows the capture of the trajectory of the writing tool [4], the user writes on a digital device using a special stylus, the system samples and records the point sequence as it is being written. Therefore, the online handwriting samples contain additional temporal data, which is not present in offline sampled data.

The main aim of this paper is to study , investigate and develop a new online Arabic Handwritten Character Recognition system using the methodology shown in Fig. 3 .

II. DATA ACQUISITION

Online handwritten data are typically a dynamic, digitized representation of the pen movement . Pen-down strokes contains a sequence of pen tip information which is sampled from the writer’s pen movement as horizontal and vertical coordinates pi=(xi, yi). Capturing the trajectory handwritten or drawn character in real time as x , y coordinates has been done after the character is drawn through the built MatLab application's painter interface , the character is then captured as an image (for offline feature extraction ), see Fig. 4.

Fig.1. Characters

handwritten with different dots

Fig. 2. Different forms of "GHYN"

character. (a) Single form, (b) Ending form, (c) Middle form, (d) Beginning

form.

1st International Conference on Advanced Technologies for Signal and Image Processing - ATSIP'2014 March 17-19, 2014, Sousse, Tunisia MSI-116

978-1-4799-4889-5/14/$31.00 ©2014 IEEE

Fig.3. Methodology of this paper

The interface in fig.4. is used in saving the training data as online form (as text files), containing x,y coordinates series of the trajectory shown on the left axes, and as offline data or image of the image shown on the right axes in the same figure (as .jpg file) . It's also used when testing a certain drawn character .

Fig. 4. Online & offline data acquisition on our matlab application

III. PRE-PROCESSING Pre-processing is a step that prepare online and offline

data to be an input to online and offline feature extraction processes and enhances the online & offline feature extraction because it enhances the quality of the image – in case of offline preprocessing , and the online trajectory – in case of online preprocessing . In this project pre-processing includes two

types as mentioned before , online preprocessing and offline preprocessing .

A. Online Preprocessing Online preprocessing steps that I used in this paper are :

translation, scaling , smoothing , re-sampling and simplifying.

1) Translation invariance: The simplest operation performed on character is

translation due to the wide area of the drawing pad of the digitizer , character may be written in different area every time so character must be normalized against translation[4]. This can be obtained by removing offset data from received data as in (1) & (2).

– (1) – (2)

Where x n , y n are (x,y) coordinates of the

received point x n and y n are point after translation . x and y are minimum value of (x,y) coordinates and n is total point index .

2) Scaling Invariance:

Due to great variation in character scales written by many users, character should be scaled to fit into fixed size . We did that with preserving the respect ratio that is the ratio between the width and the height of the character will be preserved after and before scaling .

3) Smoothing (Low –pass filtering) : The input character may include noise and disturbance ,

so it must be filtered using Low–pass filtering. Low–pass filtering can be thought of as a smoothing operation .For discrete-time sequence of points a common smoothing operation is one referred to as a moving average , where the smoothed value is the average of values in the vicinity of point [4].

This filter has bad effect on the starting and ending point because the filter range is from 4 to k-4 for total number of point of k so to reduce this bad effect we used a second stage filter of the form of

y[n]=x[n-1]+2x[n]+x[n+1] (3)

4) Re-sampling : This technique performs writing speed normalization by

re-sampling the point sequences and distributing the points uniformly over the sampled curve. Due to the variation in writing speed, the acquired points are not distributed evenly along the stroke trajectory. There are less points in under-sampling where the speed is high and more points in oversampling where the pen motion is slow. See the effect of resampling that's shown in Fig. 5 – axes no. 5 from left.

Fig. 5. Online preprocessing operations 5) Simplifying Line String (using Recursive Douglas-

Peucker algorithm): We simplified the character curve to reduce the huge

number of points – produced by resampling process- to extract only its important features later. see Fig. 5 , here we reduce it to 64 points.

B. Offline Preprocessing Offline preprocessing techniques that I used in this paper:

binarization , cropping , scaling , low pass filtering and edge detection .See Fig. 6.

1) Binarization : Traditionally, OCR systems have a binarization step that

classifies image pixels as text (black) or background (white). This rigid partitioning allows the text segmentation and character recognition steps to only analyze the regions of the image that most likely contain text. When preprocessing scene text, the binarization step is used both to isolate these candidate text regions for analysis and to provide the sharp contrast with anything considered as background needed by the later processing steps.

2) Image Cropping : The first step in image pre-processing is image

cropping. Some irrelevant parts of the image can be removed and the image region of interest is focused. This provides a user with the size information of the cropped image.

3) Image scaling : Image scaling is the process of resizing a digital image.

Scaling is a non-trivial process that involves a trade-off between efficiency, smoothness and sharpness. Apart from fitting a smaller display area, image size is most commonly decreased (or subsampled or downsampled) in order to produce thumbnails.

4) Low pass filtering : A low pass filter is the basis for most smoothing

methods. An image is smoothed by decreasing the disparity between pixel values by averaging nearby pixels .Using a low pass filter tends to retain the low frequency information within

an image while reducing the high frequency information and so remove the noise. In our system we used Gaussian and wiener filtering .

Fig. 6. Offline preprocessing

5) Edge detection

Edge Detection is a fundamental tool in image processing, machine vision and computer vision, particularly in the areas of feature detection and feature extraction, which aim at identifying points in a digital image at which the image brightness changes sharply or, more formally, has discontinuities.

IV. FEATURE EXTRACTION

Feature extraction abstracts high level information about individual patterns to facilitate recognition. Extracted features should contain the useful information carried by the character trajectory and image. The purpose of feature extraction is two-fold: to realize that not all data points are equally relevant or useful for pattern recognition and, in the case of NN, further reduction of the data input space to keep the network sizes computationally tractable[5].

After the online & offline preprocessing stage , the character trajectory and image became ready to go into the feature extraction stage .Two feature extraction phases are considered : the first is online feature extraction during writing the letter by mouse, the second is offline after the whole letter is written and captured as image.

A. Online Feature Extraction The feature extraction plays an important role in the overall

process of handwriting recognition. Many feature extraction techniques have been proposed to improve overall recognition rates; however most of them are depended on the direction and slope of handwriting characters . In our project we extract temporal features which are direction features which based on the online information of the users input stroke and computed for each point in the user pen path.

1) Directional Feature: The local writing direction at a point at instant is

described by cosine and sine.

( )co s ( ( )) =( )( )s in ( ( )) =( )

Y ttS t

X ttS t

α

α

ΔΔ

ΔΔ

(4)

where ( )X tΔ , ( )Y tΔ AND ( )S tΔ are defined as follows:

2 2

( ) = ( 1) ( 1)( ) = ( 1) ( 1)

( ) = ( ) ( )

X t X t X tY t Y t Y t

S t X t Y t

Δ − − +Δ − − +

Δ Δ + Δ

(5)

The produced feature vector approximately reaches to 63 in length.

2) Freeman chain code (8-directional Feature) Freeman code is one of the simplest and common used

features. It has eight directions where each direction has a specific code. Using this feature depends on the direction of each stroke of the input where every stroke is attributed to one Freeman direction. The final representation of the input writing will be a vector of codes of strokes that form the input writing [6].

The feature vector length produced from this technique in this paper is 63 in the beginning , but then we normalized the feature vector to minimize its length and got feature vector of length 20 only. The three cases of different online feature vector lengths - either directional or chain coding features in un-normalized and normalized form - are tested and entered to the neural network separately for classification , and all the results are compared and shown in RESULT part.

B. Offline Feature Extraction It extracts different line types that forms a particular

character. It also concentrates on the positional features of the same. The feature extraction technique explained was tested using a Neural Network which was trained with the feature vectors obtained from the system proposed.

We passed through the following steps to extract the offline geometrical features from our character preprocessed image:

1) Zoning : After the image is binaraized, scaled, cropped , smoothed

and finally skeletonized , the image is divided into windows of equal size, and the feature is done on individual windows. For the system implemented, two types of zoning were used. The image was zoned into 9 equal sized windows. Feature extraction was applied to individual zones rather than the whole image. This gives more information about fine details of character skeleton. Also positions of different line segments in a character skeleton becomes a feature if zoning is used. This is because, a particular line segment of a character occurs in a particular zone in almost cases. For instance, the horizontal line segment in character ’A’ almost occurs in the central zone of the entire character zone.

Starters, Intersections, and Minor starters.

2) Character Traversal : Character traversal starts after zoning is done on the image .Each zone is individually subjected to the process of extracting line segments. For this purpose, first the starters and intersections in the zone are found and then populated in a list. Minor starters are found along the course of traversal. Algorithm starts by considering the starters list.

3) Distinguishing Line Segments :

After line segments have been extracted from the image, they have to be classified into any one of the following line types: Horizontal line - Vertical line - Right diagonal line - Left diagonal line. After the line type of each segment is determined, feature vector is formed based on this information. Every zone has a feature vector corresponding to it. Under the algorithm proposed, every zone has a feature vector with a length of 8. The contents of each zone feature vector are: 1) Number of horizontal lines. 2) Number of vertical lines. 3) Number of Right diagonal lines. 4) Number of Left diagonal lines. 5) Normalized Length of all horizontal lines. 6) Normalized Length of all vertical lines. 7) Normalized Length of all right diagonal lines. 8) Normalized Length of all left diagonal lines. 9) Normalized Area of the Skeleton. 10) Euler Number. 11) Regional Area. 12) Eccentricity.

V. CLASSIFICATION The proposed classification stage is based on the features

extracted in real input and using a Back Propagation with a Multilayer Perceptron (3 layers of BP/MLP) neural network for recognizing the Arabic letters. The output value is a real code that relates each letter. The activation function in the BP/MLP neural network is a sigmoid function.

VI. DELAYED STROKES HANDLING The recognition strategy in our project involves recognizing

the first (primary) stroke which is one of 15 strokes as listed in Table 1. , then recognizing the second ,third and the fourth strokes .

An additional step is needed for recognizing the character for better classification results . After the BPNN finds the class that the character belongs to , a second phase or recognition is made depending on the existence of DS ,the number of DS if it exists and the position of it , the character finally recognized as one of 28 characters.

TABLE 1. The 15 classes of Arabic characters & their first and the second stroke

Secondary Stroke Primary stroke Character

Group

No HAMZA exist ا

HAMZA above أ ،أ،إ ا ا

HAMZA belowإ

One dot below ب

ن-ث-ت–ب ٮOne dot above ن

Two dot above ت

Three dot aboveث

No dots exists ح

One dot to right ج خ-ج-ح ح

One dot above خ

No dot exist د ذ-د د

One dot above ذ

No dot exist ر ز-ر ر

One dot above ز

No dots exists س ش-س س

Three dots above ش

No dots exists ص ض-ص ص

One dot aboveض

Vertical stroke ط

Vertical stroke then a ظ-ط ط

point aboveظ

No dot exist ع غ–ع ع

One dot above غ

One dot above ف ق-ف ٯ

Tow dot above ق

No HAMZA exist ل ل-ك ل

HAMZA above ك

No second stroke م م م

No second stroke ه ه ه

No second stroke و و و

No dots exists ى ي-ى ى

Two strokes below ي

VII. RESULTS & ANALYSIS The results gained after applying the Matlab system

through the five steps shown in the system methodology in Fig. 3 many times in different experiments .

A. Experimental Setup The experiments for evaluating the efficiency of the

proposed feature extraction and classification methods start with the setup of database.

1) Database In handwriting recognition, databases used to store

many samples of different writing styles for each character, in order to cope with the problem of handwriting variation from different writers. For Arabic the only publically available dataset is the IFN/ENIT offline handwritten dataset. It is a dataset of isolated words representing a limited vocabulary lexicon of 937 Tunisian city names. For on-line Arabic there are only two databases , ADAB and OHASD but they're not available for public use. Furthermore, they're not for isolated character but for cursive words(ADAB) or sentences(OHASD).

this paper , instead of using the existing databases, a new database with multiple online and offline samples for each character is created. The training and testing of our online handwriting character recognition system was conducted using our dataset to validate the system; it has used 1050 different online and offline characters from different writers. 525 character samples of online data which means 35 samples for each character are written by five users for training . Each user wrote the 15 Arabic character classes 7 times in order to get different writing variations and The writers are requested to write with stroke number and stroke order restriction , the offline data are 525 character images . As for the testing , 210 online character samples from different writers are collected plus 210 character images .

2) Neural network topology The following parameter of neural network are used in

our experiments.

TABLE 2. Parameters v/s their values used in all learning process

Parameter Value

Transfer function Sigmoid function for all the layers

'tansig'

Maximum Training Iterations 5000

Training functions Back Propagation 'trainlm'

No. of Hidden Layers 3

[8 16 32]

No. of outputs 15 "character classes"

Initial weights values Randomly generated values

Allowed maximum error for

learning 0.01

Training performance goal 0.01*mean(var(T))

~ 0.00062

B. Experimental Results In order to evaluate the performance of the proposed

feature extraction method and the designed classifier, experiment is carried out.

It's done for two main cases: (i)with online feature extraction and (ii) with combination of online & offline feature extraction.

The efficiency of the proposed recognition system is judged from three different perspective : recognition rate, network performance, processing time and storage space which are illustrated in Table 3.

VIII. CONCLUSION Recognition of handwriting isolated form of the Arabic

characters is achieved completely in both theoretical and practical experiment work with certain recognition percentage. The paper presents recognition results including neural network performance, training and testing recognition accuracy rate that's evaluated by the confusion matrix , neural networks training time and number of epochs for multiple cases of online and offline feature extraction and making a comparison between all of the cases as shown in Table 3. The results that come out from the back propagation training and testing are analyzed and shows that extracting hybrid online – offline features from the online character improves the network performance and increases the recognition accuracy .

IX. REFERENCES [1] Monji Kherallah, Lobna Haddad, Adel M. Alimi, " A new Approach for

Online Arabic Handwriting Recognition", University of Sfax, Regim: Research Group on Intelligent Machines, pp. 1.

[2] S. Mori, C.Y. Suen and K. Kamamoto, “Historical review of OCR research and development,” Proc. of IEEE, vol. 80, pp. 1029-1058, July 1992.

[3] V. K. Govindan and A.P. Shivaprasad, “Character Recognition-A review,” Pattern Recognition, vol.23, no. 7, pp. 671-683, 1990.

[4] Ammar Zahari "Handwritten Arabic Character Recognition using neural network" July , 1997 .

[5] El-Sheik, T.S. and El-Taweel, S.G., “Real-Time Arabic Handwritten Character Recognition” ,Pattern Recognition, volume 23 (1990), number 12 , pp.1323-1332.

[6] Jannoud A. I.; ―Automatic Arabic Hand Written Text Recognition System,� Damascus University,Damascus, Syria and Al-zaytoonah University, Amman, Jordan, : pp 857-864, 2007.

TABLE 3. Experimental Results

Feature Extraction Technique

Feature vector length

Training performance Training Time

Training recognition

rate

Testing recognition

rate

Epochs number

Usi

ng

Onl

ine

Feat

ure

vect

or

Direction features 63 0.001 60 min. 98.9% 58.7% 309 epochs

Chain code feature vector

63 0.0045 20 min. 98.7% 61% 159 epochs

Normalized chain code feature vector 20 0.002

12 min. 97.5% 63% 138 epochs

Usi

ng H

ybri

d Fe

atur

e ve

ctor

Chain code features + 9-zones offline

features

121 63 (of chain code) + 58 (of

offline features)

0.0005 3.5 min. 99.5% 74.8% 15 epochs

Normalized chain code features + 6-

zones offline features

78 20 (of

normalized chain code) + 58 (of offline

features)

0.0005 3 min. 99.4% 63% 19 epochs