Post on 29-Jan-2023
Online Arabic Handwritten Character Recognition using Online-Offline Feature Extraction and Back-
Propagation Neural Network
Amal Ramzi Department of Software Engineering
Faculty of Engineering & IT , Taiz University Taiz, Yemen
amalramzi88@yahoo.com
Ammar Zahary Faculty of computing & IT
University of Science & Technology Sana'a, Yemen
a.zahary@ust.edu
Abstract— The main theme of this paper is performing online handwriting recognition for Arabic character using back propagation neural network and it experiments the performance of it using online features of characters as input to the BPNN in comparison with combining online and offline character features as the input . That's done through the following stages : online data acquisition, online & offline preprocessing, online & offline feature extraction (directional & geometric features) , classification using back propagation neural network to classify the character to one of 15 character classes and finally, delayed strokes handling using logic programming to recognize the character according to the character class and its delayed strokes accounts and positions .
Keywords— online handwriting, Arabic recognition, classification , back propagation, hybrid feature extraction .
I. INTRODUCTION One of the most promising methods of interacting with
small portable computing devices, such as personal digital assistants, is the use of handwriting. In order to make this communication method more natural, we proposed to visually observe the writing process on ordinary paper and to automatically recover the pen trajectory from numerical tablet sequences[1].
Handwriting recognition has been one of the most fascinating and challenging research areas in the field of pattern recognition [2]. It contributes immensely to the advancement of an automation process and can improve the interface between human and machine in numerous applications [3]. The technology is not yet mature enough for the Arabic font written script compared with Latin-based ones [4], so it's still an open and active field. Much work has been done on the recognition of Latin characters but limited work has been done on recognizing Arabic characters. Moreover, most Arabic handwriting recognition in previous works focused on recognizing offline script and little take the online cases, it's very important for the Arabic language to keep pace with time and cope with the new technologies of automation and artificial intelligence .
Handwriting recognition of Arabic script is a difficult problem since it is naturally both cursive and unconstrained. The analysis of Arabic script is further complicated in
comparison to Latin script due to obligatory dots, additional strokes or diacritics that are placed above or below most letters, which called 'delayed strokes'(see Fig. 1), and the Arabic characters change its shapes and forms when it is placed isolated or at initial, middle or at the end of a word (see Fig. 2).
There are two distinct handwriting recognition domains; online and offline, which are differentiated by the nature of their input signals. In offline system, static representation of a digitized document is used in applications such as cheque, form, mail or document processing. On the other hand, online handwriting recognition (OHR) systems rely on information acquired during the production of the handwriting. They require specific equipment that allows the capture of the trajectory of the writing tool [4], the user writes on a digital device using a special stylus, the system samples and records the point sequence as it is being written. Therefore, the online handwriting samples contain additional temporal data, which is not present in offline sampled data.
The main aim of this paper is to study , investigate and develop a new online Arabic Handwritten Character Recognition system using the methodology shown in Fig. 3 .
II. DATA ACQUISITION
Online handwritten data are typically a dynamic, digitized representation of the pen movement . Pen-down strokes contains a sequence of pen tip information which is sampled from the writer’s pen movement as horizontal and vertical coordinates pi=(xi, yi). Capturing the trajectory handwritten or drawn character in real time as x , y coordinates has been done after the character is drawn through the built MatLab application's painter interface , the character is then captured as an image (for offline feature extraction ), see Fig. 4.
Fig.1. Characters
handwritten with different dots
Fig. 2. Different forms of "GHYN"
character. (a) Single form, (b) Ending form, (c) Middle form, (d) Beginning
form.
1st International Conference on Advanced Technologies for Signal and Image Processing - ATSIP'2014 March 17-19, 2014, Sousse, Tunisia MSI-116
978-1-4799-4889-5/14/$31.00 ©2014 IEEE
Fig.3. Methodology of this paper
The interface in fig.4. is used in saving the training data as online form (as text files), containing x,y coordinates series of the trajectory shown on the left axes, and as offline data or image of the image shown on the right axes in the same figure (as .jpg file) . It's also used when testing a certain drawn character .
Fig. 4. Online & offline data acquisition on our matlab application
III. PRE-PROCESSING Pre-processing is a step that prepare online and offline
data to be an input to online and offline feature extraction processes and enhances the online & offline feature extraction because it enhances the quality of the image – in case of offline preprocessing , and the online trajectory – in case of online preprocessing . In this project pre-processing includes two
types as mentioned before , online preprocessing and offline preprocessing .
A. Online Preprocessing Online preprocessing steps that I used in this paper are :
translation, scaling , smoothing , re-sampling and simplifying.
1) Translation invariance: The simplest operation performed on character is
translation due to the wide area of the drawing pad of the digitizer , character may be written in different area every time so character must be normalized against translation[4]. This can be obtained by removing offset data from received data as in (1) & (2).
– (1) – (2)
Where x n , y n are (x,y) coordinates of the
received point x n and y n are point after translation . x and y are minimum value of (x,y) coordinates and n is total point index .
2) Scaling Invariance:
Due to great variation in character scales written by many users, character should be scaled to fit into fixed size . We did that with preserving the respect ratio that is the ratio between the width and the height of the character will be preserved after and before scaling .
3) Smoothing (Low –pass filtering) : The input character may include noise and disturbance ,
so it must be filtered using Low–pass filtering. Low–pass filtering can be thought of as a smoothing operation .For discrete-time sequence of points a common smoothing operation is one referred to as a moving average , where the smoothed value is the average of values in the vicinity of point [4].
This filter has bad effect on the starting and ending point because the filter range is from 4 to k-4 for total number of point of k so to reduce this bad effect we used a second stage filter of the form of
y[n]=x[n-1]+2x[n]+x[n+1] (3)
4) Re-sampling : This technique performs writing speed normalization by
re-sampling the point sequences and distributing the points uniformly over the sampled curve. Due to the variation in writing speed, the acquired points are not distributed evenly along the stroke trajectory. There are less points in under-sampling where the speed is high and more points in oversampling where the pen motion is slow. See the effect of resampling that's shown in Fig. 5 – axes no. 5 from left.
Fig. 5. Online preprocessing operations 5) Simplifying Line String (using Recursive Douglas-
Peucker algorithm): We simplified the character curve to reduce the huge
number of points – produced by resampling process- to extract only its important features later. see Fig. 5 , here we reduce it to 64 points.
B. Offline Preprocessing Offline preprocessing techniques that I used in this paper:
binarization , cropping , scaling , low pass filtering and edge detection .See Fig. 6.
1) Binarization : Traditionally, OCR systems have a binarization step that
classifies image pixels as text (black) or background (white). This rigid partitioning allows the text segmentation and character recognition steps to only analyze the regions of the image that most likely contain text. When preprocessing scene text, the binarization step is used both to isolate these candidate text regions for analysis and to provide the sharp contrast with anything considered as background needed by the later processing steps.
2) Image Cropping : The first step in image pre-processing is image
cropping. Some irrelevant parts of the image can be removed and the image region of interest is focused. This provides a user with the size information of the cropped image.
3) Image scaling : Image scaling is the process of resizing a digital image.
Scaling is a non-trivial process that involves a trade-off between efficiency, smoothness and sharpness. Apart from fitting a smaller display area, image size is most commonly decreased (or subsampled or downsampled) in order to produce thumbnails.
4) Low pass filtering : A low pass filter is the basis for most smoothing
methods. An image is smoothed by decreasing the disparity between pixel values by averaging nearby pixels .Using a low pass filter tends to retain the low frequency information within
an image while reducing the high frequency information and so remove the noise. In our system we used Gaussian and wiener filtering .
Fig. 6. Offline preprocessing
5) Edge detection
Edge Detection is a fundamental tool in image processing, machine vision and computer vision, particularly in the areas of feature detection and feature extraction, which aim at identifying points in a digital image at which the image brightness changes sharply or, more formally, has discontinuities.
IV. FEATURE EXTRACTION
Feature extraction abstracts high level information about individual patterns to facilitate recognition. Extracted features should contain the useful information carried by the character trajectory and image. The purpose of feature extraction is two-fold: to realize that not all data points are equally relevant or useful for pattern recognition and, in the case of NN, further reduction of the data input space to keep the network sizes computationally tractable[5].
After the online & offline preprocessing stage , the character trajectory and image became ready to go into the feature extraction stage .Two feature extraction phases are considered : the first is online feature extraction during writing the letter by mouse, the second is offline after the whole letter is written and captured as image.
A. Online Feature Extraction The feature extraction plays an important role in the overall
process of handwriting recognition. Many feature extraction techniques have been proposed to improve overall recognition rates; however most of them are depended on the direction and slope of handwriting characters . In our project we extract temporal features which are direction features which based on the online information of the users input stroke and computed for each point in the user pen path.
1) Directional Feature: The local writing direction at a point at instant is
described by cosine and sine.
( )co s ( ( )) =( )( )s in ( ( )) =( )
Y ttS t
X ttS t
α
α
ΔΔ
ΔΔ
(4)
where ( )X tΔ , ( )Y tΔ AND ( )S tΔ are defined as follows:
2 2
( ) = ( 1) ( 1)( ) = ( 1) ( 1)
( ) = ( ) ( )
X t X t X tY t Y t Y t
S t X t Y t
Δ − − +Δ − − +
Δ Δ + Δ
(5)
The produced feature vector approximately reaches to 63 in length.
2) Freeman chain code (8-directional Feature) Freeman code is one of the simplest and common used
features. It has eight directions where each direction has a specific code. Using this feature depends on the direction of each stroke of the input where every stroke is attributed to one Freeman direction. The final representation of the input writing will be a vector of codes of strokes that form the input writing [6].
The feature vector length produced from this technique in this paper is 63 in the beginning , but then we normalized the feature vector to minimize its length and got feature vector of length 20 only. The three cases of different online feature vector lengths - either directional or chain coding features in un-normalized and normalized form - are tested and entered to the neural network separately for classification , and all the results are compared and shown in RESULT part.
B. Offline Feature Extraction It extracts different line types that forms a particular
character. It also concentrates on the positional features of the same. The feature extraction technique explained was tested using a Neural Network which was trained with the feature vectors obtained from the system proposed.
We passed through the following steps to extract the offline geometrical features from our character preprocessed image:
1) Zoning : After the image is binaraized, scaled, cropped , smoothed
and finally skeletonized , the image is divided into windows of equal size, and the feature is done on individual windows. For the system implemented, two types of zoning were used. The image was zoned into 9 equal sized windows. Feature extraction was applied to individual zones rather than the whole image. This gives more information about fine details of character skeleton. Also positions of different line segments in a character skeleton becomes a feature if zoning is used. This is because, a particular line segment of a character occurs in a particular zone in almost cases. For instance, the horizontal line segment in character ’A’ almost occurs in the central zone of the entire character zone.
Starters, Intersections, and Minor starters.
2) Character Traversal : Character traversal starts after zoning is done on the image .Each zone is individually subjected to the process of extracting line segments. For this purpose, first the starters and intersections in the zone are found and then populated in a list. Minor starters are found along the course of traversal. Algorithm starts by considering the starters list.
3) Distinguishing Line Segments :
After line segments have been extracted from the image, they have to be classified into any one of the following line types: Horizontal line - Vertical line - Right diagonal line - Left diagonal line. After the line type of each segment is determined, feature vector is formed based on this information. Every zone has a feature vector corresponding to it. Under the algorithm proposed, every zone has a feature vector with a length of 8. The contents of each zone feature vector are: 1) Number of horizontal lines. 2) Number of vertical lines. 3) Number of Right diagonal lines. 4) Number of Left diagonal lines. 5) Normalized Length of all horizontal lines. 6) Normalized Length of all vertical lines. 7) Normalized Length of all right diagonal lines. 8) Normalized Length of all left diagonal lines. 9) Normalized Area of the Skeleton. 10) Euler Number. 11) Regional Area. 12) Eccentricity.
V. CLASSIFICATION The proposed classification stage is based on the features
extracted in real input and using a Back Propagation with a Multilayer Perceptron (3 layers of BP/MLP) neural network for recognizing the Arabic letters. The output value is a real code that relates each letter. The activation function in the BP/MLP neural network is a sigmoid function.
VI. DELAYED STROKES HANDLING The recognition strategy in our project involves recognizing
the first (primary) stroke which is one of 15 strokes as listed in Table 1. , then recognizing the second ,third and the fourth strokes .
An additional step is needed for recognizing the character for better classification results . After the BPNN finds the class that the character belongs to , a second phase or recognition is made depending on the existence of DS ,the number of DS if it exists and the position of it , the character finally recognized as one of 28 characters.
TABLE 1. The 15 classes of Arabic characters & their first and the second stroke
Secondary Stroke Primary stroke Character
Group
No HAMZA exist ا
HAMZA above أ ،أ،إ ا ا
HAMZA belowإ
One dot below ب
ن-ث-ت–ب ٮOne dot above ن
Two dot above ت
Three dot aboveث
No dots exists ح
One dot to right ج خ-ج-ح ح
One dot above خ
No dot exist د ذ-د د
One dot above ذ
No dot exist ر ز-ر ر
One dot above ز
No dots exists س ش-س س
Three dots above ش
No dots exists ص ض-ص ص
One dot aboveض
Vertical stroke ط
Vertical stroke then a ظ-ط ط
point aboveظ
No dot exist ع غ–ع ع
One dot above غ
One dot above ف ق-ف ٯ
Tow dot above ق
No HAMZA exist ل ل-ك ل
HAMZA above ك
No second stroke م م م
No second stroke ه ه ه
No second stroke و و و
No dots exists ى ي-ى ى
Two strokes below ي
VII. RESULTS & ANALYSIS The results gained after applying the Matlab system
through the five steps shown in the system methodology in Fig. 3 many times in different experiments .
A. Experimental Setup The experiments for evaluating the efficiency of the
proposed feature extraction and classification methods start with the setup of database.
1) Database In handwriting recognition, databases used to store
many samples of different writing styles for each character, in order to cope with the problem of handwriting variation from different writers. For Arabic the only publically available dataset is the IFN/ENIT offline handwritten dataset. It is a dataset of isolated words representing a limited vocabulary lexicon of 937 Tunisian city names. For on-line Arabic there are only two databases , ADAB and OHASD but they're not available for public use. Furthermore, they're not for isolated character but for cursive words(ADAB) or sentences(OHASD).
this paper , instead of using the existing databases, a new database with multiple online and offline samples for each character is created. The training and testing of our online handwriting character recognition system was conducted using our dataset to validate the system; it has used 1050 different online and offline characters from different writers. 525 character samples of online data which means 35 samples for each character are written by five users for training . Each user wrote the 15 Arabic character classes 7 times in order to get different writing variations and The writers are requested to write with stroke number and stroke order restriction , the offline data are 525 character images . As for the testing , 210 online character samples from different writers are collected plus 210 character images .
2) Neural network topology The following parameter of neural network are used in
our experiments.
TABLE 2. Parameters v/s their values used in all learning process
Parameter Value
Transfer function Sigmoid function for all the layers
'tansig'
Maximum Training Iterations 5000
Training functions Back Propagation 'trainlm'
No. of Hidden Layers 3
[8 16 32]
No. of outputs 15 "character classes"
Initial weights values Randomly generated values
Allowed maximum error for
learning 0.01
Training performance goal 0.01*mean(var(T))
~ 0.00062
B. Experimental Results In order to evaluate the performance of the proposed
feature extraction method and the designed classifier, experiment is carried out.
It's done for two main cases: (i)with online feature extraction and (ii) with combination of online & offline feature extraction.
The efficiency of the proposed recognition system is judged from three different perspective : recognition rate, network performance, processing time and storage space which are illustrated in Table 3.
VIII. CONCLUSION Recognition of handwriting isolated form of the Arabic
characters is achieved completely in both theoretical and practical experiment work with certain recognition percentage. The paper presents recognition results including neural network performance, training and testing recognition accuracy rate that's evaluated by the confusion matrix , neural networks training time and number of epochs for multiple cases of online and offline feature extraction and making a comparison between all of the cases as shown in Table 3. The results that come out from the back propagation training and testing are analyzed and shows that extracting hybrid online – offline features from the online character improves the network performance and increases the recognition accuracy .
IX. REFERENCES [1] Monji Kherallah, Lobna Haddad, Adel M. Alimi, " A new Approach for
Online Arabic Handwriting Recognition", University of Sfax, Regim: Research Group on Intelligent Machines, pp. 1.
[2] S. Mori, C.Y. Suen and K. Kamamoto, “Historical review of OCR research and development,” Proc. of IEEE, vol. 80, pp. 1029-1058, July 1992.
[3] V. K. Govindan and A.P. Shivaprasad, “Character Recognition-A review,” Pattern Recognition, vol.23, no. 7, pp. 671-683, 1990.
[4] Ammar Zahari "Handwritten Arabic Character Recognition using neural network" July , 1997 .
[5] El-Sheik, T.S. and El-Taweel, S.G., “Real-Time Arabic Handwritten Character Recognition” ,Pattern Recognition, volume 23 (1990), number 12 , pp.1323-1332.
[6] Jannoud A. I.; ―Automatic Arabic Hand Written Text Recognition System,� Damascus University,Damascus, Syria and Al-zaytoonah University, Amman, Jordan, : pp 857-864, 2007.
TABLE 3. Experimental Results
Feature Extraction Technique
Feature vector length
Training performance Training Time
Training recognition
rate
Testing recognition
rate
Epochs number
Usi
ng
Onl
ine
Feat
ure
vect
or
Direction features 63 0.001 60 min. 98.9% 58.7% 309 epochs
Chain code feature vector
63 0.0045 20 min. 98.7% 61% 159 epochs
Normalized chain code feature vector 20 0.002
12 min. 97.5% 63% 138 epochs
Usi
ng H
ybri
d Fe
atur
e ve
ctor
Chain code features + 9-zones offline
features
121 63 (of chain code) + 58 (of
offline features)
0.0005 3.5 min. 99.5% 74.8% 15 epochs
Normalized chain code features + 6-
zones offline features
78 20 (of
normalized chain code) + 58 (of offline
features)
0.0005 3 min. 99.4% 63% 19 epochs