A Heuristic Approach for Firewall Policy Optimization

El-Sayed M. El-Alfy

College of Computer Sciences and Engineering King Fahd University of Petroleum and Minerals

Dhahran 31261, Saudi Arabia alfy@kfupm.edu.sa

Abstract ⎯ A primary goal of this paper is to develop a heuristic approach based on genetic algorithms to enhance the firewall performance. Typical firewall policies may have thousands of rules and determining an optimal rule order that minimizes the average number of rule comparisons while maintaining the policy integrity is proven to be NP-hard. This problem is formulated as a binary integer program for which an optimal solution is obtained using the branch-and-bound technique. Then an alternative solution approach is devised based on genetic algorithms. Several experiments are conducted to evaluate the effectiveness of the proposed approach as compared to other rule-ordering techniques. Empirical results show the potential and flexibility of the proposed approach. Keywords ⎯ Network security, firewalls, access control, and genetic algorithms.

1. Introduction

Providing solutions for network security has recently

attracted considerable attention of several network researchers and practitioners. Securing private transactions over the Internet is gaining importance everyday and several mechanisms and standards have been developed to protect message confidentiality, preserve message integrity, authenticate the sender, and ensure non-repudiation. However, these security mechanisms do not prevent an intruder from sending a harmful message to an enterprise network connected to the Internet. Firewalls [1, 2] are commonly used solutions as security barriers to stop such attacks and significantly increase the security (if they are properly configured). A firewall is usually installed between an organization’s private network and the Internet to control and prevent unauthorized access to the network resources.

Firewalls come in many different forms but the primary activity of a firewall is filtering packets that pass through it between the protected network and the Internet. The filtering operation is essentially based on an ordered set of rules. Typical firewalls may have thousands of rules. Each rule specifies whether to drop or pass a received packet based on the information contained in the packet headers. In list-based policy representation, when a packet arrives at the firewall, it is sequentially compared against rules in the policy list until a match (typically first match) is found or reaching the end of the list. To make the policy complete, i.e. always finding a

match for every possible legal packet, a deny-all rule is often inserted at the end of the list [9]. Once a match is found, the corresponding action is applied to drop or pass the packet.

The computational complexity of the filtering operation depends on the depth of finding a matched rule in the rule list. A good order of rules is one that reduces the number of comparisons. When the new policy always gives the same action for each incoming packet, it is functionally equivalent to the original policy. However, since rules are not necessarily mutually exclusive (i.e. a packet can match more than one rule), improper ordering can cause security and performance problems. Hence, it is required to reorder the rules in the policy list to minimize the average number of comparisons while maintaining the precedence relationship between intersecting rules to preserve the integrity of the original policy. The optimal rule ordering with precedence constraints is among the hardest combinatorial optimization problems for which no efficient solution algorithm is known yet to run in polynomial time. Since this problem is equivalent to scheduling jobs with precedence constraints for a single processor, it has been shown to be NP-hard [3]. Thus, a straightforward sorting is not possible. A simple heuristic along with a sorting algorithm is presented in [3]. In this method, rules are arranged in non-increasing order of matching probabilities unless precedence relationships exist between neighboring rules. Although this method can improve the performance of a list-oriented firewall, it is possible that one rule can prevent other rules from being reordered. In addition, the degree of improvement depends on the linear arrangement generated by the topological sort algorithm.

Recently, genetic algorithms have been applied to a number of engineering design problems with promising results [4]. As meta-heuristic search techniques, they provide flexibility in global optimization and do not require problem-specific knowledge in order to get good solutions. In this paper, we apply a heuristic approach based on genetic algorithms to optimize the firewall security policy. The goal is to find a rule order that minimizes the average number of comparisons while maintaining the precedence relationships between rules. To evaluate the effectiveness of the proposed solution approach, the results are compared with two other firewall optimization methods. The first method is an exact optimal solution approach for binary integer program namely the branch-and-bound (BB) method. The second method is

ISBN 978-89-5519-131-8 93560 - 1782 - Feb. 12-14, 2007 ICACT2007

an approximate solution approach using the sorting algorithm with heuristic as presented in [3].

The remainder of the paper is organized as follows. Section 2 briefly describes the firewall policy optimization problem. Section 3 reviews related work. Section 4 presents a mathematical formulation of the problem as a binary integer program for which an optimal solution is obtained using the branch-and-bound method to benchmark our results. Section 5 presents a heuristic approach based on genetic algorithms for solving the problem. Section 6 provides some numerical examples to empirically evaluate the effectiveness of the proposed approach and compare it to other approaches. Finally, Section 7 concludes the paper and summarizes the results.

2. Firewall Security Policy

Specifically a network firewall uses a list of rules for

filtering packets from one network to another. Formally, a firewall security policy, R, is an ordered list of rules R = (R1, R2, …., RN) where N is the number of rules in the list. Each rule Ri (for i =1, 2, …, N) consists of an M-ary tuple of attributes or fields (q1, q2, …., qM) and a corresponding action ai which can be either deny or pass. In TCP/IP datagram networks, 5-tuple rules are used for filtering at the network layer. The commonly used attributes are the protocol type, the source IP address and port number, and the destination IP address and port number. Other attributes can be used as well at higher layers but with additional processing delay. Each rule attribute represents either a specific value or a set of values. Multi-value attributes contain wildcards as indicated by ‘*’ which means ‘any’ or range of values. If the policy is not comprehensive (i.e. a matched rule may not exist), a deny-all rule is often inserted at the end of the list as a default rule. Table 1 illustrates an example of a firewall policy composed of five rules.

Table 1. An Example of a Firewall Security Policy.

Source Destination R# prt IP_addr port IP_addr port

Action

1 TCP 132.16.* * 193.17.12.1 80 pass 2 UDP * * 193.18.* * deny 3 * 132.* * 193.13.* * deny 4 UDP * * * * pass 5 * * * * * deny

When a packet arrives at a firewall, it is sequentially

compared against each rule in the security policy, starting with the first rule, until a match (which is typically a first match) is found or the end of the list. If a rule match is found, the corresponding action is taken; otherwise the default rule action (usually deny) is performed. A packet matches a rule when each attribute in the packet belongs to the set of values of the corresponding rule attribute.

Since rules in the list are not necessarily mutually exclusive, i.e. a general rule is a superset of one or more

other rules, it is important when constructing a security policy to pay attention to the sequence of rules in the list. Two rules are said to be intersecting if the intersection of every corresponding field is non-empty [3]. Improper ordering can result in failing the integrity of the security policy or unacceptable firewall performance. For example, in Table 1 if rule R4 is checked before rule R2, then all UDP traffic will be allowed which violates the original policy that allows UDP traffic if the destination is not 193.18.*. As the size of the rule list increases, it becomes difficult to maintain and validate the integrity. The lookup latency may increase significantly as well. Applying more complex policies does not only have negative impact on the network performance but also makes it vulnerable to denial of service (DoS) attacks [9].

Since some rules may have higher matching probabilities than others, an optimal rule order can allow larger and more sophisticated rule sets to be used without sacrificing the lookup speed or the network performance. However, the precedence relationship among rules must be maintained.

3. Related Work

Firewall policy modeling, rule conflict analysis and

performance optimization have been increasingly studied in the past few years by several network security researchers [3, 5–14]. Different models have been proposed to represent and analyze firewall policies whether for centralized or distributed firewall environments. In [6], the authors presented a tree-based model and a set of techniques to analyze the firewall policy and discover conflicts among filtering rules. Identifying anomalies in distributed firewall systems has been addressed in [7]. Gouda et al. [8] presented a method based on decision diagrams for designing the sequence of rules in a firewall to be consistent (i.e. rules are ordered correctly), complete (i.e. every packet satisfies at least one rule in the list), and compact (i.e. no redundant rules). A trie-based representation of security policies is introduced in [9] to maintain policy integrity and provide more efficient processing. Moreover, several firewall policy analyzers and management tools have been designed and implemented including Fang [10], Firmato [11], Firewall Policy Advisor [6], and Fireman [12] among others.

Having an anomaly-free policy is crucial. However, as the size of the rule-base increases, filtering packets quickly becomes of major importance, as well, for applications that demand quality-of-service (QoS) [13]. This problem can be formulated as a combinatorial optimization problem [15]. However, it has been shown that determining an optimal rule order from all the possible linear arrangements is an NP-hard problem [3]. Fulp in [3] used a directed acyclic graph (DAG) to model the rule set and introduced a heuristic method based on topological sort and a modified bubble sort to enhance the rule matching performance. Unlike previous work on optimizing packet filtering that is based on deterministic approaches, an optimization technique that uses adaptive statistical search trees is presented in [14]. Genetic algorithms have been successfully applied to different

ISBN 978-89-5519-131-8 93560 - 1783 - Feb. 12-14, 2007 ICACT2007

related combinatorial optimization problems [4, 16] including traveling salesman problem (TSP), quadratic assignment problem (QAP), production scheduling such as job shop scheduling (JSS) and vehicle routing problem (VRP). This motivates our study to investigate the application of a heuristic approach based on genetic algorithms for solving the firewall optimization problem.

4. Binary Program Formulation

Suppose a firewall policy is composed N rules (R1, R2, …,

RN). Each rule Ri has a known packet matching probability, Pr{Ri}= pi. This probability distribution can be obtained from prior measurements of network traffic. Since the performance of a firewall depends on the rule order, it is required to determine an order that minimizes computational cost to reach a decision. However rules are in general intersecting and improper order of certain rules can cause security failure. Hence, it is required that if rules Ri and Rj are intersecting, then their relative order is maintained. Formally, a firewall policy optimization problem can be formulated as a constrained binary integer program. Assume rows in the filtering table are numbered from 1 to N where the first row has index 1. Also assume that rules in the original policy list are indexed from 1 to N. Then, let xij be binary decision variables indicating whether rule i is placed at row j or not, i.e.,

⎩⎨⎧

=otherwise0

list in the rowat located is if1 jRx i

ij,

where i and j ∈ {1, 2, …, N}. Also let πij define the precedence relationship between rules Ri and Rj as follows,

⎩⎨⎧ →

=otherwise0

if1 jiij

RRπ .

where Ri → Rj indicates that Ri is a predecessor of Rj.

Table 2. Formulation of the Firewall Policy Optimization as

a Constrained Binary Integer Program.

Minimize ∑∑= =

N

i

N

jiji xjp

1 1

(1a)

Subject to,

Ni xN

jij ,...,2,1 ,1

1

==∑=

(1b)

Nj xN

iij ,...,2,1 ,1

1

==∑=

(1c)

NjNikx kx ij

N

kjk

N

kik ,...,2,1;,...,2,1,1 if 1

11===−≤− ∑∑

==

π (1d)

NjNixij ,...,2,1;,...,2,1 },1,0{ ==∈ (1e)

Table 2 summarizes the mathematical model for this problem as a constrained binary integer program. The objective function (1a) minimizes the average number of comparing packets against rules in the policy list. Constraints (1b) and (1c) state that each rule appears only once in the list and each position in the list can have only one rule (i.e. one-to-one assignment). Constraints (1d) impose the precedence relations between rules. Finally constraints (1e) enforce the decision variables to be binary.

5. The GA-Based Approach

To determine a firewall policy that maintains the

precedence relationships among rules, the filtering rules and their relationships can be represented by a directed acyclic graph (DAG) [17]. The DAG vertices represent the rules and edges represent the precedence relationships between rules. A linear arrangement of all vertices can be obtained using a topological sort algorithm such that each vertex appears only once and if there is an edge (vi, vj) then vi comes before vj. Any topological sort represents a feasible solution that maintains precedence relationships. In general, several topological sorts may be found. The goal of optimization is to find a minimum-cost arrangement (i.e. one that has minimal average number of comparisons). The proposed approach in this section is based on genetic algorithms.

Genetic algorithms are meta-heuristic adaptive search techniques for global optimization based on the principles of natural evolution and selection. Hence, they have been applied to several design problems with promising results. The GA maintains a population of chromosomes that encode candidate solutions to a problem. Chromosomes are assigned fitness values based on the objective function of the problem. Then, some chromosomes are selected according to their fitness values for reproduction and new members are generated. The population evolves until an acceptable solution is found or a maximum number of generations is reached. The main components in designing a GA-based solution are the development of a coding scheme, fitness evaluation method, crossover and mutation operators, and selection and insertion strategies.

A) Representation – Before a genetic algorithm begins evolution, a suitable encoding mechanism must be devised for the problem at hand to map solutions to the genotype space. A direct representation is to represent a solution by a binary square matrix X such that xij = 1 means that rule Ri is located at position j; otherwise xij = 0. Another encoding approach is to use an integer vector to represent a solution. Here, the gene value represents the rule number to be located at that gene position. Neither encoding can ensure that a chromosome represents a feasible solution. Although one can check that each rule is represented in the chromosome only once, verifying and fixing precedence relations will be time consuming.

Alternatively, in this paper we adopt an indirect encoding approach (known as priority-based representation) similar to [4, 16]. In this approach, a chromosome is represented using an integer vector of random numbers x = (x1, x2, …, xN)

ISBN 978-89-5519-131-8 93560 - 1784 - Feb. 12-14, 2007 ICACT2007

where xi does not indicate the index of the rule located at position i in the policy list, but it represents a priority assigned to rule Ri. An example of a priority-based representation for a randomly generated chromosome is shown in Figure 1.

Rule # R1 R2 R3 R4 R5 R6

Random 0.95 0.23 0.60 0.48 0.89 0.76

Sorted 0.23 0.48 0.60 0.76 0.89 0.95

Indices 2 4 3 6 5 1

Figure 1. Priority-based Chromosome Representation.

To decode a chromosome into a policy list, the algorithm starts by finding all rules that have no precedence relationships. If more than one rule is found, the algorithm uses their priorities to pick the highest priority rule. Then, remove this rule and its relationships and repeat. This representation guarantees that after applying genetic operators, the formed offspring is also feasible.

B) Population initialization – To generate an initial population, it is required to decide on a population size, pop_size. In addition, to ensure that the values in the chromosome are not repeated, a vector of uniform random numbers U(0, 1) is first generated then sorted and the indices of the original values in the sorted list are used to represent the chromosome.

D) Crossover – Pairs of chromosomes in the population are selected for mating and exchanging their genes to produce new chromosomes. Parent chromosomes are randomly selected based on their fitness values using a method known as roulette-wheel selection. With a pre-set crossover probability, the crossover operation is performed on the selected parents. In order to avoid generating infeasible chromosomes, crossover is performed as depicted in Figure 2. First a crossover point is selected randomly. Then, one offspring is formed from the first section of the first parent and genes from the second parent that have different values. Similarly, the second offspring is formed from genes before the crossover point in the second parent and those in the first parent that have different values. This procedure preserves the feasibility after crossover.

Parent 1 2 4 3 6 5 1 Parent2 5 2 6 4 1 3

Offspring 2 4 5 6 1 3

Figure 2. Crossover Operation.

E) Mutation – To introduce random perturbation in order

to avoid trapping in a local minimum, each chromosome is mutated by swapping two genes at random positions. Mutation is usually applied to some individuals, to guarantee population diversity. Mutation is controlled by a mutation

probability parameter that is usually set to a small value around 0.1.

F) Replacement – A new population P(t+1) is formed from the parent population P(t) and the offspring population C(t); where t is the generation index. One approach to achieve that is to select the best pop_size chromosomes in both populations.

6. Empirical Results

In this section, we present a number of examples to

demonstrate the effectiveness of the proposed GA-based approach. We also compare its performance with the original un-optimized policy list and the optimized policy obtained by the sorting algorithm presented in [3] as well as the optimal solution found using the branch-and-bound technique.

We consider a list-based firewall with ten rules with precedence relationships as depicted in Figure 3 using a directed acyclic graph (DAG) representation. The performance is studied under different rule matching probability distributions or policy profiles. A graphical illustration of the assigned probabilities for four different cases (Eq, Unif, Zipf, and r-Zipf) is depicted in Figure 4. Under Eq probability distribution, all rules are equiprobable, i.e. having the same matching probability. In Unif and Zipf, probabilities are generated from a uniform distribution U(0, 1) and Zipf [8] distribution respectively. Finally, r-Zipf is similar to Zipf but the probabilities are assigned in reversed order, which is typically the case for Internet traffic [18]. The assigned probabilities are adjusted such that their sum is equal to one (i.e., for any arriving packet, there is at least one rule matching the packet).

Although the number of rules is small in this case, using exhaustive search to find an optimal policy will be computationally intractable. There are 10! linear arrangements (topological sorts) that need to be tested for maintaining precedence relationships and evaluated for the average number of comparisons. Alternatively, we solved this problem in three different ways using the proposed GA-based approach (GA), the branch-and-bound technique (BB) and the sorting algorithm presented in [3] (Sort). Figure 5 shows the percentage of reduction in the average number of rule comparisons as compared to the original un-optimized list for different rule orders. As we can see from the results, the worst-case performance occurs when using r-Zipf distribution for the rule matching probabilities. Moreover, when rules have non-increasing order of matching probabilities (e.g. Eq and Zipf), no improvement can be attained using any of these approaches. However, when rules are uniformly distributed, the sorting approach gave some enhancement but it is not optimal and further improvement can be achieved with either the BB technique or the GA-based approach. The highest improvement is demonstrated for the r-Zipf case. For the GA-based approach, the results were obtained for a population of ten chromosomes, and using single-point crossover with crossover probability of 0.8, and swap mutation with

ISBN 978-89-5519-131-8 93560 - 1785 - Feb. 12-14, 2007 ICACT2007

mutation probability of 0.1. Although there is no general rule of thumb to set these parameters, it is found empirically that these values give good results for several cases and have been commonly in the genetic algorithms literature.

Figure 3. Precedence Relationships among Rules.

1 2 3 4 5 6 7 8 9 100

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Rule index

Pro

babi

lity

EqZipfUnifr-Zipf

Figure 4. Rule Matching Probabilities.

0

1

2

3

4

5

6

Cos

t red

uctio

n (%

) ..

....

Sort 0 1.7615 0 1.7587

GA 0 3.8566 0 5.352

BB 0 3.8566 0 5.352

Eq Unif Zipf r-Zipf

Figure 5. Percentage Improvement for Various Policies in

Terms of Reduction in Average Number of Rule Comparisons as Compared to the Original Un-optimized Policy List.

We also tested the performance of GA-based approach

when there is no precedence (NoPr) relations and when rules have complete precedence (CoPr) relations. We used the

same probability distribution for r-Zipf as in the previous case but changed the precedence constraints. The results are shown in Table 3. We found that when rules have no precedence relations, the optimal solution is to sort them in non-increasing order. Similar optimal results are attained using any of the three approaches: Sort, GA or BB. Figure 6 shows the changes of the best objective function value for the GA-based approach for this case (i.e. no precedence relations and r-Zipf). On the other hand, when rules have complete precedence relations, there is only one order possible (the original order) using any of the approaches

Table 3. Average Number of Rule Comparisons for No/Complete

Precedence Cases with r-Zipf Distribution.

Original Sort GA BB NoPr 7.7840 3.2160 3.216 3.2160 CoPr 7.7840 7.7840 7.7840 7.7840

Figure 6. Changes of the best chromosome fitness over

generations for the GA-based solution.

7. Conclusion

This paper investigates the application of a genetic

algorithm to the optimization of a firewall security policy. The adopted priority-based encoding procedure ensures feasibility hence chromosomes are evaluated using the objective function value for the corresponding solution. A modified single-point crossover and swap mutation are used during the reproduction phase. The proposed approach has been evaluated using a number of numerical examples. The results reveal that the proposed method is effective and can be used to enhance the firewall performance while maintaining the integrity of the policy.

ACKNOWLEDGMENT

The author would like to thank King Fahd University of

Petroleum and Minerals (KFUPM) for continual support and supply of computing facilities during this work.

R2 R8 R10

R4

R5 R7

R9

R3

R1

ISBN 978-89-5519-131-8 93560 - 1786 - Feb. 12-14, 2007 ICACT2007

REFERENCES [1] S. Bellovin, and W. Cheswick, “Network firewalls,” IEEE

Communications Magazine, vol. 32, no. 9, 1994, pp. 50 – 57.

[2] R. Zalenski, “Firewall technologies,” IEEE Potentials, vol. 21, no. 1, 2002, pp. 24 – 29.

[3] E. Fulp, “Optimization of network firewall policies using ordered sets and directed acyclical graphs,” In Proc. of the 9th IFIP/IEEE International Symposium on Integrated Network Management, May 2005.

[4] M. Gen and R. Cheng, Genetic Algorithms and Engineering Design. Wiley, 2000.

[5] S. Hazelhurst, A. Attar, and R. Sinnappan, “Algorithms for improving the dependability of firewall and filter rule lists,” In Proceedings of the International Conference on Dependable Systems and Networks, June 2000, pp. 576 – 585.

[6] E. Al-Shaer, and H. Hamed “Firewall policy advisor for anomaly detection and rule editing,” In Proceedings of the IFIP/IEEE 8th International Symposium on Integrated Network Management, March 2003, pp. 17 – 30.

[7] E. Al-Shaer, H. Hamed, R. Boutaba, and M. Hasan, “Conflict classification and analysis of distributed firewall policies,” IEEE Journal on Selected Areas in Communications, vol. 23, no. 10, Oct. 2005, pp. 2069 – 2084.

[8] M. Gouda and A. Liu, “Firewall design: consistency, completeness, and compactness,” In Proc. of the 24th IEEE International Conference on Distributed Computing Systems, Tokyo, Japan, March 2004, pp. 320-327.

[9] E. Fulp and S. Tarsa, “Trie-based network firewall representations for network firewalls,” In Proc. of the 10th IEEE International Symposium on Computers and Communications, June 2005, pp. 434 - 441.

[10] A. Mayer, A. Wool, and E. Ziskind, “Fang: A firewall analysis engine,” In Proc. of the 2000 IEEE Symposium on Security and Privacy (S&P 2000), May 2000, pp.177.

[11] Y. Bartal, A. Mayer, K. Nissim, and A. Wool, “Firmato: A novel firewall management toolkit,” ACM Transactions on Computer Systems, vol. 22, no. 4, Nov. 2004, pp. 381 – 420.

[12] L. Yuan, J. Mai, Z. Su, H. Chen, C-N. Chuah, and P. Mohapatra, “FIREMAN: A Toolkit for Firewall Modeling and Analysis,” In Proc. IEEE Symposium on Security and Privacy, May 2006.

[13] L. Qiu, G. Varghese, and S. Suri, “Fast firewall implementations for software and hardware-based routers,” In Proceedings of the 9th International Conference on Network Protocols, Nov. 2001, pp. 241 – 250.

[14] H. Hamed, A. El-Atawy and E. Al-Shaer, “Adaptive statistical optimization techniques for firewall packet filtering,” In Proceedings of INFOCOM, April 2006.

[15] G. Nemhauser and L. Wolsey, Integer and Combinatorial Optimization. John Wiley & Sons, 1988.

[16] T. Yamada and R. Nakano, “Genetic algorithms for job-shop scheduling problems,” In Proceedings of Modern Heuristic for Decision Support, London, March 1997.

[17] B. Preiss, Data Structures and Algorithms with Object-Oriented Design Patterns in Java. John Wiley & Sons, 1999.

[18] L. Adamic and B. Huberman, “Zipf’s law and the Internet,” Glottometrics, 2002, pp. 143 – 150.

ISBN 978-89-5519-131-8 93560 - 1787 - Feb. 12-14, 2007 ICACT2007