1

The Cosine Simplex Algorithm

H. W. Corleya, Jay Rosenbergera, Wei-Chang Yehb , and T. K. Sunga

aDepartment of Industrial and Manufacturing Systems Engineering The University of Texas at Arlington

Arlington, TX 76019-0017 corley@uta.edu, jrosenbe@uta.edu, t_k_sung@yahoo.com

be-Integration & Collaboration Laboratory

Department of Applied Mathematics National Chiayi University

P.O. Box 67-100, Taichung, Taiwan 408, R.O.C. yeh@ieee.org

Abstract: An extension of the simplex algorithm is presented. For a given linear

programming problem, a sequence of relaxed linear programming problems is solved until a

solution to the original problem is reached. Each successive relaxed problem is obtained

from the previous one by adding a single constraint chosen from the constraints violated by

the solution to the previous relaxed problem. This added constraint maximizes the cosine of

the angle that the gradient of any violated constraint forms with the gradient of the objective

function. In other words, each successive relaxed problem is obtained by adding the violated

constraint most parallel to the objective function. The proposed algorithm terminates when

no constraints are violated. Preliminary results indicate that this cosine simplex algorithm

reduces both the number of simplex iterations and the number of computations at each

iteration.

Key Words: optimization, linear programming, simplex algorithm, cosine simplex

algorithm

2

1. Introduction Consider the linear programming problem

maximize z = cTx (1)

subject to

Ax ≤ b

x ≥ 0,

where x, A, b, c, 0 are n×1, m×n, m×1, n×1, n×1 respectively. It will be assumed that (1) has

an optimal solution.

This standard linear programming model (1) is an indispensable tool in today’s

manufacturing environment. Dantzig’s simplex method [1] provided the first effective

solution technique, but numerous other algorithms have been proposed. For example,

feasible direction methods have been proposed by Brown and Koopmans [2], as well as by

Murty and Faithi [3], among others. Megiddo [4] has tried to reduce the number of

constraints through a multidimensional search technique. Khachian’s ellipsoid algorithm [5]

first established that linear programming problems can be solved in polynomial time, but it

performs poorly in practice. Karmarkar [6] later developed a polynomial projection approach

that is used in some applications.

However, the simplex algorithm remains the underlying algorithm utilized by most

commercial linear programming packages (e.g., see [7]). While the simplex method is not a

polynomial algorithm, the average running time is polynomial; and a problem such as (1)

with m constraints seldom takes more than 3m iterations [8]. Moreover, the simplex method

allows efficient post-optimality analysis and thus readily adapts to branch-and-bound

algorithms for the binary and integer models common in industrial applications. On the other

3

hand, interior-point methods such as Karmarkar’s projection algorithm are not suitable for

such uses.

A new algorithm, an extension of the simplex method, is proposed here. The essential

idea behind the algorithm comes from a class of unconstrained optimization problems in

abstract Hilbert spaces that are solved by the Schwarz inequality (see [9]) when two vectors

are collinear (i.e., the cosine of the angle between them is zero). These problems involve an

inner product and lead to a consideration of the angles formed by the objective function and

the individual constraints in (1).

The cosine simplex algorithm is summarized as follows. For a given problem it

begins by solving a relaxed problem consisting of the original objective function subject to a

single constraint yielding a nonempty bounded feasible region. At each subsequent iteration

of the algorithm, the most parallel constraint to the objective function among those

constraints violated by the solution to the current relaxed problem is appended to it. The dual

simplex algorithm is then applied. When no constraints are violated, the solution of the

current relaxed problem is optimal to the original problem.

Recent work by [10] and [11] have used a similar cosine criterion simply to get an

initial basis for the simplex algorithm, thereby reducing the number of required simplex

iterations. However, they first need to eliminate any redundant constraints. In contrast, our

approach uses the cosine criterion at each interation, and the current relaxed relaxed problem

involves only a fraction of the original constraints.

In Section 2 of this paper we describe the cosine simplex algorithm in more detail,

then give its basic properties in Section 3. Some simple illustrative examples are presented in

Section 4. Finally, in Section 5, we discuss future research.

4

2. The Cosine Simplex Algorithm Denote row i of the matrix A by ai so that constraint i of (1) becomes

ai x ≤ bi, i = 1, …, m. (2)

In the cosine simplex below, assume that (1) has a nonempty feasible region with a

constraint r in (2) for which arj > 0, j = l, …, n, and br > 0. These assumptions guarantee that

(1) has an optimal solution. For constraint i of (2), define cos θi =Ti

i

a ca c

as the cosine of

the angle θi between the normal vectors ai for constraint i and c for the objective function.

We call a constraint operative if it is part of the current relaxed problem and inoperative

otherwise. The initial relaxed problem

maximize z = cTx (3)

subject to

arx ≤ br

x ≥ 0

obviously has an optimal solution. The algorithm can now be stated.

The Cosine Simplex Algorithm.

Step 0. Compute cos θi =Ti

i

a ca c

, i = 1, …, m, i ≠ r, and order the constraints

according to decreasing cos θi, where ties are broken arbitrarily.

Step 1. Solve (3), which becomes relaxed problem 1, to obtain x1. Set k = 1.

Step 2. Check the inoperative constraints in decreasing order of cos θi. Take the first

one violated by xk and go to Step 3. If none is found, stop since xk solves problem (1).

5

Step 3. Set k = k+1. Append the violated constraint to the final tableau of relaxed

problem k to obtain relaxed problem k+1. Apply the dual simplex algorithm to obtain

a solution xk. Go to Step 2.

3. Discussion of the Cosine Simplex Algorithm The cosine simplex algorithm for (1) is an active set method [12] that fundamentally

extends the simplex algorithm by determining an entering constraint in addition to entering

and leaving basic variables. Furthermore, the cosine simplex appears to converge faster in

practice than the usual simplex method for four reasons. First, the cosine criterion for

appending a constraint reduces the total number of iterations in small examples. Second, at

each iteration only a small fraction of the original constraints will typically be involved in

the simplex computations. Third, relatively few constraints of a linear programming problem

are usually active at optimality [13]. Fourth, no artificial variables are required. In addition,

the cosine simplex algorithm eliminates the need for identifying redundant constraints as in

[10] and [11]. In fact, one may add the artificial and possibly redundant constraint x1 + … +

xn ≤ M, where M is a large positive number, if (1) has no bounding constraint arx ≤ br for the

initial problem (3).

A geometric interpretation of the new method is that Step 2 determines the

inoperative constraint violated by the current solution that is "most nearly parallel" to the

objective function. This cosine criterion appears to determine more quickly than the standard

simplex algorithm the active constraints such that the gradient of the objective function lies

in the cone generated by their gradients. Equivalently, it adds active constraints until the

Kuhn-Tucker conditions [8], which are both necessary and sufficient for (1), are satisfied.

6

4. Examples

Examples are now presented to show the efficiency of the cosine simplex algorithm

for small problems as well as its limitations. Example 1 is given in detail, while Examples 2,

3, 4, 5 are only summarized.

Example 1. Consider the problem

maximize z = 4x1 + 5x2 + 9x3 + 11x4

subject to

3x1 + 5x2 + 10x3 + 15x4 ≤ 100 (4)

x1 + x2 + x3 + x4 ≤ 15 (5)

7x1 + 5x2 + 3x3 + 2x4 ≤ 120 (6)

x1, x2, x3, x4 ≥ 0

The values to two decimals for cos θi in Step 0 for (4), (5), (6) are 0.99, 0.93, and 0.70,

respectively. Each of these constraints bound the problem, so Step (1) can be omitted. Since

the associated cos θi for (4) is a maximum, the first relaxed problem to be solved is thus

maximize z = 4x1 + 5x2 + 9x3 + 11x4 (7)

subject to

3x1 + 5x2 + 10x3 + 15x4 ≤ 100

x1, x2, x3, x4 ≥ 0

Solving problem (7) and proceeding with the cosine algorithm give the following sequence

of tableaus. Constraint (5) becomes operative in the second tableau. In the third tableau

constraint (6) is not violated by the solution z = 695/7, x1 = 50/7, x2 = 0, x3 = 55/7, so these

7

values are optimal to the original problem. The variables x5 and x6 below are the slack

variables for the entering constraints (4) and (5), respectively.

basis z x1 x2 x3 x4 x5 x6 rhs z 1 −4 −5 −9 −11 0 0

x5 0 3 5 10 15 1 100

z 1 0 5/3 13/3 9 4/3 0 400/3 x1 0 1 5/3 10/3 5 1/3 0 100/3 x6 0 0 −2/3 −7/3 4 −1/3 1 −55/3 z 1 0 9/21 0 11/7 15/21 13/7 695/7 x1 0 1 15/21 0 −5/7 -3/21 10/7 50/7 x3 0 0 2/7 1 4/7 1/7 −3/7 55/7

The standard simplex algorithm, on the other hand, requires four tableaus, including

the initial one. To compare the computational efficiency of the two algorithms for the above

problem we compute for the cosine simplex 1

C = (# of tableaus with constraints)p

k

k k=

×∑ ,

where p is the number of the number of constraints in the optimal tableau. For the standard

simplex we compute S = (# of tableaus required by the standard simplex for a solution)×m.

In this example, C = 5, while S = 12.

Example 2. This example demonstrates that the constraint most nearly parallel to the

objective function may not be active at optimality. The solution to the problem

Maximize z = −x1 − x2 + 20x3

subject to

x1 + x2 + 20x3 ≤ 120 (8)

−x1 + x2 + x3 ≤ 4

x1 − x2 + x3 ≤ 5

8

x1, x2, x3 ≥ 0

is z = 179/2, x1 = l/2, x2 = 0, x3 = 9/2. However, the nonredundant constraint (8) is most

nearly parallel to the objective function but not satisfied as equality by this solution.

Example 3. Consider the problem

maximize z = −x1 − x2 + 20x3

subject to

x1 − x2 + 20x3 ≤ 20

−x1 + x2 + x3 ≤ 4

x1 − x2 + x3 ≤ 5

−x1 − x2 ≤ −40

x1, x2, x3 ≥ 0

Applying the cosine simplex algorithm to this problem illustrates that no artificial variables

are required and that the selection criterion for constraints does not prevent the dual simplex

algorithm from sometimes requiring more than one iteration in Step 3. Thus the cosine

simplex algorithm appears to have the convergence properties of the simplex method.

Example 4. The next problem shows that the cosine algorithm may prevent cycling

without any additional analysis such as the lexicographic exiting rule. Consider

maximize z = (3/4)x1 − 150x2 + (1/50)x3 − 6x4

subject to

(1/4)x1 − 60x2 − (1/25)x3 + 9x4 ≤ 0

(1/2)x1 − 90x2 − (1/50)x3 + 3x4 ≤ 0

x3 ≤ 1

9

x1, x2, x3, x4 ≥ 0.

This problem can be shown to cycle [14] if the variable with the smallest subscript is chosen

to leave the basis in case of ties. The cosine algorithm solves this problem with no cycling to

obtain the optimal solution z = 20, x1 = l/25, x2 = 0, x3 = 1, x4 = 0.

Example 5. The final example shows the cosine simplex is substantially more efficient

than the standard simplex in solving a Klee-Minty problem [8] for which the standard

simplex requires one iteration for each extreme point of the feasible region. Consider the

following Klee-Minty problem from [15]

maximize z = 4x1 + 2x2 + x3

subject to

8x1 + 4x2 + x3 ≤ 125

4x1 + x2 ≤ 25

x1 ≤ 5

x1, x2, x3 ≥ 0

with optimal solution z = 125, x1 = x2 = 0, x3 = 125. The standard simplex algorithm solves

this problem in 8 iterations with the value S = 24, while the cosine simplex takes only 1

iteration with the value C = 1.

5. Conclusions An extension of the simplex algorithm was presented in this paper. For small linear

programming problems the cosine simplex algorithm solves them more efficiently than the

standard simplex method, eliminates the need for artificial variables, and deals with

redundant constraints. In addition, it can prevent cycling and readily handle Klee-Minty

10

problems. In subsequent work, we will investigate various modifications to the cosine

simplex algorithm that might enhance it. For example, we could include the cosine values for

the nonnegativity restrictions − xj ≤ 0, j = 1, …, n, in addition to (2). However, this change

will still not guarantee that the most parallel constraint to the objective function is active at

an optimal solution. In Example 2, for instance, the nonnegativity restriction −x3 ≤ 0 is

actually more parallel to the objective function than (8), yet it is not active at optimality

either. In addition, we shall also try to improve on the cosine criterion, which has

considerable geometric appeal for n = 2. Finally, we will apply the results to large-scale

problems.

6. Acknowledgement This research presented here was partially supported by the Texas Advanced Research

Program under Grant No. 003656-0197-2003.

References [1] G.B. Dantzig, “Maximization of a linear function of variables subject to linear

inequalities,” in T.C. Koopmans, editor, Activity Analysis of Production and Allocation, John

Wiley, New York, 339-347 (1951).

[2] G.W. Brown and T.C. Koopmans, “Computational suggestions for maximizing a linear

function subject to linear inequalities,” in T.C. Koopmans, editor, Activity Analysis of

Production and Allocation, John Wiley, New York 377−380 (1951).

[3] K.G. Murty and Y. Faithi, “A feasible direction method for linear programming,”

Operations Research Letters 3, 123-127 (1984).

[4] N. Megiddo, “Linear programming in linear time when the dimension is fixed,” Journal

11

of the Association of Computing Machinery 31, 114−127 (1984).

[5] R.G. Bland, D. Goldfarb, and M.J. Todd, “The ellipsoid method: a survey,” Operations

Research 29, 1039-1091 (1981).

[6] N. Karmarkar, “A new polynomial-time algorithm for linear programming,”

Combinatorica 4, 373−395 (1984).

[7] http://www.ilog.com/products/cplex/

[8] M.S. Bazaraa, J.J. Jarvis, and H.D. Sherali, Linear Programming and Network Flows, 3rd

edition., John Wiley, New York (2004).

[9] B. Naylor and G. Sell, Linear Operator Theory in Engineering and Science,

Springer−Verlag, New York (2000).

[10] H.V. Junior, M. P. E. Lins, “An improved initial basis for the simplex algorithm,”

Computers and Operations Research, in press, available at www.sciencedirect.com.

[11] N.V. Stojkovic, P.S. Stanimirovic, “Two direct methods in linear programming,”

European Journal of Operational Research 131, 417-439 (2001).

[12] P.E. Gill, W. Murray, and M.H. Wright, Practical Optimization, Academic Press, New

York (1981).

[13] G.B. Dantzig, Linear Programming and Extensions, Princeton University Press,

Princeton, N.J. (1963).

[14] H.M. Wagner, Principles of Operations Research, second edition, Prentice−Hall,

Englewood Cliffs, N.J. (1975).

[15] http://carbon.cudenver.edu/~hgreenbe/glossary/notes/Klee-Minty.pdf