A Framework for Privacy-Preserving ClusteringAlgorithms

I. CLUSTERING

Data clustering is an important data mining technique thatis based on grouping a set of unlabeled elements accordingto their similarity. Element similarities are defined in terms ofdistance metrics that need to be minimized within the samecluster and maximized between different clusters. Severalclustering solutions have been proposed in literature so far.Traditionally, these schemes have also been classified into:• Hierarchical

1) Agglomerative2) Divisive

• Partitioning1) Patitioning Relocation Clustering2) Density-Based Partitioning

The hierarchical schemes attempt to organize the data into ahierarchical structure, existing two approach i)bottom up oragglomerative, that creates a new cluster merging those ofprevious level and ii) top down or divisive, that create a newcluster splitting those of previous level. Instead the partitioningschemes identify conglomerated group among data. Clusteringschemes are also divided into hard and soft schemes, thehard schemes impose that an element belongs to only onecluster with membership value equal to one, the soft schemesallow that an element belongs to all cluster with differentmembership values bounded from zero to one. Soft clusteringis also referred to as fuzzy clustering because it is based onfuzzy logic.

In the following, we briefly review two of the most fa-mous clustering algorithms adopted in literature, K-Meansand Fuzzy C-Means, that can be easily adapted (as dis-cussed in II) to work in a privacy-preserving way. The datapoints considered in these schemes are addressed as vectorsdi, i = 1 · · ·N , where each vector component (coordinate)represents a different data feature. The clustering problem isformulated in terms of allocation of each point di into a clusterCj j = 1 · · · k, such that the distance (i.e. the similarity)among all the points in the cluster are as small as possible.For this purpose, a cluster centroid cj ∈ Rm is defined foreach cluster Cj .

A. K-Means Algorithm

K-Means is an hard partitioning algorithm whose goal is theidentification of the centroids of the clusters grouping the dataelements. To this purpose, the scheme works iteratively on theminimization of a function quantifying the distances betweenthe data elements and the centroid. Assuming to represent each

i-th data element in terms of a vector di ∈ Rm and to use theclassical L2-Norm for expressing the distances between data,the function to be minimized is given by:

k∑j=1

∑di∈Cj

‖di − cj‖2 (1)

where cj the centroid of the data cluster Cj , and k is thenumber of clusters in which elements have to be grouped.

Starting from k random centroids cj(0) selected at the first

iteration, the iterative algorithm can be summarized as follows:1) Update of the clusters: at each step t, each data element

i is assigned to the cluster corresponding to the closestcentroid (i.e. the centroid j-th for which distance ‖di−cj

(t)‖ is minimal;2) Update of the cluster centroids: for each cluster Cj

whose cardinality is |Cj |, the cluster centroid is updatedby averaging the data assigned to the cluster:

cj(t+1) = 1

|Cj |

∑d∈Cj

d (2)

3) Convergence verification: if the distance between cj(t+1)

and cj(t) is lower than the desired convergence level ε

for each cluster j the scheme is stopped.Although the scheme is straightforward and easily to be

implemented, it has some shortcomings since the final resultsmay depend on the initial choice of the centroid (i.e. the localoptimum found by the iterative scheme can be different fromthe global one). These problems can be particularly evidentwhen working with data strongly clustered.

B. Fuzzy C-Means Algorithm

Fuzzy C-means [1] is a soft partitioning algorithm whosefinal results are not very sensitive to the scheme initialization.While in the previous scheme each data element is assignedto a given cluster, in this scheme the beloning relationship isrepresented by a membership matrix U ∈ [0, 1]n·k, where thegeneric element uij gives the probability (also called mem-bership degree) that the i-th data belongs to the j-th cluster.The matrix U satisfies two conditions: i)

∑ni=1 uij > 0, i.e.

each cluster j has a non-null probability to include at leastone data element, and ii)

∑kj=1 uij = 1, i.e. each element i is

certainly contained in one of the considered clusters. On thebasis of this matrix, a new objective function is defined as:

k∑j=1

∑di∈Cj

ufij‖di − cj‖2 (3)

1

where f is the fuzzyfication factor affecting the shape of theclusters in the range [1,∞], which is typically set to 2. Startingfrom a random selection of the clusters’ centroid, the schemesteps are the following:

1) Update of the membership matrix: at each step t, eachdata element i is probabilistically assigned to eachcluster j by specifying the whole vector of belongingprobability ui ∈ [0, 1]k:

uij(t+ 1) = 1∑kl=1(‖

di−cj(t)di−cl(t)

‖)2

f−1(4)

2) Update of the cluster centroids: for each cluster Cj , thecluster centroid is updated by minimizing the objectivefunction:

cj(t+ 1) =∑n

i=1 uij(t+1)f ·di∑ni=1 uij(t+1)f

(5)

3) Convergence verification: if the metric ‖U (t+1) −U (t)‖is lower than the desired convergence level ε, the schemeis stopped.

C. Cross-Entropy Algorithm

In this section we show how the clustering problem can beaddressed through combinatorial problem algorithms.

We assume that a set of points D, defined over the spaceRm, must be assigned to a set of clusters C.

Let X be the matrix of one possible points allocation, withcolumn the number of cluster k and rows the number of pointsN , the entry’s matrix xij is 1 if the point di belongs tothe cluster j, otherwise is 0. Again, let C the matrix of thecluster’s centroids, where its column j−th holds the centroid’scoordinates.

Then, the problem stated above can be mathematicallywritten as:

P1 Given the set X of all possible allocations X

To find that one

minX∈X

D(cj , [(d1, · · · ,dN ) ·X]

(j))∀ j = 1 · · · k

s.t∑j

∑i

xi,j ≤ N (6)

D(cj , [(d1, · · · ,dN ) ·X]

(j))

is the distance between thevector of the centroids and the coordinates of the nodes in Cj .The constraint (6) assures that each node di belongs to onlyone cluster, and that the number of nodes clustered is N .

The problem P1 becomes an NP-Hard problem because ofthe dimension of X , given that for small values of N and kits dimension rapidly increases.

A possible approach to explore the space of solutions X ,can be that one involves the random methods. In this, casethe solution of the optimization problem is the most probableallocation X that minimize the objective function O(X).The study of deterministic problems through them sthochastic

formulation it is a consolidated techniques. In this case,we need to define the Associated Stochastic Problem (ASP)linked to the initial deterministic problem. Usually, a stochasticproblem is more difficult to solve than a deterministic problem,but in this case helps us to probe the space of solution avoidingexhaustive investigations.

Let γ be the min value of the objective function O(X) inthe optimal value X, to study the ASP we need to definethe density distribution of probability (d.d.p) f(·;u), over theoptimization domain X , and define the new objective:

Pu {O(X) ≤ γ} = Eu{IO(X) ≤ γ

}; X ∈ X . (7)

Through the eq. (7) we find all the possible values ofX, with d.d.p f(·;u), so that the probability of the eventO(X) ≤ γ is maximized. This is equivalent to find the Xswhich optimize, in stochastic sense, the objective functionavoiding extensive researches of the X values. Through theformulation in the equation (7) our objective becomes to findthe f(·;u) that verify the eq. (7). Hence, we need to addresstwo problems: define the ASP for our starting problem andthen to calculate the d.d.p f(·;u).

The starting problem was to find the best allocation of thepoints D, so that the the distance among all the nodes ofthe cluster j is minimized, subject to some constraints. Thisproblem can be viewed again as a Stochastic Node Network(SNN) problem with set of nodes {1, · · · , N} and set of nodecharacteristics {1, · · · , k}. In this way the allocation matrix Xcan be replaced by the allocation vector x. The objective is toassign the node characteristics, so that the objective functionis minimized.

Without loss of generality, we assume that the nodes of thenetwork are independent random variables, with d.d.p. Π =[Π1 · · ·Πk]′, the column Πi = [π1i · · ·πki] is the d.d.p ofthe node i to hold the characteristics {1, · · · , k}. Then, x =[x1, · · · , xN ] is the allocation of characteristics vector, wherexi is the most probable value of characteristic for the node i.Hence, if we have the set of points D and the set of clustersC we need to find the d.d.p for the vector x = [x1, · · · , xN ]where xi is the most probable cluster j where the user i isassigned, according to the d.d.p Π = [Π1, · · · ,Πk]′.

These kind of SNN problems can be resolved throughalgorithms which are able to partition N nodes into k subsets.By means of the Cross-Entropy (CE) method we propose toaddress this kind of problems through an algorithm which findsthe solution iteratively in the time t starting from an initiald.d.p Π(0) i.e uniformly distributed.

In the following the main structure of the algorithm isshowed.

The (1−α)-quantile γ̂(t) is evaluated through the equation

γ̂ = max {o | P(O(x(s)) ≥ o) ≥ α} . (8)

over the clustering samples x(1), · · · , x(S).The updating of Π̂(t−1), at step .3 by the S samples, can

Algorithm 1 Main Optimal Allocation Algorithm

1. Generate the initial distribution Π̂(0) =[Π̂

(0)1 , · · · , Π̂(0)

k ]′ with entries π̂(0)ji = 1

k ; i=1,· · · ,N ;j=1,· · · ,k;

while (|γ̂(t) − γ̂(t−1)| ≤ ε) do2. Generate the random vectors x(1), · · · ,x(S) of pos-sible allocations with Π = Π̂(t−1); evaluate the sample(1-α)-quantile γ̂(t) of the objective function

3. Update Π̂(t−1) to Π̂(t) by the sample vectorsx(1), · · · ,x(S)

4. Filter Π̂(t) by Π̂(t) = αΠ̂(t) + (1− α)Π̂(t−1)

t=t+1end while

be performed by the following equation:

π̂(t)ji =

S∑s=1

I{O(x(s))≥γ̂(t)}I{xi(s)=j}

S∑s=1

I{O(x(s))≥γ̂(t)}

(9)

D. Incremental Cross-Entropy Algorithm

Clustering techniques can be used to classify a big data setcoming from a long observation period, many days or manymouths, during which a certain amount of data is availableevery t time, for example ones or twice in a day. In this kindscenario and for an application that do not support big storageresources perform the clustering computation via traditionalalgorithm can be a problem. For this reason we present anincremental form of the clustering solution based on the CrossEntropy method explained above(view I-C).Suppose that every day are available N data vector, from anequal number of users, referring to some of their behavioror habit. If so, it is reasonable think that the users will notrevolutionize their practices from day to day and therefored(t)i

1 probably will not equal to d(t+1)i but it will be slightly

different. Staring form this assumption at first day we calculatek centroids over N data vectors d

(t)i by the Cross Entropy

Algorithm, if really the new data are close to the previous atthe second day the d

(t+1)i data vectors will move the centroids

position of a small amount, called Dc; so we assume that thenew centroids will be equal to the previous plus this Dc thatwe identify like a normal variable with mean zero e covariancematrix Σ.

c(t+1)i = |c(t)i + Dc

(t+1)i | (10)

1In this section t is not referred to an algorithm iteration but specify differentsubperiod, i.e. one day, into observation period

Now, from (10), we can rewrite the object function (1)proposed in I-A in this way:

k∑j=1

N∑i=1

‖d(t+1)i − c

(t+1)j ‖2 =

=

k∑j=1

N∑i=1

‖d(t+1)i − c

(t)j + Dc

(t+1)i ‖2 =

=

k∑j=1

N∑i=1

‖d(t+1)i − D̃c

(t+1)

i ‖2

(11)

But how compute each Dc? We use the CE method toestimate them supposing S possible choices and updating the Σmatrices following the Algorithm 1. If cj ∈ Rm j = 1 · · · kthen we need to m Dc vectors k-dimensionally one for eachcomponent (or coordinate) of all centroids: the j-th value of thei-th Dc vector will be the variation to which will be subjectedthe i-th component of j-th centroid.At the end of the observation period we will have k cen-

Algorithm 2 Incremental Cross Entropy Algorithm

1. At t=1 run Algorithm1 and generate k centroids c(t) m-dimensional;

while 1 < t ≤ tmax do

2. Create m diagonal matrix Σ(t)(0) k × k and set each

entry σjj ∈ Σ(t)i to i-th component of j-th centroids c

(t)j .

while (|Dc(t)(iter) −Dc

(t)(iter−1)| ≤ ε

′) do

3.1. Generate m Dc from a normal distribution withcovariance matrix Σ

(t)(iter−1); repeat until S Dc sam-

ples have been compute.3.2. Following (10) calculate S D̃c and through (11)S values of object function γ̂(t). Evaluate the sample(1-α)-quantile γ̂(t)

3.3. Update each Σ(t)(iter−1) to Σ

(t)iter by the sample

vectors Dc who satisfy the condition over γ̂(t)

3.4. Filter Σ(t)iter by Σ

(t)iter = αΣ

(t)iter + (1−α)Σ

(t)iter−1

iter=iter+1end while

4. (cj1, cj2, · · · , cjm)(t+1) = (cj1 + Dc1j , cj2 +Dc2j , · · · , cjm +Dcmj)

(t)

t=t+1end while

troids called c(incr). Otherwise we denote by c(opt) (optimalsolution) those generate running the Algorithm1 over theentire data set, probably the first centroids set will be slightlydifferent of the latter but a trade-off is unavoidable: in thisway you reduce the storage requirement but lost something in

accuracy.

II. PRIVACY-PRESERVING GENERAL APPROACH

As stated above, we need to cluster a data set, ensuring theprivacy of the users. Precisely, the clustering algorithm mustupdate the clusters’ centroids ensuring that no one can accessto the user data. In most of the clustering algorithms the updateof the centroids is obtained by the linear combination of thedata set by the iterative way. Hence, the algorithm performssimply a sequence of additions as in the case of the in K-meansalgorithm[2]. To preserve the privacy of the data during thecluster algorithm, hence any addend of the sum can not bedisclosed.

This kind of problem is the typical case of Secure Multi-Party Computation MPC, which is one of the most importantproblems addressed in cryptography [3]. In the MPC problemthe objective is to compute the function y = f(x1, ..., xn),where x1,.....,xn are the private inputs of n users P1,...,Pn,named parties. The function f must be evaluated under theconditions stated below:• The parties do not trust in each others;• No Trusted Third Party(TTP) is available to compute the

function;Even if in [4], [5] is proved that for both the problems two-

party computation and multi-party computation the functionf can be evaluated under the conditions II, [6] shows thatthese solutions can be impractical. Precisely, the computationcost as well as the communication cost can become heavy.Then the author suggests some ad-hoc solutions for some givenproblems.

Driven by the efficiency requirement in terms of computa-tional overhead, in this paper we propose a method of MPCbased on the Secret Sharing scheme (SSS) [7], [8]. Throughthe SSSs the Dealer can share the secret s with K parties,giving to each one a share of the secret, without reveal secretitself. All the subsets of the K parties authorized to recoverthe secret are named Access Structure (A). Any subset B ⊆ Awith cardinality |B| = t ≤ K can recover the secret s, througha set of shares r1,.....,rt. Note that the single share can not usedto recover the secret.

The SSSs were investigated independently by [7] and [8] in1979. Both the authors have developed two different schemesfor the threshold secret sharing, also known as (t,K)-scheme2.For these schemes A is the set of all the subsets of parties,with cardinality t at least.

In the last years the SSSs have been extensively studied,and in particularly in [9] a survey on the SSSs is showed, andin [10] the authors prove some features of these schemes.

Following the literature the most important features of theSSSs can be summarized as below:• Perfect Secrecy: the SSS hods the perfect secrecy when

any subset of parties, each one with given share, andsuch that the subset not belongs to the Access Structure,can not recover anything about the secret s through the

2t <= K and K=cardinality of entire set of parties

Fig. 1. Network scenario

collecting of its shares. Precisely, by this property thesecret is uniformly distributed in its membership group,for the set of shares discussed above.

• Homomorphic property: the SSS verifies the homomor-phic property if given two secrets s1 and s2, the SSSgenerates the shares r11,...,r1K for the first secret andthe shares r21,...,r2K for the second one, then the sumof shares r1i + r2i is one of the shares of the sum of thesecrets s1 + s2.

A. Problem Definition

The reference scenario for the problem addressed is showedin figure 1. The scenario holds a set of N customers, for whichthe consumer profile and the service price must be generated,guaranteeing the customer’s privacy. We assume that eachcustomer is endowed with a device to log its behaviors. Hence,we assume that any node in the network can be identified byits device.

The two privacy servers in the scenario emulate the TTPfor the MPC problem addressed here. We assume that privacyservers can exchange only the data sequence according tothe prefixed protocol, showed in the following 3. Finally, weassume the privacy servers honest but curious. According tothis model the involved servers follow the protocol steps, butthey try to obtain additional information from all the datahandled.

We assume that the data exchange among the privacy serversand network nodes is achieved by secure wireless channels.

Our objective is characterize the consumer profile for thecustomers through privacy-preserving clustering algorithms.

As discussed above we need to define a protocol for thedata exchanging among nodes and the privacy servers. In thefollowing are showed the step of the protocol developed forthis application.

3This assumption does not appear to be critical in our target scenario

1) The user i = 1...N gives one share of his own secretsi to each privacy-server. This step must be performedfor the data vector and for the cluster membershipinformation;

2) Both privacy-servers through the homomorphic propertycan obtain one share of the sum of the all secretssimply summing all the collected shares

∑ni=1 ri1 and∑n

i=1 ri2;3) The privacy-server whic perform the update of clusters’

centroids receive from the other one its own sum result.4) Privacy-server which perform the update of the clusters’

centroids send the results to the users.The protocol discussed above is based on the SSS chosen for

our application. Precisely, for our problem we apply a schemebased on that one proposed by [11], where a Dealer identifiedby the network node, in order to share his secret s follows thesteps showed below:• The dealer chooses K − 1 random value r1,.....,rK−1• The dealer computes the K − th share as rK = s− r1-

...-rK−1• The dealer sends ri to the i− th privacy server

III. APPLYING SSS ON CLUSTERING ALGORITHMS

A. SSS on K-Means

In this section we will show how it’s possible to apply aSSS on a clustering algorithm, in particular we will focus onthe K-Means algorithm as an example. As shown above theupdating law (1) consists, for a given cluster j, of a sum of allelements wich have slightest distance from j−th centroid overthe cardinality of this set. Starting from this observation, servermaking the update, to be able to perform such a compute,needs know this sum and this cardinality, but without havingaccess to any singular addend.To make server realizing suchtask, users work as below:• Create a matrix D with all null value except the column

indexed by the belonging cluster, in which they put themdata value di, so they can send to the two servers oneshare of they own matrix.

• Create a vector I with all null value excpet the onecorresponding to the index of belonging cluster whichis set to 1, they then send one share of this vector to thetwo servers.

It’s straightforward to note that A =∑ni=1 Ii =

[|C1| · · · |Ck|], and that each column of the matrix B =∑ni=1

Di holds the sum of all and just that data which belong tothe cluster indexed by the column; these consideration makesimple updating the centroids just by dividing each columnby the value in the correspondent position of the vector ofcardinality, or better:

cj(t+1) = 1

A(j)B:j (12)

B. SSS on Cross Entropy Method

Here we explain how to apply a SSS on the clusteringalgorithm based on CE-method. The updating law for thecentroids and the object function are the same of the previous

Fig. 2. Probabilities distributions of centroids alteration

section I-A where we describe K-Means, but here we haveto manage N vector sample x S-dimensional of possibleallocations (view Algorithm1). Initially each user select arandom vector x on Π̂(0) and then works as follow:• Generate S null matrices, D1,D2,...Di,...DS , except the

column indexed by the value read in x(i) where she putsher data;

• Generate S null vectors, I1,I2,...Ii,...IS , except the col-umn indexed by the value read in x(i) where she putsthe value 1.

Subsequently she creates two shares, both each matrix D andeach vector I , and sends them to each server. As well as forK-Means also in this case the server, making the updating,holds the sum of all N matrices D and all N vectors I, butnow it has S sums, obtained from the S shares coming fromeach one user, and employs them for compute S samples fork centroids. To updating the probability distribution Π̂(0) likeAlgorithm1the user needs to evaluate the objective function,for this reason the server sends to her S centroids samples.Each user compute the Euclidean Distance between her dataand the centrodis suggested in x, by this value organizes Smatrices again at the same manner, creates the shares andsends them to the two servers. Now the servers can compute Svalue of the object function and sends them back to the users.Will be the user herself to evaluate the (1 − α)-quantile γ̂(t)

and update the initial distribution like Algorithm 1. The stopcriterion is the same of K-Means. When the convergence isachieved among the S samples we choose the set of k centroidscorresponding to the sample that minimize the object function.

IV. PERFORMANCE EVALUATION

A. Algorithms’ precision

First of all we need to evaluate performance of the algo-rithms explained above in terms of how centroids changein several runs on the same data-set, that is quantify thesensitivity related to the scheme initialization. As we alreadysaid, K-Means is the one with high sensitivity on the schemeinitialization then the others here investigated; to prove thisfeature we ran each one algorithm on the same data-set n times(with n = 100), where data-set is made by 800 bidimensionalvectors where each component belongs to a specific range.

In figure 2 we then report the probabilities distributions ofthe discance by which centroids could range from differentalgorithm’s running;

REFERENCES

[1] W. Pedrycz, Knowledge-Based Clustering: From Data to InformationGranules. Hoboken, NJ, USA: Wiley, 2005.

[2] J. B. MacQueen, “Some Methods for classification and Analysis ofMultivariate Observations,” in Proc. of 5-th Berkeley Symposium onMathematical Statistics and Probability, vol. 1, 1967, pp. 281–297.

[3] C. Orlandi, “Is multiparty computation any good in practice?” in Proc.IEEE Intl. Conf. ICASSP, Prague, Czech Republic, May 2011, pp. 5848–5851.

[4] A. C. Yao, “Protocols for secure computations,” in Proc. IEEE Intl.Conf. SFCS’08, Chicago, IL, USA, 1982.

[5] O. Goldreich, S. Micali, A. Wigderson, “How to play any mental gameor a completeness theorm for protocols with honest majority,” in Proc.ACM Symposium STOC’87, New York, NY, USA, 1987, pp. 218–229.

[6] O. Goldreich, Foundations of Cryptography:Volume 2-Basic Applica-tions. Cambridge University Press, 2004.

[7] A. Shamir, “How to share a secret,” in Communications of the ACM,vol. 22, November 1979, pp. 612–613.

[8] G. R. Blakley, “Safeguarding cryptographic keys,” in AFIPS Conf. Proc.,vol. 48, 1979, pp. 313–317.

[9] A. Beimel, “Secret-Sharing Schemes: a Survey,” in Coding and Cryp-tology, Third International Workshop IWCC 2011, Lecture Notes inComput. Sci., 2011, pp. 11–46.

[10] E. D. Karnin, “On secret sharing systems,” in IEEE Transactions onInformation Theory, vol. IT-29, no. 1, January 1983, pp. 35–41.

[11] M. Ito, A. Saito, T. Nishizeki, “Secret sharing schemes realizing generalaccess structures,” in Proc. of the IEEE Global TelecommunicationConf., Globecom 87, 1987, pp. 99–102.