Opportunistic Spectrum Access for Energy-constrained
Cognitive Radios
Anh Tuan Hoang, Ying-Chang Liang, David Tung Chong Wong,
Yonghong Zeng, and Rui Zhang
Institute for Infocomm Research, Singapore
Abstract
This paper considers a scenario in which a secondary user makes opportunistic use of a channel allocated
to some primary network. The primary network operates in a time-slotted manner and switches between idle and
active states according to a stationary Markovian process.At the beginning of each time slot, the secondary user
can choose to stay idle or to carry out spectrum sensing to detect if the primary network is idle or active. If the
primary network is detected as idle, the secondary user can carry out data transmission. Spectrum sensing consumes
time and energy and introduces false alarms and mis-detections. Given the delay cost associated with staying idle,
the energy costs associated with spectrum sensing and data transmission, and the throughput gain associated with
successful transmissions, the objective is to decide, for each time slot, whether the secondary user should stay idle
or carry out sensing, and if so, for how long, to maximize the expected net reward. We formulate this problem
as a partially observable Markov decision process (POMDP) and prove several structural properties of the optimal
spectrum sensing/accessing policies. Based on these properties, heuristic control policies with low complexity and
good performance are proposed.
I. INTRODUCTION
The traditional approach of fixed radio spectrum allocationleads to under-utilization. It has been reported
in recent studies by the US Federal Communications Commission (FCC) that there are vast temporal and
spatial variations in the usage of allocated spectrum [1]. This motivates the concepts of opportunistic
spectrum access (OSA), which allows secondary cognitive radio (CR) systems to opportunistically exploit
the under-utilized spectrum.
One of the core components of an OSA system is the spectrum-sensing module, which examines a
spectrum of interest to determine whether the primary network which owns the spectrum is currently
active or idle. Spectrum sensing, therefore, is a binary hypothesis test or a series of binary hypothesis
tests. Spectrum sensing can have several effects on the OSA system, which are highlighted as follows.
2
• Energy consumption: carrying out spectrum sampling and subsequent signal processing consumes
energy. In general, the longer the sensing time, the more theconsumed energy.
• Time consumption: to avoid self interfering with the sensing process, the OSAsystem may need to
suspend its data communications when carrying out spectrumsensing. This means spectrum sensing
consumes communication time.
• False alarms: a false alarm occurs when the spectrum sensing module mistakes an idle primary
network as active. This leads to a spectrum access opportunity being overlooked.
• Mis-detections: a mis-detection occurs when the spectrum sensing module mistakes an active primary
network as idle. This leads to possible collision between secondary and primary transmissions.
In this paper, we take the above effects into account when designing an energy-constrained OSA system.
A. General Control Problem
We consider a scenario in which a secondary user makes opportunistic use of a channel allocated to
some primary network. The primary network operates in a time-slotted manner and switches between
idle and active states according to a stationary Markovian process. Within each time slot, the state of the
primary network remains unchanged. At the beginning of eachtime slot, the instantaneous state of the
primary network is not directly observable and the secondary user needs to decide whether to stay idle
or to carry out spectrum sensing. If the secondary user chooses to carry out spectrum sensing, it needs to
decide the duration of the sensing period and to configure related parameters to meet a minimum detection
probability. Subsequently, if spectrum sensing indicatesthat the primary network is idle, the secondary
user proceeds to transmit data during the rest of the time slot.
There are important trade-offs when the secondary user makes the above control decisions. By staying
idle in a particular time slot, the secondary user conservesenergy, but at the same time suffers increase in
delay and reduction in throughput. By carrying out spectrumsensing, the secondary user consumes time
and energy to acquire knowledge of the state of the primary network, and stands a chance to transmit
data if the primary network is idle. Furthermore, there are trade-offs involving energy consumption,
sensing accuracy, and transmission time when the duration of sensing periods is varied. When the required
probability of detection is fixed, increasing the sensing time can reduce the probability of false alarms and
therefore increase the probability of transmission for thesecondary user. However, increasing the sensing
time also reduces the time available for transmission.
3
For the secondary user, given the delay cost associated withstaying idle in a time slot, the energy
costs associated with spectrum sensing and data transmission, and the throughput gain associated with
a successful transmission, we consider the problem of finding an optimal policy which decides the idle
and sensing modes, together with spectrum sensing time, to maximize the expected net reward. Here, the
reward is defined as a function of delay and energy costs and throughput gain.
B. Contributions
The main contributions of this paper are as follows.
• We formulate the control problem that captures important throughput and delay/energy trade-offs for
OSA systems. The problem involves important decisions for the secondary user, i.e., to stay idle or
to carry out spectrum sensing, and to determine the optimal duration of each sensing period.
• We analyze the problem using the framework of partially observable Markov decision processes
(POMDP) and prove important structural properties of the optimal control policies that maximize the
expected net reward.
• Based on theoretical characterization of the optimal policies, we propose heuristic control policies
that can be obtained at lower complexity while achieving good performance. One of these policies
is based on grid approxmimation of POMDP solutions.
• Finally, we obtain numerical results to support our theoretical analysis.
C. Related Work
There has been a series of recent works on optimizing spectrum-sensing activities in OSA systems
[2]–[6]. These works can roughly be classified into two groups, i.e., those that focus on the control within
each time slot, when the status of a primary network is more orless static [2]–[4] and those that focus
on the time dynamics of the control problem [5], [6].
In [2]–[4], the sensing duration within each time slot can bevaried, i.e., the spectrum-sensing module
can operate at different ROC curves. The objective then is totrade off between sensing accuracy and time
available for communications. Assuming the mis-detectionprobability is fixed, the longer the sensing
duration, the lower the false alarm probability. However, the longer the sensing duration, the less time is
available for communications. As the focus is on control within each time slot, the dynamics of primary
networks is not taken into account in [2]–[4].
4
In [5], Zhao et al. consider a spectrum access scenario similar to ours, where primary networks switch
between idle and active states in a Markovian manner and a secondary user carries out spectrum sensing
prior to opportunistic data transmission. An important result of [5] is the separation principle, which
decouples the sensing policy from that of the sensor and spectrum access policy. Unlike in our model,
in [5], the energy cost of sensing is not of concern and the secondary user carries out sensing in every
time slot. Furthermore, in [5], determining the sensing duration is not part of the control problem, rather,
it is assumed that a fixed Receiver Operating Characteristic(ROC) that defines the relationship between
false-alarm and mis-detection probabilities is given. In our model, varying the spectrum-sensing duration
results in different ROC curves. The work in [6] does take into account the energy and power consumption
when scheduling spectrum sensing. However, in [6], spectrum sensing is assumed perfect, i.e., there are
no false alarms and mis-detections.
To some extend, this paper bridges the gap between the two classes of problems considered in [2]–[4]
and in [5], [6]. In particular, our control problem recognizes the energy costs of sensing and transmission,
allows the variation of spectrum-sensing duration within each time slot, and incorporates the dynamics
of the primary network over time. All these factors are takeninto account for the final objective of
maximizing the expected long term net reward received by thesecondary user.
It is also interesting to note that some important results inthis paper bear significant similarities to
those in [7] and [8], even though the control scenarios are totally different. In [7], [8], the problem of
scheduling packet transmission over time-varying channels with memory is considered, with the objective
of balancing throughput and energy consumption. The state of the channel is not directly observed and
the authors also formulate and analyze the problem using a POMDP framework.
D. Paper Organization
The rest of this paper is organized as follows. In Section II,we describe the system model and the
control problem. Important properties of the optimal control policies are proved and discussed in Section
III. In Section IV, heuristic policies are proposed. Numerical results and discussion are presented in
Section V. Finally, we conclude the paper and highlight future directions in Section VI.
5
II. SYSTEM MODEL
A. Primary Network
We consider a channel being allocated to a primary network operating in a time-slotted manner. In each
time slot of durationT , the primary network is either active (on) or idle (off). From one time slot to the
next, the primary network switches between active and idle states according to a stationary Markovian
process specified by the following state-transition matrix:
M =
1 − b b
g 1 − g
, 0 < g, b < 1, (1)
whereb is the probability that the primary network becomes active in the next time slot, given that it is
idle in the current slot andg is the probability that the primary network becomes idle in the next time
slot, given that it is active in the current time slot. The stationary probabilities of being idle and active
for the primary network areπi = g/(b + g) andπa = b/(b + g), respectively.
From the secondary user’s point of view, when the primary network is idle, the user has a ‘good’ channel
to exploit. On the other hand, an active primary network results in a ‘bad’ channel for the secondary user.
This leads to an interesting observation thatM can be regarded as the state-transition matrix for a virtual
Gilbert-Elliot (GE) channel of the secondary user. In [9], for a GE channel with state-transition matrix
M , the channel memory is defined asµ = 1− b− g. Whenµ > 0, it is said that the channel has positive
memory, i.e., the probability of remaining in a particular state is greater than or equal to the stationary
probability of that state. In this paper, we also assume thatµ = 1 − b − g > 0.
B. Opportunistic Spectrum Access
A secondary user opportunistically accesses the channel when the primary network is idle by first
synchronizing with the slot structure of the primary network and then carrying out the following mechanism
(illustrated in Fig. 1).
1) Spectrum Sensing: If the secondary user wishes to transmit in a particular slot, it will first spend a
time durationτ at the beginning of the slot to carry out spectrum sensing. This basically involves sampling
the channel and carrying out a binary hypothesis test:
H0 : the primary network is idle,
versus H1 : the primary network is active.(2)
Let θ denote the outcome of the above binary hypothesis test, where θ = 0 meansH0 is detected andθ = 1
otherwise. Associated with the spectrum sensing activity are probability of false alarm, i.e., mistakingH0
6
for H1, and probability of mis-detection, i.e., mistakingH1 for H0. In this paper, we assume that the
secondary network must carry out spectrum sensing to meet a fixed probability of detectionPd. Then,
the probability of false alarm is a function of the sensing time τ and is denoted byPfa(τ). The sensing
durationτ must be within the interval[τmin, τmax], where0 < τmin ≤ τmax < T . It is assumed that for the
given range ofτ , 0 < Pfa(τ) < Pd < 1. This is in fact a reasonable assumption as in practical cognitive
radio systems [10], we normally havePd > 90% andPfa < 10%. Furthermore, we assume thatPfa(τ) is
continuous, differentialbe, and decreasing inτ ∈ [τmin, τmax].
2) Data transmission: If the spectrum sensing results inθ = 0, the secondary user proceeds to transmit
data in the rest of the time slot. Otherwise, ifθ = 1, the secondary user must stay quiet and wait until
the next time slot to try again.
3) Acknowledgment: Even though the spectrum sensing outcome indicatesθ = 0, this can be due to
a mis-detection. Mis-detections result in collisions between primary and secondary transmissions. In this
paper, we assume that if collision happens due to mis-detection, a negative acknowledgment (NAK) is
returned. On the other hand, if the secondary transmission is carried out when the primary network is
actually idle, a positive acknowledgment (ACK) is returned.
C. POMDP Formulation
At the beginning of each time slot, the secondary user decides whether or not to carry out spectrum
sensing, and if so, for how long. As the instantaneous state of the primary network is not directly observed,
our control problem can be classified as a discrete-time POMDP with the following components.
1) Belief State: In a discrete-time POMDP, the decision maker selects an action and receives some
reward, together with some observation which reveal information about the actual system state. It is well
known ( [11]) that for each POMDP, all information that is useful for making decisions can be encapsulated
in the posterior distribution vector of the system states. In our control problem, at the beginning of each
time slot, based on previous actions and observations, the secondary user can calculate the probability
that the primary network is idle in the time slot. We denote this probability byp and name it the ‘belief
state’. After each time slot, depending on the action taken by the secondary user and the corresponding
outcome, the belief statep can be updated according to one of the following four cases.
Case 1: The secondary user stays idle and does not carry out spectrum sensing. Then, the next belief
state, i.e., the probability that the primary network is idle in the next time slot can be derived as:
L1(p) = p(1 − b) + (1 − p)g = p(1 − b − g) + g. (3)
7
Case 2: The secondary user senses the channel for the durationτ , obtains the outcomeθ = 1, i.e., the
primary network is active, and therefore needs to keep quietin the rest of the time slot. Using Bayes’
rule, the belief state in the next time slot can be derived as:
L2(p, τ) =pPfa(τ)(1 − b) + (1 − p)Pdg
pPfa(τ) + (1 − p)Pd. (4)
Case 3: The secondary user senses the channel for the durationτ , obtainsθ = 0, i.e., the primary
network is idle, carries out transmission and subsequentlyreceives an ACK at the end of the slot. The
ACK implies that the primary network is actually idle duringthe current time slot and therefore, the belief
state in the next time slot isL3 = 1 − b.
Case 4: All the same as Case 3, except that an NAK is received at the end of the slot. This implies that
mis-detection happens during spectrum sensing, the primary network is actually active during the current
time slot and therefore, the belief state in the next time slot is L4 = g.
As can be noted,L3 and L4 do not depend onp and τ . However, in the rest of this paper, in order
to simplify the notation, we useL1(p, τ), L3(p, τ), L4(p, τ) interchangeable forL1(p), L3, L4 defined
above. Also, lettingQi(p, τ), i = 2, 3, 4 denote the probabilities that Casei above happens, we have:
Q2(p, τ) = pPfa(τ) + (1 − p)Pd, Q3(p, τ) = p(1 − Pfa(τ)), and Q4(p, τ) = (1 − p)(1 − Pd). (5)
2) Properties of Li(p, τ): From (3) and the assumption that the state-transition matrix M has positive
memory, i.e.,1 − b − g > 0, it follows that L1(p) is increasing inp. Also, it can be verified that:
∂L2(p, τ)
∂p=
(1 − b − g)PdPfa(τ)
[Pd − p(Pd − Pfa(τ))]2. (6)
As 1− b−g > 0 andPd > Pfa(τ) > 0, ∂L2(p,τ)∂p
is positive and increasing inp. Therefore,L2(p) is convex
and increasing inp. At the same time:
L2(p) =(1 − b)(pPfa(τ) + (1 − p)Pd) − (1 − p)Pd(1 − b − g)
pPfa(τ) + (1 − p)Pd≤ (1 − b). (7)
Similarly, it can be shown thatL2(p) ≥ g. So
L4(p) ≤ L2(p) ≤ L3(p). (8)
3) Costs, Reward, and Control Objective: When carrying out spectrum sensing, the secondary user
spends energy in channel sampling and signal processing. Weassume that the energy cost of carrying
out spectrum sensing forτ units of time is a continuous, non-negative and increasing function in τ and
is denoted bycs(τ). If the sensing outcome isθ = 0, the secondary user proceeds to transmit during the
8
rest of the time slot. Letrt and ct respectively be the gain in throughput and the energy cost, both are
measured per unit of transmission time. It is reasonable to assume thatrt > ct ≥ 0, otherwise, there is
no justification for the secondary user to carry out transmission.
Note that the secondary user can also choose to stay idle, i.e., neither carry out spectrum sensing nor
data transmission, during a time slot to conserve energy. However, doing so results in negative effects
such as lower throughput and longer latency. We assume that,for the secondary user, the cost of staying
idle during each time slot isci, ci ≥ 0.
In a particular time slot, if the probability of the the primary network being idle isp and the secondary
user carries out spectrum sensing forτ units of time, then the expected net gain can be calculated as:
G(p, τ) =p(
1 − Pfa(τ))
(T − τ)rt − cs(τ) −(
p(1 − Pfa(τ)) + (1 − p)(1 − Pd))
(T − τ)ct
=p(1 − Pfa(τ))(T − τ)(rt − ct) − (1 − p)(1 − Pd)(T − τ)ct − cs(τ).
(9)
As, by assumption,rt > ct, T > τ , and Pfa(τ) < Pd < 1, G(p, τ) is continuous and increasing inp.
G(p, τ) is also continuous inτ .
To simplify the notation, we also useτ = 0 to represent that the secondary user chooses to stay idle.
We then have the following expected reward when the sensing decision is set toτ .
R(p, τ) =
−ci τ = 0
G(p, τ), τmin ≤ τ ≤ τmax.(10)
We are interested in the following problem.
Definition 1: Let pn denote the probability that the primary network is idle during time slot n, select
the sensing time τn, τn ∈ {0, [τmin, τmax]}, to maximize the following discounted reward function
E{
N−1∑
n=0
αnR(pn, τn)∣
∣ p0 = p}
, (11)
where 0 < α < 1 is a discounting factor, and 1 ≤ N ≤ ∞ is the control horizon.
III. STRUCTURE OFOPTIMAL POLICIES
A. Monotonicity and Convexity of Value Functions
WhenN < ∞, let V N(p) denote the maximum achievable discounted reward function in Definition 1,
V N(p) satisfies the following Bellman equation ( [11]):
V N(p) = maxτ∈[τmin,τmax]
{
−ci+αV N−1(L1(p)), G(p, τ)+α∑
i=2,3,4
Qi(p, τ)V N−1(Li(p, τ))}
, N > 1, (12)
9
where
V 1(p) = maxτ∈[τmin,τmax]
{
− ci, G(p, τ)}
. (13)
When N = ∞, let V (p) denote the maximum achievable discounted reward function in Definition 1,
V (p) satisfies the following Bellman equation ( [11]):
V (p) = maxτ∈[τmin,τmax]
{
− ci + αV (L1(p)), G(p, τ) + α∑
i=2,3,4
Qi(p, τ)V (Li(p, τ))}
. (14)
Note that in (14),G(p, τ) is the immediate gain obtained by sensing for durationτ while the expected
discounted future gain given this sensing duration isα∑
i=2,3,4 Qi(p, τ)V (Li(p, τ)). As can be seen, both
immediate and future gains are dependent onτ .
It can be shown ( [11]) thatlimn→∞ V n(p) = V (p). It can also be verified thatV N (p) and V (p) are
continuous inp. Let us now prove some important structural results forV N (p) andV (p).
Proposition1: V N(p) and V (p) are nondecreasing in p.
Proof: First, let us prove the property forV N(p), N < ∞. The proof proceeds by induction. As
G(p, τ) is increasing inp, from (13), it follows thatV 1(p) is nondecreasing inp. Now assuming that
V n(p) is nondecreasing inp for some value ofn ≥ 1, we have
V n+1(p) = maxτ∈[τmin,τmax]
{
− ci + αV n(L1(p)), G(p, τ) + α∑
i=2,3,4
Qi(p, τ)V n(Li(p, τ))}
. (15)
Let E1 = −ci + αV n(L1(p)), andE2 = G(p, τ) + α∑
i=2,3,4 Qi(p, τ)V n(Li(p, τ)). As L1(p) andV n(p)
are both nondecreasing inp, it follows that E1 is nondecreasing inp. The first term inE2, i.e., G(p, τ),
is nondecreasing inp. For the second term inE2, letting 0 < q < p, we have:∑
i=2,3,4
Qi(p, τ)V n(Li(p, τ)) −∑
i=2,3,4
Qi(q, τ)V n(Li(q, τ))
>∑
i=2,3,4
(Qi(p, τ) − Qi(q, τ))V n(Li(p, τ)), asL2(p, τ) > L2(q, τ),
= V n(L3) (Q3(p, τ) − Q3(q, τ)) + V n(L2(p, τ)) (Q2(p, τ) − Q2(q, τ))
+ V n(L4) (Q3(q, τ) + Q2(q, τ) − Q3(p, τ) − Q2(p, τ)) , as∑
i=2,3,4
Qi(p, τ) =∑
i=2,3,4
Qi(q, τ) = 1,
=(
V n(L3) − V n(L2(p, τ)))(
Q3(p, τ) − Q3(q, τ))
+(
V n(L2(p)) − V n(L4))(
(Q3(p, τ) + Q2(p, τ)) − (Q3(q, τ) + Q2(q, τ)))
≥ 0,
(16)
where the last inequality is due toQ3(p, τ) = p(1 − Pfa(τ)) is increasing inp and
Q3(p, τ) + Q2(p, τ) = p(1 − Pd) + Pd (17)
10
is also increasing inp. So the second term inE2 is increasing inp, which implies thatE2 is also increasing
in p. As bothE1 andE2 are nondecreasing inp, it follows thatV n+1(p) is nondecreasing inp.
As limn→∞ V n(p) = V (p), it follows thatV (p) is also nondecreasing inp.
Proposition2: V N(p) and V (p) are convex in p.
Please refer to the Appendix A for the proof.
Remark1: Proposition 1 states intuitively that, the higher the probability p that the primary network is
idle at the beginning of the control process, the higher the maximum achievable expected rewardV N(p)
andV (p). Proposition 2 then indicates how fastV N(p) andV (p) increase inp. As V N(p) andV (p) are
convex,V N (p) andV (p) increase at least linearly inp.
B. Properties of Optimal Policies
Let us explore some useful structural properties of the optimal control policies. Letting
G∗(p) = maxτ∈[τmin,τmax]
G(p, τ), (18)
asG(p, τ) is increasing inp, so isG∗(p). We state the following property of the optimal control policies.
Proposition3: Let p∗ be the minimum value of p such that G∗(p∗) > −ci. If, in a particular time slot,
the probability of the primary network being idle is p such that p ≥ p∗, then an optimal policy must carry
out spectrum sensing in that time slot.
Proof: Let us prove for the case whenN < ∞. We haveV 1(p) = maxτ∈[τmin,τmax]
{
−ci, G(p, τ)}
=
max{−ci, G∗(p)}, whereG∗(p) ≥ G∗(p∗) > −ci, therefore, spectrum sensing should be carried out when
N = 1. For N > 1 we have
V N(p) = maxτ>0
{
− ci + αV N−1(L1(p)), G(p, τ) + α∑
i=2,3,4
Qi(p, τ)V N−1(Li(p, τ))}
. (19)
Notice that∑
i=2,3,4 Qi(p, τ) = 1 and∑
i=2,3,4 Qi(p, τ)Li(p, τ) = L1(p). Then, asV N−1(p) is convex in
p, it follows that
V N−1(L1(p)) ≤∑
i=2,3,4
Qi(p, τ)V N−1(Li(p, τ)), ∀ τmin ≤ τ ≤ τmax. (20)
This, together with the fact thatG∗(p) ≥ G∗(p∗) > −ci, implies the system should carry out spectrum
sensing in the current time slot. The proof forN = ∞ is similar.
Remark2: Proposition 3 gives the sufficient condition on the value ofp, i.e., p ≥ p∗, for carrying out
spectrum sensing in a particular time slot. However, this may not be a necessary condition. In general,
11
the optimal control policies for our POMDP model may not possess the threshold-based structure (in
the value ofp). To the best of our knowledge, there have been a limited number of works that prove
the threshold-based characteristic of the optimal controlpolicies for some specific POMDP models (see
[12]–[14]). Unfortunately, our problem does not directly fit into these models.
A natural question to ask is, given that sensing is carried out, how the optimal sensing timeτ would
vary with p. Let
F N(p, τ) = α∑
i=2,3,4
Qi(p, τ)V N−1(Li(p, τ)), N > 1, τmin ≤ τ ≤ τmax, (21)
be the expected discounted future reward if the probabilitythat the primary network being active in a
particular time slot isp and sensing is carried out for a durationτ . Similarly, for the case of infinite
control horizon, define
F (p, τ) = α∑
i=2,3,4
Qi(p, τ)V (Li(p, τ)), τmin ≤ τ ≤ τmax. (22)
The following result highlights the effect of increasing the spectrum sensing timeτ .
Proposition4: F N(p, τ) and F (p, τ ) are nondecreasing in τ .
Please refer to the Appendix B for the proof.
Remark3: Essentially, Proposition 4 makes concrete the intuition that the more sensing being carried
out in the current time slot, the better the expected reward in the future time slots. This is because
increasing the sensing time gives the secondary user more accurate knowledge of the state of the primary
network, which in turn improves future control.
To further study the effect of varying the sensing time, we need to make the following assumptions.
• A1: The probability of false alarm, i.e.,Pfa(τ), is convex and decreasing inτ , τmin ≤ τ ≤ τmax.
• A2: The energy cost of sensing, i.e.,cs(τ), is convex and increasing inτ .
For the justifications of assumptions A1 and A2, please referto the Appendix E.
Lemma1: Given assumptions A1 and A2, function G(p, τ) is concave in τ , τmin ≤ τ ≤ τmax.
Please refer to the Appendix C for the proof.
As the functionG(p, τ) is concave, continuous, and has a strictly decreasing first order derivative for
all τ in the interval[τmin, τmax], there exists an unique maximum point for this function. Letting
τ ∗(p) = arg maxτmin≤τ≤τmax
G(p, τ), (23)
12
the following proposition relates the optimal sensing timeto the value ofτ ∗(p) defined in (23).
Proposition5: Given assumptions A1 and A2, if at the beginning of a particular time slot, the prob-
ability of the primary network being idle is p and sensing is carried out, then the optimal sensing time
τ opt is greater than or equal to τ ∗(p).
Proof: We prove forN = ∞, the case whenN < ∞ is similar. The proof is by contradiction.
Suppose the optimal sensing timeτ opt < τ ∗(p). Due to the concavity ofG(p, τ) in τ we haveG(p, τ opt) <
G(p, τ ∗(p)). Furthermore, from Proposition 4, we haveF (p, τ opt) ≤ F (p, τ ∗(p)), therefore
G(p, τ opt) + F (p, τ opt) < G(p, τ ∗(p)) + F (p, τ ∗(p)), (24)
which contradicts to the fact thatτ opt is the optimal sensing time givenp. This completes the proof.
Proposition 5 gives the lower bound for the optimal sensing time τ opt. To see how this lower bound
varies withp, consider the first order derivative ofG(p, τ) with respect toτ :
∂G
∂τ= p
(
(rt − ct)(
Pfa(τ) − (T − τ)P ′fa(τ) − 1
)
+ (1 − Pd)ct
)
−(
c′s(τ) − (1 − Pd)ct
)
= pu(τ) − v(τ),
(25)
whereu(τ) = (rt − ct)(
Pfa(τ)− (T − τ)P ′fa(τ)−1
)
+(1−Pd)ct andv(τ) = c′s(τ)− (1−Pd)ct. We have
argued in the proof of Lemma 1 thatu(τ) is strictly decreasing inτ . Furthermore, ascs(τ) is convex,
v(τ) is nondecreasing inτ . The following lemma characterizes howτ ∗(p) varies inp.
Lemma2: Given assumptions A1 and A2, we have the following cases.
• Case 1: v(τmin) ≥ 0 then τ ∗(p) is nondecreasing in p.
• Case 2: v(τmax) < 0 then τ ∗(p) is nonincreasing in p.
• Case 3: v(τmin) < 0 ≤ v(τmax). Letting v(τ v) = 0, τmin < τ v ≤ τmax,
a) if u(τ v) > 0 then τ ∗(p) is nondecreasing in p.
b) if u(τ v) ≤ 0 then τ ∗(p) is nonincreasing in p.
Please refer to the Appendix D for the proof.
Remark4: Proposition 5 states that when the probability of the primary network being idle isp and
sensing is carried out, then the lower bound of the optimal sensing time isτ ∗(p). Lemma 2 further shows
that the lower boundτ ∗(p) is always monotonic inp.
13
Remark5: If the energy costs of sensing and transmission are ignorable, then the gain of sensing for
τ unit of time when the probability of primary network being idle is p can be simplified to:
G(p, τ) = p(
1 − Pfa(τ))
(T − τ)rt. (26)
Then the value ofτ = τ that maximizesG(p, τ) does not depends onp. Therefore, it can be verified
that the policy that carries out sensing forτ unit of time for every time slot maximizes the expected
reward. Our optimization problem is then equivalent to the problem considered in [2], [3], which focus
on maximizing throughput within each time slot.
IV. HEURISTIC POLICIES
Directly solving the POMDP described in Section III can be computational challenging. In this section,
suboptimal control policies that can be obtained at lower complexity are discussed.
A. Grid-based Approximation
Grid-based approximation is a widely-used approach for approximating solutions to POMDPs. In this
approach, the value function is approximated at a finite number of belief points on a grid. The value
function at belief points not belonging to the grid is evaluated using interpolation. In this paper, we
employed the fixed-resolution, regular-grid approach proposed by Lovejoy [15]. Applying to our POMDP,
the range of belief state is first divided intoP equally spaced pointsp0, p1, . . . pP−1. Then, the value
function at these grid points are calculated using the following iteration.
v(pj) = maxτ∈[τmin,τmax]
{
− ci + αV (L1(pj)), G(p, τ) + α
∑
i=2,3,4
Qi(p, τ)V (Li(pj, τ))
}
, (27)
where, for each value ofp, we find j such thatpj ≤ p ≤ pj+1 and calculate
V (p) =p − pj
pj+1 − pjv(pj+1) +
pj+1 − p
pj+1 − pjv(pj). (28)
As pointed out in [15], the iteration in (27) is guaranteed toconverge and the valueV (p) in (28) is an
upper-bound of the optimal value functionV (p) (asV (p) is convex). After obtaining the approximation
function V (p), we can substitute into the Bellman equation (14) to obtain the corresponding sensing time.
B. Myopic Policy ζm(.)
Proposition 3 identifies the sufficient condition on the probability that the primary network is idle for
sensing to be carried out. Furthermore, Proposition 5 givesthe lower bound on the optimal sensing time,
given that sensing is carried out. Based on these two propositions, we consider the following policy:
τn = ζm(pn) =
0, if pn < p∗
τ ∗(pn), if pn ≥ p∗,(29)
14
where ζm(.) maps the belief statep into the sensing time for the secondary user. Note that setting the
sensing timeτ = ζm(p) myopically maximizes the instantaneous gain when the belief state isp.
C. Static Policy ζs(.)
Consider a static spectrum access policy that always carries out sensing with a fixed sensing duration.
Note that the stationary probability of the primary networkbeing idle isπi = bb+g
. We can calculate
τ ∗ (πi) = arg maxτmin≤τ≤τmax
G (πi, τ). (30)
Then, for every time slot, sensing is carried out for the duration of τ ∗(πi). We term this policyζs(.).
D. Genie’s Policy
To facilitate comparison, we are going to normalize the performance of different control policies to
that of the following so called ‘Genie’s policy’. Suppose that at the beginning of each time slot, a genie
tells the secondary user exactly what the state of the primary network is. Then, the best action for the
secondary user is to carry out transmission if the primary network is idle, and to stay idle if the primary
network is active. The expected discounted reward for the secondary user can be calculated as:
V genie =∞
∑
n=0
αn
(
g
b + gT (rt − ct) −
b
b + gci
)
=gT (rt − ct) − bci
(1 − α)(b + g). (31)
V. NUMERICAL RESULTS AND DISCUSSION
In this section, we present numerical results that illustrate our theoretic analysis. We also compare the
performance of the optimal policies to that of the suboptimal policies described in Section IV. We focus
on the infinite horizon control scenario, i.e., whenN = ∞.
A. Model for Numerical Studies
We assume that the primary network operates on a channel withbandwidth ofB = 6 MHz. The time
slot duration isT = 10 msecs. The secondary user employs energy detection for spectrum sensing, with
channel being oversampled at ratefs = 7/8B. Given the required probability of detectionPd, the SNR
of the active primary signal at the receiver of the secondaryuser beingγ, and the sensing time ofτ , the
probability of false alarm can be calculated by ( [2]):
Pfa(τ) = Q(√
2γ + 1Q−1(Pd) +√
τfsγ), (32)
15
whereQ(.) is the complementary distribution function of a standard Gaussian variable. We setγ = −15
dB andPd = 10%. We assume that the sensing cost is linear in sensing time, i.e., cs(τ) = sτ , s > 0. This
satisfies the assumption thatcs(τ) is convex and increasing inτ . The memory of the primary network
switching process, i.e.,µ = 1 − b − g, is varied from0.1 to 0.95. While doing so, we always set
b = g = (1 − µ)/2.
B. Characteristics of Optimal Policies
To obtain optimal solutions to our POMDP, we use the solver package provided by A. Cassandra [16].
1) Optimal reward function is convex and nondecreasing: In Fig. 2, we plot the optimal reward function
V (p) versus the initial belief statep. As proved in Propositions 1 and 2,V (p) is convex and nondecreasing
in p. As can be seen, for high value ofp, V (p) is close to a linear function. This can be explained by the
fact that the optimal control action does not vary much for high value ofp (refer to Figs. 5 and 6).
2) Convexity of Pfa(τ) and concavity of G(p, τ): In Figs. 3 and 4, we respectively plot the probability
of false alarmPfa(τ) and the instantaneous gainG(p, τ) as functions of sensing timeτ when other
parameters are fixed. As shown in Appendix E, for energy detection, Pfa(τ) is convex and decreasing
in the region where0 < Pfa(τ) ≤ 0.5. Also, from Lemma 1, whenPfa(τ) is convex and decreasing
while cs(τ) is convex and increasing, the instantaneous gainG(p, τ) is concave inτ . These effects are
illustrated in Figs. 3 and 4.
3) Optimal sensing time τ opt(p): In Figs. 5 to 8, given different system parameters, we plot the sensing
time versus the probability of the primary network being idle for different control policies, i.e., optimal,
grid-based approximation, myopic, and static. From these plots, it can be observed that there is a threshold
value for the belief statep above which it is optimal for the secondary user to carry out spectrum sensing
and below which it is optimal to stay idle. In other words, theoptimal sensing strategies tend to exhibit
the ‘threshold-based’ characteristic. Note that this conjecture is stronger than Proposition 3, where it is
shown that there is a minimum value ofp above which sensing should be carried out.
From Fig. 5, it can also be observed that for a relatively low value of memory of the primary network
switching process (µ = 0.5), both the grid-based and myopic policies approximate the optimal sensing
durations well. However, in Fig. 6, when the memory increases (µ = 0.9), only the grid-based policy is
close to optimal, while the myopic policy is significantly different from the optimal policy. This difference
can be explained by the fact that the myopic policy only focuses on instantaneous gain and ignores future
effects. As a result, it is not able to exploit the memory (correlation) in the primary network states. We note
16
also that the grid-based policy plotted in Figs. 5 and 6 is obtained using only10 discretized points. This
shows that, for our POMDP control problem, small-sized grid-based approximations are good enough.
Comparing Fig. 5 and 6, it is evident that when the memory of ofthe primary network switching
process increases fromµ = 0.5 to µ = 0.9, the optimal sensing time also increases. This is because the
higher the memory, the slower the change in status of the primary network and therefore, the more useful
the sensing activity for predicting the future state of the primary network. In Fig. 7, the cost of staying
idle in a time slot is increased toci = 1000. As can be seen, this discourages the secondary user from
staying idle, i.e., sensing is carried out given smaller value of probability that primary network is idle.
In Figs. 5 to 7, it can be observed thatτ ∗(p) is nondecreasing inp. This is because by settings = 1
and ct = 2, we havev(τ) = c′s(τ) − (1 − Pd)ct = 0.8 > 0, therefore, Case 1 in Lemma 2 applies. On
the other hand, in Fig. 8, it can be observed thatτ ∗(p) is nonincreasing inp. This is because by setting
s = 1 andct = 50, we havev(τ) = c′s(τ)− (1− Pd)ct = −4 < 0, therefore, Case 2 in Lemma 2 applies.
C. Performance Comparison
To facilitate comparison, we normalize the performance of different control policies to that of the
‘Genie’s policy’ described in Section IV-D.
1) Rewards: In Fig. 9, we plot the normalized expected reward versus the memoryµ of the primary
network switching process for optimal, grid-based, myopic, and static policies. As can be seen, the
performance of the optimal, grid-based and myopic polices all improves with increase in memoryµ.
This is because the higher the memory, the slower the primarynetwork switches state, which implies the
more useful sensing activities for future control. On the other hand, the performance of the static policy
(ζs(.)) does not change with memoryµ. This is because whenµ is varied, we always keepsb = g, which
makes the stationary probability of the primary network being active or idle unchanged (πi = 0.5). This
means that when the memory changes, in every time slot,ζs(.) always carries out sensing for a fixed
duration. Therefore, the performance ofζs(.) does not change with increase in memory.
In Fig. 9, it can be observed that for the low range of memory (µ < 0.7), the performance of both
grid-based and myopic policies are close to that of the optimal policy. On the other hand, for higher
value of memory, i.e., when primary network switches state more slowly, only grid-based policy can
approximate the performance of the optimal policy. The explanation for this is the same as what we have
given for the difference in the sensing time in Figs. 5 and 6. In particular, for low memory, both myopic
17
and grid-based policies can approximate the optimal decisions well. However, for high value of memory,
only grid-based policy has decisions close to optimal.
In Fig. 9, it is evident that the performance of a10-point grid-based approximation is almost the
same as that of the optimal policy. The performance of a5-point grid-based approximation is also very
close to optimal. This shows that we can employ the grid-based approximation approach without loss of
performance in our POMDP problem.
In Fig. 10, we setci = 1000 to examine the effect of introducing the cost of staying idle. As can be
seen, the performance trends are similar to those in Fig. 9. The extra effect that can be observed is that
the performance of myopic policy (ζm(.)) is closer to that of the optimal and grid-based policies in this
scenario. This can be explained by referring back to Fig. 7, where the sensing time of the myopic policy
is quite close to that of the optimal and grid-based policies.
VI. CONCLUSION
In this paper, we study spectrum-sensing policies that takeinto account the dynamics of the primary
networks and determine the spectrum sensing duration in order to optimize secondary users’ performance.
As such, this paper bridges the gap between the two groups of existing spectrum-sensing/control literature
that either focus on adapting to the dynamics of the primary networks or on optimizing the spectrum-
sensing duration, but not on both. We present detailed theoretical analysis and numerical studies for the
optimization problem.
We are currently extending this work in two directions. One is to study different trade-offs when some
constraints in this paper are relaxed, e.g., instead of enforcing a fixed probability of detection, a time-
average interference constraint can be introduced to protect primary networks’ operation. Another direction
is to extend this work to multi-user scenarios and consider cooperative and/or distributed spectrum-sensing
policies.
APPENDIX
A. Proof of Proposition 2
Proof: We proceed by induction. From (13),V 1(p) = maxτ∈[τmin,τmax]
{
− ci, G(p, τ)}
, where
G(p, τ) is linear inp. As maximum of convex functions is a convex function,V 1(p) is convex inp. Now,
assume thatV n(p) is convex inp for some valuen ≥ 1. From (12), if we can prove thatV n(L1(p)) and
Qi(p, τ)V n(Li(p, τ)), i = 2, 3, 4 are all convex. Then it follows thatV n+1(p) is convex.
18
As L1(p) is linear,L1(p) is convex. By assumption,V n(p) is convex and from Proposition 1,V n(p) is
nondecreasing. It can be shown that ( [17]), ifh(x) andg(x) are two functions such thath(x) is convex
and nondecreasing andg(x) is convex, then the compositionf(x) = h(g(x)) is also convex. Applying
here, it follows thatV n(L1(p)) is convex.
We haveQ3(p, τ)V n(L3(p, τ)) = p(1 − Pfa)(τ)V n(1 − b), i.e. linear and convex inp. Similarly,
Q4(p, τ)V n(L4(p, τ)) = (1 − p)(1 − Pd)Vn(g) is linear and convex inp.
To check ifQ2(p, τ)V n(L2(p, τ)) is convex inp, we look at its second order derivative with respect to
p. It can be verified that
∂2(Q2(p, τ)V n(L2(p, τ)))
∂p2=
V n′′(L2(p, τ))(1 − b − g)2P 2d P 2
fa(τ)
[Pd − p(Pd − Pfa(τ))]3(33)
We have shown earlier thatL2(p, τ) is convex and increasing inp, this, coupled with the fact thatV n(p) is
convex and nondecreasing, leads toV n(L2(p, τ)) is convex and nondecreasing. Therefore,V n′′(L2(p, τ)) ≥
0. It then follows from (33) that the second order derivative of Q2(p, τ)V n(L2(p, τ)), with respect top,
is nonnegative, which implies thatQ2(p, τ)V n(L2(p, τ)) is convex.
We have proved thatV n(L1(p)) and Qi(p, τ)V n(Li(p, τ)), i = 2, 3, 4 are all convex inp, therefore,
V n+1(p) is convex.
As limn→∞ V n(p) = V (p), it follows thatV (p) is also convex inp. This completes the proof.
B. Proof of Proposition 4
Proof: We prove the property ofF (p, τ), the proof forF N(p, τ) is similar. Letτmin ≤ τ1 < τ2 ≤ τmax.
It can be verified that
(F (p, τ2) − F (p, τ1))/α =(Q3(p, τ2) − Q3(p, τ1))V (L3) + Q2(p, τ2)V (L(p, τ2))
− Q2(p, τ1)V (L2(p, τ1)).(34)
From (5) and the fact thatPfa(τ) is decreasing inτ , we have:
Q3(p, τ2) − Q3(p, τ1) > 0 and Q2(p, τ2) > 0, and
Q3(p, τ2) − Q3(p, τ1) + Q2(p, τ2) = Q2(p, τ1).(35)
Furthermore
(Q3(p, τ2) − Q3(p, τ1))L3 + Q2(p, τ2)L2(p, τ2) = (pPfa(τ1) − pPfa(τ2))(1 − b)
+ (pPfa(τ2) + (1 − p)Pd)pPfa(τ2)(1 − b) + (1 − p)Pdg
pPfa(τ2) + (1 − p)Pd
= pPfa(τ1)(1 − b) + (1 − p)Pdg = Q2(p, τ1)L2(p, τ1).
(36)
19
From (35) and (36) and the fact thatV (p) is convex inp, it follows thatF (p, τ2)−F (p, τ1) is non-negative.
This completes the prove.
C. Proof of Lemma 1
Proof: We have:
G(p, τ) = p(1 − Pfa(τ))(T − τ)(rt − ct) − (1 − p)(1 − Pd)(T − τ)ct − cs(τ). (37)
As cs(τ) is convex,−cs(τ) is concave. Also, the second term in (37) is linear inτ . So all we have to
prove is that the first term in (37) is concave inτ . Let w(p, τ) = p(1 − Pfa(τ))(T − τ)(rt − ct).
∂w
∂τ= p(rt − ct)
(
Pfa(τ) − (T − τ)P ′fa(τ) − 1
)
, (38)
whereP ′fa(τ) is the first order derivative ofPfa(τ). From assumption A1, it follows thatP ′
fa(τ) is negative
and nondecreasing inτ . Furthermore,T − τ is positive and decreasing inτ . That leads to(T − τ)P ′fa(τ)
being negative and nondecreasing inτ . This, together with the fact thatPfa(τ) is decreasing inτ and
rt > ct, implies that∂w/∂τ is decreasing inτ . Therefore,w(p, τ) is concave inτ and so isG(p, τ).
D. Proof of Lemma 2
Proof: The proof is straightforward and based mainly on the fact that u(τ) is decreasing inτ while
v(τ) is nondecreasing inτ . Due to limited space, we only give the proof for Case 3a.
Given thatv(τmin) < 0 ≤ v(τmax), v(τ v) = 0, u(τ v) > 0, and letting0 < q < p < 1, we need to prove
that τ ∗(p) ≥ τ ∗(q). There are the following scenarios to consider:
Scenario 1: τ ∗(q) = τmin. It immediately follows thatτ ∗(p) ≥ τ ∗(q).
Scenario 2: τ ∗(p) = τmax. It immediately follows thatτ ∗(p) ≥ τ ∗(q).
Scenario 3: τ ∗(q) = τmax. This implies thatqu(τmax) − v(τmax) ≥ 0. Furthermore, asv(τmax) ≥ 0 by
assumption, we must haveu(τmax) ≥ 0. This leads to:
pu(τmax) − v(τmax) ≥ qu(τmax) − v(τmax) ≥ 0, (39)
which implies thatτ ∗(p) = τmax.
Scenario 4: τ ∗(p) = τmin. This implies thatpu(τmin)− v(τmin) < 0. Furthermore, from the assumption
that u(τ v) > 0 and the fact thatu(τ) is decreasing inτ , we must haveu(τmin) > 0. Therefore
qu(τmax) − v(τmax) < pu(τmax) − v(τmax) < 0, (40)
20
which implies thatτ ∗(q) = τmin.
Scenario 5: τmin < τ ∗(p), τ ∗(q) < τmax. We then have
pu(τ ∗(p)) − v(τ ∗(p)) = qu(τ ∗(q)) − v(τ ∗(q)) = 0. (41)
If τ ∗(p) ≤ τ v then from the assumption thatu(τ v) > 0 = v(τ v) and the fact thatu(.) is decreasing while
v(.) is nondecreasing, we havepu(τ ∗(p)) − v(τ ∗(p)) > 0, contradicts to (41). Therefore,τ ∗(p) > τ v.
Similarly, τ ∗(q) > τ v. Then from (41) we have:
u(τ ∗(p)) =v(τ ∗(p))
p<
v(τ ∗(q))
q= u(τ ∗(q)), (42)
which implies thatτ ∗(p) > τ ∗(q).
Combining the arguments in Scenarios 1 - 5, completes the proof for Case 3a. The proof for other
cases is similar.
E. Convexity of Pfa(τ) and cs(τ)
Let us justify the assumption thatPfa(τ) is convex and decreasing andcs(τ) is convex and increasing
for the case of spectrum sensing based on energy detection.
1) Convexity of Pfa(τ): Assume that the primary signal is complex-valued PSK and noise is circular
symmetric complex Gaussian. Given required probability ofdetectionPd and the SNR of primary signal
at the secondary receiver ofγ, the probability of false alarm can be calculated by ( [2]):
Pfa(τ) = Q(√
2γ + 1Q−1(Pd) +√
τfsγ). (43)
Differentiating with respect toτ gives:
P ′fa(τ) =
dPfa
dτ= − γ
√fs
2√
2πτ−1/2 exp
(
− (√
2γ + 1Q−1(Pd) + γ√
τfs)2/2
)
. (44)
For τ > 0, it is clear thatP ′fa(τ) < 0 soPfa(τ) is decreasing inτ . Furthermore, whenPfa(τ) ≤ 0.5, from
(43) we have√
2γ + 1Q−1(δ)+ γ√
τfs ≥ 0. This, together with (44), imply thatP ′fa(τ) is monotonically
increasing inτ whenPfa(τ) ≤ 0.5, i.e., Pfa(τ) is convex inτ whenPfa(τ) ≤ 0.5.
2) Convexity of cs(τ): Energy detection comprises of three main steps, i.e., i) to take K samples of
the signal; ii) to calculate the average power of theK samples; and iii) to compare the average power to
a certain threshold. The complexity of the first and second steps is linear inK. Furthermore,K is linear
in τ , so cs(τ) can be assumed linear inτ when spectrum sensing is based on energy detection.
21
REFERENCES
[1] FCC, “Spectrum policy task force report, FCC 02-155.” Nov. 2002.
[2] Y. C. Liang, Y. H. Zeng, E. Peh, and A. T. Hoang, “Sensing-throughput tradeoff for cognitive radio networks,” inProc. of IEEE
ICC’07, Glasgow, Jun. 2007.
[3] A. Ghasemi and E. Sousa, “Optimization of spectrum sensing for opportunistic spectrum access in cognitive radio networks,” in Proc.
of 4th IEEE Consumer Communications and Networking Conference (CCNC), Las Vegas, USA, Jan. 2007.
[4] Y. Pei, A. T. Hoang, and Y.-C. Liang, “Sensing-throughput tradeoff in cognitive radio networks: How frequently should spectrum sensing
be carried out?” inProc. of 18th Annual IEEE International Symposium on Personal, Indoor and Mobile Radio Communications, Athens,
Greece, Sep. 2007.
[5] Q. Zhao, L. Tong, A. Swami, and Y. Chen, “Decentralized cognitive mac for opportunistic spectrum access in ad hoc networks:
A pomdp framework,”IEEE Journal on Selected Areas in Communications (JSAC): Special Issue on Adaptive, Spectrum Agile and
Cognitive Wireless Networks, vol. 25, no. 3, pp. 589–600, Apr. 2007.
[6] A. T. Hoang and Y.-C. Liang, “Adaptive scheduling of spectrum sensing periods in cognitive radio networks,” into appear in Proc. of
50th IEEE Global Telecommunications Conference (Globecom), Washington DC, USA, Nov. 2007.
[7] D. Zhang and K. Wasserman, “Transmission schemes for time-varying wireless channels with partial state observations,” in Proceedings
of IEEE INFOCOM’02, New York, Jun. 2002, pp. 467–476.
[8] L. A. Johnston and V. Krishnamurthy, “Opportunistic filetransfer over a fading channel: A pomdp search theory formulation with
optimal threshold policies,”IEEE Transactions on Wireless Communications, vol. 5, no. 2, pp. 394–405, Feb. 2006.
[9] M. Mushkin and I. Bar-David, “Capacity and coding for thegilbert-elliott channels,”IEEE Transactions on Information Theory, vol. 35,
no. 6, pp. 1277–1290, Nov. 1989.
[10] IEEE 802.22 Wireless RAN, “Functional requirements for the 802.22 WRAN standard, IEEE 802.22- 05/0007r46,” Oct. 2005.
[11] D. P. Bertsekas,Dynamic Programming and Optimal Control: 2nd edition, vols. 1 and 2. Athena Scientific, 2001.
[12] S. C. Albright, “Structural results for partially observable markov decision processes,”Operations Research, vol. 27, no. 5, pp. 1041–
1053, Sep.-Oct. 1979.
[13] W. S. Lovejoy, “Some monotonicity results for partially observable markov decision processes,”Operations Research, vol. 35, no. 5,
pp. 736–743, Sep.-Oct. 1987.
[14] I. MacPhee and B. Jordan, “Optimal search for a moving target,” Probability in Engineering and Information Sciences, vol. 9, pp.
159–182, Sep.-Oct. 1995.
[15] W. Lovejoy, “Computationally feasible bounds for partially observed markov decision processes,”Operations Research, vol. 39, no. 1,
pp. 162–175, Jan./Feb 1991.
[16] A. R. Cassandra, “Tony’s pomdp page,”Website http://www.cs.brown.edu/research/ai/pomdp/.
[17] S. Boyd and L. Vandenberghe,Convex Optimization, 1st ed. Cambridge University Press, 2004.
22
PU
Slots
SU
idle idle active idle
s trans. ack s quiet s quiet s nak trans. idle
1 2 3 4 5
active
Fig. 1. Operations of a primary network and secondary user. The primary network switches between active and idle according to a Markovianprocess. The secondary user must carry out spectrum sensingbefore transmitting in each time slot. Sensing can introduces false alarms (e.g.in slot 2) and mis-detections (e.g. in slot5). Positive acknowledgments (ACK) and negative acknowledgments (NAK) are returned forsuccessful and failed transmissions respectively.
0 0.2 0.4 0.6 0.8 10.2
0.25
0.3
0.35
0.4
0.45
0.5
0.55
0.6
0.65
p (prob. of primary network being idle)
Nor
mal
ized
V(p
)
Fig. 2. Normalized optimal reward function. Optimal rewardfunction V (p) is convex and increasing inp. Pd = 10%, ci = 0, s = 1,ct = 2, rt = 3, andµ = 0.9. The optimal reward function is normalized by the reward of Genie’s Policy.
23
0.05 0.1 0.15 0.2 0.25 0.30
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
τ/T
Pfa
(τ)
Convexity of Pfa
(τ)
Fig. 3. Probability of false alarm for energy detection. Given that energy detection is used for spectrum sensing, for the range ofτ inwhich 0 ≤ Pfa(τ ) ≤ 0.5, Pfa(τ ) is convex and decreasing inτ . Parameters: Channel bandwidth =6 MHz, oversampling ratio =8/7, SNR= −15 dB. Pd = 10%, s = 1, ct = 2, rt = 3.
0.05 0.1 0.15 0.2 0.25 0.3−0.04
−0.02
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
τ/T
G(0
.5, τ
)/T
Concavity of G(0.5, τ)
Fig. 4. Instantaneous gain as a function of sensing time. Forthe range ofτ in which Pfa(τ ) is convex and decreasing,G(p, τ ) is concavein τ . Parameters: Channel bandwidth =6 MHz, oversampling ratio =8/7, SNR =−15 dB. Pd = 10%, s = 1, ct = 2, rt = 3.
24
0 0.2 0.4 0.6 0.8 10
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
p (prob. of primary network being idle)
τ/T
OptimalGrid
ζm(.)
ζs(.)
µ=0.5
Fig. 5. Sensing times for different policies when the memoryof primary user switching process is set toµ = 0.5. Pd = 10%, ci = 0,s = 1, ct = 2, rt = 3.
0 0.2 0.4 0.6 0.8 10
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
p (prob. of primary network being idle)
τ/T
OptimalGrid
ζm(.)
ζs(.)
µ=0.9
Fig. 6. Sensing times for different policies when the memoryof primary user switching process is set toµ = 0.9. Pd = 10%, ci = 0,s = 1, ct = 2, rt = 3.
25
0 0.2 0.4 0.6 0.8 10
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
p (prob. of primary network being idle)
τ/T
OptimalGrid
ζm(.)
ζs(.)
µ=0.9
Fig. 7. Effect of delay cost to sensing time. Sensing times for different policies when the memory of primary user switching process is setto µ = 0.9, Pd = 10%, ci = 1000, s = 1, ct = 2, rt = 3.
0 0.2 0.4 0.6 0.8 10
0.05
0.1
0.15
0.2
0.25
0.3
0.35
p (prob. of primary network being idle)
τ/T
Optimal
ζm(.)
ζs(.)
µ=0.9
Fig. 8. A scenario whenτ∗(p) (sensing time of myopic policyζm(.)) is non-increasing inp when sensing is carried out.µ = 0.9,Pd = 10%, ci = 0, s = 1, ct = 50, rt = 80.
26
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90.24
0.26
0.28
0.3
0.32
0.34
0.36
0.38
0.4
0.42
0.44
µ (memory of primary network switching process)
Nor
mal
ized
rew
ard
OptimalGrid (10 points)Grid (5 ponts)
ζm(.)
ζs(.)
10−point grid approximation
5−point grid approximation
Fig. 9. Normalized rewards achieved by different control policies. The rewards achieved by different policies are normalized by that of theGenie’s Policy.ci = 0, cs = 1, ct = 2, rt = 3.
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90.26
0.28
0.3
0.32
0.34
0.36
0.38
0.4
µ (memory of primary network switching process)
Nor
mal
ized
rew
ard
OptimalGrid (20 points)Grid (10 points)
ζm(.)
ζs(.)10−point grid approximation
20−point grid approximation
Fig. 10. Effect of increasing delay cost to rewards achievedby different control policies. The rewards achieved by different policies arenormalized by that of the Genie’s Policy.ci = 1000, cs = 1, ct = 2, rt = 3.
Top Related