Sampling irregularity perturbs visual reconstruction

628 J. Opt. Soc. Am. A/Vol. 5, No. 4/April 1988

Sampling irregularity perturbs visual reconstruction

Gote Nyman

Department of General Psychology, University of Helsinki, Ritarikatu 5, SF-00170 Helsinki 17, Finland

Mark W. Greenlee

Neurologische Universitdtsklinik, Hansastrasse 9, 7800 Freiburg, West Germany

Pentti Laurinen

Department of General Psychology, University of Helsinki, Ritarikatu 5, SF-00170 Helsinki 17, Finland

Received August 12, 1987; accepted December 10, 1987

We explored the human observer's ability to detect and discriminate sine-wave and square-wave gratings that weresampled at intervals varying from 4.7 to 9.4 arcmin. To study the effect of sampling irregularity on visual

performance, we varied the position of each line sample on the basis of a Gaussian probability distribution, thestandard deviation of which varied from 0 (regular sampling) to 4.7 arcmin (highly irregular sampling). The resultsindicate that irregular sampling has no systematic effect on the observer's ability merely to detect the presence of asine- or square-wave grating. In contrast, sampling irregularity strongly impairs the subject's ability to discrimi-nate between these waveforms. A model based on the convolution of difference-of-Gaussians-type weightingprofiles predicted that sampling irregularity should have little to no effect on the output of a channel tuned to thethird harmonic of the square-wave grating. The findings thus suggest the existence of a sampling scheme in thevisual system. This scheme is based on local feature-selective mechanisms, probably edge detectors, that are highlysensitive to the relative position of the sample points in the space domain.

INTRODUCTION

An important interface between the objective world and oursensory brain is the retinal photoreceptor lattice. The fidel-ity of the neural imaging in the visual system depends on thequality and structure of this initial sampling stage. Al-though most sampling schemes are based on the output ofregularly spaced sample elements, studies of the retinal lat-tice have also disclosed irregularity in the photoreceptormatrix.1-4 The functional significance of this sampling ir-regularity for the visual performance is, however, notknown.5,

6 The absolute sampling density of the cones in thecentralmost foveola matches well the quality of the imagespassed by the optics of the human eye,7 but at 1-deg eccen-tricity considerable undersampling takes place. The irregu-lar and suboptimal sampling may well be decisive con-straints on the visual filtering occurring at later stages of thevisual system.

Image sampling in the primate visual system is also per-formed at the level of simple and complex cells of the visualcortex. Their receptive-field weighting profiles play a cru-cial role in the representation of image signals by using a setof elementary functions, such as difference-of-Gaussians(DOG), sin x/x, and Gabor functions.A1 0 The density, ex-tent of overlap, and regularity of the cortical sampling lat-tice1 1' 12 will determine the completeness of image represen-tation in the visual system.

SAMPLING SCHEMES AND NYQUISTrFREQUENCY

According to the classical sampling scheme,1 ' 4 reconstruc-tion of a sampled signal is possible only if it was sampled at a

minimum rate of twice the frequency of its highest frequen-cy component. Hence the completeness of image represen-tation is limited by the sampling scheme applied. Severalmethods for relaxing the sampling constraints exist, howev-er, and even signals that were undersampled according to theclassical sampling theorem can be successfully recovered,provided that in addition to the sample points the slope ofthe image signal at the sample points is known or can becalculated by the system. Furthermore, two displaced sam-pling lattices, which separately would undersample the sig-nal, can form in combination an interlaced sampling latticethat allows the signal to be recovered completely. 15

Another method by which an efficient reconstruction orrepresentation of the image information is possible is thecase of bunched sampling. In this scheme, the total set ofsample points is divided into smaller sets that consist ofsample bunches that are equidistantly spaced. In otherwords, the interval between the bunches can be larger thanthe Nyquist interval without making the reconstruction im-possible.1 6 As shown by these examples, a variety ofschemes exist for image representation that the visual sys-tem might use to preserve as much of the sampled imageinformation as possible.

It has been suggested that the visual brain is able to per-form spatial interpolation by using the receptive-field prop-erties of single cells that fullfill some requirements of idealreconstruction by sin x/x- (sinc-) type filtering.8 The Gaborscheme especially, which Omse Symmetric and antisymmetric

elementary signals closely resembling the properties of sin-gle cortical cells, has been considered as a realistic candidatefor the representation of visual information in the cortex.9

The mechanisms of spatial interpolation and filtering in

0740-3232/88/040628-08$02.00 © 1988 Optical Society of America

Nyman et al.

Vol. 5, No. 4/April 1988/J. Opt. Soc. Am. A 629

human vision can be studied by using sampled test targetswhose detection or discrimination depends on the type ofinterpolation that the visual system can apply to the testtargets.1 7-19 The neurophysiological properties of singlecells and their considerable overlap in each point of thevisual field would make them ideal filters whose propertiescould be expected also to affect the psychophysical perfor-mance in tasks that are critically dependent on the obser-ver's ability to perform spatial interpolation of the images.

RATIONALE OF PRESENT EXPERIMENTS

We studied the effects of randomly spaced samples in one-dimensional visual grating stimuli on the observer's abilityto detect and discriminate gratings having different spatialwaveforms. By holding the average number of sample ele-ments constant and by varying the sampling locations sto-chastically, we could study the effect of randomness inde-pendent of the overall sampling density. Figure 1 shows thetype of stimuli used in our experiments. The spatial fre-quency of the spatial waveforms and the sampling rate used

- I -1 I I I L _ ~, t

in the figure differ somewhat from the values used in theactual psychophysical experiments; however, these parame-ters were not critical for the occurrence of the perturbingeffects observed for the random sampling.

The upper half of Fig. 1 shows a sine- (left) and square-wave (right) grating sampled every sixth raster line andpresented against a uniform gray background. The lowerhalf of Fig. 1 shows the same two gratings sampled at thesame average interval but on an irregular basis. The exactposition of each line sample was determined by the output ofa Gaussian random-number generator. In such sampling,each sample value is a correct representation of the signal,and only the location from which it was taken was random-ized. In the regular sampling case, although the samplingdensity is rather low, the waveform can be discriminated.In contrast, in the irregular sampling case, although thegratings are visible, their waveforms become indiscrimina-ble.

In Fig. 1 the number of sample lines for each dark or brightbar of the waveform does not remain constant because thesampling interval was not an even multiple of the cycle

I I 1 __

.f, tr 11U :T.

I! 1I "As 1 -I --Fig. 1. Demonstration of the type of stimuli used in the experiments. Four gratings are shown: (upper left) a sine-wave grating systematical-ly sampled at an interval of every sixth raster line, (upper right) a square-wave grating systematically sampled at the same interval, (lower left) asine-wave grating sampled at the same interval as above but now done randomly, and (lower right) a square-wave grating randomly sampled atthe same interval as above. In the actual psychophysical experiments the sampling rates used differed somewhat from that shown in thedemonstration.

. _AT

Nyman et al.

L � _--i T I 7 I T U


length of the waveform. This could introduce a dc shift inthe mean luminance of the sampled grating. This effectwas, however, averaged out in the psychophysical experi-ments in which the sample locations were varied randomlyfrom trial to trial. Separate control experiments showedalso that in the case of regular sampling this effect did nothave a significant effect on the waveform discriminationthresholds presented here.

METHOD

The sine- and square-wave gratings used in the present ex-periments were produced on a high-resolution cathode-raytube (Joyce Electronics). The spatial frequency, spatialwaveform, spatial phase, and contrast of the gratings werecontrolled by a microprocessor. Grating contrast was de-fined by

c = (Lmax - Lmin)/(Lmax + Lmin)1l, (1)

where Lmax and Lmin are the maximum and minimum lumi-nance levels in the continuous gratings, respectively. Thestimuli were windowed in space and time by Gaussian enve-lopes. The spatial window was one dimensional and had astandard deviation (SD) of 2.5 deg. The time window had arise time of 30 msec and a total exposure duration of 150msec. The gratings had a space-averaged luminance level of60 cd/M2. The display was masked to a rectangular field of30 deg X 22 deg, with an evenly illuminated (60-cd/M2)surround. Viewing was binocular with natural pupils at adistance of 57 cm. Head orientation was controlled by usinga chin and forehead rest.

The grating stimuli were sampled spatially by using asampling interval Ax that varied from 4.7 to 9.4 arcmin,which in our display meant every fourth to every eighthraster line. The regularly sampled waveform L 8(x) thus hadthe form

512

L,(x) = LoL1 + m f(x)(x - iAX) X (2)

where Lo is the mean luminance, f(x) is the continuous wave-form (either a sine or a square wave), and 5(x - iAx) is thedelta sampling function, which is zero outside the samplingpoints.

Because the distance between the raster lines of our dis-play was 1.18 arcmin, each cycle of a 1-cycle-per-degree (c/deg) grating consisted of 51 raster lines. In what follows thesampling interval is given in minutes of arc and by indicatingthe number of raster lines (dx) covered by the samplinginterval. The sampling intervals selected to produce thesampled gratings were thus multiples of the minimum inter-val Ax = 1.18 arcmin (dx = 1), being 4.7 arcmin (dx = 4), 7.1arcmin (dx = 6), and 9.4 arcmin (dx = 8). The correspond-ing sampling rates were 12.7, 8.5, and 6.4 samples/deg.

The first sidelobe frequency of a sampled sinusoid is locat-ed at the difference frequency s - fl, where s is the samplingfrequency in samples per degree and fi is the fundamentalfrequency. Thus the first sidelobe component of a 1-c/degsampled sinusoid will be located at 11.7 c/deg (dx = 4), 7.5c/deg (dx = 6), and 5.4 c/deg (dx = 8). For the square wave,the situation is quite different because lowering the sam-pling rate increases the number of undersampled harmonic

components. However, the fifth harmonic was sampled cor-rectly at all sampling rates used in this study.

The extent of sampling randomness was varied by aGaussian random-number generator, which determined theposition of each sample location. The probability p(x) thatsampling line i was positioned at x is described by the ex-pression

p(x) = (1/Aa2 )exp[-(x -Xi)2/22], (3)

where x, is the location of the ith sample line when regularsampling was applied and a is the standard deviation of theGaussian distribution. a = 2 lines means, for example, thatthere is a 67% probability that the sample will be takenwithin the lines xi i 2, i.e., within xi + 2.4 arcmin. Theprobability density function was truncated to ensure that agiven sample did not cross over or coincide with the locationof one of its neighbors. Thus there was always an integralnumber of available locations for each randomly positionedsample line.

In the first two experiments to be presented, we measureddetection thresholds separately for sampled sine- andsquare-wave gratings that were shown randomly in one ofthe two temporal intervals that were delineated by auditorysignals. The subject's task was to press the appropriateresponse button, indicating when the grating was presented.In the next experiment, a sine- and a square-wave grating ofequal fundamental (1-c/deg) amplitude were presented ran-domly in either of the two intervals, and the subject had tosignal in which interval the sine-wave grating was presented.The fundamental components of the two waveforms wereequated to create a situation in which the discriminationbetween them had to be based on signal components otherthan the fundamental one, i.e., the sidelobe frequencies gen-erated by the sampling and the harmonic components of thesquare wave.

To rule out the use of fixed local cues in the sampledgrating stimuli, the absolute spatial phase of the cosine grat-ing, the absolute position of the Gaussian window, and thecontrast of the gratings were randomly jittered from trial totrial. For each trial this jittering was done by adding apositive or negative random constant-to these stimulus pa-rameters. Grating contrast was varied using a maximum-likelihood search algorithm 2 0 in a single staircase. The posi-tion of the psychometric function along the contrast axishaving the maximum likelihood of representing the truepsychometric function was found after 30 trials. The con-trast value for which the subjects responded correctly 70% ofthe time, based on this function, was taken as the threshold.The average standard error for a threshold estimate wasapproximately 0.7 dB for detection and 1 dB for discrimina-tion thresholds. The jitter introduced to the test contrastaveraged to zero over trials, thus eliminating any systematiceffect on contrast thresholds.

Subjects were two of the authors (GN and MWG), whowere both corrected for myopia.

RESULTS

Detection of Regularly and Irregularly Sampled GratingsFirst we explored the effect of random sampling on theobserver's ability to detect the gratings. The results of these

Nyman et al.


MWG GN 42dx 4 fundamental signal frequency and high enough to pass all) dx:6 the frequency components that could be expected to affecto odx:8 detection. In the case of a square wave, this frequency

- a ^ ~ z. ^ passes also the third- and fifth-harmonic components (3 and- 8Ba 0 73-- 5 c/deg).

0 -0 aThe output of this low-pass filtering was analyzed by a256-point fast-Fourier-transform (FFT) procedure by usinga raised cosine Hanning window to suppress spurious side-lobes. Figure 3 shows the relative amplitude of the funda-

* -. ~ " >*<^-j ~mental component of a sine wave as a function of the ran-. . . . . _ _. *ffi. - domness of sampling as calculated by the FFT of the sam-

pled and filtered signal. Values for two different samplingrates (open squares, dx = 6; open circles, dx = 8) and fivelevels of randomness are shown. The data were obtainedfrom 30 simulation runs for each point. Although not shownin Fig. 3, the fundamental component of the square wave didnot change much when the standard deviation of the Gauss-

o 1 2 3 4 0 1 2 3 4 ian random-number generator, which determined the sam-ple-line locations, was increased from 0 to 4.

SD Gaussian Sampler (arcmin) To estimate the effects of random sampling on the averageg contrast threshold is plotted as a function of the SD of spectra of our stimuli, we performed a set of FFT runs atin random sampler used to determine the position of the three different levels of randomness. The runs consisted ofations. Open symbols present the values based on the the sampling of a signal (a sine or a square wave), low-passto sine-wave gratings, and filled symbols show the data filtering with the cutoff set at six times the fundamentala-wave gratings. Square-wave thresholds have been ar- frequency of the waveform, and the FFT. The normalized3placed by 1 log unit for sake of clarity. The parametersampling interval in terms of the number of raster lines amplitude of each component was then calculated by using

the sampling interval. The corresponding angular mea- the amplitude of the fundamental component as a reference.7 arcmin (dx = 4), 7.1 arcmin (dx = 6), and 9.4 arcmin (dx The average normalized amplitudes as obtained from theshown for subjects (left) MWG and (right) GN. simulations are shown in Fig. 4. The output of the FFT was

scaled so that the fundamental component corresponds ex-its are shown in Fig. 2. The contrast thresholds actly to the 1-c/deg grating that we used in our psychophysi-d as a function of the SD of the Gaussian random- cal measurements. Hence all the average components seen3nerator that determined the position of the sam- in Fig. 4 correspond to the actual components of our grating

A SD of zero indicates the regular sampling case. stimuli.-he two observers are shown for both waveforms The results show that the randomness of sampling pro-'he spatial frequency of the fundamental was 1 c/ duced spread of spectral energy over all frequencies, but theparameter is the sampling interval dx, as defined fundamental component was well preserved in the averagere. spectra. The spectral spread caused by the sampling noisealts from both subjects indicate that the extent of was negligible when compared with the energy of the funda-

irregularity in the samples' location had little or no effect onthe ability of the observer to detect the sampled gratings.The average sampling density, however, does have a system-atic effect on thresholds: the higher the average samplingdensity, the lower the threshold. This result could be ex-pected because the amplitude of the fundamental of a sam-pled grating is inversely proportional to the size of the sam-pling interval used. For subject MWG, for example, thecontrast of the fundamental component (1 c/deg) of thesampled grating at threshold was 0.48 and 0.60% for thesampling intervals dx = 4 (4.7 arcmin) and dx = 8 (9.4arcmin), respectively. These values correspond well to thethreshold values obtained for continuous gratings of thesame frequency. 2'

According to the multiple-channel model proposed byCampbell and Robson,2 2 detection of a sampled spatialwaveform could be possible when any of its spectral compo-nents is above its threshold. To see if the randomness ofsampling produces strong noise components that could me-diate detection, we calculated the spectral contents of oursampled signals after first low-pass filtering them by usingan approximation of the sin x/x interpolation that set thecutoff frequency at 6 c/deg. This is 2.6 octaves above the

0'a

E-7

0

a)

cc

.2

.1

00 1 2 3 4

SD Gaussian Sampler (arcmin)

Fig. 3. Relative amplitude of the fundamental component of sam-pled sine waves is plotted as a function of the SD of the Gaussiansampler. The data are based on the FFT of the low-pass filtered(cutoff corresponding to 6 c/deg in our psychophysical stimuli)versions of the sampled waveforms. The parameter is the samplinginterval dx(o, dx = 6; O, dx = 8). Thirty runs were performed foreach data point, and the bars indicate the standard deviations of theamplitudes.

0

'0

0

Ca

0)0)

-J

1

2

Fig. 2. Loqthe Gaussiesample locEthresholdsfrom squarebitrarily di,dx gives thecovered bysures are 4.'= 8). Data

experimerare plotteinumber gopled lines.Data for Itested. Tdeg. Thein the figu

The resi

Nyman et al.


1.0

0 1 2 3 4 5

0 1 2 3 4 5

L I I I I I I I I I II I I I I a

0 1 2 3 4 5

1.0 S.D=O

.11 I

0 1 2 3 4 5

S.D.=1

L 'III III IIII IIII I LL I0 1 2 3 4 5

S. D.=2

I I I I d 1IIIIIa 0 1 2 3 4 5

Spatial frequency

Fig. 4. Examples of the average (N = 50) spectra of sampled and low-pass filtered (left) sine and (right) square waves. The sampled signals

were low-pass filtered before they were subjected to the 256-point FFT. One sampling interval (dx = 6) was used with three different levels of

randomness as indicated in the figure. The component frequencies were multiplied by a constant so that they are comparable to the actual psy-

chophysical stimuli used. Spatial frequency 1.0 corresponds to the fundamental (1 c/deg) of the actual grating signals used in the experiments.

The noise that occurs also in the spectra of the regularly sampled square wave was caused by the undersampling of the highest harmonics, as the

low-pass filtering was performed after the sampling. Some distortion was also caused at the fundamental and the third-harmonic bands.

mental component of the original signals. Hence, at all

levels of randomness, the most detectable component wasthe fundamental of the sampled waveform.

Some spectral noise occurred in the sampled square waveat other frequencies than its harmonics even when the sam-pling was regular, i.e., SD = 0. This was caused by the

folding of the high-frequency harmonics of the square waveto the lower frequencies because the low-pass filtering wasperformed to the sampled version of the signal and not to theoriginal continuous signal. This noise was apparent in the

amplitudes of the components located near and within thefundamental and third-harmonic bands.

The average amplitude spectra show that the third har-monic was well preserved when random sampling was used.This can be readily observed by comparing the spectra of thetwo different randomly sampled waveforms. However, asnoted above, the undersampling of the higher-spatial-frequency components of the square wave caused some re-flection of spectral energy into the third-harmonic band,which could, in fact, improve the discriminability of the twodifferent waveforms.

Waveform Discrimination for Regularly and IrregularlySampled GratingsNext we asked whether sampling irregularity affects an ob-server's ability to discriminate between sampled gratingshaving two different spatial waveforms. The results ofthese experiments are shown for both observers in Fig. 5. Inthe figure, log discrimination threshold is plotted as a func-tion of the SD of the Gaussian random sampler. As in Fig. 2,

the parameter is the sampling interval (in number of linescovered by the sampling interval) depicted in the insets. Incontrast to simple detection, spatial waveform discrimina-tion is impaired strongly by random sampling. Increasing

the standard deviation of the random sampler from 0 to 4raises the discrimination threshold by approximately onelogarithmic unit in contrast. This large change in discrimi-nability occurs although the average sampling density, andthus the average energy under the contrast envelope, variesnegligibly.

Convolution by Difference-of-Gaussians-Type SpatialFilters and Sine/Square Wave DiscriminationIn an attempt to model our results shown in Fig. 5, weperformed computer simulations by convolving the sampledone-dimensional stimuli with a DOG weighting functioncentered at f3 to map the sensitivity of such filters to thesampled stimuli. The Fourier transform of the output of

M

0un

9

F_r_0

._

.E_

._

C',0c

-J

0

1

0 1 2 3 4 0 1 2 3 4

SD Gaussian Sampler (aromin)

Fig. 5. Log waveform discrimination threshold is plotted as a func-

tion of the SD of the Gaussian random sampler as in Fig. 2. Thesampling interval is depicted by the different symbols, as defined inthe figure.

Nyman et al.

......


the convolution between the DOG filter and the sampledsine and square waveforms showed that the random Gauss-ian positioning of the sample lines had caused some spread-ing of spectral energy, which was strong enough to be reflect-ed in the filter responses. Hence it was possible that thedifficulty in discriminating between the randomly sampledsine and square waves was caused not by suboptimal visualinterpolation but by the aliasing of the harmonics originat-ing from the random sampling. Such aliased componentscould have affected the frequency bands normally used fordiscriminating between the two different waveforms.

Campbell and Robson22 showed that the discrimination ofa medium-spatial-frequency square wave was possible whenthe contrast of the third harmonic was above its individualcontrast threshold. Hence a channel tuned to the thirdharmonic could be used for the discrimination between thesine and square waves. To see if this is true also for sampledgratings, we simulated the output of the mechanism tuned tothe third (3-c/deg) harmonic by the following DOG weight-ing function, which was adopted from the model of Wilsonand Bergen2 3:

DOG 3(x) = klexp[-(x -XO)IIU2]

- 0.57 exp[-(x - x0)2 /(1.75U)2 ]J. (4)

Here the space constants of the center and the surroundhave a ratio of 1.75. The tuning to the third harmonic wasachieved by suitably scaling the space constants. To esti-mate the response of these two mechanisms to sampledgratings, the sampled sine and square waveforms were con-volved with the DOG3 function separately, which producedthe convolution output

s(x) = DOG3(x)*L,(x), (5)

where L,(x) is the sampled version of the sine or the squarewave. The convolution output s(x) was then Fourier trans-formed by using a FFT algorithm. To estimate the responseof the filter to the sampled signal, the total power of theFourier spectrum of the convolution was calculated. Thistotal power reflects the response of the mechanism to allfrequency components that were produced by the samplingand the noise generated from the randomness of sampling.Assuming that the output of the channel tuned to the third-harmonic component determines the discriminability of thetwo waveforms, the sampling noise should be an effectivemasker if it produced equal activity in this channel for bothsine and square waves.

Figure 6 shows the results of the simulations as well as thepsychophysical data of both observers. For the psycho-physical data, the two axes represent the relative sensitiv-ities of the observer in discriminating between a sampledsine wave and a sampled square wave. The vertical axisgives the relative sensitivity at discrimination threshold forsystematically sampled gratings, and the horizontal axisgives the corresponding threshold for randomly sampledwaveforms.

The data were scaled so that the maximum sensitivity wasassigned the value of 1.0 for systematic and random sam-pling separately. All other data points were expressed rela-tive to these reference values. Each data point (xi, yi) in the

E

.'

CU

.EU)

.0

a)C

0 0.5 1.0Relative Discriminability (random)

Fig. 6. Relative discriminability of systematically sampled sine-and square-wave gratings based on the data in Fig. 5 (squares) ofboth observers is compared with the discriminability of irregularlysampled gratings. The location of each data point is determined bythe relative sensitivity at one systematic sampling rate (ordinate)and the same average sampling rate for the irregular case (abscissa).Filled circles present the simulation results based on the relativeoutput of the DOG filter tuned to the third harmonic of the squarewave. The filled squares show the psychophysical data obtainedunder stimulus parameters identical to those used in the computersimulations, and open squares show the data for the remainingconditions. Both sets of data should fall on the diagonal if randomsampling had no effect on waveform discrimination. The psycho-physical data are typically above the diagonal, suggesting that thesystematically sampled waveforms are discriminated more easily.

plot thus represents the results from two experiments: yi isthe relative sensitivity obtained with one sampling intervaland systematic sampling, and xi is the result from the mea-surement with the same average sampling interval but withrandom sampling. The data of the two subjects are shown.

As can be seen from Fig. 6, the psychophysical data fallalmost exclusively above the diagonal toward the verticalaxis. This clustering represents the finding shown also inFig. 5; the observers were more sensitive in the waveformdiscrimination task when the sampling was systematic.

The selectivity of the f3 channel to the type of waveformwas estimated by the ratio Psq/Psi, where Psi is its response toa sampled sine wave and Psq is the response to a sampledsquare wave. Typically, the channel tuned to the thirdharmonic gave 1.5-4 times stronger responses to the sam-pled square wave than to the sine wave. To make the simu-lation data comparable with the psychophysical data, thisratio was scaled, again separately for each axis, by giving thebest discrimination performance the relative value of 1.0. Ifthe modeled f3 channel were insensitive to the randomnessof sampling, the data from the simulations would fall on thediagonal of the plot. The simulations were run using thesame sampling intervals dx = 4 (4.7 arcmin), 6 (7.1 arcmin),and 8 (9.4 arcmin) as in the psychophysical experiments;however, for clarity, only one level of randomness (SD = 2.0)was used. The results are depicted by the filled circles,which represent the average of 30 simulation runs, each ofwhich consisted of generating the randomly sampled squareand sine waveforms and convolving them with the DOG3filter tuned to the third harmonic. The convolution outputwas subsequently subjected to the FFT.

The simulation data fall on the diagonal or below it, sug-

I I

Nyman et al.


gesting that the observed loss of spatial waveform discrimi-nation for irregularly sampled gratings cannot be explainedsolely on the basis of the spatial filtering characteristics of

the channels tuned to the third harmonic of the square wave.Thus it appears that some other mechanism than that tunedto the third harmonic is required to perform the sine/squarewaveform discrimination. Similar conclusions were drawn

elsewhere 2 l and were based on the adaptation to the higher-harmonic components of the square wave.

DISCUSSION

The introduction of positional variation to the sample loca-tions of our sampled test stimuli proved to have a detrimen-tal effect on waveform discrimination. Contrast detection,on the other hand, was not impaired by this kind of samplingnoise, as evidenced by the finding that the detection thresh-olds were independent of the statistical distribution of thesample locations. Thus it appears that the visual system isable to perform effective integration of contrast energy,which may be done by an interpolation process based on

elementary functions that have spatial bandpass character-istics. The calculations of the average fundamental ampli-tude of sampled sine waves (Fig. 3) support this finding.

Spatial pattern discrimination was, however, strongly im-paired by relatively small, approximately 1 arcmin or more,random displacements of the sample locations. This resultis rather surprising because it shows that the pattern-dis-criminating mechanisms of the visual system cannot toleratedisplacements, the amplitude of which is near the visualacuity limit. Because the spatial frequency of the wave-forms studied was relatively low (1 c/deg), a 1-arcmin dis-placement in the sample location corresponds to only one

twentieth of the cycle length of the third harmonic, whichshould not have strong detrimental effects on the waveformspectrum.

The regularity/irregularity of spatial sampling is criticalfor unambiguous representation of image information in thevisual system. Although it seems reasonable to assume thatthe visual system must somehow adapt to inherent biologicalirregularities, whenever strong irregularities are introducedartificially to the visual stimuli, the system apparently is nolonger able to discriminate among them properly. In the

crude pattern-discrimination task that we used, the distort-ing effects of irregularity occurred first when the irregularitywas of the order of spatial resolution limit. This is relativelylarge compared with the natural cone lattice irregularity,whose magnitude is near the hyperacuity range.

In the human fovea, for example, the cone mosaic irregu-larity becomes apparent within an area having a diameter ofapproximately 1 deg or less. This can readily be observed

from the flat-mount preparations published by Polyak.2

Data of Hirsch and Miller2 4 obtained from a monkey retinashow that the standard deviation of the cone row separationswithin a small retinal patch is only about 10% of its averagevalue, which corresponds to only about 4 arcsec. This valueis 1 order of magnitude less than the minimum positionaljitter that we used for random sampling. It seems possiblethat the variation in the packing of cones might offer a betterscale than their absolute density for estimating the toleranceof the visual system to normal positional jitter in stimulussamples.

PATTERN RECOGNITION UNDER CONDITIONSOF SAMPLING IRREGULARITY

The main finding of the present study was that samplingirregularity affects the sine/square waveform discriminationbut fails to affect the detection of these patterns. Thisfinding indicates that irregular sampling disturbs neuralmechanisms that perform spatial pattern discrimination butdoes not disturb the mechanisms that perform effective en-ergy integration. If the visual system is made up of a set ofmultiple spatial filters that have relatively narrow spatial-frequency bandwidths, i.e., large integration windows, thepositional variations in the samples of our stimuli should notdisturb waveform discrimination. This effect could be dem-onstrated readily by the convolutions between the sampledwaveforms and the spatial-frequency-specific DOG func-tions (Fig. 6). For pattern discrimination between a sam-pled sinusoid and a sampled square-wave grating, a simplemultiple-channel hypothesis would suggest that discrimina-tion is based on the detection of the component frequenciesby different channels. For the discrimination of a simplesinusoidal from complex pattern, the activity of any otherchannel in addition to the channel tuned to the fundamentalshould be sufficient for discrimination. The simulationsbased on DOG3 filters and shown in Fig. 6 did not indicate,however, that irregular sampling would decrease the detect-ability of the third harmonic in the square wave. Thisobservation suggests that the effects of sampling irregularitycannot be accounted for by masking by noise2 5 in the chan-nel sensitive to this component. This finding is also in

accordance with the finding that at low sampling rates thedetection threshold for a regularly sampled sine wave cannotbe fully predicted on the basis of masking by the spuriousharmonics generated by the sampling.26

It could be argued that the subjects could use a strategy inwhich they respond "square" if the neighboring lines appearto have the same luminance and "sine" otherwise. Howev-er, the randomness of sampling should have no effect on thediscrimination thresholds if this were the case. Subjective-ly, we can also notice that in Fig. 1 the equal luminance of thebright lines of the randomly sampled square wave is notimmediately apparent.

To account for our findings we would like to proposetentatively the existence of mechanisms locally sensitive tothe polarity and position of contrast steps. Such filterswould be characterized by narrow integration areas (i.e.,narrow receptive fields) and would thus be spectrally broad-band. These mechanisms would respond optimally to asingle contrast edge falling within their receptive field. Wepropose that such a theoretical edge detector should bemade up of a series of odd-symmetric Gabor-type receptivefields, the zero crossings of which would coincide in spacesuch that a contrast edge positioned at this crossing wouldlead to an optimal response in the entire filter set. The largeeffect of irregularity on the sine/square waveform discrimi-nation performance in our experiments suggests that ran-dom sampling perturbs the activity of this type of mecha-nism by locally scrambling the information about the spatialphases of the square wave's harmonics. In the regularlysampled case, the positions of the light-dark and dark-lightsample pairs apparently are sufficient to stimulate such odd-symmetric mechanisms, thus providing the required edge

Nyman et al.

Nyman et al.Vol. 5, No. 4/April 1988/J. Opt. Soc. Am. A 635

information. Subjectively, our observers reported that theyperformed the task by attending to the presence or absenceof sharp edges in the sampled gratings.

ACKNOWLEDGMENTS

This research was supported by the Deutsche Forschungsge-meinschaft, SFB 325.B4, and by the Finnish Academy. Wethank L. Spillmann, D. Pelli, and K. Arnold for helpfulcomments on an earlier version of this manuscript.

REFERENCES

1. G. Osterberg, "Topography of the layer of rods and cones in thehuman retina," Acta Ophthalmol. Suppl. 6, 1-103 (1935).

2. S. L. Polyak, The Vertebrate Visual System (U. Chicago Press,Chicago, Ill., 1957).

3. J. I. Yellott, Jr., "Spectral analysis of spatial sampling by photo-receptors: topological disorder prevents aliasing," Vision Res.22, 1205-1210 (1982).

4. J. Hirsch and R. Hylton, "Quality of the primate photoreceptorlattice and limits of spatial vision," Vision Res. 24, 347-355(1984).

5. D. R. Williams and R. Collier, "Consequences of spatial sam-pling by a human photoreceptor mosaic," Science 221, 385-387(1983).

6. A. W. Snyder, T. R. J. Bossomaier, and A. Hughes, "Opticalimage quality and the cone mosaic," Science 231, 499-501(1986).

7. A. W. Snyder and W. H. Miller, "Photoreceptor diameter andspacing for highest resolving power," J. Opt. Soc. Am. 67, 696-698 (1977).

8. H. B. Barlow, "Reconstructing the visual image in space andtime," Nature 279, 189-190 (1979).

9. S. J. Marcelja, "Mathematical description of the responses ofsimple cortical cells," J. Opt. Soc. Am. 70, 1297-1300 (1980).

10. J. G. Daugman, "Two-dimensional spectral analysis of corticalreceptive field profiles," Vision Res. 20, 847-856 (1980).

11. D. J. Hubel and T. N. Wiesel, "Receptive fields, binocular inter-action, and functional architecture in the cat's visual cortex," J.Physiol. (London) 160, 106-154 (1962).

12. B. Fischer, "Overlap of receptive field centers and representa-tion of the visual field in the cat's optic tract," Vision Res. 13,2113-2120 (1973).

13. C. E. Shannon, "Communication in the presence of noise," Proc.IRE 37, 10-21 (1949).

14. E. T. Whittaker, "On the functions which are represented by theexpansions of the interpolation theory," Proc. R. Soc. Edin-burgh Sect. A 35, 181-194 (1915).

15. R. R. Bracewell, The Fourier Transform and Its Applications(McGraw-Hill, New York, 1978).

16. A. Papoulis, Signal Analysis (McGraw-Hill, Singapore, 1984).17. G. Nyman and P. Laurinen, "Reconstruction of spatial informa-

tion in the human visual system," Nature 279, 324-325 (1982).18. M. J. Morgan and R. J. Watt, "Mechanisms of interpolation in

human spatial vision," Nature 299, 553-554 (1982).19. D. M. Levi and S. A. Klein, "Sampling in spatial vision," Nature

320, 360-362 (1986).20. H. R. Liberman and A. P. Pentland, "Microcomputer-based

estimation of psychophysical thresholds: the best-PEST," Be-hav. Res. Methods Instrum. 14, 21-25 (1982).

21. M. W. Greenlee and S. Magnussen, "Higher harmonic adapta-tion and the detection of squarewave gratings," Vision Res. 27,249-255 (1987).

22. F. W. Campbell and J. G. Robson, "Application of Fourieranalysis to the visibility of gratings," J. Physiol. (London) 197,551-566 (1968).

23. H. R. Wilson and J. R. Bergen, "A four mechanism model forthreshold spatial vision," Vision Res. 19, 19-32 (1979).

24. J. Hirsch and W. H. Miller, "Does cone positional disorder limitresolution?" J. Opt. Soc. Am. A 4, 1481-1492 (1987).

25. C. F. Stromeyer and B. Julesz, "Spatial-frequency masking invision: critical bands and spread of masking," J. Opt. Soc. Am.62, 1221-1232 (1972).

26. D. C. Burr, J. Ross, and M. C. Morrone, "Local regulation ofluminance gain," Vision Res. 25, 717-727 (1985).

Sampling irregularity perturbs visual reconstruction

Documents

Transcript of Sampling irregularity perturbs visual reconstruction