Real-time Hyperspectral Imaging in Hardware via Trained MetasurfaceEncoders

Maksim Makarenko1†, Arturo Burguete-Lopez1, Qizhou Wang1, Fedor Getman1, Silvio Giancola1,Bernard Ghanem1, Andrea Fratalocchi1

1King Abdullah University of Science and Technology (KAUST)Thuwal, 23955-6900, KSA

†maksim.makarenko@kaust.edu.sa

Abstract

Hyperspectral imaging has attracted significant attentionto identify spectral signatures for image classification andautomated pattern recognition in computer vision. State-of-the-art implementations of snapshot hyperspectral imagingrely on bulky, non-integrated, and expensive optical elements,including lenses, spectrometers, and filters. These macro-scopic components do not allow fast data processing for, e.g.real-time and high-resolution videos. This work introducesHyplex™, a new integrated architecture addressing the limi-tations discussed above. Hyplex™ is a CMOS-compatible,fast hyperspectral camera that replaces bulk optics withnanoscale metasurfaces inversely designed through artificialintelligence. Hyplex™ does not require spectrometers butmakes use of conventional monochrome cameras, opening upthe possibility for real-time and high-resolution hyperspec-tral imaging at inexpensive costs. Hyplex™ exploits a model-driven optimization, which connects the physical metasur-faces layer with modern visual computing approaches basedon end-to-end training. We design and implement a pro-totype version of Hyplex™ and compare its performanceagainst the state-of-the-art for typical imaging tasks such asspectral reconstruction and semantic segmentation. In allbenchmarks, Hyplex™ reports the smallest reconstructionerror. We additionally present what is, to the best of ourknowledge, the largest publicly available labeled hyperspec-tral dataset for semantic segmentation. 1

1. IntroductionHyperspectral imaging is gaining considerable interest in

many areas including civil, environmental, aerial, military,and biological sciences for estimating spectral features thatallow the identification and remote sensing of complex mate-rials [10, 29]. Ground-based hyperspectral imaging enables

1Dataset available on https://github.com/makamoa/hyplex.

Figure 1. Hardware implemented Hyplex™ imaging system.(a) Example of metasurface pixel arrays (blue squares). (b)Schematic of meta-pixel array on top of a camera sensor. (c)Closeup showing the metasurface projectors as subpixels of thearray. (d) Scanning electron microscope image of a fabricated meta-surface pixel. (e) Optical micrograph of the metasurface pixel array.(f) Illustration of the barcode generated by (e).

automated classification for food inspection, surgery, biol-ogy, dental and medical diagnosis [1, 21, 34, 42]. Likewise,aerial and submarine hyperspectral imaging are currentlyopening new frontiers in agriculture and marine biology forthe taxonomic classification of fauna, and through aerialdrone footage for precision agriculture [2, 10, 13, 14]. Thepresent state-of-the-art in hyperspectral imaging, however,is still affected by problems of expensive setup costs, time-consuming post-data processing, low speed of data acquisi-tion, and the needs of macroscopic optical and mechanical

arX

iv:2

204.

0208

4v1

[cs

.CV

] 5

Apr

202

2

components [41, 58]. A single hyperspectral image obtainedfrom a high-resolution camera typically requires gigabytesof storage space, making it impossible to perform real-timevideo analysis with today’s computer vision techniques [28].

Computational hyperspectral reconstruction from a singleRGB image is a promising technique to overcome some ofthe challenges mentioned above [4, 7, 18, 22, 25, 26, 38, 44,54, 63]. Heidrich et al. [24] proposed hyperspectral camerasbased on integrated diffractive optical elements, while othergroups [12,60] leveraged deep neural networks for designingspectral reconstruction filters. While these approaches couldhelp address the problem of speed, they are not yet able totackle the issues of high cost and slow data processing. Otherbottlenecks are the use of elementary filter responses, whichare not optimized beyond primitive thin-film interferencepatterns, and the lack of integrated structures that couldexploit the modern footprint of CCD/CMOS sensors.

We here introduce the Hyplex™ system (Fig. 1), adata-driven hyperspectral imaging camera (Fig. 1, a-b),which uses state-of-the-art metasurfaces to replace macro-scopic components with highly integrated dielectric nanores-onators that manipulate light as a feed-forward neural net-work [9, 17, 19]. Metasurfaces have successfully demon-strated the ability to integrate various basic optical compo-nents for different applications [48–50]. Hyplex™ leveragesthis technology to compress high-dimensional spectral datainto a low-dimensional space via suitably defined projectors(Fig. 1, c-d), designed with end-to-end learning of large hy-perspectral datasets. ALFRED [16, 19, 37], an open-source,inverse-design software exploiting artificial intelligence (AI),provides the means to design the metasurface projectors.These nanostructures encode broadband information carriedby incoming spectra into a barcode composed of a discretepattern of intensity signals (Fig. 1, e-f). A physical model-aware framework finds the optimal projectors’ response withvarious learning schemes, designed based on user end tasks.

We summarize our contribution as follows: (i) We pro-pose and implement an inexpensive and fast-processing data-driven snapshot hyperspectral camera that uses two inte-grated components: inverse-designed spectral encoders anda monochrome camera. (ii) We implement an end-to-endframework for hyperspectral semantic image segmentationand spectral reconstruction, and benchmark it against thestate-of-the-art, reporting the highest performance to date.(iii) We create FVgNET, the largest publicly available datasetof 317 samples of labeled hyperspectral images for semanticsegmentation and classification.

2. Related WorkHyperspectral reconstruction is an ill-posed problem

demanding the inverse projection from low-dimensionalRGB images to densely sampled hyperspectral images(HSI) [5, 39]. Metamerism [15], which projects different

spectral distributions to similar activation levels of visualsensors, represents a significant challenge. Traditional RGBcameras project the entire visible spectra into only three pri-mary colors. This process eliminates critical informationmaking it challenging to distinguish different objects [38].For the specific task of hyperspectral reconstruction, we canpartially recover such lost information. Spectral projectionsare similar to autoencoders in the sense that they downsam-ple the input to a low-dimensional space. If we design asuitable algorithm that explores this space efficiently, wecould retrieve sufficient data to reconstruct the initial input.

Reconstruction by sparse coding and deep learning:Sparse coding [32, 45] represents perhaps the most intu-itive approach to this idea. These methods statically discovera set of basis vectors from HSI datasets known a priori.Arad et al. [5] implemented the K-SVD algorithm to cre-ate overcomplete HSI and RGB dictionaries. The HSI isreconstructed by decomposing the input image into a linearcombination of basis vectors, then transferred into the hy-perspectral dictionary. A limit of sparse-coding methods istheir applied matrix decomposition algorithms, which arevulnerable to outliers and show degraded performance [27].Recently, research groups extended the capabilities of sparsecoding by investigating deep learning. Galliani et al. [18]demonstrated a supervised learning method, where a UNet-like architecture [46] is trained to predict HSI out of singleRGB images. Nguyen [38] trained a radial basis functionnetwork to translate white-balanced RGB values to reflec-tion spectra. In another work, Xiong et al. [55] introduced a2-stage reconstruction approach comprising an interpolation-based upsampling method on RGB images. The end-to-endtraining proposed recovers true HSI from the upsampledimages. Wug et al. [40] used different RGB cameras toacquire non-overlapping spectral information to reconstructthe HSI. These approaches reconstruct spectral informationfrom highly non-linear prediction models, limited by theirsupervised learning structure. The models constrain datadownsampling to non-optimal RGB images by applying acolor generation function on HSI or generic RGB cameras.With Hyplex™, we avoid all the issues of the sparse codingand deep-learning reconstruction methods by exploring anew concept, which performs spectral downsampling withoptimally designed metasurface projectors.

Hyperspectral imaging with trainable projectors: Opti-cal projectors in cameras mimic the chromatic vision ofhumans based on primary colors [23]. In hyperspectral imag-ing, however, the design of projectors requires further studyto identify their optimal number and response. Human eyesare not the best imaging apparatus for every possible real-world scenario. The works of [6, 47, 53] expand the conceptof RGB cameras to arbitrary low-dimensional sampling ofreflectance spectra. These works employ different variantsof optimization routines, which converge to a set of optimal

projectors from an initial number of candidates. The selectedprojectors provide a three-channel reconstruction of the HSIwith superior performance. Nie et al. [39] demonstrated thata 1×1 convolution operation achieves similar functionalityto optical projectors while processing multi-spectral dataframes. The network is like an autoencoder, where the inputHSI is downsampled and then reconstructed by a decoder net-work. Zhang et al. [62] designed and fabricated a broadbandencoding stochastic camera containing 16 trainable projec-tors that map high-dimensional spectra to lower-dimensionalintensity matrices. Recently, Liutao et al. [56] proposedFS-Net, a filter-selection network for task-specific hyper-spectral image analysis. In [59], the authors showcased anidea of filter optimization for hyperspectral-informed imagesegmentation tasks.Inverse design of metasurface projectors: Optimizingbest-fit filters is a dimensionality reduction problem, whichrequires finding the principal component directions that en-code eigenvectors showing the lowest loss. The state-of-the-art results are generated either from theoretical calculation orexperimental measurement on thin-film filters, representinga rough approximation of the precise principal components.In hyperspectral imaging, these components typically exhibitfrequency-dependent irregular patterns composed of com-plex distributions of sharp and broad resonances, indicatingthe need for more dedicated control of material structures,e.g. metasurface technology. Modern metasurface designapproaches [51,52] usually rely on a library of pre-computedmetasurface responses and polynomial fitting to further gen-eralize the relationship between design parameters and thedevice performance. We, instead, design our metasurfaceoptical projectors via ALFRED [20], a hybrid inverse de-sign scheme that combines classical optimization and deeplearning [52]. In this work, we significantly extend the ca-pabilities of the original code by adding differentiability,physical-model regularization, and complex decoder pro-jectors able to tackle different computer vision tasks andperform thousands of parameter optimizations through thesupervised end-to-end learning process.

3. MethodologyThe Hyplex™ hyperspectral imaging system consists of

two parts: a hardware linear spectral encoder E and a soft-ware decoder (Fig. 2). The encoder compresses an inputhigh-dimensional HSI β to a lower multispectral image ten-sor S = E(β), while the decoder maps the tensor S touser-defined task-specific outputs. In this work, we considertwo types of tasks: hyperspectral reconstruction and seman-tic segmentation. Spectral reconstruction aims to reconstructwith minimum losses the input HSI tensor. We define the lossvia the Root Mean Squared Error (RMSE) β = Drec(E(β))between reconstructed and input spectra. Semantic segmen-tation, conversely, provides a pixel-by-pixel classification

hardware encoder software decoder

reconstruction

segmentation

HSI Image tensor

Figure 2. Conceptual sketch of Hyplex™ system. The system isconstructed by a hardware optical encoder E that is implemented viatrainable metasurface arrays and a software decoder D optimizedfor two different tasks, including hyperspectral reconstruction andspectral-informed semantic segmentation.

of HSI. In this task, we use as decoder Dseg the U-Net ar-chitecture, with adjusted input and output layers to meetthe dimensionality of the HSI tensor. The decoder outputssoftmax logits y, representing the probability of observingeach pixel ground-truth label y. We assess these predictionsquantitatively by using the Cross-Entropy loss function Lseg .

3.1. Hardware encoder

Recent work demonstrates that the transfer function of anarray of sub-micron nanostructured geometries can approx-imate arbitrarily defined continuous functions [19, 36]. InHyplex™, we exploit such universal approximation abilityto design and implement an optimal linear spectral encoderhardware for a specific hyperspectral information-relatedimaging task.

Figure 3 summarizes the data workflow of Hyplex™ fora generic linear encoder operator E = Λ

†. Panel (a) shows

an example hyperspectral image. The data is represented asa tensor β with three dimensions: two spatial dimensions(x, y), corresponding to the camera virtual image plane, andone frequency axis ω, measuring the power density spec-tra retrieved at one camera pixel (Fig. 3b). Following adata-driven approach, we implement a linear dimensional-ity reduction operator that finds a new equivalent encodedrepresentation of β (Fig. 3c). The hyperspectral tensor of adataset of images is flattened to a matrix B that contains, oneach column, the power density spectra of a set of camerapixels. We then the apply the linear encoding Λ† to obtain anapproximation of B [8] via a set of linear projectors Λ(ω),which map pixel-by-pixel the spectral coordinate βij to a setof scalar coefficients Sijk:

Sij = Λ(ω)βij(ω), Sijk =

∫Λk(ω)βij(ω) dω . (1)

Figure 3. Metasurface subpixel array as a linear spectral encoder. (a) A spectral image tensor (β) is captured by a hyperspectral camera.(b) The corresponding pixel spectra (βij) at position ij in the xy camera plane. (c) Example of dimensional reduction linear operator Λ†

of a flattened matrix B with the resulted projected encoder barcode for a pixel spectra at the ij position. (d) Optimal encoder functionsΛ†. (e) Non-differentiable inverse design optimization framework implemented via ALFRED utilized to find a set of metasurfaces L withdesired response Λi. (f) Differentiable backbone enabling simultaneous optimization of responses Λ and structures L. Metasurface pixel (g)composed of a two-dimensional array of resonant metapixels with corresponding fitted transmission responses Λ. (h) Conceptual sketch ofthe Hyplex™ system with an enlarged spectral-specific barcode (i) produced by an imaging-based readout of the metasurface’s transmissionresponse.(j) Recovered pixel spectra through decoder Drec projection βij .

The spectral information contained in βij(ω) is embed-ded into an equivalent barcode Sijk of a few components.To implement the Λ encoder projectors into hardware, Hy-plex™ uses two different engineering lines (Fig. 3e-f). Whenthe user end task does not require additional constraints, suchas in, e.g. spectral reconstruction, Hyplex™ implements theprojector by utilizing optimization frameworks to minimizethe norm between the physical metasurface response Λ andthe target Λ (Fig. 3e). Conversely, in tasks that requirefurther conditions such as, e.g. hyperspectral semantic seg-mentation, Hyplex™ uses a learnable backbone (Fig. 3f).This optimization exploits d-ALFRED, a new version ofALFRED that creates a differentiable physical model that istrained with an end-to-end approach. d-ALFRED designsmetasurface geometries with an iterative process that mini-mizes the loss function Lseg by optimizing simultaneously

the projector responses Λ and the vector L containing allthe parameters defining the metasurface. A single Hyplex™pixel (Fig. 3g) integrates various metasurface projectors ina two-dimensional array of sub-pixels, which are replicatedin space to form the Hyplex™ hardware encoder (Fig. 3h).The encoder transforms a reflection spectra arising from ascene into a barcode Sij (Fig. 3i), composed of a set of in-tensity signals proportional to the overlap between the inputspectra and each projector’s response as defined in Eq. (1).A standard monochromatic camera, placed behind the meta-surfaces, acts as an imaging readout layer. Each pixel of thecamera matches the sub-pixel of the hardware encoder andretrieves one intensity signal of the barcode Sij (Fig. 3j).

PCA projectors engineered with ALFRED: We use a lin-ear encoder Λ obtained through an unsupervised learningtechnique via principal component analysis (PCA). The PCA

performs hardware encoding E by selecting the k strongest(k = 9 for this work) principal components Λ† from thesingular value decomposition of B = ΛΣV† [8], and ap-proximating B as follows:

B ≈ ΛΣVVV † (2)

Equation (2) offers the closest linear approximation of Bin least square sense. We implement the decoder D with thelinear projector βij = ΛSij , which recovers the best leastsquare approximation of the pixel spectra βij(ω) ≈ βij(ω)(Fig. 3j) from the selected PCA component.

3.2. Learnable backbone via differentiable physicalmodel

In this approach, we represent the decoder operator D asa set of hierarchical nonlinear operators F , which projectthe input tensor S into an output measurement tensor y.This process is iteratively trained via supervised learning,comparing the measurement y with some ground-truth tensory. This end-to-end training finds the optimal feature space Sand the associated linear projectors Λ. To train Hyplex™ inthis framework with backpropagation, the encoder E needsto be differentiable.

In the inverse-design of projectors, the encoder E = H,with H(ω) representing the output transmission functionof the metasurface response, which is obtained from thesolution of the following set of coupled-mode equations [36]:

a(ω) = K

i(ω−W )+ KK†2

s+

s−(ω) = C(ω) ·(s+ − K† · a

) (3)

where W is a diagonal matrix with resonant frequencies ωnof the modes Wnn = ωn, C(ω) is a scattering matrix mod-eling the scattering of impinging waves s+ on the resonatorspace, and K is a coupling matrix representing the inter-action between traveling waves s±(t) and resonator modesa(t). Equations (3) describe the dynamics of a network ofresonator modes a = [a1(ω), . . . , an(ω)], interacting withs± = [s1±(ω), . . . , sm±(ω)] incoming (+) and reflected(−) waves. Section 1 of the Supplementary Material pro-vides more details on the quantities appearing in Eq. (3).

The input-output transfer function H = s−/s+ resultingfrom the solution of Eq. (3) is the superposition of two mainterms: a propagation term defined by the scattering matrixC(ω) and a nonlinear term containing the rational function

Kσ(ω−W ) . Equation (3) represents a differentiable function ofW through which it is possible to backpropagate (Fig. 4 b).d-ALFRED: To project the resonator quantities in Eq. (3)to metasurface input parameters L, we use a supervised opti-mization process. We train a deep neural network to learn therelationship between L and the resonator variables in Eq. (3).Following the same approach of [37], we train the network

Figure 4. Coupled mode network as a differentiable metasur-face physical model. (a) Coupled-mode photonic network as afeedback-loop with skip connection. (b) trainable coupled reso-nance layer. (c) d-ALFRED: trained differentiable projections fromparametric geometry shapes to resonances.

with a supervised spectral prediction task by using arraysof silicon boxes with simulated transmission/reflection re-sponses (see Sec. 2 of Supplementary Material).

4. DatasetsTo train and validate the Hyplex™ system, we use three

publicly available datasets: the CAVE dataset, consistingof 32 indoor images covering 400 nm to 700 nm, and theHarvard and KAUST sets, which contain both indoor andoutdoor scenes, and amount to 75 and 409 images, respec-tively, with spectral bands covering 420 nm to 720 nm and400 nm to 700 nm respectively. We create an additional hy-perspectral dataset FVgNET. FVgNET is comprised of 317scenes showing fruits and vegetables, both natural and artifi-cial, taken indoors under controlled lighting conditions, andcovering the 400 nm to 1000 nm range. We acquired the im-ages using a setup consisting of a white paper sheet arrangedin an infinity curve, a configuration employed in photogra-phy to isolate objects from the background. We achieve goodspectral coverage while minimizing the presence of shadowsin the final images by illuminating the objects with overheadwhite LED indoor lighting, a 150 W halogen lamp (OSL2from Thorlabs) equipped with a glass diffuser and a 100 Wtungsten bulb mounted in a diffuse reflector.

Figure 5a-b shows the distribution of object classes inthe dataset. For each class of objects (e.g., apple, orange,pepper), we generated an approximately equal number ofscenes showing: natural objects only and artificial objectsonly. The dataset consists of 12 classes, represented in theimages proportionally to their chromatic variety. Further-more, we annotated 80% of our images with addititional

Figure 5. Example and statistical analysis on our dataset (a) Overview of the composition of the dataset. There exist a near equal numberof natural and artificial objects in the scenes, 80% of the images are with segmentation masks and the rest with labels only. (b) Distributionof scene objects in classes. Each class has a roughly equal number of instances in the dataset with the exception of apples and peppers, asthey have more chromatic variety. (c) Left: RGB visualization of hyperspectral image. Right: Segmentation mask and labels for each object.

segmentation masks. We incorporate semantic segmenta-tion masks into the dataset by processing the RGB imagesgenerated from the 204 spectral channels. We acquired theimages in such a way to avoid the intersection of objects,allowing for automatic generation of masks for the areasoccupied by each object. We then annotated each markedobject, identifying each object class and whether they arenatural or artificial. Figure 5c illustrates the implementa-tion of the semantic segmentation mask on an image of thedataset. For more details about the FVgNET dataset pleaserefer to Sec. 3 of Supplementary Material.

5. Results

5.1. Hardware implementation

We fabricate arrays of metasurface projectors by pattern-ing thin layers of amorphous silicon deposited on opticalgrade fused silica glass slides. Figure 6a shows a scanningelectron microscope (SEM) image of a manufactured meta-surface pixel, detailing the nanoscale structure of each of thenine projectors. We produce each projector of the 3× 3 sub-array, so it occupies the area of a 2.4 µm wide square, a sizetypical for the pixels present in modern digital camera sen-sors, which allows integration with the camera in the schemeof Fig. 1b. We characterize the optical response of eachprojector by using linearly polarized light with wavelengthsfrom 400 nm to 1000 nm. Figure 3 in the SupplementaryMaterial shows the experimentally measured responses ofthe metasurfaces, illustrating excellent agreement with theexpected theoretical responses. We utilize the fabricatedprojector as a fixed encoder to optimize the reconstructionability of the neural network decoders.

Figure 6. Spectral reconstruction. (a) Scanning electron micro-scope image of the array of projectors. (b) The output of the sceneprocessed by our projectors. (c) Comparison between acquiredand recovered hyperspectral image using the theoretical (middlerow) and experimental (lower row) responses of our projectors. (d)Comparison between the original spectra and their reconstructionusing the projectors and the reconstruction algorithm by Nie etal. [39] for random pixels of the scene in (c).

Model Dataset

CAVE [57] Harvard(out) [11] Harvard(in) [11] KAUST [33] FVgNET

Nguyen et al. [38] 14.91±11.09 9.06±9.69 15.61±8.76 - -Arad and Ben-Shahar [5] 8.84±7.23 14.89±13.23 9.74±7.45 - -Jia et al. [25] 7.92±3.33 8.72 ±7.40 9.50±6.32 - -Nie et al. [39] 4.48 ± 2.97 7.57±4.59 8.88 ± 4.25 - -Hyplex™ 2.05± 1.82 2.13 ± 1.81 6.65 ± 5.88 2.23 ± 3.35 1.73 ± 1.35

Table 1. Comparison of baselines. We report the RMSE from spectral reconstruction in multiple hyperspectral datasets

5.2. Spectral Reconstruction

We perform spectral reconstruction from the barcodes ob-tained from both the theoretical and experimental responsesof the fabricated metasurface projectors. Figure 6b shows ascene from the FVgNET dataset as perceived through eachof the projectors based on experimental data. In Fig. 6c wepresent a qualitative comparison between the hyperspectralreconstruction of this scene based on both the simulatedand experimental barcodes against the original. Figure 6dillustrates a quantitative comparison between the originalspectra and its reconstructions as obtained from the experi-mental implementation Hyplex™ and the algorithm by Nieet al. [39]. The reconstruction is carried out through the useof the connected MLP decoder introduced in Sec. 3. Wedesignate 80% of our dataset for training the decoder andthe remainder for validation purposes.

Table 1 presents a performance comparison of Hyplex™against state-of-the-art reconstruction approaches. Wepresent the results of the reconstruction from the datasetsdescribed in Sec. 4, as well as for the validation part of ourown dataset. For the consistency of the comparison, weadapted the metrics and data reported in [39], where thecalculated RMSE is normalized into the range [0, 255] toapproximately represent the error in pixel intensity. Thereconstruction error of Hyplex™ is the lowest value amongCAVE and both indoor and outdoor images in the Harvarddataset, showing superior performance against all state-of-the-art models. We further tested our model on the KAUSTdataset and FVgNET dataset by using the optical responseof the fabricated metasurfaces.

5.3. Hyperspectral Semantic Segmentation

Here we present labeling of artificial and real fruits fromscenes of the FVgNET dataset. Artificial and real fruits havesimilar RGB colors. However, they differ significantly intheir reflection spectra. Supplementary Fig. 4 provides anexample of this. We showcase the learning ability of theproposed physical encoders by training two classificationnetworks. One model uses the spectral encoders for semanticsegmentation labeling, and the second the RGB channels.Both models use an identical U-Net-like decoder and identi-

RMSE mIoU

Simulation 4.23 0.812Experiment 5.41 0.741

Table 2. Simulation and experiment results. We report RMSEand mIoU seperately for reconstruction and segmentation tasks

Object classHyperspectralsegmentation

RGBsegmentation

IoU F1 IoU F1

real orange 0.979 0.989 0.935 0.966artificial orange 0.954 0.976 0.609 0.757real grape 0.829 0.907 0.009 0.017artificial grape 0.897 0.946 0.494 0.661

Table 3. Quantitative comparison. We report 4 examples ofobject classes segmented with HSI and RGB images.

cal parameters (number of epochs, batch size, learning rate).The results are summarized in Fig. 7, where the panel ashows a qualitative comparison of the segmentation predic-tion quality for both models against the ground-truth mask.

While the mask quality is similar for both methods, themean Intersection over Union (IoU) score for the spectral-informed model is significantly higher compared to the RGBone. The mIoU computed with the theoretical and exper-imental responses of encoders reaches 81%, and 74%, asshown in Tab. 2. With the RGB model, conversely, the mIoUdecreases to 68%. The confusion matrix of the RGB trainedmodel shows that the RGB model struggles to predict correctresults for real-artificial pairs of fruits with similar colors(Fig. 7b). The spectral-informed model, conversely, gener-ates correct labels for most real-artificial pairs (Fig. 7c) andoutperforms the RGB model in IoU and F1 (Tab. 3). Theseresults demonstrate that the small-sized barcodes generatedby Hyplex™ efficiently compress spectral features that con-vey key information about the objects imaged. Table 1 and 2in Supplementary Material provide detailed metrics for eachobject type (apple, potato, etc.) on both models.

Figure 7. Spectral and RGB-based semantic segmentations. (a) Comparison between segmentation masks generated from a spectral-informed model, an RGB-only model, and the ground truth. (b) Confusion matrix for RGB only model. (c) Confusion matrix for thespectral-informed model. Each value in the confusion matrix represents the number of pixels of the segmentation mask of the item in thecolumn that was classified as the item in the row.

6. Discussion and Limitations

In this work, we designed and implemented Hyplex™, anew hardware system for real-time and high-resolution hy-perspectral imaging. We validated Hyplex™ against currentstate-of-the-art approaches and proved it to be outperformingin all benchmarks. Additionally, we demonstrated the supe-riority of hyperspectral features and trainable encoders bydesigning a model for spectral-informed semantic segmenta-tion and comparing its performance against RGB models.

One of the limitations in the current implementation ofHyplex™ is the linear structure of the physical encoder. Thestudy of nonlinear encoders [3] could enable more com-plex feature embeddings. This topic may stimulate futureresearch that could generalize the Hyplex™ framework toinclude nonlinear metasurfaces, an essential area of researchin the field of meta-optics [30, 35]. The second area of im-provement is the spectral sparsity assumption at the coreidea of efficient dimensionality reduction. While this as-

sumption is practically verified in the majority of computervision problems [43,61], it may not hold for specialized tasks.Fabrication errors are also an essential aspect that, if not ad-equately considered, can limit performance. In this work,we mitigate this effect by tuning the software decoder tobest use the experimental response of the projectors. Futurework could investigate techniques from robustness controlin inverse design, a new promising area of research [31, 37].

Improved results could also be obtained if we augment thepublicly available hyperspectral datasets with more scenesobtained at different wavelengths and in different settingssuch as, e.g., medical. Such study could generalize the re-sults of Hyplex™ to provide high impact systems for person-alized healthcare and precision medicine. Hyplex™ couldprovide a game-changer technology in this field, leveragingits vast capacity to fast-process high-resolution hyperspec-tral images (see Sec. 7 of Supplementary Material) at speedcomparable with current RGB cameras.

Acknowledgements. This work was supported by the King

Abdullah University of Science and Technology (KAUST)through the Artificial Intelligence Initiative (AII) funding.This research received funding from KAUST (Award OSR-2016-CRG5-2995). Parallel simulations are performed onKAUST’s Shaheen supercomputer.

References[1] M.A. Afromowitz, J.B. Callis, D.M. Heimbach, L.A. DeSoto,

and M.K. Norton. Multispectral imaging of burn wounds:A new clinical instrument for evaluating burn depth. IEEETransactions on Biomedical Engineering, 35(10):842–850,Oct. 1988. 1

[2] Airobot. Hyperspectral drone and software for agriculture,2021. https://airobot.eu/solutions/hyperspectral-drone-and-software-for-agriculture/. 1

[3] Xavier Alameda-Pineda, Elisa Ricci, Yan Yan, and Nicu Sebe.Recognizing emotions from abstract paintings using non-linear matrix completion. In Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR),June 2016. 8

[4] Aitor Alvarez-Gila, Joost Van De Weijer, and Estibaliz Gar-rote. Adversarial networks for spatial context-aware spectralimage reconstruction from rgb. 2017 IEEE International Con-ference on Computer Vision Workshops (ICCVW), Oct 2017.2

[5] Boaz Arad and Ohad Ben-Shahar. Sparse recovery of hy-perspectral signal from natural rgb images. In EuropeanConference on Computer Vision, pages 19–34. Springer, 2016.2, 7

[6] B. Arad and O. Ben-Shahar. Filter selection for hyperspectralestimation. 2017 IEEE International Conference on Com-puter Vision (ICCV), pages 3172–3180, 2017. 2

[7] Arad B. and Ben-Shahar O. Sparse recovery of hyperspectralsignal from natural rgb images. In Leibe B., Matas J., SebeN., and Welling M, editors, Computer Vision – ECCV 2016.ECCV 2016. Lecture Notes in Computer Science, volume9911. Springer, Cham, 2016. 2

[8] Christopher M. Bishop. Pattern Recognition and MachineLearning (Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg, 2006. 3, 5

[9] Marcella Bonifazi, Valerio Mazzone, Ning Li, Yi Tian, andAndrea Fratalocchi. Free-Electron Transparent Metasurfaceswith Controllable Losses for Broadband Light Manipulationwith Nanometer Resolution. Advanced Optical Materials,8(1):1900849, 2020. 2

[10] Renfu Lu Bosoon Park. Hyperspectral Imaging Technologyin Food and Agriculture. Springer, 2015. 1

[11] A. Chakrabarti and T. Zickler. Statistics of Real-World Hy-perspectral Images. In Proc. IEEE Conf. on Computer Visionand Pattern Recognition (CVPR), pages 193–200, 2011. 7

[12] Wei Ting Chen, Alexander Y Zhu, Vyshakh Sanjeev, Moham-madreza Khorasaninejad, Zhujun Shi, Eric Lee, and FedericoCapasso. A broadband achromatic metalens for focusing andimaging in the visible. Nature Nanotechnology, 13(3):220–226, Jan. 2018. 2

[13] Arjun Chennu, Paul Farber, Glenn De’ath, Dirk de Beer,and Katharina E. Fabricius. A diver-operated hyperspectralimaging and topographic surveying system for automatedmapping of benthic habitats. Scientific Reports, 7(1):7122,Aug. 2017. 1

[14] Ines Dumke, Autun Purser, Yann Marcon, Stein M. Nornes,Geir Johnsen, Martin Ludvigsen, and Fredrik Søreide. Un-derwater hyperspectral imaging as an in situ taxonomic tool

for deep-sea megafauna. Scientific Reports, 8(1):12860, Aug.2018. 1

[15] David H Foster, Kinjiro Amano, Sergio MC Nascimento, andMichael J Foster. Frequency of metamerism in natural scenes.Josa a, 23(10):2359–2372, 2006. 2

[16] Andrea Fratalocchi, Fedor Getman, Maksim Makarenko, andArturo Burguete-Lopez. Flat optics polarizer beam splitter. 2

[17] Henning Galinski, Gael Favraud, Hao Dong, Juan S. ToteroGongora, Gregory Favaro, Max Dobeli, Ralph Spolenak,Andrea Fratalocchi, and Federico Capasso. Scalable, ultra-resistant structural colors based on network metamaterials.Light: Science & Applications, 6(5):e16233–e16233, May2017. 2

[18] Silvano Galliani, Charis Lanaras, Dimitrios Marmanis, Em-manuel Baltsavias, and Konrad Schindler. Learned spectralsuper-resolution. arXiv preprint arXiv:1703.09470, 03 2017.2

[19] F. Getman, M. Makarenko, A. Burguete-Lopez, and A. Frat-alocchi. Broadband vectorial ultrathin optics with experimen-tal efficiency up to 99% in the visible region via universalapproximators. Light: Science & Applications, 10(1):1–14,Mar. 2021. 2, 3

[20] Fedor Getman, Maksim Makarenko, Arturo Burguete-Lopez,and Andrea Fratalocchi. Broadband vectorial ultrathin opticswith experimental efficiency up to 99% in the visible regionvia universal approximators. Light: Science & Applications,10(1):1–14, 2021. 3

[21] Aoife A. Gowen, Yaoze Feng, Edurne Gaston, and VasilisValdramidis. Recent applications of hyperspectral imaging inmicrobiology. Talanta, 137:43–54, May 2015. 1

[22] Qiong He, Shulin Sun, Shiyi Xiao, and Lei Zhou. High-efficiency metasurfaces: Principles, realizations, and appli-cations. Advanced Optical Materials, 6(19):1800415, 2018.2

[23] Noor A Ibraheem, Mokhtar M Hasan, Rafiqul Z Khan, andPramod K Mishra. Understanding color models: a review.ARPN Journal of science and technology, 2(3):265–275, 2012.2

[24] Daniel S. Jeon, Seung-Hwan Baek, Shinyoung Yi, QiangFu, Xiong Dun, Wolfgang Heidrich, and Min H. Kim. Com-pact snapshot hyperspectral imaging with diffracted rotation.ACM Transactions on Graphics (Proc. SIGGRAPH 2019),38(4):117:1–13, 2019. 2

[25] Yan Jia, Yinqiang Zheng, Lin Gu, Art Subpa-Asa, AntonyLam, Yoichi Sato, and Imari Sato. From rgb to spectrum fornatural scenes via manifold-based mapping. In Proceedingsof the IEEE International Conference on Computer Vision(ICCV), Oct 2017. 2, 7

[26] Jun Jiang, Dengyu Liu, Jinwei Gu, and S. Susstrunk. Whatis the space of spectral sensitivity functions for digital colorcameras? 2013 IEEE Workshop on Applications of ComputerVision (WACV), pages 168–179, 2013. 2

[27] Rei Kawakami, Yasuyuki Matsushita, John Wright, MosheBen-Ezra, Yu-Wing Tai, and Katsushi Ikeuchi. High-resolution hyperspectral imaging via matrix factorization. InCVPR 2011, pages 2329–2336. IEEE, 2011. 2

[28] Nasser Kehtarnavaz and Mark Gamadia. Real-Time Imageand Video Processing: From Research to Reality. Synthesis

Lectures on Image, Video, and Multimedia Processing, 2(1):1–108, Jan. 2006. 2

[29] Muhammad Jaleed Khan, Hamid Saeed Khan, Adeel Yousaf,Khurram Khurshid, and Asad Abbas. Modern trends in hy-perspectral image analysis: A review. IEEE Access, 6:14118–14129, 2018. 1

[30] Yuri Kivshar. All-dielectric meta-optics and non-linearnanophotonics. National Science Review, 5(2):144–158, Mar2018. 8

[31] Julius Kuhne, Juan Wang, Thomas Weber, Lucca Kuhner,Stefan A Maier, and Andreas Tittl. Fabrication robustness inbic metasurfaces. Nanophotonics, 2021. 8

[32] Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Y Ng.Efficient sparse coding algorithms. In Advances in neuralinformation processing systems, pages 801–808, 2007. 2

[33] Yuqi Li, Qiang Fu, and Wolfgang Heidrich. Multispectralillumination estimation using deep unrolling network. In 2021IEEE International Conference on Computer Vision(ICCV),pages 1–8. IEEE, 2021. 7

[34] Guolan Lu and Baowei Fei. Medical hyperspectral imaging:A review. Journal of Biomedical Optics, 19(1):010901, Jan.2014. 1

[35] Stefan A. Maier. Dielectric and low-dimensional-materialsnanocavities for non-linear nanophotonics and sensing. In Ad-vanced Photonics 2018 (BGPP, IPR, NP, NOMA, Sensors,Networks, SPPCom, SOF) (2018), paper SeW2E.4, pageSeW2E.4. Optical Society of America, Jul 2018. 8

[36] M. Makarenko, A. Burguete-Lopez, F. Getman, and A. Frat-alocchi. Generalized maxwell projections for multi-modenetwork photonics. Scientific Reports, 10(1):9038, Dec 2020.3, 5

[37] Maksim Makarenko, Qizhou Wang, Arturo Burguete-Lopez,Fedor Getman, and Andrea Fratalocchi. Robust and scalableflat-optics on flexible substrates via evolutionary neural net-works. Advanced Intelligent Systems, page 2100105, Aug2021. 2, 5, 8

[38] Rang MH Nguyen, Dilip K Prasad, and Michael S Brown.Training-based spectral reconstruction from a single rgb im-age. In European Conference on Computer Vision, pages186–201. Springer, 2014. 2, 7

[39] Shijie Nie, Lin Gu, Yinqiang Zheng, Antony Lam, NobutakaOno, and Imari Sato. Deeply learned filter response functionsfor hyperspectral reconstruction. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition,pages 4767–4776, 2018. 2, 3, 6, 7

[40] Seoung Wug Oh, Michael S Brown, Marc Pollefeys, andSeon Joo Kim. Do it yourself hyperspectral imaging witheveryday digital cameras. In Proceedings of the IEEE Con-ference on Computer Vision and Pattern Recognition, pages2461–2469, 2016. 2

[41] Yaniv Oiknine, Isaac August, Vladimir Farber, DanielGedalin, and Adrian Stern. Compressive Sensing Hyperspec-tral Imaging by Spectral Multiplexing with Liquid Crystal.Journal of Imaging, 5(1):3, Jan. 2019. 2

[42] Svetlana V. Panasyuk, Shi Yang, Douglas V. Faller, DuyenNgo, Robert A. Lew, Jenny E. Freeman, and Adrianne E.

Rogers. Medical hyperspectral imaging to facilitate resid-ual tumor identification during surgery. Cancer Biology &Therapy, 6(3):439–446, Mar. 2007. 1

[43] Arun P.V., Krishna Mohan B., and Porwal A. Spatial-spectralfeature based approach towards convolutional sparse codingof hyperspectral images. Computer Vision and Image Under-standing, 188:102797, 2019. 8

[44] Antonio Robles-Kelly. Single image spectral reconstructionfor multimedia applications. In Proceedings of the 23rdACM International Conference on Multimedia, MM ’15, page251–260, New York, NY, USA, 2015. Association for Com-puting Machinery. 2

[45] Antonio Robles-Kelly. Single image spectral reconstructionfor multimedia applications. In Proceedings of the 23rdACM international conference on Multimedia, pages 251–260, 2015. 2

[46] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net:Convolutional networks for biomedical image segmentation.In International Conference on Medical image computingand computer-assisted intervention, pages 234–241. Springer,2015. 2

[47] Hui-Liang Shen, Jian-Fan Yao, Chunguang Li, Xin Du, Si-JieShao, and John H Xin. Channel selection for multispectralcolor imaging using binary differential evolution. Appliedoptics, 53(4):634–642, 2014. 2

[48] Andreas Tittl, Aurelian John-Herpin, Aleksandrs Leitis, Ed-uardo R. Arvelo, and Hatice Altug. Metasurface-BasedMolecular Biosensing Aided by Artificial Intelligence. Ange-wandte Chemie International Edition, 58(42):14810–14822,Oct. 2019. 2

[49] Andreas Tittl, Aleksandrs Leitis, Mingkai Liu, Filiz Yesilkoy,Duk-Yong Choi, Dragomir N Neshev, Yuri S Kivshar, andHatice Altug. Imaging-based molecular barcoding with pixe-lated dielectric metasurfaces. Science, 360(6393):1105–1109,2018. 2

[50] Andreas Tittl, Aleksandrs Leitis, Mingkai Liu, Filiz Yesilkoy,Duk-Yong Choi, Dragomir N. Neshev, Yuri S. Kivshar, andHatice Altug. Imaging-based molecular barcoding with pixe-lated dielectric metasurfaces. Science, 360(6393):1105–1109,2018. 2

[51] Ethan Tseng, Shane Colburn, James Whitehead, LuochengHuang, Seung-Hwan Baek, Arka Majumdar, and Felix Heide.Neural nano-optics for high-quality thin lens imaging. Naturecommunications, 12(1):1–7, 2021. 3

[52] Qizhou Wang, Maksim Makarenko, Arturo Burguete Lopez,Fedor Getman, and Andrea Fratalocchi. Advancing statisticallearning and artificial intelligence in nanophotonics inversedesign. Nanophotonics, 2021. 3

[53] Renjie Wu, Yuqi Li, Xijiong Xie, and Zhijie Lin. Optimizedmulti-spectral filter arrays for spectral reconstruction. Sensors,19(13):2905, 2019. 2

[54] Zhiwei Xiong, Zhan Shi, Huiqun Li, Lizhi Wang, Dong Liu,and Feng Wu. Hscnn: Cnn-based hyperspectral image recov-ery from spectrally undersampled projections. 2017 IEEEInternational Conference on Computer Vision Workshops (IC-CVW), pages 518–525, 2017. 2

[55] Zhiwei Xiong, Zhan Shi, Huiqun Li, Lizhi Wang, Dong Liu,and Feng Wu. Hscnn: Cnn-based hyperspectral image recov-ery from spectrally undersampled projections. In Proceedingsof the IEEE International Conference on Computer VisionWorkshops, pages 518–525, 2017. 2

[56] Liutao Yang, Zhongnian Li, Zongxiang Pei, and DaoqiangZhang. Fs-net: Filter selection network for hyperspectralreconstruction. In 2021 IEEE International Conference onImage Processing (ICIP), pages 2933–2937. IEEE, 2021. 3

[57] F. Yasuma, T. Mitsunaga, D. Iso, and S.K. Nayar. GeneralizedAssorted Pixel Camera: Post-Capture Control of Resolution,Dynamic Range and Spectrum. Technical report, ColumbiaUniversity, Nov 2008. 7

[58] Seung Chul Yoon, Bosoon Park, Kurt C. Lawrence, William R.Windham, and Gerald W. Heitschmidt. Line-scan hyper-spectral imaging system for real-time inspection of poultrycarcasses with fecal material and ingesta. Computers andElectronics in Agriculture, 79(2):159–168, Nov. 2011. 2

[59] Jun Yu, Toru Kurihara, and Shu Zhan. Optical filter net:A spectral-aware rgb camera framework for effective greenpepper segmentation. IEEE Access, 9:90142–90152, 2021. 3

[60] Nanfang Yu and Federico Capasso. Flat optics with designermetasurfaces. Nature Materials, 13(2):139–150, Feb. 2014. 2

[61] Lei Zhang, Wei Wei, Yanning Zhang, Chunhua Shen, Antonvan den Hengel, and Qinfeng Shi. Cluster sparsity field:An internal hyperspectral imagery prior for reconstruction.International Journal of Computer Vision, 126(8):797–821,Aug 2018. 8

[62] Wenyi Zhang, Hongya Song, Xin He, Longqian Huang, XiyueZhang, Junyan Zheng, Weidong Shen, Xiang Hao, and Xu Liu.Deeply learned broadband encoding stochastic hyperspectralimaging. Light: Science & Applications, 10(1):1–7, 2021. 3

[63] Yuzhi Zhao, Lai-Man Po, Qiong Yan, Wei Liu, and TingyuLin. Hierarchical regression network for spectral reconstruc-tion from rgb images. In Proceedings of the IEEE/CVF Con-ference on Computer Vision and Pattern Recognition Work-shops, pages 422–423, 2020. 2

Supplementary information for: Real-time hyperspectral imaging in hardwarevia trained metasurface encoders

M. Makarenko†, A. Burguete-Lopez, Q. Wang, F. Getman, Silvio Giancola,Bernard Ghanem & A. Fratalocchi

King Abdullah University of Science and Technology (KAUST)Thuwal, 23955-6900, KSA

†maksim.makarenko@kaust.edu.sa

1. Details on time-domain coupled-mode the-ory TDCMT

Figure 1. Splitting of metasurface geometry space Ω into resonancespace Ωr and exterior (propagation) space Ωe

In this section, we describe a set of exact coupled-modeequations that are fully equivalent to Maxwell’s equations.The main idea of the coupled-mode approach is to divide thegeometrical space Ω where light propagation into a resonatorspace Ωr and an external space Ωe (Fig. 1). We assume thatthe external space does not contain sources or charges. Underthis formulation, the set of Maxwell equations reduces to thefollowing set of exact coupled-mode equations [?]:

a(ω) = K

i(ω−W )+ KK†2

s+

s−(ω) = C(s+ − K† · a

) (1)

with 1/X the inverse matrix X−1. Power conservation im-plies that the matrix σ :

σ = 1− K1

i(ω −W ) + KK†2

K† (2)

defined from the solution of the coupled mode equations, isunitary σ† · σ = 1.

Equations (1) show that the dynamics of the system de-pend only on three independent matrices: the coupling ma-trix K, the scattering matrix C, and the resonance matrixW .

2. Network training for supervised spectralprediction

Figure 2. Differentiable ALFRED spectral predictor. a Conceptualsketch of d-alfred neural network shape-to-resonance mapper. bQualitative results of spectra prediction for the dataset samples.

Figure 2 illustrates the model of the proposed differen-tiable spectral predictor (d-ALFRED). It consists of severalfully connected (FC) blocks connected sequentially. EachFC block consists of multi-layer perceptrons (MLP) of differ-ent sizes, batch normalization layer, and dropout. To processseparately categorical variables in input (period, thickness),we design an alternative branch consisting of a linear em-bedding layer and FC block connected sequentially. Theprimary purpose of this branch is to balance categorical andcontinuous variable’s weights in the model. Then we con-catenate both continuous and categorical features and feed

1

arX

iv:2

204.

0208

4v1

[cs

.CV

] 5

Apr

202

2

them into readout blocks, consisting of multiple FC blocks.We use the training dataset provided by [?], which con-

tains over 600 000 simulation results of pure silicon struc-tures on top of glass under a Total-Field Scattered-Field(TFSF) simulation. Each simulation has periodic boundaryconditions with one of the three different periods (250 nm,500 nm or 750 nm) and one of the ten different discretethicknesses from 50 to 300 nm with a 25 nm step. Eachstructure consists of a random combination (up to 5) ofcuboid resonators. We split the dataset into test and trainingparts comprising 20% and 80% of the total, respectively,then take 10% of the training set as a validation set.

For the training part, we use the Adam optimizer [?] witha learning rate 1 × 10−5 and a step learning rate schedulerwith stepsize = 50 and γ = 0.1 hyperparameters. Forthe desired system response in either transmission or re-flection, we apply a sigmoid activation function at the toplayer of FCN. This function maps the output spectrum tothe range [0,1], which is beneficial for convergence at thebeginning of the training stage. Since we use periodic bound-ary conditions, we used random translation and rotations asdata-augmentation.

As a result, we obtain 0.008 validation mean squared er-ror, which is slightly higher than the previous result withconvolution-based model [?]. Figure 2 provides a qualita-tive comparison between trained and ground truth spectralresponses.

3. Dataset detailsIn the main dataset we provide hyperspectral images of

real and artificial fruits. The miscellaneous class correspondsto fruits and vegetables that do not have a natural counterpart.Approximately 40% of the scenes consist of a single rowof objects located at the camera’s focal plane. The remain-ing scenes show two rows of objects, with the focal planelocated in between. We keep the position of the white ref-erence panel approximately constant throughout the datasetfor easy normalization. The hyperspectral images have aspatial resolution of 512×512 pixels and 204 spectral bands.We also provide an RGB image as seen through the lens ofthe camera for each scene with the same spatial resolution.

To validate the generalization ability of our framework,we augmented the dataset with 20 additional images in thewild (examples can be seen in Fig. 3). The resulting re-construction error for these images is 2.54 ± 2.72, a valueconsistent with the results obtained with the KAUST datasetused to train the encoder.

4. Nanofabrication detailsWe produce the devices using 15 mm wide and 0.5 mm

thick square pieces of fused silica glass as the substrate. Us-ing plasma-enhanced vapor deposition, we deposit a thin

Figure 3. Additional samples captured in a real-world setting.

layer of amorphous silicon on the glass, the thickness ofwhich is controlled on each sample to match the designrequirements. We then spin coat 200 nm of the resist ZEP-520A (ZEON corporation and 40 nm of the resist AR-PC5090 (ALLRESIST) and pattern the shapes of the nanos-tructures using an electron beam lithography system with a100 kV acceleration voltage. Following this, we remove theAR-PC 5090 by submersing each sample for 60 s in deion-ized water. We develop the samples by submerging themin ZED-50 (ZEON corporation) for 90 s and rinse for 60 sin isopropyl alcohol. We then deposit 22 nm of chromiumusing electron beam evaporation to create a hard mask andperform liftoff followed by ultrasonic agitation for 1 min.Following this, we remove the unprotected silicon using re-active ion etching, submerge the devices in TechniEtch Cr01(Microchemicals) for 30 s to remove the metal mask, andrinse with deionized water to obtain the final device.

5. CharacterizationWe measure the spectral response of our devices in trans-

mission. For accurate characterization, we fabricate eachfilter separately as a uniform square 500 µm wide. A setupwith two 10x microscope objectives allows us to focus broad-band (400 nm-1000 nm) linearly polarized light on our sam-ples. A spectrometer then processes the transmitted light.The resulting transmission curves, and their comparison tothe theoretical ones, are shown in Fig. 4.

6. Additional resultsIn this section, we provide additional computational re-

sults. Fig. 6 provides additional qualitative comparisonsbetween RGB trained and Hyplex™ model on the segmen-tation quality on the FVgNET dataset. Fig. 7 illustratesreconstruction efficiency in simulations of Hyplex™ on thesamples from KAUST dataset.

7. Real-time processingThe “first layer” of the learning model is purely opti-

cal and acquires data at the speed of light. Therefore, the

data acquisition speed of Hyplex™ is limited only by thesensor frame rate (30 FPS in this work). For real-time clas-sification/segmentation tasks, the remaining layers of thenetwork will create delays between the real-time process-ing of the hyperspectral images and the output for the task.We chose a shallow network implemented in a GPU in thiswork, which resulted in real-time (> 20 FPS) processing.For better validation purposes, we match the specificationsof the dataset we used in training and designed the system towork from 400 nm to 700 nm with 10 nm spectral resolutionand 512 × 512 spatial resolution. In general, our spectralresolution can achieve up to 2 nm, covering the wavelengthrange from 400 nm to 700 nm. Using a high-resolution cam-era sensor currently available in the market (> 12 MP),we could produce a > 2 MP hyperspectral camera with anacquisition speed close to 1 Tb/s. We will provide theseadditional details in the suppl. material.

.

Validation stats IoU F1 Prec recall Acc

background 0.9940 0.9970 0.9966 0.9974 0.9948real potato 0.9626 0.9809 0.9877 0.9743 0.9999artificial potato 0.9655 0.9824 0.9877 0.9772 0.9999real apple 0.9522 0.9754 0.9645 0.9867 0.9997artificial apple 0.9021 0.9485 0.9714 0.9267 0.9988real orange 0.9786 0.9891 0.9948 0.9836 0.9999artificial orange 0.9541 0.9764 0.9866 0.9666 0.9996real grape 0.8294 0.9067 0.8530 0.9677 0.9978artificial grape 0.8971 0.9457 0.9331 0.9588 0.9990real lemons 0.8673 0.9289 0.9881 0.8764 0.9992artificial lemons 0.0009 0.0018 0.7143 0.0009 0.9968real avocado 0.0365 0.0704 1.0000 0.0365 0.9981artificial avocado 0.9436 0.9709 0.9764 0.9656 0.9999real pepper 0.9586 0.9788 0.9808 0.9769 0.9993artificial pepper 0.9426 0.9704 0.9583 0.9828 0.9996real unknown 0.9243 0.9606 0.9491 0.9726 0.9980artificial unknown 0.7016 0.8246 0.7215 0.9621 0.9957

total 0.8124 0.8476 0.9391 0.8537 0.9986total(-background) 0.8011 0.8382 0.9355 0.8447 0.9988

Table 1. Validation stats on spectral segmentation

Validation stats IoU F1 Prec recall Acc

background 0.9950 0.9975 0.9968 0.9983 0.9957real potato 0.9688 0.9841 0.9980 0.9707 0.9999artificial potato 0.0000 0.0000 0.0000 0.0000 0.9961real apple 0.9469 0.9727 0.9860 0.9598 0.9993artificial apple 0.9186 0.9575 0.9301 0.9867 0.9992real orange 0.9351 0.9664 0.9940 0.9404 0.9994artificial orange 0.6094 0.7572 0.6239 0.9633 0.9959real grape 0.0088 0.0174 0.9949 0.0088 0.9917artificial grape 0.4935 0.6608 0.5015 0.9687 0.9904real lemons 0.8241 0.9035 0.8387 0.9794 0.9997artificial lemons 0.0000 0.0000 0.0000 0.0000 0.9973real avocado 0.5467 0.7069 0.9891 0.5500 0.9992artificial avocado 0.9708 0.9851 0.9992 0.9716 0.9999real pepper 0.9068 0.9511 0.9954 0.9107 0.9988artificial pepper 0.8945 0.9443 0.9081 0.9836 0.9988real unknown 0.9524 0.9756 0.9762 0.9751 0.9989artificial unknown 0.7274 0.8421 0.7605 0.9435 0.9967

total 0.6882 0.7425 0.7937 0.7712 0.9975total(-background) 0.6690 0.7265 0.7810 0.7570 0.9976

Table 2. Validation stats on RGB segmentation

400 500 600 700 800 900 1000Wavelength (nm)

0.0

0.2

0.4

0.6

0.8

1.0

Tran

smiss

ion

Filter 1

ExperimentTheory

400 500 600 700 800 900 1000Wavelength (nm)

0.0

0.2

0.4

0.6

0.8

1.0

Tran

smiss

ion

Filter 2

ExperimentTheory

400 500 600 700 800 900 1000Wavelength (nm)

0.0

0.2

0.4

0.6

0.8

1.0

Tran

smiss

ion

Filter 3

ExperimentTheory

400 500 600 700 800 900 1000Wavelength (nm)

0.0

0.2

0.4

0.6

0.8

1.0

Tran

smiss

ion

Filter 4ExperimentTheory

400 500 600 700 800 900 1000Wavelength (nm)

0.0

0.2

0.4

0.6

0.8

1.0

Tran

smiss

ion

Filter 5ExperimentTheory

400 500 600 700 800 900 1000Wavelength (nm)

0.0

0.2

0.4

0.6

0.8

1.0

Tran

smiss

ion

Filter 6ExperimentTheory

400 500 600 700 800 900 1000Wavelength (nm)

0.0

0.2

0.4

0.6

0.8

1.0

Tran

smiss

ion

Filter 7ExperimentTheory

400 500 600 700 800 900 1000Wavelength (nm)

0.0

0.2

0.4

0.6

0.8

1.0

Tran

smiss

ion

Filter 8

ExperimentTheory

400 500 600 700 800 900 1000Wavelength (nm)

0.0

0.2

0.4

0.6

0.8

1.0

Tran

smiss

ion

Filter 9ExperimentTheory

Figure 4. Comparison between theoretical and experimental response of designed filters.

Figure 5. Comparison between spectral responses and RGB images of real and artificial grapes.a Reflection spectra of real and artificialgrapes. b RGB image of real grapes. c RGB image of artificial grapes.

Figure 6. Comparison between RGB and spectral-informed models on semantic fruit segmentation task

Figure 7. Simulation reconstruction results on KAUST dataset. a Image spectral recovery at different wavelengths. b Simulated barcode ofthe scene as it would be perceived by Hyplex™ through each of the nine projectors.