Optical Satellite Data Compression and Implementation

410 Pages·2013·24.701 MB·English

by  Shen-en Qian

This book provides a global review of optical satellite image and data
compression theories, algorithms, and system implementations. Consisting of
nine chapters, it describes a variety of lossless and near-lossless
data-compression techniques and three international satellite-data-compression
standards. The author shares his firsthand experience and research results in
developing novel satellite-data-compression techniques for both onboard and
on-ground use, user assessments of the impact that data compression has on
satellite data applications, building hardware compression systems, and
optimizing and deploying systems. Written with both postgraduate students and
advanced professionals in mind, this handbook addresses important issues of
satellite data compression and implementation, and it presents an end-to-end
treatment of data compression technology.
WWant to learn more about the calibration and
eenhancement of spaceborne optical sensors and
mmethods for image enhancement and fusion? SPIE
PPress presents this book's companion text, Optical
SSatellite Signal Processing and Enhancement, also
wwritten by Shen-En Qian.
QIAN
P.O. Box 10
Bellingham, WA 98227-0010
ISBN: 9780819497871
SPIE Vol. No.: PM241

Optical
Satellite
Data Compression
and Implementation
Shen-En Qian
SPIE PRESS
Bellingham, Washington USA
Library of Congress Cataloging-in-Publication Data
Qian, Shen-En.
Optical satellite data compression and implementation / Shen-En Qian.
pages cm
Includes bibliographical references and index.
ISBN 978-0-8194-9787-1
1. Datacompression(Computerscience). 2. Imagingsystems Imagequality.
3. Signalprocessing. 4. Codingtheory. 5. Opticalimages. I. Title.
QA76.9.D33 2013
629.43'7 dc23
2013944363
Published by
SPIE The International Society for Optical Engineering
P.O. Box 10
Bellingham, Washington 98227-0010 USA
Phone: [phone]
Fax: [phone]
Email: [e-mail]
Web: http://spie.org
Copyright © 2013 Society of Photo-Optical Instrumentation Engineers (SPIE)
All rights reserved. No part of this publication may be reproduced or distributed in
any form or by any means without written permission of the publisher.
Thecontentofthisbookreflectstheworkandthoughtoftheauthor(s).Everyefforthas
beenmadetopublishreliableandaccurateinformationherein,butthepublisherisnot
responsible for the validity of the information or for any outcomes resulting from
reliance thereon.
Printed in the United States of America.
First printing

Contents
Preface xiii
List of Terms and Acronyms xvii
1 Needs for Data Compression and Image Quality Metrics 1
1.1 Needs for Satellite Data Compression 1
1.2 Quality Metrics of Satellite Images 4
1.3 Full-Reference Metrics 5
1.3.1 Conventional full-reference metrics 6
1.3.1.1 Mean-square error (MSE) 6
1.3.1.2 Relative-mean-square error (ReMSE) 7
1.3.1.3 Signal-to-noise ratio (SNR) 7
1.3.1.4 Peak signal-to-noise ratio (PSNR) 7
1.3.1.5 Maximum absolute difference (MAD) 7
1.3.1.6 Percentage maximum absolute difference (PMAD) 8
1.3.1.7 Mean absolute error (MAE) 8
1.3.1.8 Correlation coefficient (CC) 8
1.3.1.9 Mean-square spectral error (MSSE) 9
1.3.1.10 Spectral correlation (SC) 9
1.3.1.11 Spectral angle (SA) 9
1.3.1.12 Maximum spectral information divergence (MSID) 10
1.3.1.13 ERGAS for multispectral image after pan-sharpening 10
1.3.2 Perceived-visual-quality-based full-reference metrics 11
1.3.2.1 Universal image-quality index 11
1.3.2.2 Multispectral image-quality index 12
1.3.2.3 Quality index for multi- or hyperspectral images 14
1.3.2.4 Structural similarity index 15
1.3.2.5 Visual information fidelity 17
1.4 Reduced-Reference Metrics 18
1.4.1 Four RR metrics for spatial-resolution-enhanced images 20
1.4.2 RR metric using the wavelet-domain natural-image statistic
model 22
1.5 No-Reference Metrics 24
1.5.1 Statistic-based methods 24
v
vi Contents
1.5.1.1 Entropy 24
1.5.1.2 Energy compaction 25
1.5.1.3 Coding gain 25
1.5.2 NR metric for compressed images using JPEG 26
1.5.3 NR metric for pan-sharpened multispectral image 27
1.5.3.1 Spectral distortion index 28
1.5.3.2 Spatial distortion index 29
1.5.3.3 Jointly spectral and spatial quality index 29
References 29
2 Lossless Satellite Data Compression 33
2.1 Introduction 33
2.2 Review of Lossless Satellite Data Compression 35
2.2.1 Prediction-based methods 35
2.2.2 Transform-based methods 38
2.3 Entropy Encoders 40
2.3.1 Adaptive arithmetic coding 40
2.3.2 Golomb coding 41
2.3.3 Exponential-Golomb coding 42
2.3.4 Golomb power-of-two coding 42
2.4 Predictors for Hyperspectral Datacubes 44
2.4.1 1D nearest-neighboring predictor 45
2.4.2 2D/3D predictors 45
2.4.3 Predictors within a focal plane image 45
2.4.4 Adaptive selection of predictor 47
2.4.5 Experimental results of the predictors 48
2.4.5.1 Compressionresultsusingfixedcoefficientpredictors 49
2.4.5.2 Compression results using variable coefficient
predictors 50
2.4.5.3 Compression results using adaptive selection of
predictor 51
2.5 Lookup-Table-Based Prediction Methods 53
2.5.1 Single-lookup-table prediction 53
2.5.2 Locally averaged, interband-scaling LUT prediction 54
2.5.3 Quantized-index LUT prediction 56
2.5.4 Multiband LUT prediction 56
2.6 Vector-Quantization-Based Prediction Methods 57
2.6.1 Linear prediction 57
2.6.2 Grouping based on bit-length 58
2.6.3 Vector quantization with precomputed codebooks 58
2.6.4 Optimal bit allocation 59
2.6.5 Entropy coding 59
2.7 Band Reordering 60
2.8 Transform-Based Lossless Compression Using the KLT and DCT 61
Contents vii
2.9 Wavelet-Transform-Based Methods 62
2.9.1 Wavelet decomposition structure 63
2.9.2 Lossy-to-lossless compression: 3D set-partitioning
embedded block 63
2.9.3 Lossy-to-lossless compression: 3D embedded
zeroblock coding 66
References 68
3 International Standards for Spacecraft Data Compression 75
3.1 CCSDS and Three Data Compression Standards 75
3.2 Lossless Data Compression Standard 76
3.2.1 Preprocessor 76
3.2.2 Adaptive entropy encoder 78
3.2.2.1 Variable-length coding 78
3.2.2.2 Coding options 80
3.2.2.3 Coded dataset format 81
3.2.3 Performance evaluation 81
3.2.3.1 1D data: Goddard High-Resolution Spectrometer 82
3.2.3.2 1D data: Acousto-Optical Spectrometer 83
3.2.3.3 1D data: Gamma-Ray Spectrometer 83
3.2.3.4 2D image: Landsat Thematic Mapper 84
3.2.3.5 2D image: Heat-Capacity-Mapping Radiometer 84
3.2.3.6 2D image: Wide-Field Planetary Camera 85
3.2.3.7 2D image: Soft X-Ray Solar Telescope 85
3.2.3.8 3D image: hyperspectral imagery 85
3.3 Image Data Compression Standard 86
3.3.1 Features of the standard 86
3.3.2 IDC compressor 87
3.3.3 Selection of compression options and parameters 91
3.3.3.1 Segment headers 92
3.3.3.2 Integer or float DWT 93
3.3.3.3 Parameters for controlling compression ratio
and quality 93
3.3.3.4 Parameters for lossless compression 93
3.3.3.5 Segment size S 94
3.3.3.6 Golomb code parameter 94
3.3.3.7 Custom subband weight 95
3.3.4 Performance evaluation 95
3.3.4.1 Lossless compression results 95
3.3.4.2 Lossy compression results 97
3.4 Lossless Multispectral/Hyperspectral Compression Standard 98
3.4.1 Compressor composition 98
3.4.2 Adaptive linear predictor 99
3.4.3 Encoder 102
3.4.4 Performance evaluation 103
References 104
viii Contents
4 Vector Quantization Data Compression 107
4.1 Concept of Vector Quantization Compression 107
4.2 Review of Conventional Fast Vector Quantization Algorithms 110
4.3 Fast Vector-Quantization Algorithm Based on Improved
Distance to MDP 112
4.3.1 Analysis of the generalized Lloyd algorithm for fast training 113
4.3.2 Fast training based on improved distance to MDP 115
4.3.3 Experimental results 117
4.3.4 Assessment of preservation of spectral information 120
4.4 Fast Vector Quantization Based on Searching Nearest Partition Sets 123
4.4.1 Nearest partition sets 124
4.4.2 Upper-triangle matrix of distances 126
4.4.3 p-least sorting 127
4.4.4 Determination of NPS sizes 128
4.4.5 Two fast VQ search algorithms based on NPSs 130
4.4.5.1 Algorithm 1 130
4.4.5.2 Algorithm 2 132
4.4.6 Experimental results 133
4.4.7 Comparison with published fast search methods 136
4.5 3D VQ Compression Using Spectral-Feature-Based Binary Code 138
4.5.1 Spectral-feature-based binary coding 138
4.5.2 Fast 3D VQ using the SFBBC 140
4.5.3 Experimental results of the SFBBC-based VQ compression
algorithm 141
4.6 Correlation Vector Quantization 143
4.6.1 Process of CVQ 143
4.6.2 Performance of CVQ 146
4.7 Training a New Codebook for a Dataset to Be Compressed 147
4.8 Multiple-Subcodebook Algorithm Using Spectral Index 149
4.8.1 Spectral indices and scene segmentation 149
4.8.1.1 Manual multithresholding 150
4.8.1.2 Isoclustering 151
4.8.1.3 Histogram-based segmentation with same-size
regions 151
4.8.1.4 Modified histogram-based segmentation 152
4.8.2 Methodology of MSCA 153
4.8.3 Improvement in processing time 154
4.8.4 Experimental results of the MSCA 154
4.8.5 MSCA with training set subsampling 157
4.8.6 MSCA with training set subsampling plus SFBBC
codebook training 160
4.8.7 MSCA with training set subsampling plus SFBBC for both
codebook training and coding 162
Contents ix
4.9 Successive Approximation Multistage Vector Quantization 162
4.9.1 Compression procedure 162
4.9.2 Features 164
4.9.3 Test results 167
4.10 Hierarchical Self-Organizing Cluster Vector Quantization 168
4.10.1 Compression procedure 168
4.10.2 Features 170
References 171
5 Onboard Near-Lossless Data Compression Techniques 177
5.1 Near-Lossless Satellite Data Compression 177
5.2 Cluster SAMVQ 178
5.2.1 Organizing continuous data flow into regional datacubes 178
5.2.2 Solution for overcoming the blocking effect 180
5.2.3 Removing the boundary between adjacent regions 181
5.2.4 Attaining a fully redundant regional datacube for preventing
data loss in the downlink channel 182
5.2.5 Compression performance comparison between SAMVQ
and cluster SAMVQ 184
5.3 Recursive HSOCVQ 185
5.3.1 Reuse of codevectors of the previous region to attain
a seamless conjunction between regions 185
5.3.2 Training codevectors for a current frame and applying them
to subsequent frames 186
5.3.3 Two schemes of carrying forward reused codevectors
trained in the previous region 188
5.3.4 Compression performance comparison between baseline
and recursive HSOCVQ 190
5.4 Evaluation of Near-Lossless Performance of SAMVQ and HSOCVQ 191
5.4.1 Evaluation method and test dataset 191
5.4.2 Evaluation of a single spectrum 192
5.4.3 Evaluation of an entire datacube 194
5.5 Evaluation of SAMVQ with Regard to the Development of
International Standards of Spacecraft Data Compression 197
5.5.1 CCSDS test datasets 198
5.5.2 Test results of hyperspectral datasets 199
5.5.3 Compression of multispectral datasets using SAMVQ 203
References 209
6 Optimizing the Performance of Onboard Data Compression 211
6.1 Introduction 211
6.2 The Effect of Raw Data Anomalies on Compression Performance 212
6.2.1 Anomalies in the raw hyperspectral data 212
6.2.2 Effect of spikes on compression performance 213