Applied Optimum
Signal Processing

Sophocles J. Orfanidis

ECE Department
Rutgers University
94 Brett Road
Piscataway, NJ 08854-8058
Tel: 848-445-5017
e-mail: orfanidi@rutgers.edu

Any feedback from readers is welcome.

This book is an updated and much enlarged 2018 edition of Optimum Signal Processing, which was published in 2007 as a republication of the second edition published by McGraw-Hill Publishing Company, New York, NY, in 1988 (ISBN 0-07-047794-9), and also published earlier by Macmillan, Inc., New York, NY, 1988 (ISBN 0-02-389380-X). All copyrights to this work reverted to Sophocles J. Orfanidis in 1996.

Initially posted online in August 2018. Latest revision date - August 1, 2018.

The entire book is freely available in PDF 2-up format, and in PDF 1-up format. Individual chapters are available below.

A printed version is also available as a two-volume 6x9 paperback set ( vol.1, vol.2).

Table of Contents
Ch.1: Review of Random Signals: Probability concepts, Chebyshev's inequality, joint and conditional densities, Bayes' rule, correlation canceling and optimum estimation, regression lemma, Gram-Schmidt orthogonalization of random variables, partial correlations, random signals, power spectrum, sample autocorrelation, periodogram, filtering of stationary random signals, random signal models, stability and stationarity, parameter estimation, linear prediction and signal modeling, Cramér-Rao bound and maximum likelihood, minimum-phase signals and filters, spectral factorization theorem, minimum-phase property of the prediction-error filter.
Ch.2: Signal Extraction Basics: Noise reduction and signal enhancement concepts and tradeoffs, first-order exponential smoother, FIR averaging filters.
Ch.3: Local Polynomial Filters: Local polynomial filters (Schiaparelli-DeForest-Gram-Hardy-Sheppard-Henderson-Whittaker-Savitzky- Golay), local polynomial fitting, geometric interpretation, orthogonal polynomial bases, discrete Chebyshev/Gram polynomials, polynomial predictive and interpolation filters, minimum variance filters, predictive differentiation filters, filtering implementations.
Ch.4: Minimum Roughness Filters: Weighted local polynomial filters, Henderson minimum-Rs filters, Hahn orthogonal polynomials, maximally-flat filters and Krawtchouk polynomials, missing data and outliers.
Ch.5: Local Polynomial Modeling: Weighted local polynomial modeling, bandwidth selection, local polynomial interpolation, variable bandwidth, repeated observations, loess smoothing.
Ch.6: Exponential Smoothing: Mean tracking, forecasting and state-space models, higher-order polynomial smoothing filters, linear trend FIR filters, higher-order exponential smoothing, steady-state exponential smoothing, smoothing parameter selection, single, double, and triple exponential smoothing, Tukey's twicing operation, twicing and zero-lag filters, local level, local slope, local acceleration filters, basis transformations and EMA initialization, Holt's exponential smoothing, state-space models for Holt's method, filtering methods and technical analysis in financial market trading: moving average filters (SMA, WMA, TMA), integrated linear regression slope indicator (ILRS), predictive moving average filters, single, double, and triple EMA indicators (EMA, DEMA, TEMA, WEMA), linear regression, linear regression slope, and R-square indicators, R-squared critical values, Student's t-distribution, standard deviation index, time series forecast, initialization schemes, Butterworth moving average filters, reduced-lag level and slope filters (GDEMA, ZEMA, HMA, EHMA, SHMA, T3), envelopes, bands, and channels (Bollinger, standard-error, projection, fixed-width, Donchian, Keltner, Starc bands, average true range, parabolic SAR), delays, momentum, oscillators, relative strength index (RSI), Chande momentum oscillator, vertical-horizontal filter, directional movement system, momentum, price rate of change, price oscillator and MACD, stochastic, percent-K, percent-D, accumulation/distribution line, Chaikin oscillator-money flow-volatility, positive/negative volume indices, commodity channel index, detrended price oscillator, dynamic momentum index, forecast oscillator, TRIX oscillator, variable-length EMA, Open-High-Low-Close bar charts.
Ch.7: Smoothing Splines: Interpolation versus smoothing, variational approach, natural cubic smoothing splines, optimality of natural splines, generalized cross validation, repeated observations, equivalent filter, stochastic model, computational aspects.
Ch.8: Whittaker-Henderson Smoothing: Whittaker-Henderson smoothing methods, regularization filters, Hodrick-Prescott filters, Wiener filter interpretation, regularization and kernel machines, sparse Whittaker-Henderson methods, L₁ trend filtering, iterative reweighted L₁ and L₀ regularization, total variation minimization.
Ch.9: Periodic Signal Extraction: Notch and comb filters for periodic signals, seasonal fractional delay filters, signal averaging, ideal seasonal decomposition filters, classical seasonal decomposition, seasonal moving-average filters, census X-11 decomposition filters, Musgrave asymmetric filters, seasonal Whittaker-Henderson decomposition with L₂ and L₁ regularization.
Ch.10: Wavelets: Multiresolution analysis, dilation equations, wavelet filter properties, multiresolution and filter banks, fast discrete wavelet transform, multiresolution decomposition, wavelet denoising, undecimated wavelet transform.
Ch.11: Wiener Filtering: Linear and nonlinear estimation of signals, orthogonality and normal equations, stationary Wiener filter, construction of the Wiener filter by prewhitening, Wiener filter as Kalman filter, construction of the Wiener filter by the gapped function, construction of the Wiener filter by covariance factorization, Kalman filter.
Ch.12: Pure Prediction and Signal Modeling: Autoregressive models, linear prediction and the Levinson recursion, Levinson's algorithm in matrix form, autocorrelation sequence extensions, split Levinson algorithm, analysis and synthesis lattice filters, alternative proof of the minimum-phase property, orthogonality of backward prediction errors, Cholesky factorization, Schur algorithm, lattice realizations of FIR Wiener filters, autocorrelation, covariance, and Burg's methods, dynamic predictive deconvolution, waves in layered media, least-squares waveshaping and spiking filters, ARIMA modeling example.
Ch.13: Kalman Filtering: State-space models, Kalman filter and its derivation, forecasting and missing observations, Kalman filter with deterministic inputs, time-invariant models, steady-state Kalman filters, continuous-time Kalman filter, equivalence of Kalman and Wiener filtering, fixed-interval smoothing, square-root algorithms, maximum likelihood parameter estimation, parameter estimation with the EM algorithm.
Ch.14: Spectrum Estimation and Array Processing: Spectrum estimation by autoregressive modeling, spectral analysis of sinusoids in noise, superresolution array processing, eigenvector methods, MUSIC method, minimum-norm method, reduced-order method, maximum mikelihood method, ESPRIT method, spatial smoothing, asymptotic properties, linearly-constrained minimum-variance beamforming (LCMV), generalized sidelobe canceler (GSC), Markowitz mean variance portfolio theory.
Ch.15: SVD and Signal Processing: Vector and matrix norms, subspaces, bases, and projections, fundamental theorem of linear algebra, solving linear equations, singular value decomposition, Moore-Penrose pseudoinverse, least-squares problems and the SVD, condition number, reduced-rank approximation, regularization of ill-conditioned problems, sparse regularization, L₁ and L₀ regularization, iterative reweighted least squares, SVD and signal processing, least-squares linear prediction, MA and ARMA modeling, Karhunen-Loève transform, principal component analysis (PCA), SVD signal enhancement, singular spectrum analysis (SSA), structured matrix approximations, matrix pencil methods, QR factorization, canonical correlation analysis (CCA).
Ch.16: Adaptive Filters: Adaptive implementation of Wiener filters, correlation canceler loop (CCL), Widrow-Hoff LMS adaptation algorithm, adaptive linear combiner, adaptive FIR Wiener filter, speed of convergence, adaptive channel equalizers, adaptive echo cancelers, adaptive noise canceling, adaptive line enhancer, adaptive linear prediction, adaptive implementation of pisarenko's method, gradient adaptive lattice filters, adaptive Gram-Schmidt preprocessors, rank-one modification of covariance matrices, RLS adaptive filters, fast RLS filters, RLS lattice filters.
Appendices, References, Index

Copyright Notices

Copyright (c) 2018 by Sophocles J. Orfanidis, All Rights Reserved. The book exists in online form through the web page Applied Optimum Signal Processing. Links to this page may be placed on any web site.

Any part of this book may be downloaded and printed for personal or educational use only, as long as the printed or photocopied pages are not altered in any way from the original PDF files posted on the book's web page.

No part of this book may be reproduced, altered in any way, or transmitted in any form for commercial, profit, sale, or marketing purposes.

MATLAB (R) is a registered trademark of The MathWorks, Inc.

OSP Toolbox by S. J. Orfanidis, August 2018.

CVX software by Michael Grant and Stephen Boyd, September 2013.

MATLAB Toolbox

The OSP toolbox contains about 270 MATLAB functions for carrying out all of the computations and simulation examples in the book. Code segments illustrating the usage of these functions are found throughout the book, and serve as a user manual. Please read the license agreement before using the toolbox. The zipped file osp_toolbox.zip (last revised on August 1, 2018) contains all the MATLAB functions. Note that just typing the name of any function will produce a help/usage comment for that function. The detailed list of the MATLAB functions is as follows:

Local Polynomial Smoothing Filters ---------------------------------- binom - vector of binomial coefficients bkfilt - Baxter-King bandpass filter cldec - classical decomposition method combfd - comb fractional-delay filter design compl - complement of an odd-length symmetric filter diffb - backward difference operator diffmat - difference convolution matrix diffpol - differentiate polynomial diffs - seasonal backward difference operator ecg - ECG generator ecgsim - ECG simulation filtdbl - filtering with double-sided FIR filter hahnbasis - Hahn orthogonal polynomials hahncoeff - coefficients of Hahn orthogonal polynomials hahnpol - Hahn orthogonal polynomial evaluation hahnrec - Hahn orthogonal polynomials hend - Henderson weighting function kmat - difference convolution matrix kraw - Krawtchouk binomial weighting function kwindow - Kaiser window for spectral analysis lagrfd - Lagrange-interpolation fractional-delay filter lpbasis - local polynomial basis lpdiff - weighted local polynomial differentiation filters lpfilt - local polynomial filtering - fast version lpfilt2 - local polynomial filtering - slower version lpinterp - local polynomial interpolation and differentiation filters lpmat - local polynomial smoothing matrix lpmissing - weighted local polynomial filters for missing data lprs - local polynomial minimum-Rs smoothing filters lprs2 - local polynomial minimum-Rs smoothing filters (closed-form) lpsm - weighted local polynomial smoothing and differentiation filters minrev - minimum revision asymmetric filters polval - polynomial evaluation in factorial power series rlpfilt - robust local polynomial filtering sigav - signal averaging smadec - decomposition using seasonal moving-average filters smafilt - impulse responses of seasonal decomposition moving average filters smat - seasonal moving-average filtering matrix smav - seasonal moving average filter stirling - Stirling numbers of first or second kind, signed or unsigned swhdec - seasonal Whittaker-Henderson decomposition trendma - trend moving-average filter, 2xD if D is even, 1xD if D is odd upmat - upsample matrix of smoothing filters whkdec - Whittaker-Henderson-Kaiser seasonal decomposition x11dec - US Census X-11 decomposition method for seasonal adjustment x11filt - impulse responses of the US Census X-11 seasonal adjustment filters Local Linear Regression ----------------------- avobs - average repeated observations locband - bandwidth for local polynomial regression locgcv - local polynomial GCV and CV evaluation locgrid - uniform grid for local polynomial evaluation locpol - local polynomial regression locval - evaluation/interpolation of local polynomial regression locw - local weighting functions for local polynomial regression loess - Cleveland's robust locally weighted scatterplot smoothing (loess) loess2 - Cleveland's robust locally weighted scatterplot smoothing (loess) Spline and Whittaker-Henderson Smoothing ---------------------------------------- splambda - find optimum lambda for spline smoothing using GCV splav - averaged repeated observations at spline knots splcoeff - spline coefficients splgcv - evaluate GCV(lambda) splmat - spline smoothing matrices Q,T splsm - spline smoothing using Reinsch's algorithm splsm2 - spline smoothing using Reinsch's algorithm - robust version splval - evaluate spline smoothing polynomials whgcv - Whittaker-Henderson GCV whgen - generalized Whittaker-Henderson whimp - Whittaker-Henderson filter impulse response whsm - Whittaker-Henderson smoothing method whsm1 - Whittaker-Henderson smoothing method - L1 version Exponentially Weighted Average ------------------------------ binmat - binomial boost matrices for exponential smoothers ema - exponential moving average - exact version emaerr - calculate MAE, MSE, and MAPE for a range of lambda's emap - map equivalent lambdas between d=0 EMA and d=1 EMA emat - polynomial to cascaded transformation matrix holt - Holt's exponential smoothing holterr - calculate MAE, MSE, and MAPE for a range of lambda's mema - multiple exponential moving average stema - steady-state exponential moving average Linear Prediction & Wiener and Kalman Filtering Functions --------------------------------------------------------- acext - autocorrelation sequence extension using Levinson recursion acf - sample auto-correlation function acmat - construct autocorrelation Toeplitz matrix from autocorrelation lags acsing - sinusoidal representation of singular autocorrelation matrices aicmdl - estimates dimension of signal subspace from AIC and MDL criteria argen - generate a zero-mean segment of an AR process bkwlev - backward Levinson recursion burg - Burg's method of linear prediction dir2nl - direct form to normalized lattice dpd - dynamic predictive deconvolution dwf - sample processing algorithm of direct-form Wiener filter dwf2 - direct-form Wiener filter using circular delay-line buffer dwfilt - direct-form Wiener filtering of data dwfilt2 - circular-buffer direct-form Wiener filtering of data faest - sample processing algorithm of adaptive lattice Wiener filter firw - FIR Wiener filter design flipv - flip a vector, column, row, or both for a matrix frwlev - forward Levinson recursion glwf - sample processing algorithm of lattice Wiener filter kfilt - Kalman filtering ksmooth - Kalman smoothing latt - sample processing algorithm of analysis lattice filter lattfilt - lattice filtering of a data vector lattsect - sample processing algorithm of a single lattice section lattsynth - sample processing algorithm of synthesis lattice filter lev - Levinson-Durbin recursion lms - sample processing LMS algorithm of direct-form Wiener filter lpf - extract linear prediction filter from matrix L lpg - extract reflection coefficients from matrix L lpspec - compute LP spectrum of a prediction-error filter lwf - sample processing algorithm of lattice Wiener filter lwfilt - lattice Wiener filtering of data mgs - adaptive modified Gram-Schmidt mgslms - adaptive Gram-Schmidt using LMS minorm - minimum-norm noise subspace eigenvector music - MUSIC spectrum computation nlfilt - filtering in the normalized lattice form obmat - observability matrix for canonical or transposed realizations obmatc - observability matrix for continuous-time rlev - reverse of Levinson's algorithm rls - RLS algorithm for adaptive linear combiner rlsl - sample processing algorithm of lattice Wiener filter rmusic - minimum-norm noise subspace eigenvector scatt - direct scattering problem schur1 - Schur algorithm for linear prediction schur2 - Schur algorithm for Cholesky factorization spike - spiking filter design yw - Yule-Walker method of linear prediction SVD, Subspace, and ARMA Modeling Functions ------------------------------------------ arma2imp - ARMA impulse response armaacf - ARMA autocorrelation function armachol - ARMA covariance matrix Cholesky factorization armafit - fitting an ARMA(p,q) model to covariance lags armainf - ARMA asymptotic Fisher information matrix armainnov - ARMA modeling using the innovations method armamf - Mayne-Firoozan ARMA modeling method armamyw - ARMA modeling by the modified Yule-Walker method armasim - simulate a zero-mean segment of a gaussian ARMA process armasim2 - simulate a zero-mean segment of a gaussian ARMA process bwidth - beamwidth mapping from psi-space to phi-space cca - Canonical Correlation Analysis ccacov - CCA applied to a covariance matrix cholgs - Cholesky factorization by Gram-Schmidt orthogonalization cholinnov - Cholesky factorization by innovations representation crb - calculate Cramer-Rao bounds for sinusoids in noise crb2 - calculate Cramer-Rao bounds for sinusoids in noise datamat - convolution data matrix of a signal vector datasig - extract data signal from a Toeplitz or Toeplitz/Hankel data matrix dolph - Dolph-Chebyshev array weights fisher - calculate Fisher information matrix for sinusoids in noise imp2arma - impulse response to ARMA coefficients irls - Lp_regularized iteratively reweighted least squares irls_wh - Lp_regularized IRLS Whittaker-Henderson lpls - construct least-squares linear prediction filter from data matrix madurbin - MA modeling by Durbin's method mafit - Wilson's method of fitting an MA(q) model to covariance lags mainnov - MA modeling by the innovations method mpencil - matrix-pencil method of extracting sinusoids in noise poly2 - specialized version of poly scan - scan array with given scanning phase setrank - reduce the rank of a diagonal matrix of singular values sigsub - construct reduced-rank signal subspace of a data matrix sines - generate sum of real or complex decaying sinusoids in noise snap - generate snapshot matrix for array problems snapshot - generate data matrix of snapshots for array problems snr - magnitude to SNR in dB, and conversely steer - steer array towards given angle steering - construct steering matrix of multiple sinusoids/plane-waves steermat - construct steering matrix of multiple sinusoids/plane-waves svdenh - SVD signal enhancement toepl - Toeplitz, Hankel, or Toeplitz/Hankel approximation of data matrix varper - percentage variances Wavelet Functions ----------------- advance - circular time-advance (left-shift) of a vector casc - cascade algorithm for phi and psi wavelet functions circonv - circular convolution cmf - conjugate mirror of a filter convat - convolution a trous convmat - sparse convolution matrix convmat2 - sparse convolution matrix (simplified version) daub - Daubechies scaling filters (daublets, symmlets, coiflets) dn2 - downsample by a factor of 2 dwtcell - cell array of sparse discrete wavelet transform matrices dwtdec - DWT decomposition into orthogonal multiresolution components dwtmat - discrete wavelet transform matrices dwtmat2 - discrete wavelet transform matrices dwtwrap - wrap a DWT matrix into a lower DWT matrix flipv - flip a vector, column, row, or both for a matrix fwt - fast wavelet transform using convolution and downsampling fwtm - fast wavelet transform in matrix form fwtmat - overall DWT orthogonal matrix ifwt - inverse fast wavelet transform using upsampling and convolution ifwtm - inverse fast wavelet transform in matrix form iuwt - inverse undecimated wavelet transform iuwtm - inverse undecimated wavelet transform modwrap - wrap matrix column-wise mod-N phinit - eigenvector initialization of phi plotdec - plot DWT/UWT decomposition or DWT/UWT coefficients up2 - upsample a vector by factor of two upr - upsample a vector by factor of 2^r uwt - undecimated wavelet transform uwtdec - UWT multiresolution decomposition uwtm - undecimated wavelet transform uwtmat - undecimated wavelet transform matrices uwtmat2 - undecimated wavelet transform matrices w2V - wavelet vector to wavelet matrix wcoeff - extract wavelet coefficients from DWT at given level wdenoise - Donoho & Johnstone's VisuShrink denoising procedure wduwt - wavelet denoising with UWT wthr - soft/hard level-dependent wavelet thresholding Technical Analysis Functions ---------------------------- accdist - accumulation/distribution line atr - true range & average true range bbands - Bollinger bands bma - Butterworth moving average cci - commodity channel index chosc - Chaikin oscillator chvol - Chaikin volatility cmflow - Chaikin money flow cmo - Chande momentum oscillator delay - lag or delay or advance by d samples dema - steady-state double exponential moving average dirmov - directional movement system dmi - dynamic momentum index (DMI) donch - Donchian channels dpo - detrended price oscillator ehma - exponential Hull moving average fbands - fixed-envelope bands forosc - forecast oscillator gdema - generalized dema hma - Hull moving average ilrs - integrated linear regression slope indicator kbands - Keltner bands or channels lreg - linear regression, slope, and R-squared indicators mom - momentum and price rate of change ohlc - make Open-High-Low-Close bar chart ohlcyy - OHLC plot with other indicators on the same graph pbands - Projection Bands and Projection Oscillator pma - predictive moving average, linear fit pma2 - predictive moving average, polynomial order d=1,2 pmaimp - predictive moving average impulse response pmaimp2 - predictive moving average impulse response, d=1,2 pnvi - positive and negative volume indices (PVI & NVI) prosc - price oscillator & MACD psar - Wilder's parabolic SAR r2crit - R-squared critical values rsi - relative strength index (RSI) sebands - standard-error bands sema - single exponential moving average shma - SMA-based Hull moving average sma - simple moving average stbands - STARC bands stdev - standard deviation index stoch - stochastic oscillator t3 - Tillson's T3 indicator, triple gdema tcrit - critical values of Student's t-distribution tdistr - cumulative t-distribution tema - triple exponential moving average tma - triangular moving average trix - TRIX oscillator vema - variable-length exponential moving average vhfilt - Vertical Horizontal filter wema - Wilder's exponential moving average wma - weighted or linear moving average yylim - adjust left/right ylim & ticks Miscellaneous Functions ----------------------- canfilt - IIR filtering in canonical form using linear delay-line buffer ccan - IIR filter in canonical form using circular delay-line buffer ccanfilt - IIR filtering in canonical form using circular delay-line buffer frespc - frequency response of a cascaded IIR filter at a frequency vector w loadfile - load data file ignoring any text lines taxis - define time axis up - upsample by a factor of L ustep - unit-step or rising unit-step function xaxis - set x-axis limits and tick marks yaxis - set y-axis limits and tick marks zmean - zero mean of each column of a data matrix (or row vector)

Applied Optimum Signal Processing

Sophocles J. Orfanidis

ECE Department Rutgers University 94 Brett Road Piscataway, NJ 08854-8058 Tel: 848-445-5017 e-mail: orfanidi@rutgers.edu

Copyright Notices

MATLAB Toolbox

Applied Optimum
Signal Processing

ECE Department
Rutgers University
94 Brett Road
Piscataway, NJ 08854-8058
Tel: 848-445-5017
e-mail: orfanidi@rutgers.edu