ISBN: 3-540-67921-9
TITLE: Self-Organizing Maps
AUTHOR: Kohonen, Teuvo
TOC:

1. Mathematical Preliminaries 1
1.1 Mathematical Concepts and Notations 2
1.1.1 Vector Space Concepts 2
1.1.2 Matrix Notations 8
1.1.3 Eigenvectors and Eigenvalues of Matrices 11
1.1.4 Further Properties of Matrices 13
1.1.5 On Matrix Differential Calculus 15
1.2 Distance Measures for Patterns 17
1.2.1 Measures of Similarity and Distance in Vector Spaces 17
1.2.2 Measures of Similarity and Distance Between Symbol Strings 21
1.2.3 Averages Over Nonvectorial Variables 28
1.3 Statistical Pattern Analysis 29
1.3.1 Basic Probabilistic Concepts 29
1.3.2 Projection Methods 34
1.3.3 Supervised Classification 39
1.3.4 Unsupervised Classification 44
1.4 The Subspace Methods of Classification 46
1.4.1 The Basic Subspace Method 46
1.4.2 Adaptation of a Model Subspace to Input Subspace 49
1.4.3 The Learning Subspace Method (LSM) 53
1.5 Vector Quantization 59
1.5.1 Definitions 59
1.5.2 Derivation of the VQ Algorithm 60
1.5.3 Point Density in VQ 62
1.6 Dynamically Expanding Context 64
1.6.1 Setting Up the Problem 65
1.6.2 Automatic Determination
of Context-Independent Productions 66
1.6.3 Conflict Bit 67
1.6.4 Construction of Memory
for the Context-Dependent Productions 68
1.6.5 The Algorithm for the Correction of New Strings 68
1.6.6 Estimation Procedure for Unsuccessful Searches 69
1.6.7 Practical Experiments 69
2. Neural Modeling 71
2.1 Models, Paradigms, and Methods 71
2.2 A History of Some Main Ideas in Neural Modeling 72
2.3 Issues on Artificial Intelligence 75
2.4 On the Complexity of Biological Nervous Systems 76
2.5 What the Brain Circuits Are Not 78
2.6 Relation Between Biological and Artificial Neural Networks 79
2.7 What Functions of the Brain Are Usually Modeled? 81
2.8 When Do We Have to Use Neural Computing? 81
2.9 Transformation, Relaxation, and Decoder 82
2.10 Categories of ANNs 85
2.11 A Simple Nonlinear Dynamic Model of the Neuron 87
2.12 Three Phases of Development of Neural Models 89
2.13 Learning Laws 91
2.13.1 Hebb's Law 91
2.13.2 The Riccati-Type Learning Law 92
2.13.3 The PCA-Type Learning Law 95
2.14 Some Really Hard Problems 96
2.15 Brain Maps 99
3. The Basic SOM 105
3.1 A Qualitative Introduction to the SOM 106
3.2 The Original Incremental SOM Algorithm 109
3.3 The "Dot-Product SOM" 115
3.4 Other Preliminary Demonstrations of Topology-Preserving Mappings 116
3.4.1 Ordering of Reference Vectors in the Input Space 116
3.4.2 Demonstrations of Ordering of Responses in the Output Space 120
3.5 Basic Mathematical Approaches to Self-Organization 127
3.5.1 One-Dimensional Case 128
3.5.2 Constructive Proof of Ordering of Another One-Dimensional SOM 132
3.6 The Batch Map 138
3.7 Initialization of the SOM Algorithms142
3.8 On the "Optimal" Learning-Rate Factor 143
3.9 Effect of the Form of the Neighborhood Function 145
3.10 Does the SOM Algorithm Ensue
from a Distortion Measure? 146
3.11 An Attempt to Optimize the SOM 148
3.12 Point Density of the Model Vectors 152
3.12.1 Earlier Studies 152
3.12.2 Numerical Check of Point Densities in a Finite One-Dimensional SOM 153
3.13 Practical Advice for the Construction of Good Maps 159
3.14 Examples of Data Analyses Implemented by the SOM 161
3.14.1 Attribute Maps with Full Data Matrix 161
3.14.2 Case Example of Attribute Maps Based on Incomplete Data Matrices (Missing Data):
"Poverty Map" 165
3.15 Using Gray Levels to Indicate Clusters in the SOM 165
3.16 Interpretation of the SOM Mapping 166
3.16.1 "Local Principal Components" 166
3.16.2 Contribution of a Variable to Cluster Structures 169
3.17 Speedup of SOM Computation 170
3.17.1 Shortcut Winner Search 170
3.17.2 Increasing the Number of Units in the SOM 172
3.17.3 Smoothing 175
3.17.4 Combination of Smoothing, Lattice Growing, and SOM Algorithm 176
4. Physiological Interpretation of SOM 177
4.1 Conditions for Abstract Feature Maps in the Brain 177
4.2 Two Different Lateral Control Mechanisms 178
4.2.1 The WTA Function, Based on Lateral Activity Control 179
4.2.2 Lateral Control of Plasticity 184
4.3 Learning Equation 185
4.4 System Models of SOM and Their Simulations 185
4.5 Recapitulation of the Features of the Physiological SOM Model 188
4.6 Similarities Between the Brain Maps and Simulated Feature Maps 188
4.6.1 Magnification 189
4.6.2 Imperfect Maps 189
4.6.3 Overlapping Maps 189
5. Variants of SOM 191
5.1 Overview of Ideas to Modify the Basic SOM 191
5.2 Adaptive Tensorial Weights 194
5.3 Tree-Structured SOM in Searching 197
5.4 Different Definitions of the Neighborhood 198
5.5 Neighborhoods in the Signal Space 200
5.6 Dynamical Elements Added to the SOM 204
5.7 The SOM for Symbol Strings 205
5.7.1 Initialization of the SOM for Strings 205
5.7.2 The Batch Map for Strings 206
5.7.3 Tie-Break Rules 206
5.7.4 A Simple Example: The SOM of Phonemic Transcriptions 207
5.8 Operator Maps 207
5.9 Evolutionary-Learning SOM 211
5.9.1 Evolutionary-Learning Filters 211
5.9.2 Self-Organization According to a Fitness Function 212
5.10 Supervised SOM 215
5.11 The Adaptive-Subspace SOM (ASSOM) 216
5.11.1 The Problem of Invariant Features 216
5.11.2 Relation Between Invariant Features and Linear Subspaces 218
5.11.3 The ASSOM Algorithm 222
5.11.4 Derivation of the ASSOM Algorithm by Stochastic Approximation 226
5.11.5 ASSOM Experiments 228
5.12 Feedback-Controlled Adaptive-Subspace SOM (FASSOM) 242
6. Learning Vector Quantization 245
6.1 Optimal Decision 245
6.2 The LVQ1 246
6.3 The Optimized-Learning-Rate LVQ1 (OLVQ1) 250
6.4 The Batch-LVQ1 251
6.5 The Batch-LVQ1 for Symbol Strings 252
6.6 The LVQ2 (LVQ2.1) 252
6.7 The LVQ3 253
6.8 Differences Between LVQ1, LVQ2 and LVQ3 254
6.9 General Considerations 254
6.10 The Hypermap-Type LVQ 256
6.11 The "LVQ-SOM" 261
7. Applications 263
7.1 Preprocessing of Optic Patterns 264
7.1.1 Blurring 265
7.1.2 Expansion in Terms of Global Features 266
7.1.3 Spectral Analysis 266
7.1.4 Expansion in Terms of Local Features (Wavelets) 267
7.1.5 Recapitulation of Features of Optic Patterns 267
7.2 Acoustic Preprocessing 268
7.3 Process and Machine Monitoring 269
7.3.1 Selection of Input Variables and Their Scaling 269
7.3.2 Analysis of Large Systems 270
7.4 Diagnosis of Speech Voicing 274
7.5 Transcription of Continuous Speech 274
7.6 Texture Analysis 280
7.7 Contextual Maps 281
7.7.1 Artifically Generated Clauses 283
7.7.2 Natural Text 285
7.8 Organization of Large Document Files 286
7.8.1 Statistical Models of Documents 286
7.8.2 Construction of Very Large WEBSOM Maps by the Projection Method 292
7.8.3 The WEBSOM of All Electronic Patent Abstracts 296
7.9 Robot-Arm Control 299
7.9.1 Simultaneous Learning of Input and Output Parameters 299
7.9.2 Another Simple Robot-Arm Control 303
7.10 Telecommunications 304
7.10.1 Adaptive Detector for Quantized Signals 304
7.10.2 Channel Equalization in the Adaptive QAM 305
7.10.3 Error-Tolerant Transmission of Images by a Pair of SOMs 306
7.11 The SOM as an Estimator 308
7.11.1 Symmetric (Autoassociative)Mapping 308
7.11.2 Asymmetric (Heteroassociative)Mapping 309
8. Software Tools for SOM 311
8.1 Necessary Requirements 311
8.2 Desirable Auxiliary Features 313
8.3 SOM Program Packages 315
8.3.1 SOM PAK 315
8.3.2 SOM Toolbox 317
8.3.3 Nenet (Neural Networks Tool) 318
8.3.4 Viscovery SOMine 318
8.4 Examples of the Use of SOM PAK 319
8.4.1 File Formats 319
8.4.2 Description of the Programs in SOM PAK 322
8.4.3 A Typical Training Sequence 326
8.5 Neural-Networks Software with the SOM Option 327
9. Hardware for SOM 329
9.1 An Analog Classifier Circuit 329
9.2 Fast Digital Classifier Circuits 332
9.3 SIMD Implementation of SOM 337
9.4 Transputer Implementation of SOM 339
9.5 Systolic-Array Implementation of SOM 341
9.6 The COKOS Chip 342
9.7 The TInMANN Chip 342
9.8 NBISOM 25 Chip 344
10. An Overview of SOM Literature 347
10.1 Books and Review Articles 347
10.2 Early Works on Competitive Learning 348
10.3 Status of the Mathematical Analyses 349
10.3.1 Zero-Order Topology (Classical VQ)Results 349
10.3.2 Alternative Topological Mappings 350
10.3.3 Alternative Architectures 350
10.3.4 Functional Variants 351
10.3.5 Theory of the Basic SOM 352
10.4 The Learning Vector Quantization 358
10.5 Diverse Applications of SOM 358
10.5.1 Machine Vision and Image Analysis 358
10.5.2 Optical Character and Script Reading 360
10.5.3 Speech Analysis and Recognition 360
10.5.4 Acoustic and Musical Studies 361
10.5.5 Signal Processing and Radar Measurements 362
10.5.6 Telecommunications 362
10.5.7 Industrial and Other Real-World Measurements 362
10.5.8 Process Control 363
10.5.9 Robotics 364
10.5.10 Electronic-Circuit Design 364
10.5.11 Physics 364
10.5.12 Chemistry 365
10.5.13 Biomedical Applications Without Image Processing 365
10.5.14 Neurophysiological Research 366
10.5.15 Data Processing and Analysis 366
10.5.16 Linguistic and AI Problems 367
10.5.17 Mathematical and Other Theoretical Problems 368
10.6 Applications of LVQ 369
10.7 Survey of SOM and LVQ Implementations 370
11. Glossary of "Neural" Terms 373
References 403
Index 487
END
