MATLAB Machine Learning Recipes: A Problem-Solution Approach

Michael Paluszek and Stephanie Thomas

MATLAB Machine Learning RecipesA Problem-Solution Approach2nd ed.

../images/420697_2_En_BookFrontmatter_Figa_HTML.png

Michael Paluszek

Plainsboro, NJ, USA

Stephanie Thomas

Plainsboro, NJ, USA

ISBN 978-1-4842-3915-5e-ISBN 978-1-4842-3916-2

https://doi.org/10.1007/978-1-4842-3916-2

Library of Congress Control Number: 2018967208

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.

Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.

While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.

Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com, or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.

Introduction

Machine learning is becoming important in every engineering discipline. For example:

1.
Autonomous cars. Machine learning is used in almost every aspect of car control systems.
2.
Plasma physicists use machine learning to help guide experiments on fusion reactors. TAE Systems has used it with great success in guiding fusion experiments. The Princeton Plasma Physics Laboratory has used it for the National Spherical Torus Experiment to study a promising candidate for a nuclear fusion power plant.
3.
It is used in finance for predicting the stock market.
4.
Medical professionals use it for diagnoses.
5.
Law enforcement, and others, use it for facial recognition. Several crimes have been solved using facial recognition!
6.
An expert system was used on NASA’s Deep Space 1 spacecraft.
7.
Adaptive control systems steer oil tankers.

There are many, many other examples.

Although many excellent packages are available from commercial sources and open-source repositories, it is valuable to understand how these algorithms work. Writing your own algorithms is valuable both because it gives you an insight into the commercial and open-source packages and because it gives you the background to write your own custom machine learning software specialized for your application.

MATLAB ^® had its origins for that very reason. Scientists who needed to do operations on matrices used numerical software written in FORTRAN. At the time, using computer languages required the user to go through the write-compile-link-execute process, which was time-consuming and error-prone. MATLAB presented the user with a scripting language that allowed the user to solve many problems with a few lines of a script that executed instantaneously. MATLAB has built-in visualization tools that helped the user to better understand the results. Writing MATLAB was a lot more productive and fun than writing FORTRAN.

The goal of MATLAB Machine Learning Recipes: A Problem–Solution Approach is to help all users to harness the power of MATLAB to solve a wide range of learning problems. The book has something for everyone interested in machine learning. It also has material that will allow people with an interest in other technology areas to see how machine learning, and MATLAB, can help them to solve problems in their areas of expertise.

Using the Included Software

This textbook includes a MATLAB toolbox, which implements the examples. The toolbox consists of:

1.
MATLAB functions
2.
MATLAB scripts
3.
html help

The MATLAB scripts implement all of the examples in this book. The functions encapsulate the algorithms. Many functions have built-in demos. Just type the function name in the command window and it will execute the demo. The demo is usually encapsulated in a sub-function. You can copy out this code for your own demos and paste it into a script. For example, type the function name PlotSet into the command window and the plot in Figure 1 will appear.

>> PlotSet

../images/420697_2_En_BookFrontmatter_Fig1_HTML.png — Figure 1
Example plot from the function PlotSet.m.

If you open the function you will see the demo:

%%% PlotSet>Demo

function Demo

x = linspace (1,1000);

y = [ sin (0.01*x); cos (0.01*x); cos (0.03*x)];

disp ( ’PlotSet:␣One␣x␣and␣two␣y␣rows’)

PlotSet( x, y, ’figure␣title’, ’PlotSet␣Demo’,...

’plot␣set’,{[2 3], 1}, ’legend’,{{ ’A’ ’B’},{}}, ’plot␣title’,

{ ’cos’, ’sin’});

You can use these demos to start your own scripts. Some functions, such as right-hand side functions for numerical integration, don’t have demos. If you type:

>> RHSAutomobileXY

Error using RHSAutomobileXY (line 17)

a built-in demo is not available.

The toolbox is organized according to the chapters in this book. The folder names are Chapter_01, Chapter_02, etc. In addition, there is a general folder with functions that support the rest of the toolbox. You will also need the open-source package GLPK (GNU Linear Programming Kit) to run some of the code. Nicolo Giorgetti has written a MATLAB MEX interface to GLPK that is available on SourceForge and included with this toolbox. The interface consists of:

1.
glpk.m
2.
glpkcc.mexmaci64, or glpkcc.mexw64, etc.
3.
GLPKTest.m

which are available from https://sourceforge.net/projects/glpkmex/ . The second item is the MEX file of glpkcc.cpp compiled for your machine, such as Mac or Windows. Go to https://www.gnu.org/software/glpk/ to get the GLPK library and install it on your system. If needed, download the GLPKMEX source code as well and compile it for your machine, or else try another of the available compiled builds.

1 An Overview of Machine Learning 1

1.1 Introduction 1

1.2 Elements of Machine Learning 2

1.2.1 Data 2

1.2.2 Models 3

1.2.3 Training 3

1.2.3.1 Supervised Learning 3

1.2.3.2 Unsupervised Learning 4

1.2.3.3 Semi-Supervised Learning 4

1.2.3.4 Online Learning 4

1.3 The Learning Machine 4

1.4 Taxonomy of Machine Learning 6

1.5 Control 8

1.5.1 Kalman Filters 8

1.5.2 Adaptive Control 9

1.6 Autonomous Learning Methods 9

1.6.1 Regression 10

1.6.2 Decision Trees 13

1.6.3 Neural Networks 14

1.6.4 Support Vector Machines 15

1.7 Artificial Intelligence 16

1.7.1 What is Artificial Intelligence? 16

1.7.2 Intelligent Cars 16

1.7.3 Expert Systems 17

1.8 Summary 18

2 Representation of Data for Machine Learning in MATLAB 19

2.1 Introduction to MATLAB Data Types 19

2.1.1 Matrices 19

2.1.2 Cell Arrays 20

2.1.3 Data Structures 21

2.1.4 Numerics 23

2.1.5 Images 23

2.1.6 Datastore 25

2.1.7 Tall Arrays 26

2.1.8 Sparse Matrices 27

2.1.9 Tables and Categoricals 27

2.1.10 Large MAT-Files 29

2.2 Initializing a Data Structure Using Parameters 30

2.2.1 Problem 30

2.2.2 Solution 30

2.2.3 How It Works 30

2.3 Performing MapReduce on an Image Datastore 33

2.3.1 Problem 33

2.3.2 Solution 33

2.3.3 How It Works 33

2.4 Creating a Table from a File 35

2.4.1 Problem 35

2.4.2 Solution 36

2.4.3 How It Works 36

2.5 Processing Table Data 37

2.5.1 Problem 37

2.5.2 Solution 38

2.5.3 How It Works 38

2.6 Using MATLAB Strings 41

2.6.1 String Concatenation 41

2.6.1.1 Problem 41

2.6.1.2 Solution 41

2.6.1.3 How It Works 41

2.6.2 Arrays of Strings 41

2.6.2.1 Problem 41

2.6.2.2 Solution 41

2.6.2.3 How It Works 41

2.6.3 Substrings 42

2.6.3.1 Problem 42

2.6.3.2 Solution 42

2.6.3.3 How It Works 42

2.7 Summary 43

3 MATLAB Graphics 45

3.1 2D Line Plots 45

3.1.1 Problem 45

3.1.2 Solution 45

3.1.3 How It Works 46

3.2 General 2D Graphics 48

3.2.1 Problem 48

3.2.2 Solution 48

3.2.3 How It Works 48

3.3 Custom Two-Dimensional Diagrams 50

3.3.1 Problem 50

3.3.2 Solution 50

3.3.3 How It Works 50

3.4 Three-Dimensional Box 51

3.4.1 Problem 52

3.4.2 Solution 52

3.4.3 How It Works 52

3.5 Draw a 3D Object with a Texture 54

3.5.1 Problem 54

3.5.2 Solution 54

3.5.3 How It Works 55

3.6 General 3D Graphics 56

3.6.1 Problem 56

3.6.2 Solution 56

3.6.3 How It Works 56

3.7 Building a GUI 58

3.7.1 Problem 58

3.7.2 Solution 58

3.7.3 How It Works 58

3.8 Animating a Bar Chart 63

3.8.1 Problem 64

3.8.2 Solution 64

3.8.3 How It Works 64

3.9 Drawing a Robot 67

3.9.1 Problem 67

3.9.2 Solution 67

3.9.3 How It Works 67

3.10 Summary 71

4 Kalman Filters 73

4.1 A State Estimator Using a Linear Kalman Filter 74

4.1.1 Problem 74

4.1.2 Solution 75

4.1.3 How It Works 75

4.2 Using the Extended Kalman Filter for State Estimation 92

4.2.1 Problem 92

4.2.2 Solution 93

4.2.3 How It Works 93

4.3 Using the Unscented Kalman Filter for State Estimation 97

4.3.1 Problem 97

4.3.2 Solution 97

4.3.3 How It Works 99

4.4 Using the UKF for Parameter Estimation 104

4.4.1 Problem 104

4.4.2 Solution 104

4.4.3 How It Works 104

4.5 Summary 108

5 Adaptive Control 109

5.1 Self Tuning: Modeling an Oscillator 110

5.2 Self Tuning: Tuning an Oscillator 112

5.2.1 Problem 112

5.2.2 Solution 112

5.2.3 How It Works 112

5.3 Implement Model Reference Adaptive Control 117

5.3.1 Problem 117

5.3.2 Solution 117

5.3.3 How It Works 117

5.4 Generating a Square Wave Input 121

5.4.1 Problem 121

5.4.2 Solution 121

5.4.3 How It Works 121

5.5 Demonstrate MRAC for a Rotor 123

5.5.1 Problem 123

5.5.2 Solution 123

5.5.3 How It Works 123

5.6 Ship Steering: Implement Gain Scheduling for Steering Controlof a Ship 126

5.6.1 Problem 126

5.6.2 Solution 126

5.6.3 How It Works 126

5.7 Spacecraft Pointing 130

5.7.1 Problem 130

5.7.2 Solution 130

5.7.3 How It Works 131

5.8 Summary 133

6 Fuzzy Logic 135

6.1 Building Fuzzy Logic Systems 136

6.1.1 Problem 136

6.1.2 Solution 136

6.1.3 How It Works 136

6.2 Implement Fuzzy Logic 139

6.2.1 Problem 139

6.2.2 Solution 139

6.2.3 How It Works 139

6.3 Demonstrate Fuzzy Logic 142

6.3.1 Problem 142

6.3.2 Solution 142

6.3.3 How It Works 143

6.4 Summary 146

7 Data Classification with Decision Trees 147

7.1 Generate Test Data 148

7.1.1 Problem 148

7.1.2 Solution 148

7.1.3 How It Works 148

7.2 Drawing Decision Trees 151

7.2.1 Problem 151

7.2.2 Solution 151

7.2.3 How It Works 151

7.3 Implementation 155

7.3.1 Problem 155

7.3.2 Solution 155

7.3.3 How It Works 155

7.4 Creating a Decision tree 158

7.4.1 Problem 158

7.4.2 Solution 158

7.4.3 How It Works 159

7.5 Creating a Handmade Tree 162

7.5.1 Problem 162

7.5.2 Solution 162

7.5.3 How It Works 163

7.6 Training and Testing 165

7.6.1 Problem 165

7.6.2 Solution 165

7.6.3 How It Works 166

7.7 Summary 169

8 Introduction to Neural Nets 171

8.1 Daylight Detector 171

8.1.1 Problem 171

8.1.2 Solution 172

8.1.3 How It Works 172

8.2 Modeling a Pendulum 173

8.2.1 Problem 173

8.2.2 Solution 174

8.2.3 How It Works 174

8.3 Single Neuron Angle Estimator 177

8.3.1 Problem 177

8.3.2 Solution 177

8.3.3 How It Works 178

8.4 Designing a Neural Net for the Pendulum 182

8.4.1 Problem 182

8.4.2 Solution 182

8.4.3 How It Works 182

8.5 Summary 186

9 Classification of Numbers Using Neural Networks 187

9.1 Generate Test Images with Defects 188

9.1.1 Problem 188

9.1.2 Solution 188

9.1.3 How It Works 188

9.2 Create the Neural Net Functions 192

9.2.1 Problem 192

9.2.2 Solution 193

9.2.3 How It Works 193

9.3 Train a Network with One Output Node 197

9.3.1 Problem 197

9.3.2 Solution 197

9.3.3 How It Works 198

9.4 Testing the Neural Network 202

9.4.1 Problem 202

9.4.2 Solution 202

9.4.3 How It Works 203

9.5 Train a Network with Many Outputs 203

9.5.1 Problem 203

9.5.2 Solution 203

9.5.3 How It Works 204

9.6 Summary 207

10 Pattern Recognition with Deep Learning 209

10.1 Obtain Data Online for Training a Neural Net 211

10.1.1 Problem 211

10.1.2 Solution 211

10.1.3 How It Works 211

10.2 Generating Training Images of Cats 211

10.2.1 Problem 211

10.2.2 Solution 211

10.2.3 How It Works 212

10.3 Matrix Convolution 215

10.3.1 Problem 215

10.3.2 Solution 215

10.3.3 How It Works 215

10.4 Convolution Layer 217

10.4.1 Problem 217

10.4.2 Solution 217

10.4.3 How It Works 217

10.5 Pooling to Outputs of a Layer 218

10.5.1 Problem 218

10.5.2 Solution 218

10.5.3 How It Works 219

10.6 Fully Connected Layer 220

10.6.1 Problem 220

10.6.2 Solution 220

10.6.3 How It Works 220

10.7 Determining the Probability 222

10.7.1 Problem 222

10.7.2 Solution 222

10.7.3 How It Works 222

10.8 Test the Neural Network 223

10.8.1 Problem 223

10.8.2 Solution 223

10.8.3 How It Works 223

10.9 Recognizing a Number 225

10.9.1 Problem 225

10.9.2 Solution 225

10.9.3 How It Works 226

10.10 Recognizing an Image 228

10.10.1 Problem 228

10.10.2 Solution 228

10.10.3 How It Works 228

10.11 Summary 230

11 Neural Aircraft Control 231

11.1 Longitudinal Motion 232

11.1.1 Problem 233

11.1.2 Solution 233

11.1.3 How It Works 233

11.2 Numerically Finding Equilibrium 238

11.2.1 Problem 238

11.2.2 Solution 238

11.2.3 How It Works 239

11.3 Numerical Simulation of the Aircraft 240

11.3.1 Problem 240

11.3.2 Solution 240

11.3.3 How It Works 240

11.4 Activation Function 242

11.4.1 Problem 242

11.4.2 Solution 242

11.4.3 How It Works 242

11.5 Neural Net for Learning Control 243

11.5.1 Problem 243

11.5.2 Solution 243

11.5.3 How It Works 243

11.6 Enumeration of All Sets of Inputs 248

11.6.1 Problem 248

11.6.2 Solution 248

11.6.3 How It Works 248

11.7 Write a Sigma-Pi Neural Net Function 249

11.7.1 Problem 249

11.7.2 Solution 249

11.7.3 How It Works 250

11.8 Implement PID Control 251

11.8.1 Problem 251

11.8.2 Solution 251

11.8.3 How It Works 252

11.9 PID Control of Pitch 256

11.9.1 Problem 256

11.9.2 Solution 256

11.9.3 How It Works 256

11.10 Neural Net for Pitch Dynamics 258

11.10.1 Problem 258

11.10.2 Solution 258

11.10.3 How It Works 258

11.11 Nonlinear Simulation 261

11.11.1 Problem 261

11.11.2 Solution 262

11.11.3 How It Works 262

11.12 Summary 264

12 Multiple Hypothesis Testing 265

12.1 Overview 265

12.2 Theory 267

12.2.1 Introduction 267

12.2.2 Example 269

12.2.3 Algorithm 269

12.2.4 Measurement Assignment and Tracks 270

12.2.5 Hypothesis Formation 271

12.2.6 Track Pruning 272

12.3 Billiard Ball Kalman Filter 274

12.3.1 Problem 274

12.3.2 Solution 274

12.3.3 How It Works 274

12.4 Billiard Ball MHT 280

12.4.1 Problem 280

12.4.2 Solution 280

12.4.3 How It Works 280

12.5 One-Dimensional Motion 285

12.5.1 Problem 285

12.5.2 Solution 285

12.5.3 How It Works 285

12.6 One-Dimensional Motion with Track Association 287

12.6.1 Problem 287

12.6.2 Solution 287

12.6.3 How It Works 287

12.7 Summary 289

13 Autonomous Driving with Multiple Hypothesis Testing 291

13.1 Automobile Dynamics 292

13.1.1 Problem 292

13.1.2 Solution 292

13.1.3 How It Works 292

13.2 Modeling the Automobile Radar 295

13.2.1 Problem 295

13.2.2 Solution 295

13.2.3 How It Works 295

13.3 Automobile Autonomous Passing Control 297

13.3.1 Problem 297

13.3.2 Solution 297

13.3.3 How It Works 298

13.4 Automobile Animation 299

13.4.1 Problem 299

13.4.2 How It Works 299

13.4.3 Solution 299

13.5 Automobile Simulation and the Kalman Filter 303

13.5.1 Problem 303

13.5.2 Solution 303

13.5.3 How It Works 303

13.6 Automobile Target Tracking 306

13.6.1 Problem 306

13.6.2 Solution 306

13.6.3 How It Works 306

13.7 Summary 309

14 Case-Based Expert Systems 311

14.1 Building Expert Systems 312

14.1.1 Problem 312

14.1.2 Solution 312

14.1.3 How It Works 313

14.2 Running an Expert System 313

14.2.1 Problem 313

14.2.2 Solution 313

14.2.3 How It Works 313

14.3 Summary 316

A A Brief History of Autonomous Learning 317

A.1 Introduction 317

A.2 Artificial Intelligence 317

A.3 Learning Control 320

A.4 Machine Learning 322

A.5 The Future 323

B Software for Machine Learning 325

B.1 Autonomous Learning Software 325

B.2 Commercial MATLAB Software 326

B.2.1 MathWorks Products 326

B.2.1.1 Statistics and Machine Learning Toolbox 326

B.2.1.2 Neural Network Toolbox 327

B.2.1.3 Computer Vision System Toolbox 327

B.2.1.4 System Identification Toolbox 327

B.2.1.5 MATLAB for Deep Learning 328

B.2.2 Princeton Satellite Systems Products 328

B.2.2.1 Core Control Toolbox 328

B.2.2.2 Target Tracking 328

B.3 MATLAB Open Source Resources 329

B.3.1 DeepLearnToolbox 329

B.3.2 Deep Neural Network 329

B.3.3 MatConvNet 329

B.4 Non- MATLAB Products for Machine Learning 329

B.4.1 R 330

B.4.2 scikit-learn 330

B.4.3 LIBSVM 330

B.5 Products for Optimization 330

B.5.1 LOQO 331

B.5.2 SNOPT 331

B.5.3 GLPK 331

B.5.4 CVX 332

B.5.5 SeDuMi 332

B.5.6 YALMIP 332

B.6 Products for Expert Systems 332

B.7 MATLAB MEX files 333

B.7.1 Problem 333

B.7.2 Solution 333

B.7.3 How It Works 333

Bibliography 337

Index 341

About the Authors

Michael Paluszek

../images/420697_2_En_BookFrontmatter_Figb_HTML.jpg

is President of Princeton Satellite Systems, Inc. (PSS) in Plainsboro, New Jersey. Mr. Paluszek founded PSS in 1992 to provide aerospace consulting services. He used MATLAB to develop the control system and simulations for the Indostar-1 geosynchronous communications satellite. This led to the launch of Princeton Satellite Systems first commercial MATLAB toolbox, the Spacecraft Control Toolbox, in 1995. Since then he has developed toolboxes and software packages for aircraft, submarines, robotics, and nuclear fusion propulsion, resulting in Princeton Satellite Systems current extensive product line. He is working with the Princeton Plasma Physics Laboratory on a compact nuclear fusion reactor for energy generation and space propulsion.

Prior to founding PSS, Mr. Paluszek was an engineer at GE Astro Space in East Windsor, NJ. At GE he designed the Global Geospace Science Polar despun platform control system and led the design of the GPS IIR attitude control system, the Inmarsat-3 attitude control systems and the Mars Observer delta-V control system, leveraging MATLAB for control design. Mr. Paluszek also worked on the attitude determination system for the DMSP meteorological satellites. Mr. Paluszek flew communication satellites on over twelve satellite launches, including the GSTAR III recovery, the first transfer of a satellite to an operational orbit using electric thrusters. At Draper Laboratory Mr. Paluszek worked on the Space Shuttle, Space Station and submarine navigation. His Space Station work included designing of Control Moment Gyro based control systems for attitude control.

Mr. Paluszek received his bachelors degree in Electrical Engineering, and master’s and engineers degrees in Aeronautics and Astronautics from the Massachusetts Institute of Technology. He is author of numerous papers and has over a dozen U.S. Patents. Mr. Paluszek is the author of “MATLAB Recipes” and “MATLAB Machine Learning” both published by Apress.

Stephanie Thomas

../images/420697_2_En_BookFrontmatter_Figc_HTML.jpg

is Vice President of Princeton Satellite Systems, Inc. in Plainsboro, New Jersey. She received her bachelors and masters degrees in Aeronautics and Astronautics from the Massachusetts Institute of Technology in 1999 and 2001. Ms. Thomas was introduced to the PSS Spacecraft Control Toolbox for MATLAB during a summer internship in 1996 and has been using MATLAB for aerospace analysis ever since. In her nearly 20 years of MATLAB experience, she has developed many software tools including the Solar Sail Module for the Spacecraft Control Toolbox; a proximity satellite operations toolbox for the Air Force; collision monitoring Simulink blocks for the Prisma satellite mission; and launch vehicle analysis tools in MATLAB and Java,. She has developed novel methods for space situation assessment such as a numeric approach to assessing the general rendezvous problem between any two satellites implemented in both MATLAB and C++. Ms. Thomas has contributed to PSS Attitude and Orbit Control textbook, featuring examples using the Spacecraft Control Toolbox, and written many software Users Guides. She has conducted SCT training for engineers from diverse locales such as Australia, Canada, Brazil, and Thailand and has performed MATLAB consulting for NASA, the Air Force, and the European Space Agency. Ms. Thomas is the author of “MATLAB Recipes” and “MATLAB Machine Learning” both published by Apress. In 2016, Ms. Thomas was named a NASA NIAC Fellow for the project “Fusion-Enabled Pluto Orbiter and Lander”.

Previous Chapter

Cover

Next Chapter

1. An Overview of Machine Learning

Table of Contents for
MATLAB Machine Learning Recipes: A Problem-Solution Approach

Using the Included Software

Contents

About the Authors

Table of Contents for MATLAB Machine Learning Recipes: A Problem-Solution Approach

Using the Included Software

Contents

About the Authors

Table of Contents for
MATLAB Machine Learning Recipes: A Problem-Solution Approach