Table of Contents for
Mastering Geospatial Analysis with Python

Version ebook / Retour

Cover image for bash Cookbook, 2nd Edition Mastering Geospatial Analysis with Python by Paul Crickard Published by Packt Publishing, 2018
  1. Python Scripting for GIS
  2. Title Page
  3. Copyright and Credits
  4. Mastering Geospatial Analysis with Python
  5. Packt Upsell
  6. Why subscribe?
  7. PacktPub.com
  8. Contributors
  9. About the authors
  10. About the reviewer
  11. Packt is searching for authors like you
  12. Table of Contents
  13. Preface
  14. Who this book is for
  15. What this book covers
  16. To get the most out of this book
  17. Download the example code files
  18. Download the color images
  19. Conventions used
  20. Get in touch
  21. Reviews
  22. Package Installation and Management
  23. Introducing Anaconda
  24. Installing Python using Anaconda
  25. Running a Jupyter Notebook
  26. Running a Notebook
  27. Creating a new Notebook
  28. Adding code
  29. Managing Python packages
  30. Managing packages with Anaconda Navigator
  31. Online searching for packages using Anaconda Cloud
  32. Managing Python packages with conda
  33. Managing Python packages using pip
  34. Upgrading and uninstalling the package with pip
  35. Python virtual environments
  36. Virtual environments using Anaconda
  37. Managing environments with conda 
  38. Virtual environments using virtualenv
  39. Summary
  40. Introduction to Geospatial Code Libraries
  41. Geospatial Data Abstraction Library (GDAL) and the OGR Simple Features Library
  42. Installing GDAL
  43. Installing GDAL using Anaconda3
  44. Installing GDAL using conda
  45. Installing GDAL using pip
  46. Installing a second GDAL version using pip
  47. Other recommended GDAL resources
  48. GEOS
  49. Installing GEOS
  50. Shapely
  51. Installing Shapely
  52. Fiona
  53. Installing Fiona
  54. Python shapefile library (pyshp)
  55. Installing pyshp
  56. pyproj
  57. Installing pyproj
  58. Rasterio
  59. Rasterio dependencies
  60. Installation of Rasterio
  61. GeoPandas
  62. GeoPandas installation
  63. GeoPandas dependencies
  64. How it all works together
  65. Summary
  66. Introduction to Geospatial Databases
  67. Installing PostgreSQL and PostGIS on Windows
  68. Installing PostgreSQL and PostGIS on Mac
  69. Working with PostgreSQL and PostGIS using Python
  70. Connecting to PostgreSQL using psycopg2
  71. Installing psycopg2
  72. Connecting to the database and creating a table
  73. Adding data to the table
  74. Shapely
  75. Querying the data
  76. Changing the CRS
  77. Buffer
  78. Distance and near
  79. Lines in the database
  80. Length of a line
  81. Intersecting lines
  82. Polygons in the database
  83. Point in polygon
  84. Summary
  85. Data Types, Storage, and Conversion
  86. Raster and vector data
  87. Shapefiles
  88. GeoJSON
  89. KML
  90. GeoPackage
  91. Raster data formats
  92. Reading and writing vector data with GeoPandas
  93. Reading and writing vector data with OGR
  94. Reading and writing raster data with Rasterio
  95. Reading and writing raster data using GDAL
  96. Summary
  97. Vector Data Analysis
  98. OGR Simple Features Library
  99. OGR batch commands
  100. ogrmerge
  101. The OGR library and Python bindings
  102. OGR's main modules and classes
  103. Creating polygon geometry with OGR
  104. Creating polygon geometry from GeoJSON
  105. Basic geometric operations
  106. Writing polygon data to a newly created shapefile
  107. Using a spatial filter to select features
  108. Shapely and Fiona
  109. Shapely objects and classes
  110. Shapely methods for geospatial analysis
  111. Fiona's data model
  112. Creating geometries with Shapely
  113. Applying geometrical methods with Shapely
  114. Reading JSON geometries with Shapely
  115. Reading data with Fiona
  116. Accessing vector geometry in shapefiles using Shapely and Fiona
  117. GeoPandas
  118. Geospatial analysis with GeoPandas
  119. Selecting and plotting geometry data with GeoPandas and Matplotlib
  120. Mapping wildfire data with GeoPandas
  121. Why data inspection matters
  122. Summary
  123. Raster Data Processing
  124. Raster operations using GDAL
  125. Using the GDAL library to load and query rasters
  126. Using GDAL to create rasters
  127. Raster operations using PostgreSQL
  128. Loading rasters into PostgreSQL
  129. Performing queries on rasters using PostgreSQL
  130. Querying raster metadata
  131. Queries returning geometry
  132. Queries returning values
  133. Summary
  134. Geoprocessing with Geodatabases
  135. A crime dashboard
  136. Building a crime database
  137. Creating the tables
  138. Populating the data
  139. Mapping queries
  140. Incidents by date
  141. Incidents in a polygon
  142. Buffers
  143. Nearest neighbor
  144. Interactive widgets
  145. Charts
  146. Triggers
  147. Summary
  148. Automating QGIS Analysis
  149. Working in the Python console
  150. Loading layers
  151. Processing a layer
  152. Layer properties
  153. Feature properties
  154. Drawing a layer from PostGIS
  155. Drawing points
  156. Drawing polygons from PostGIS
  157. Adding, editing, and deleting features
  158. Adding features to an existing layer
  159. Deleting features from an existing layer
  160. Editing features from an existing layer
  161. Selecting features using expressions
  162. Using toolboxes in Python
  163. Writing custom toolboxes
  164. Summary
  165. ArcGIS API for Python and ArcGIS Online
  166. Introducing the ArcGIS API for Python and ArcGIS Online
  167. A Pythonic web API
  168. Installing the API
  169. Testing the API
  170. Troubleshooting
  171. Authenticating your Esri user accounts
  172. Different Esri user accounts
  173. Different modules of the ArcGIS API for Python
  174. Exercise 1 – importing the API and using the map widget
  175. Creating a personalized ArcGIS Online account
  176. Exercise 2 – searching, displaying, and describing geospatial content
  177. Exercise 3 – working with raster data and the API's geoprocessing functions
  178. Summary
  179. Geoprocessing with a GPU Database
  180. Cloud geodatabase solutions
  181. Big data processing
  182. MapD architecture
  183. Cloud versus local versus combined
  184. Creating a MapD instance in the cloud
  185. Finding the AMI
  186. Opening an AWS account
  187. Creating a key pair
  188. Launching an instance
  189. Picking a version
  190. Searching for an instance
  191. Setting up a security group
  192. Immerse environment
  193. Logging in to Immerse
  194. Default dashboards
  195. NYC taxi dataset
  196. Importing a CSV
  197. Creating a chart
  198. Selections with the SQL EDITOR
  199. Use geospatial data
  200. Connecting to the database using a terminal
  201. PuTTYgen
  202. Connection configuration
  203. Using the private key
  204. Installing pymapd
  205. The conda install command
  206. The pip install command
  207. Creating a connection
  208. User and password
  209. Data cursor
  210. Creating a table
  211. Insert statements
  212. Using Apache Arrow to load data
  213. Contains queries
  214. Other available spatial SQL commands
  215. Summary
  216. Flask and GeoAlchemy2
  217. Flask and its component modules
  218. Setup
  219. Installing modules using pip
  220. Installing Flask using pip
  221. Installing Flask-SQLAlchemy via pip
  222. Installing GeoAlchemy2 using pip
  223. Installing Flask-WTForms and WTForms using pip
  224. Installing psycopg2 using pip
  225. Installing SQLAlchemy-Utils using pip
  226. Installing pyshapefile (or pyshp) using pip
  227. Installing pygeoif using pip
  228. Writing a Flask application
  229. Downloading the data from a data source
  230. County, district, state, and arena shapefiles
  231. Creating the database and data tables
  232. Adding the PostGIS extension tables to the new database
  233. Defining the database tables
  234. The declarative base
  235. Database table model classes 
  236. Table properties
  237. Creating the tables
  238. Inserting data into the new data tables
  239. Importing the required modules
  240. Locating and reading the shapefiles
  241. Accessing shapefile data
  242. Using queries
  243. Components of the Flask application
  244. Folder structure and the controller object
  245. Models
  246. Forms
  247. Views
  248. Dissecting the view
  249. Using forms
  250. Evaluating the request method
  251. POST requests
  252. Spatial queries
  253. Relationship query
  254. The web map template
  255. Running the web application locally
  256. Summary
  257. GeoDjango
  258. Installing and configuring Django and GeoDjango
  259. Steps from Django to GeoDjango
  260. Installing Django
  261. Installing PostGIS and psycopg2
  262. Creating the database
  263. GDAL/OGR
  264. Modifying Windows environment variables
  265. Creating a project and application
  266. Command-line argument – startproject
  267. What is created by startproject?
  268. Creating an application using manage.py
  269. What is created by manage.py
  270. Configuring settings.py
  271. Adding a new database connection
  272. Adding new installed apps
  273. Creating the application
  274. manage.py
  275. Generating data models
  276. Multipolygons 
  277. Database migrations
  278. makemigrations 
  279. sqlmigrate
  280. migrate
  281. LayerMapping
  282. Running the layer mapping 
  283. Administrative panel
  284. GeoDjango administrative panel
  285. admin.py
  286. createsuperuser
  287. runserver
  288. URLs
  289. URL patterns
  290. Views
  291. Required folders and files
  292. forms.py
  293. templates folder
  294. Writing views
  295. index view
  296. queryarena function
  297. arena view
  298. Running the application
  299. Summary
  300. Geospatial REST API
  301. Writing a REST API in Python
  302. REST
  303. JSON
  304. Python for REST API
  305. Flask
  306. REST modules
  307. Other frameworks
  308. Variables in Flask URLs
  309. Number converters
  310. Other data converters
  311. Request methods
  312. GET
  313. POST
  314. Other available request methods
  315. PUT
  316. DELETE
  317. The REST API application
  318. Application components
  319. Application folder and file structure
  320. app.py
  321. __init__.py
  322. The database
  323. models.py
  324. Importing required modules
  325. Declaring the session
  326. Declaring the models
  327. forms.py
  328. views.py
  329. Importing modules
  330. Base URL
  331. Arenas
  332. Getting all arenas
  333. Getting arenas by ID
  334. Getting arenas by name
  335. A geospatial query
  336. States
  337. Getting all states
  338. Getting a state by ID
  339. Getting a state by name
  340. Getting arenas by state
  341. Counties
  342. Getting a county by ID
  343. Getting a county by name
  344. Districts
  345. Getting all districts
  346. Getting a district by ID
  347. Getting a district by name
  348. API POST endpoints  
  349. New arenas
  350. The view function
  351. The addarena.html head
  352. The addarena.html script
  353. The addarena.html form
  354. Sending a POST request using the requests library
  355. Deleting an arena
  356. Running the REST API locally
  357. Deploying Flask to IIS
  358. Flask and web servers
  359. WSGI
  360. Installing the WFastCGI module and FastCGI
  361. Configuring FastCGI
  362. Root server settings and Environment Variables
  363. Summary
  364. Cloud Geodatabase Analysis and Visualization
  365. How to install CARTOframes
  366. Additional resources
  367. Jupyter Notebooks
  368. The CARTO API key
  369. Package dependencies
  370. The CARTO Data Observatory
  371. Signing up for a CARTO account
  372. A free trial of CARTO
  373. Adding a dataset
  374. The API key
  375. Adding a dataset
  376. Virtual environments
  377. Installing virtualenv
  378. Running virtualenv
  379. Activating the virtual environment
  380. Installing modules in virtualenv
  381. Modules to use
  382. Using Jupyter Notebook
  383. Connecting to an account
  384. Saving credentials
  385. Accessing a dataset
  386. Selecting individual rows
  387. Loading a CSV dataset
  388. Loading a shapefile
  389. Installing GeoPandas
  390. Writing to CARTO
  391. Loading CSV with geometry
  392. Geospatial analysis
  393. Editing and updating datasets
  394. overwrite=True
  395. Creating a map
  396. Summary
  397. Automating Cloud Cartography
  398. All things cartographic
  399. How to integrate Mapbox into your GIS 
  400. Mapbox tools
  401. MapboxGL.js
  402. Mapbox Python SDK
  403. Installing the Python SDK
  404. Getting started with Mapbox
  405. Signing up for a Mapbox account
  406. Creating an API token
  407. Adding data to a Mapbox account
  408. Tilesets
  409. Datasets
  410. Example – uploading a GeoJSON dataset
  411. Example – uploading data as a tileset
  412. Mapbox Studio
  413. Customizing a basemap
  414. Adding a tileset
  415. Virtual environment
  416. Installing MapboxGL – Jupyter
  417. Installing Jupyter Notebooks
  418. Installing Pandas and GeoPandas
  419. Using the Jupyter Notebook server
  420. Importing data using GeoPandas
  421. Creating point data from polygons
  422. Data cleanup 
  423. Saving the points as GeoJSON
  424. Adding the points to a map
  425. Creating a graduated color visualization
  426. Automatically setting colors, sizes, and breaks
  427. Creating a choropleth map
  428. Saving the map
  429. Creating a heat map
  430. Uploading data using the Mapbox Python SDK
  431. Creating the dataset
  432. Loading the data into the dataset
  433. Reading data from a dataset
  434. Deleting a row
  435. Summary
  436. Python Geoprocessing with Hadoop
  437. What is Hadoop?
  438. Installing the Hadoop framework
  439. Installing Linux
  440. Installing Docker
  441. Install Hortonworks
  442. Hadoop basics
  443. Connecting via Secure Shell
  444. Ambari
  445. Esri GIS tools for Hadoop
  446. HDFS and Hive in Python
  447. Summary
  448. Other Books You May Enjoy
  449. Leave a review - let other readers know what you think

Cloud geodatabase solutions

Cloud storage of geospatial data has become a common part of many GIS architectures. Whether it is used as a backup to an on-premises solution, replaces an on-premises solution, or is combined with a local solution to provide internet support for an intranet-based system, the cloud is a big part of the future of GIS. 

With ArcGIS Online, CARTO, MapBox, and now MapD, the options for a cloud data store that support geospatial data are more numerous than ever. Each offers a visualization component and a different type of data storage and each will integrate with your data and software in different ways.

ArcGIS Online, while also offering stand-alone options (that is, direct data upload), integrates with ArcGIS Enterprise (formerly ArcGIS Server) to consume enterprise REpresentational State Transfer (REST) web services that are stored on a local geodatabase. ArcGIS Online is built on top of Amazon Web Services (AWS) and all of the server architecture is hidden from users. Enterprise integration requires a high-level of licensing (cost), which includes a number of cloud tokens (that is credits), and storage and analysis within the cloud account itself can use lots of those tokens.

CARTO offers cloud PostGIS storage, allowing for geospatial data files to be uploaded. With the release of the Python package CARTOframes (covered in Chapter 14, Cloud Geodatabase Analysis and Visualization), the cloud datasets can be uploaded and updated using scripting. Using Python, a CARTO account can become a part of an enterprise solution that maintains up-to-date datasets while allowing them to be quickly deployed as custom web maps using the builder application. CARTO offers two tiers of paid accounts which have different levels of storage.

MapBox is focused on map tools for creating custom basemaps for mobile apps, but it also offers cloud data storage of datasets and map creation tools such as MapBox GL, the JavaScript library for maps built on the Web Graphics Library (WebGL). With the new MapBox GL—Jupyter module, the data can be accessed using Python. 

MapD, while offering similar solutions to those mentioned, is different in a number of respects. It has an open source version of the database (MapD Core Community Edition) which can be used locally or on the cloud, and has an enterprise version for large customers. While MapD Core has a relational database schema and uses SQL for queries like a traditional RDBMS, it uses GPUs to accelerate queries. MapD Core can be cloud-deployed on AWS, Google Cloud Platform, and Microsoft Azure. MapD can be installed on servers without GPUs as well, though this reduces its effective speed gains over other geodatabases.

All of the geodatabases support Jupyter Notebook environments for data queries, but MapD has them integrated into the SQL EDITOR within the Immerse visualization platform. MapD uses Apache Arrow to upload data when using pymapd and also supports INSERT statements while allowing for data to be loaded using the Immerse data importer (including SHPs, GeoJSONs, and CSVs).