Table of Contents for
Mastering PostGIS

Version ebook / Retour

Cover image for bash Cookbook, 2nd Edition Mastering PostGIS by Tomasz Nycz Published by Packt Publishing, 2017
  1. Mastering PostGIS
  2. Title Page
  3. Copyright
  4. Credits
  5. About the Authors
  6. About the Reviewers
  7. www.PacktPub.com
  8. Customer Feedback
  9. Table of Contents
  10. Preface
  11. What this book covers
  12. What you need for this book
  13. Who this book is for
  14. Conventions
  15. Reader feedback
  16. Customer support
  17. Downloading the example code
  18. Downloading the color images of this book
  19. Errata
  20. Piracy
  21. Questions
  22. Importing Spatial Data
  23. Obtaining test data
  24. Setting up the database
  25. Importing flat data
  26. Importing data using psql
  27. Importing data interactively
  28. Importing data non-interactively
  29. Importing data using pgAdmin
  30. Extracting spatial information from flat data
  31. Importing shape files using shp2pgsql
  32. shp2pgsql in cmd
  33. The shp2pgsql GUI version
  34. Importing vector data using ogr2ogr
  35. Importing GML
  36. Importing MIF and TAB
  37. Importing KML
  38. ogr2ogr GUI (Windows only)
  39. Importing data using GIS clients
  40. Exporting a shapefile to PostGIS using QGIS and SPIT
  41. Exporting shapefile to PostGIS using QGIS and DbManager
  42. Exporting spatial data to PostGIS from Manifold GIS
  43. Importing OpenStreetMap data
  44. Connecting to external data sources with foreign data wrappers
  45. Connecting to SQL Server Spatial
  46. Connecting to WFS service
  47. Loading rasters using raster2pgsql
  48. Importing a single raster
  49. Importing multiple rasters
  50. Importing data with pgrestore
  51. Summary
  52. Spatial Data Analysis
  53. Composing and decomposing geometries
  54. Creating points
  55. Extracting coordinates from points
  56. Composing and decomposing Multi-geometries
  57. Multi-geometry decomposition
  58. Composing and decomposing LineStrings
  59. LineString composition
  60. LineString decomposition
  61. Composing and decomposing polygons
  62. Polygon composition
  63. Polygon decomposition
  64. Spatial measurement
  65. General warning - mind the SRID!
  66. Measuring distances between two geometries
  67. Measuring the length, area, and perimeter of geometries
  68. Line length
  69. Polygon perimeter
  70. Polygon area
  71. Geometry bounding boxes
  72. Accessing bounding boxes
  73. Creating bounding boxes
  74. Using bounding boxes in spatial queries
  75. Geometry simplification
  76. Geometry validation
  77. Simplicity and validity
  78. Testing for simplicity and validity
  79. Checking for validity
  80. Repairing geometry errors
  81. Validity constraint
  82. Intersecting geometries
  83. Nearest feature queries
  84. Summary
  85. Data Processing - Vector Ops
  86. Primer - obtaining and importing OpenStreetMap data
  87. Merging geometries
  88. Merging polygons
  89. Merging MultiLineStrings
  90. Slicing geometries
  91. Splitting a polygon by LineString
  92. Splitting a LineString with another LineString
  93. Extracting a section of LineString
  94. Buffering and offsetting geometries
  95. Offsetting features
  96. Creating convex and concave hulls
  97. Computing centroids, points-on-surface, and points-on-line
  98. Reprojecting geometries
  99. Spatial relationships
  100. Touching
  101. Crossing
  102. Overlapping
  103. Containing
  104. Radius queries
  105. Summary
  106. Data Processing - Raster Ops
  107. Preparing data
  108. Processing and analysis
  109. Analytic and statistical functions
  110. Vector to raster conversion
  111. Raster to vector conversion
  112. Spatial relationship
  113. Metadata
  114. Summary
  115. Exporting Spatial Data
  116. Exporting data using \COPY in psql
  117. Exporting data in psql interactively
  118. Exporting data in psql non-interactively
  119. Exporting data in PgAdmin
  120. Exporting vector data using pgsql2shp
  121. pgsql2sph command line
  122. pgsql2shp gui
  123. Exporting vector data using ogr2ogr
  124. Exporting KML revisited
  125. Exporting SHP
  126. Exporting MapInfo TAB and MIF
  127. Exporting to SQL Server
  128. ogr2ogr GUI
  129. Exporting data using GIS clients
  130. Exporting data using QGIS
  131. Exporting data using Manifold.
  132. Outputting rasters using GDAL
  133. Outputting raster using psql
  134. Exporting data using the PostgreSQL backup functionality
  135. Summary
  136. ETL Using Node.js
  137. Setting up Node.js
  138. Making a simple Node.js hello world in the command line
  139. Making a simple HTTP server
  140. Handshaking with a database using Node.js PgSQL client
  141. Retrieving and processing JSON data
  142. Importing shapefiles revisited
  143. Consuming JSON data
  144. Geocoding address data
  145. Consuming WFS data
  146. Summary
  147. PostGIS – Creating Simple WebGIS Applications
  148. ExtJS says Hello World
  149. Configuring GeoServer web services
  150. Importing test data
  151. Outputting vector data as WMS services in GeoServer
  152. Outputting raster data as WMS services in GeoServer
  153. Outputting vector data as WFS services
  154. Making use of PgRaster in a simple WMS GetMap handler
  155. Consuming WMS
  156. Consuming WMS in ol3
  157. Consuming WMS in Leaflet
  158. Enabling CORS in Jetty
  159. Consuming WFS in ol3
  160. Outputting and consuming GeoJSON
  161. Consuming GeoJSON in ol3
  162. Consuming GeoJSON in Leaflet
  163. Outputting and consuming TopoJSON
  164. Consuming TopoJSON in ol3
  165. Consuming TopoJSON in Leaflet
  166. Implementing a simple CRUD application that demonstrates vector editing via web interfaces
  167. WebGIS CRUD server in Node.js
  168. WebGIS CRUD client
  169. Layer manager
  170. Drawing tools
  171. Analysis tools - buffering
  172. Summary
  173. PostGIS Topology
  174. The conceptual model
  175. The data
  176. Installation
  177. Creating an empty topology
  178. Importing Simple Feature data into topology
  179. Checking the validity of input geometries
  180. Creating a TopoGeometry column and a topology layer
  181. Populating a TopoGeometry column from an existing geometry
  182. Inspecting and validating a topology
  183. Topology validation
  184. Accessing the topology data
  185. Querying topological elements by a point
  186. Locating nodes
  187. Locating edges
  188. Locating faces
  189. Topology editing
  190. Adding new elements
  191. Creating TopoGeometries
  192. Splitting and merging features
  193. Splitting features
  194. Merging features
  195. Updating edge geometry
  196. Topology-aware simplification
  197. Importing sample data
  198. Topology output
  199. GML output
  200. TopoJSON output
  201. Summary
  202. pgRouting
  203. Installing the pgRouting extension
  204. Importing routing data
  205. Importing shapefiles
  206. Importing OSM data using osm2pgrouting
  207. pgRouting algorithms
  208. All pairs shortest path
  209. Shortest path
  210. Shortest path Dijkstra
  211. A-Star (A*)
  212. K-Dijkstra
  213. K-Shortest path
  214. Turn restrictions shortest path (TRSP)
  215. Driving distance
  216. Traveling sales person
  217. Handling one-way edges
  218. Consuming pgRouting functionality in a web app
  219. Summary

Making use of PgRaster in a simple WMS GetMap handler

As mentioned previously, PgRaster support in GeoServer is not there yet. This is good, because we can learn how to consume it ourselves!

Let's import some data first:

raster2pgsql -s 4326 -C -l 2,4,6,8,10,12,14 -I -F -t 256x256 NE2_HR_LC_SR_W_DR.tif webgis.ne_raster | psql -h localhost -p 5434 -U postgres -d mastering_postgis  

In Chapter 5, Exporting Spatial Data, we used PostgreSQL's large object support to export the raster from the database. We will now build on what we achieved there, so we can come up with a simple raster extractor query for our WMS handler. The interesting bit is the query we used for assembling the tiles of the imported raster back into one raster:

select  
ST_Union(rast) as rast
from
data_import.gray_50m_partial
where
filename = 'gray_50m_partial_bl.tif'

Our slightly extended query looks like this:

select 
--3. union our empty canvas with the extracted raster
ST_Union(rast) as rast
from (
--1. empty raster based on the passed bounds and raster settings of
the raster data is extracted from;
--this is our 'canvas' we will paint the extracted raster on.
--this lets us always output a raster that extends to the requested
bounds
select ST_AsRaster(ST_MakeEnvelope(14,85,24,95,4326), (select rast
from data_import.gray_50m_partial limit 1)) as rast

--2. extract the tiles of the raster that interset with bounds of out
request and clip them to the requested bound
union all select
ST_Clip(
ST_Union(rast),
ST_MakeEnvelope(14,85,24,95,4326)
)as rast
from
data_import.gray_50m_partial

where
ST_Intersects(rast, ST_MakeEnvelope(14,85,24,95,4326))

) as preselect

As a matter of fact, we do not do much more. Basically, what happens here is:

  • Generation of an empty raster that has the extent of the requested bounds and the parameters of the source raster we read from
  • Extraction of the source tiles that intersect with the requested bounds; tiles are further cropped with the very same bounds
  • Data assembly - we paint the natural earth raster on top of the empty canvas

At this stage, we have a raster we can output. To do so, we'll need a simple HTTP handler that can deal with our WMS requests. This will be a simplistic handler that is supposed to present the idea rather than be bullet-proof, production-ready code. Our WMS handler will only support a GetMap request.

Let's start with disassembling the WMS request itself into separate parameters, so it is clear what we are about to deal with. You may remember an example of a WMS request presented a few pages back - basically its query string will be similar to the following: ?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&FORMAT=image%2Fjpeg&TRANSPARENT=true&STYLES&LAYERS=mastering_postgis%3Ane_coastline&&WIDTH=256&HEIGHT=256&SRS=EPSG%3A4326&BBOX=-11.25%2C33.75%2C0%2C45.

It will be a bit easier when we look at the parameters one by one:

  • SERVICE=WMS: Service type
  • VERSION=1.1.1: Version of the WMS service
  • REQUEST=GetMap: Request type
  • FORMAT=image/jpeg: Output format
  • TRANSPARENT=true: Whether or not the background should be transparent; important when outputting vector data, but also when there are voids in the raster data
  • STYLES: Styles to be applied to the requested data
  • LAYERS=mastering_postgis:ne_coastline: Layers to be extracted
  • WIDTH=256: Width of the image
  • HEIGHT=256: Height of the image
  • SRS=EPSG:4326: Coordinate system of the request
  • BBOX=-11.25,33.75,0,45: Bounding box in the form of minx,miny,maxx,maxy
The bounding box described previously is specific to version < 1.3.0 of the WMS specification. Starting with version 1.3.0, the order of the bounding box coordinates depends on the order of coordinates defined by the SRS itself. Our WMS handler will only support v 1.1.1 of the specs.

Now, as we fully understand the WMS request parameters, we can move on to coding a simple handler in Node.js. We will build upon a web Hello World example that we wrote in a previous chapter:

The code for this example can be found in the code/wms directory.
const http = require('http'); 
const url = require('url');

const server = http.createServer((req, res) => {

console.warn('Processing WMS request...', req.url);

var params = url.parse(req.url, true).query; // true to get query
as object
//fix param casing; url param names should not be case sensitive!...
let pLowerCase = p.toLowerCase();
if(p !== pLowerCase){
params[pLowerCase] = params[p];
delete params[p];
}
}

//validate the request
if(vaidateRequest(res, params)){
processRequest(res, params);
}
});

const port = 8081; //another port so we can have it working with geoserver
server.listen(port, () => {
console.warn('WMS Server listening on http://localhost:%s', port);
});

As mentioned, our handler is going to be a simple one; therefore we will hardcode some logic that otherwise should be made dynamic, and thanks to that we'll keep the example clear.

Let's perform a simple checkup on the request parameter first so we can ensure that the submitted request is valid. According to the OGC specification, WMS should output exceptions in a specified form driven by the exception format parameter. In this case though, we will simply output 400, as handling exceptions the way that complies to the specs is not our task at this time. Our validation handler will therefore look like this:

/** 
* validates the WMS request; returns true if the request is valid and false otherwise. if request is nod valid response writes 400 and closes
*/
const validateRequest = (res, params) => {
var valid = true;
try {
for(var validator of validationRules){
validator(params);
}
}
catch(e){
valid = false;
handleError(res, e);
}
return valid;
}

/**
* handles exception response
*/
const handleError = (res, msg) => {
res.statusCode = 400;
res.end(msg);
};

We'll need some validation rules too:

const validationRules = [ 
(params) => {validateParamPresence(params, 'service')},
(params) => {if(params.service !== 'WMS'){throw 'This service only
supports WMS'}},
(params) => {validateParamPresence(params, 'version')},
(params) => {if(params.version !== '1.1.1'){throw 'The only supported version
is 1.1.1';}}, (...)
];
Do review the source code, as there are more validation rules applied to a request before it can be processed.

Once our request is validated, we should be able to render a map image safely (without errors). In order to do so, we have to talk to our database and for this, we need the pg module first:

npm install --pg save

Let's extract the params we need to process the request:

/** 
* generates wms output based on the params. params should be validated prior to calling this method
*/
const processRequest = (res, params) => {

//prepare some params first
let w = parseInt(params.width);
let h = parseInt(params.height);
let bb = params.bbox.split(',');
let minX = parseFloat(bb[0]);
let minY = parseFloat(bb[1]);
let maxX = parseFloat(bb[2]);
let maxY = parseFloat(bb[3]);
let format = getGdalFormat(params.format);

//get table name based on tile resolution expressed in map units
let tableName = getTableName(Math.abs(maxX - minX) / w);
}

The database connection skeleton is not very complex and actually it is pretty much what we have seen before:

//init client with the appropriate conn details 
const client = new pg.Client({
host: 'localhost',
port: 5434,
user: 'postgres',
password: 'postgres',
database: 'mastering_postgis'
});

//connect to the database
client.connect(function(err){
if(err){
handleError(err);
return;
}

let query = `TODO`;

client.query(query, function(err, result){
client.end();

if(err){
handleError(err);
return;
}

//handle response
res.statusCode = 200;
res.setHeader('content-type', params.format);
res.end(result.rows[0].rast);
});
});

The last part we have left is the actual query. We have experimented with it a bit, so it is now time to make it dynamic:

let query = ` 
select
--3. union our empty canvas with the extracted raster and resize it to the requested tile size
ST_AsGDALRaster(
ST_Resample(
ST_Union(rast),
$1::integer,
$2::integer,
NULL,NULL,0,0,'Cubic',0.125
),
$3
)as rast
from (
--1. empty raster based on the passed bounds and raster settings of the raster data is extracted from;
--this is our 'canvas' we will paint the extracted raster on.
--this lets us always output a raster that extends to the requested bounds
select ST_AsRaster(ST_MakeEnvelope($4,$5,$6,$7,4326), (select rast from webgis.${tableName} limit 1)) as rast

--2. extract the tiles of the raster that interset with bounds of out request and clip them to the requested bound
union all select
ST_Clip(
ST_Union(rast),
ST_MakeEnvelope($4,$5,$6,$7,4326)
)as rast
from
webgis.${tableName}
where
ST_Intersects(rast, ST_MakeEnvelope($4,$5,$6,$7,4326))
) as preselect
;

As you can see, the preceding query is almost the same as the one we saw already; the main difference is its parameterization. Also, I have added a ST_Resample call with the Cubic resampling algorithm, so the resized images look smooth, and an ST_AsGDALRaster call so we get the binary data that we can pipe straight into the response.

At this stage, our WMS handler should be ready, so let's launch it via the node index.js command and paste the following URL into the browser's address bar: http://localhost:8081/?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&STYLES&LAYERS=ne_raster&&WIDTH=256&HEIGHT=256&SRS=EPSG%3A4326&BBOX=-11.25%2C33.75%2C0%2C45.

The expected output is the Iberian Peninsula cut out of an o_2_ne_raster table:

Congratulations! You have just created your very own Geo-Server capable of serving WMS GetMap requests. Obviously, our implementation is quite limited and should be considered a rather basic one--our point though was to consume PgRaster, not to create a full- blown service.