Archive for the ‘wktraster’ Category

GDAL/OGR 1.7.0 Released

Saturday, January 30th, 2010

GDAL logoFrank has just posted announcement about freshly released GDAL/OGR 1.7.0:

This is the first major new release since the 1.6.0 release approximately one year ago

This new version brings quite a nice collection of new drivers for raster and vector data formats:

  • New Raster Drivers: BAG, EPSILON, Northwood/VerticalMapper, R, Rasterlite, SAGA GIS Binary, SRP (USRP/ASRP), EarthWatch .TIL, WKT Raster
  • GDAL PCIDSK driver using the new PCIDSK SDK by default
  • New Vector drivers : DXF, GeoRSS, GTM, PCIDSK and VFK
  • New utilities: gdaldem, gdalbuildvrt now compiled by default
  • Add support for Python 3.X. Compatibility with Python 2.X preserved
  • Remove old-generation Python bindings
  • Significantly improved raster drivers: GeoRaster, GeoTIFF, HFA, JPEG2000 JasPer, JPEG2000 Kakadu, NITF
  • Significantly improved vector drivers: CSV, KML, SQLite/SpataiLite, VRT

Crunching overviews

Wednesday, June 3rd, 2009

Continuing my tale about loading big raster datasets into PostGIS database with WKT Raster extension, I’d like to post an update about experience with processing overviews.

For testing purposes, I built excessive number of overviews for japan.tif dataset using gdaladdo utility:

$ gdaladdo -r average japan.tif 2 4 8 16 32 64 128

The command above produced 7 overviews with the following dimensions:

$ gdalinfo japan_2_4_8_16_32_128.tif | grep -m 1 Ov
Overviews: 7000x7000, 3500x3500, 1750x1750, 875x875, 438x438, 219x219, 110x110

(more…)

PostGIS In Action

Thursday, May 7th, 2009

It really must be very hot and fresh news, so the virtual devil spirit of social networking hasn’t fished it out yet and also Google (check this if you haven’t heard yet about this local family company) lists less than 15 pages.

PostGIS provides over 300 spatial operators, spatial functions, spatial data types and spatial indexing enhancements. If you add to the mix the complimentary features that PostgreSQL and other PostgreSQL related projects provide, then you’ve got one jam-packed powerhouse at your disposal well suited for hardcore work as well as a valuable training tool for spatial concepts.

Three words: PostGIS In Action. The first book about PostGIS spatial database being written by Regina O. Obe and Leo S. Hsu with release planned for the beginning of 2010. First chapter of the book has been published and is freely available as PDF file through the Manning Early Access Program. Chapter two and three are also available for MAEP subscribers.

Next to the early access, another cool thing about the way Manning Publications release their books is possibility to comment chapters and discuss with authors directly through Manning Sandbox forums. There is no exception for the PostGIS in Action :-)

Update 2009-05-08T23:08:21+00:00: The book official announcement has been posted on postgis-devel and postgresqlonline.com.

I’m looking forward to grab the book!

Pierre, check the TOC for chapter thirteen. Cool, isn’t it?

WKT Raster Cruncher

Monday, March 30th, 2009

I’ve been playing with some nice amount of pixels using WKT Raster engine recently. Today, I tiled the japan.tif (gdalinfo output is here) using 3 different sizes of tile block. All the work was done using gdal2wktraster.py loader available from WKT Raster repository. Below, I included more interesting numbers which may be helpful for others who will decide to juggle rasters in PostGIS database.

Tiling schemes applied on japan.tif file:

  1. Using natural block size reported by GDAL gives 700 tiles, 14000 x 20 pixels each:
    gdal2wktraster.py -r japan.tif -t japan -o japan.sql -k
  2. Using block size 100 x 100 pixels gives 140 x 140 = 19600 tiles:
    gdal2wktraster.py -r japan.tif -t japan_100 -o japan_100.sql -k -m 100x100
  3. Using block size 200 x 200 pixels gives 70 x 70 = 4900 tiles:
    gdal2wktraster.py -r japan.tif -t japan_100 -o japan_200.sql -k -m 200x200

Sizes of all .sql text files produced are close to 1.2 GB. The timing for gdal2wktarster falls to around 40 minutes on Ubuntu 8.10 (i386) running under VirtualBox 2.with assigned single CPU core of Intel Xeon 3.2 GHz and 1024 MB RAM (host machine: 4 x Intel Xeon 3.2 GHz with 16 GB RAM). psql loads single table in about 6 minutes:

psql -d japan -f japan_100.sql

Interestingly, memory usage during processing WKT Raster by gdal2wktraster.py (Python run-time) seems not high (10-20%) and was on constant level of ~85 MB. However, 100% of CPU power was being eaten. It’s got me interested in comparing these results vector-brother of gdal2wktraster, to the shp2pgsql from PostGIS.

Run WKT Raster, run!

Friday, March 27th, 2009

I have a not-so-small GeoTIFF raster dataset. Here is what GDAL has to say about it:

$ gdalinfo japan.tif
Driver: GTiff/GeoTIFF
Files: japan.tif
Size is 14000, 14000
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.2572235630016,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (137.386330630776030,38.325757833006122)
Pixel Size = (0.000256020461176,-0.000256020461176)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_DOCUMENTNAME=ER Mapper 6.4
  TIFFTAG_IMAGEDESCRIPTION=ER Mapper GeoTiff raster translator V1.0: Band 1 = Red, Band 2 = Green, Band 3 = Blue
  TIFFTAG_SOFTWARE=ER Mapper 6.4
  TIFFTAG_XRESOLUTION=0
  TIFFTAG_YRESOLUTION=0
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  ( 137.3863306,  38.3257578) (137d23'10.79"E, 38d19'32.73"N)
Lower Left  ( 137.3863306,  34.7414714) (137d23'10.79"E, 34d44'29.30"N)
Upper Right ( 140.9706171,  38.3257578) (140d58'14.22"E, 38d19'32.73"N)
Lower Right ( 140.9706171,  34.7414714) (140d58'14.22"E, 34d44'29.30"N)
Center      ( 139.1784739,  36.5336146) (139d10'42.51"E, 36d32'1.01"N)
Band 1 Block=14000x20 Type=Byte, ColorInterp=Red
Band 2 Block=14000x20 Type=Byte, ColorInterp=Green
Band 3 Block=14000x20 Type=Byte, ColorInterp=Blue

I loaded it to PostGIS/WKT Raster table with regular blocking enabled in PostgreSQL 8.3 database running on Ubuntu 8.10 (32-bit) installed as a guest system under VirtualBox:

$ gdal2wktraster.py -r japan.tif -t japan_rb -o japan_rb.sql -k
$ psql -d test -f japan_rb.sql

The loader generated output file japan_rb.sql of size of 1.1 GB and it makes 700 records (raster tiles or blocks) in the database. The disk usage reported for the raster table is:

sistest=# SELECT relfilenode, relpages FROM pg_class WHERE relname = 'japan_rb';
 relfilenode | relpages
-------------+----------
       71700 |        5
(1 row)

I run simple test query:

$ psql -d test
test=# SET log_statement_stats TO 1;
SET
test=# SELECT rid FROM japan_rb WHERE RT_Width(rast) != 14000 OR RT_Height(rast) != 20;
 rid
-----
(0 rows)

In the PostgreSQL log I got dumped a bunch of interesting statistics:

2009-03-27 15:23:38 GMT LOG:  QUERY STATISTICS
2009-03-27 15:23:38 GMT DETAIL:  ! system usage stats:
 ! 5.084838 elapsed 2.544159 user 2.468154 system sec
 ! [4.960310 user 5.204325 sys total]
 ! 0/0 [16/28160] filesystem blocks in/out
 ! 0/320190 [0/647479] page faults/reclaims, 0 [0] swaps
 ! 0 [0] signals rcvd, 0/0 [0/0] messages rcvd/sent
 ! 0/155 [9/327] voluntary/involuntary context switches
 ! buffer usage stats:
 ! Shared blocks:      74192 read,          0 written, buffer hit rate = 51.71%
 ! Local  blocks:          0 read,          0 written, buffer hit rate = 0.00%
 ! Direct blocks:          0 read,          0 written

The virtual machine has assigned 1024 MB of RAM and one CPU Intel Xeon 3.2 GHz.

You are strongly welcome to share your comments?

WKT Raster crash course #1

Sunday, March 8th, 2009

Recently, a good friend of mine Sandro has started spreading the story of WKT Raster project. Here, I’d like to join him and post a bunch of technical notes about how to start using tandem of PostGIS and WKT Raster. This is the first post that hopefully will start a series discussing various aspects of use and development of WKT Raster extension.

What

From the overview of WKT Raster project:

WKT Raster is an ongoing project aiming at developing raster support in PostGIS. (…) WKT Raster’s goal is to implement the RASTER type as much as possible like the GEOMETRY type is implemented in PostGIS and to offer a single set of overlay SQL functions (like ST_Intersects) operating seamlessly on vector and raster coverages.

The idea of WKT Raster extension was presented (PDF 1.1 MB) in December 2008 by Pierre Racine from University Laval. Pierre’s presentation made foundation of the WKT Raster Specification.

Hacker #1: Why WKTRaster? sounds a bit silly :D
Hacker #2: Apparently the initial thoughts on it were expressed using a textual representation for the rasters and the thinking was that this would be core to the effort, though it has proven not so important. I also find the name unhelpful.

Actually, this is not as trivial problem as it may seem. I am not sure myself that the name is WKT Raster or WKTRaster, nor I’m confident if the name is fixed and will be valid in future.

Update 2009-03-08 14:05: See Pierre’s comment below.

Where

At the moment, WKT Raster project does not have a single home, but different parts of it live in different places:

Who

Currently, there is a small team of people who contribute their time, skills and money to move the project forward. Here is a summary of contributors everyone can collect from the scattered homepage(s) of WKT Raster:

  • Pierre Racine – originator and author of the WKT Raster idea and specification and implementation developer.
  • Sandro Santilli – well-known member of the PostGIS core team who is the architect of RASTER type definition, canonical and Well-Known-Binary format of stored raster, developer of input/output operations, and more.

There are also prime financial contributors who established development of the WKT Raster project: Steve Cumming, Martin Daly (from Cadcorp, where I work as member of core development team) and Tyler Erickson.

Recently, I’ve also joined the WKT Raster team and being led by Martin Daly I’m focused on programming new features and core testing.

News

Today and during next 3 days, a few people interested in the PostGIS / WKT Raster project are meeting together for the Toronto Code Sprint 2009. I’m also present there, though in spirit by joining the event on #tosprint IRC channel. I have a hope that during this meeting, we will discuss and agree about a couple of outstanding issues:

  • meta data tables: Do we need them? How to define schema of meta data?
  • review of specification of RT functions
  • improvements in the solution architecture and design to handle common raster use cases, for both categories, visualizations and analysis
  • how to answer needs of pyramids/overviews
  • does the project need an infrastructure, a new home?
  • and last but not least, how to achieve sustainability and rise more funds.

By the way, isn’t it good time to create WKT Raster article on Wikipedia? Writers are welcome!