Fellow LizardTech developer Kirk and I just got back from the third annual OSGeo Code Sprint; this time in Montreal, Canada. Daniel Morissette did a great job of organizing this event and has already posted an excellent review of the sprint (here) including pictures (here), so I’ll simply add that that it was great fun to meet personally with the guys who build great libraries like PostGIS, libLAS/libPC and GDAL.

Kirk and I worked on adding MG4 (MrSID Generation 4) support to the new libPC. The “PC” stands for “point cloud”. libPC is the next evolution of libLAS and aims to be the GDAL of point cloud formats. We didn’t finish, but once this work is done it will enable decode support in libPC for compressed LiDAR data in MrSID format. For more info, see http://libpc.org/notes/goals.html.

Fun fact:  The pronunciation of GDAL is the subject of some disagreement. It seems obvious to me and Kirk that “GeeDAL” is correct (like “Vi”, the editor or “Unix”, the OS).  I was amused to hear that in French-speaking Montreal it’s sometimes “J’DAL.” For reasons that are not clear to me some of the inner circle (Frank W and Hobu) usually pronounce it “GuuDAL.”

Enjoying a break after sprinting. Image courtesy of OSGeo Code Sprint 2011.

On a more serious note, much of the GIS industry’s foundation code depends on some of these libraries. That’s good for all of us: quality libraries allow us all to write valuable applications quickly with high confidence in the underlying stability and reliability. They allow us to focus on what we’re good at and (hopefully) what is important to our users. That’s really valuable and it doesn’t cost very much to give a little back.

1. If you work around a limitation, return the patch to the community. You don’t need to be a committer, just send it in. It’s in everyone’s interest to consider your patch.
2. If you can’t work around it, offer to pay the community to fix it. Some of these guys make a living doing this and our experience is that the costs are often quite reasonable.
3. Consider direct financial support of key projects and/or the OSGeo Foundation (http://www.osgeo.org).

Several of us just returned from the ILMF 2010 conference in Denver, Colorado. What was near and dear to my heart there was the interest in LiDAR compression. Several presentations focused on this (linked items below are PDF documents):

• “Storing and Managing LiDAR Data” by Jon Skiffington (LizardTech)
• “Compressing LiDAR Waveform Data” by Dr. Charles Toth (Ohio State University)
• “laszip: Lossless Compression for LiDAR in LAS Format” (poster) by Dr Martin Isenburg (Lawrence Livermore National Laboratory)
• “LiDAR Compression with MrSID Generation 4” by Michael Rosen (LizardTech)

My sense from talking with the other presenters is that this is something for which the industry is hungry. Dr. Toth is researching ways to compress waveform data (as opposed to the spatial and other attributes of the point cloud itself). He presented two approaches:  one based on a wavelet transform and another based on compressive sensing.

Dr. Isenburg presented a poster session describing laszip, a compression mechanism for LAS files. He described it to me as being primarily focused on archiving LAS data and so rather orthogonal to our efforts, which emphasize accessibility and integration. Also, laszip is free.

Mr. Skiffington’s presentation focused on our LiDAR Compressor product while your author’s presentation gave an outline of the motivation for and an algorithmic explanation of the underlying MG4 technology.

I had several people asked for copies of my presentation so I’ve provided the link above.

Michael Rosen
LizardTech, Engineering Mgr
mrosen at lizardtech dot com

This morning, July 13, 2009, LizardTech unveiled LiDAR Compressor. This application allows consumers of LiDAR data the same benefits LT has provided raster consumers for over 10 years. The press release describes some of the product’s overall capabilities and benefits. I was privileged to be the project manager for this effort and in this post will describe the product from an engineering perspective.

At the lowest level, the application leverages our experience doing wavelet-compression. Our specific approach is the subject of a patent application and I won’t attempt to describe it in detail. However, a concise summary is that we do a 1D wavelet transform on each of the channels of interest (x, y, z, intensity, etc) and then use the same compression techniques we use in our MrSID and JP2 implementations (bitplaning, entropy encoding, appropriate quality-weighting of the bitplanes). 

At the SDK implementation level, we depend on several packages from the OSGeo stack. We use liblas to read data out of the LAS files. LT is proud to be the first (to my knowledge) commercial application of this industrial-strength library. (Regular readers of our blog may recall that last March we supported the OSGeo Code Sprint in Toronto; liblas was certainly part of our interest there!). Liblas, in turn, uses libgeotiff to decode the CRS information in the LAS files and the now venerable GDAL to translate that into a WKT that the rest of us can understand. The result is a C++ SDK available to our application … and soon to yours too (decode only).  We’ve got distributions for both Windows and Linux which will be available on our developer’s site any day now.
 

At the application level, the underlying technology has changed as well. Our flagship product, GeoExpress, is written in managed C++ using the Windows Forms library. Our most recent products (GeoViewer and Lidar Compressor) are built in C# using WPF. Our experience was that this change allowed much greater developer productivity and the creation of more flexible (not to mention prettier) applications. For example, I suspect we would not even have attempted a ListView with dropdown boxes in the header (all dynamically populated based on the selected input text file — see Figure 1 below) if we had to implement it without XAML support:
 

Figure 1: Importing a text file.

Additionally, the application makes extensive use of the WPF/.NET threading support. This allows the application to achieve two design goals: (a)it provides first class support for text-based LiDAR files and (b) it can run several jobs at once. 

Before we can do any kind of processing (e.g. compressing, viewing) we need to tell the file’s reader how many points there are and what the overall extent is. For LAS files (and MG4 files) this is no big deal because everything we need to know is stored explicitly in a compact header. For text files, it’s not stored at all and must be calculated which involves reading the entire input file. This “initialization” (see figures 2 and 3 below) can take a long time and needs to happen in the background.

Figure 2: Text file initializing (in the background, so the application is still responsive.)

A few minutes later …

Figure 3: We’ve read the 321 Mb text file and now know the extent and number of points.

Running several jobs concurrently in and of itself is not a hard problem. The difficulty comes when we consider how to provide feedback (progress bar, log messages), the ability to cancel and deal with failures. For historical reasons, GeoExpress runs its compression work in a separate process. This provides excellent isolation (uh,… that’s part of the ‘history’) but complicates tight integration with the parent application. This is part of the reason we could not include the ability to run concurrent jobs in GeoExpress 7. Lidar Compressor uses WPF’s built in BackgroundWorker class to manage the worker threads.  This vastly simplifies the integration tasks. The result is an application that leverages the multi-processor capabilities of modern hardware as well as a UI that is simple to understand and use (see Figure 4 below).
 

Figure 4: Five jobs running simultaneously.

Everything the user sees is pure WPF except for one thing:  the viewer. The viewer is a WPF wrapper around DirectX. There’s a lot of math that goes into manipulating 3D objects. DirectX provides an API to do this for us and, very significantly, runs it on the graphics card (i.e. no burden on the CPU). So, the good news is that rendering is blazing fast. The bad news is that DirectX 9 D3D (which is what we use) gets very finicky when it comes to older OSs (that is, XP), drivers and hardware. We were not able to get a really satisfying resolution to when DX would fail and why, but we were able to guard against it so that the application handles the failure gracefully.

Checkout the free trial and let us know what you think.

Glen Thompson and I were fortunate to attend the ESRI Developer’s Summit in Palm Springs last month. In addition to being a great excuse to get out of the rain here in Seattle, the Dev Summit provides a great opportunity to catch up with our friends from Redlands and see what’s new with ArcGIS.

Those who missed it can look over the presentations.

One way to think about an event like this is in terms of what’s cool and exciting and what’s getting the “business as usual” treatment. This year what’s cool and exciting is most definitely client-side web mapping. We’re talking Flex, Javascript and the newly released Silverlight API for ArcGIS Server. These were accompanied by cheering, contests, prizes and “Playful” (?) references to glitzy, “silverlighty” (get it?) applications.

There were no contests or prizes for the backend stuff on which Glen and I spent most of our time. Of particular note, GDAL is definitely on the rise here. GDAL is the open source project that (among other things) provides the ability to convert between raster formats. Principal maintainer Frank Warmerdam presented a technical session on “Custom Raster Format Support in ArcGIS through GDAL”. Guys, start thinking about migrating those Erdas Imagine / RDO extensions to GDAL. The writing is on the wall.

And, speaking of items that will eventually need migrating, I gave a short talk on “Implementing a Custom Image Server Raster Format.” It describes the technical architecture of the project and includes some server benchmarking work that we did in Seattle highlighting the impact of using Express Server with ArcGIS Image Server Extension.

ESRI is even hosting a video of the presentation.

Thanks, guys!

Lizard watchers will have noticed the recent LizardTech announcement regarding the Express Server sale to North Carolina. I spent an afternoon pointing Google Earth over there and reflected about the need for better public access for public GIS aerial imagery.

Like most people (certainly most people who read LizardTech’s blog) I’m really excited by the advent of feature-rich GIS clients like Google Earth and very extensive base map that they make available for free, non-commercial use. That said, there are lots of different kinds of imagery you might want to view and Google can’t provide them all. (For an interesting introduction to some of what’s behind the Google Earth base map imagery, see this Google Earth Blog posting.

Despite the availability of free or nearly free commercial data from Google, there are myriad public programs that acquire new imagery each year.

• USDA’s Aerial Photography Field Office runs several image collection programs the best known of which is probably the National Agricultural Image Progam (NAIP) which provides farm-related orthophotos
• USGS’s National Aerial Photography Program (NAPP) supports various federal agencies
• The National Geodetic Survey’s Aeronautical Survey Program supports aerial photography of Airports

That’s a by-no-means exhaustive list of federal collections programs. There are similar programs that work at the State and Local level. In general, there’s a very good reason for this: no single base map, even one as well-funded and maintained as Google’s, can hope to meet everyone’s needs.

Happily, most of that publicly-acquired data is publicly available for little or no cost to US taxpayers. The rub is the means by which it’s made accessible.

In many cases you can view maps via a web application like this NAIP 2006 Viewer and this MDOQ Viewer hosted by USDA’s Geospatial Data Warehouse. That’s great for basic access and a pan-and-zoom style of data discovery but limits the user to exactly what web application has built for you. For most non-trivial work, it’s not what you need (for example comparing / aggregating diverse datasets).

More frequently, you download the data or have it mailed to you on a DVD. Like this. That’s OK if you’re not in a rush and you have industrial strength GIS tools and the necessary technical sophistication to use them. Even then, you’re stuck with this same problem for every public dataset you need to access. Consider briefly that each of those datasets are frequently bigger than 1 gigabyte in size.

What we need are better tools and better integration. Increasingly, Google Earth is being picked as an easy, relatively inexpensive ($400/year for commercial use, free for personal use) tool and OGC standards like WMS are serving the integration need.

All of that brings me back to the the North Caroline site. Randolph County is a growing community. If you want to see the latest growth (here from 2007), you’ll need to ask the locals, it is not in the Google Earth base map, which shows this:

Here’s the same area, again inside of Google Earth, using the WMS feed from NC OneMap.

Try it yourself: Click this KML file to open it in Google Earth (Note: In Firefox, you’ll be asked first whether you want to open or save the file).

Directions magazine explored the question of the commercial data Google publishes as a public trust. My view is that, by making their real public data accessible via a common WMS interface, North Carolina has made it, well, really public.