You’ve heard of Open Street Map, right? You can think of it as Wikipedia for maps: people from all over the world map their own neighborhoods, and the results are made available for free, to all, with no copyright restrictions. For most of the US the road network has been pretty well mapped already, via free data available from the government, but it can get out of date surprisingly quickly and of course it doesn’t list the really important stuff like where to get the best ice cream.

These brave souls are about to redefine the leisurely walk.

On March 13th, nineteen people from CUGOS (including three Lizards) and one dog met up over on sunny Bainbridge Island for an Open Street Map mapping party. It was a diverse group — academics, kids, hackers, spouses. We met in the morning at the Bainbridge ferry dock, took a picture to commemorate the event, and divided up into four groups. Each group then spent the next few hours walking around a different part of Bainbridge’s downtown core, recording the coordinates of any “features of interest”:

• roads that had changed or been renamed
• “street furniture”: bus stops, park benches, fire hydrants
• points of interest: pharmacies, libraries, museums, pubs, ice cream parlors
• walking trails
• and much, MUCH more

Holly reviews her route while Dane gets “walking orders” from Michael.

Altogether we collected a few hundred different way points, using everything from handheld GPS units to iPhones to laser range finders. Over a leisurely late lunch by the waterfront, and in a subsequent CUGOS meeting, we worked to get all our GPS tracks and waypoints uploaded to the OSM databases.

Leaving no avenue uncharted, no point of interest unheralded, Michael, Walter and Roger investigate.

A walk through Bainbridge on a sunny day with friends — and GPS gear. What more could a GIS geek want?

Roger and Tifani display the elements of their GPS walking kit (bus stop sold separately).

Photos courtesy of CUGOS’ own Roger and Tifani.

This morning, July 13, 2009, LizardTech unveiled LiDAR Compressor. This application allows consumers of LiDAR data the same benefits LT has provided raster consumers for over 10 years. The press release describes some of the product’s overall capabilities and benefits. I was privileged to be the project manager for this effort and in this post will describe the product from an engineering perspective.

At the lowest level, the application leverages our experience doing wavelet-compression. Our specific approach is the subject of a patent application and I won’t attempt to describe it in detail. However, a concise summary is that we do a 1D wavelet transform on each of the channels of interest (x, y, z, intensity, etc) and then use the same compression techniques we use in our MrSID and JP2 implementations (bitplaning, entropy encoding, appropriate quality-weighting of the bitplanes). 

At the SDK implementation level, we depend on several packages from the OSGeo stack. We use liblas to read data out of the LAS files. LT is proud to be the first (to my knowledge) commercial application of this industrial-strength library. (Regular readers of our blog may recall that last March we supported the OSGeo Code Sprint in Toronto; liblas was certainly part of our interest there!). Liblas, in turn, uses libgeotiff to decode the CRS information in the LAS files and the now venerable GDAL to translate that into a WKT that the rest of us can understand. The result is a C++ SDK available to our application … and soon to yours too (decode only).  We’ve got distributions for both Windows and Linux which will be available on our developer’s site any day now.
 

At the application level, the underlying technology has changed as well. Our flagship product, GeoExpress, is written in managed C++ using the Windows Forms library. Our most recent products (GeoViewer and Lidar Compressor) are built in C# using WPF. Our experience was that this change allowed much greater developer productivity and the creation of more flexible (not to mention prettier) applications. For example, I suspect we would not even have attempted a ListView with dropdown boxes in the header (all dynamically populated based on the selected input text file — see Figure 1 below) if we had to implement it without XAML support:
 

Figure 1: Importing a text file.

Additionally, the application makes extensive use of the WPF/.NET threading support. This allows the application to achieve two design goals: (a)it provides first class support for text-based LiDAR files and (b) it can run several jobs at once. 

Before we can do any kind of processing (e.g. compressing, viewing) we need to tell the file’s reader how many points there are and what the overall extent is. For LAS files (and MG4 files) this is no big deal because everything we need to know is stored explicitly in a compact header. For text files, it’s not stored at all and must be calculated which involves reading the entire input file. This “initialization” (see figures 2 and 3 below) can take a long time and needs to happen in the background.

Figure 2: Text file initializing (in the background, so the application is still responsive.)

A few minutes later …

Figure 3: We’ve read the 321 Mb text file and now know the extent and number of points.

Running several jobs concurrently in and of itself is not a hard problem. The difficulty comes when we consider how to provide feedback (progress bar, log messages), the ability to cancel and deal with failures. For historical reasons, GeoExpress runs its compression work in a separate process. This provides excellent isolation (uh,… that’s part of the ‘history’) but complicates tight integration with the parent application. This is part of the reason we could not include the ability to run concurrent jobs in GeoExpress 7. Lidar Compressor uses WPF’s built in BackgroundWorker class to manage the worker threads.  This vastly simplifies the integration tasks. The result is an application that leverages the multi-processor capabilities of modern hardware as well as a UI that is simple to understand and use (see Figure 4 below).
 

Figure 4: Five jobs running simultaneously.

Everything the user sees is pure WPF except for one thing:  the viewer. The viewer is a WPF wrapper around DirectX. There’s a lot of math that goes into manipulating 3D objects. DirectX provides an API to do this for us and, very significantly, runs it on the graphics card (i.e. no burden on the CPU). So, the good news is that rendering is blazing fast. The bad news is that DirectX 9 D3D (which is what we use) gets very finicky when it comes to older OSs (that is, XP), drivers and hardware. We were not able to get a really satisfying resolution to when DX would fail and why, but we were able to guard against it so that the application handles the failure gracefully.

Checkout the free trial and let us know what you think.

Glen Thompson and I were fortunate to attend the ESRI Developer’s Summit in Palm Springs last month. In addition to being a great excuse to get out of the rain here in Seattle, the Dev Summit provides a great opportunity to catch up with our friends from Redlands and see what’s new with ArcGIS.

Those who missed it can look over the presentations.

One way to think about an event like this is in terms of what’s cool and exciting and what’s getting the “business as usual” treatment. This year what’s cool and exciting is most definitely client-side web mapping. We’re talking Flex, Javascript and the newly released Silverlight API for ArcGIS Server. These were accompanied by cheering, contests, prizes and “Playful” (?) references to glitzy, “silverlighty” (get it?) applications.

There were no contests or prizes for the backend stuff on which Glen and I spent most of our time. Of particular note, GDAL is definitely on the rise here. GDAL is the open source project that (among other things) provides the ability to convert between raster formats. Principal maintainer Frank Warmerdam presented a technical session on “Custom Raster Format Support in ArcGIS through GDAL”. Guys, start thinking about migrating those Erdas Imagine / RDO extensions to GDAL. The writing is on the wall.

And, speaking of items that will eventually need migrating, I gave a short talk on “Implementing a Custom Image Server Raster Format.” It describes the technical architecture of the project and includes some server benchmarking work that we did in Seattle highlighting the impact of using Express Server with ArcGIS Image Server Extension.

ESRI is even hosting a video of the presentation.

Thanks, guys!

LizardTech was one of the sponsors of an OSGeo code sprint in Toronto last week. Mike Rosen and I were both fortunate enough to attend and spend some quality time with twenty of the brightest minds of the open source geo world, folks working on everything from GDAL to MapServer to PostGIS to OpenLayers.

Perhaps the most visible result from the LizardTech side of things was some performance analysis that Frank Warmerdam, Chris Schmidt, and I did on GDAL’s use of MrSID – which should eventually help everyone downstream of GDAL, notably MapServer.

Paul Ramsey (mine host) and Chris both blogged daily about the event. Perry Nacionales and Paul have also posted some pix on flickr, including some of us all unwinding at Le Hockey (Hey, look! Real live Canadians, in their natural habitat!).

And, though we might have the grace to blush just a little, we’re not too modest to nonetheless link to Chris’ stirring post on using GDAL with our D-SDK.

Update: 13 July 2009. Read an article about the OSGeo code sprint in Toronto by Michael Gerlek here. – mdf

Image courtesy of Perry Nacionales

It is an oft-invoked stereotype that engineers prefer to work alone in dark caves, and there’s certainly some truth there. Here at LizardTech, for example, each development team member works in a well-ventillated but cozy and earthy burrow that our ops team constructed to individual specifications out of papier-mache. Some of these workspaces have convincing stalactites, or narrow entrances lined with lichens. A few are strewn with bones.*

But in mass emergences similar to those of the 13- and 17-year cicada, engineers periodically gather together in high-energy events called “code sprints”, which last several days and whose purpose is to resolve bugs, churn out new code, share information and ideas, and dispatch untold wedges of pizza. Ethnologists now suspect that a form of socialization is also carried on.

In March, LizardTech will be cosponsoring such an event in Toronto hosted by OSGeo. A couple of lizards will be attending and will work on GDAL/MrSID performance issues. Please consider joining us there!

*By contrast, our sales team is housed in a single, large spherical room partially filled with rubber balls and water toys.

Seriously, image of New Mexico’s Lechuguilla Cave courtesy of Wikipedia.