Data.gov Adds Geoviewer

Today, Data.gov added a new capability to its growing arsenal of tools that allow for using the data the website makes accessible. The so-called Data.gov GEO Viewer has some interesting capabilities:

• Data loaded into viewer in Real-time through web URLs – the viewer downloads data directly from the authoritative source. An ArcGIS Server Geoprocessing service uncompresses data if needed (.zip, .gz, .tar), transforms data to JSON, and streams this back to the flex viewer.
• The GEO Viewer loads data in Web Mercator (if data or service supports it).Otherwise the GEO Viewer changes its basemap projection to Geographic Coordinate System and loads the data.
• The viewer supports the following data types:
• Map Services: OGC WMS, ArcGIS
• Feeds: GeoRSS
• Files: KML/KMZ, Shapefile
• The GEO Viewer allows for mashing up multiple datasets, map services, and feeds in one view. It supports basic navigation using the keyboard (without the need to use the "shift+alt+F7+drag the mouse+release alt and mouse button at the same time"-like features...).
• Set a basic color for the added data layer, set transparency for the layers, and use a swipe/see-through feature.
• Basic identify operation on the added data.
• Switching the basemap.

There are some limitations with this viewer, most of which are due to the fact that it downloads data from the source every time someone wants to see it:

• File size limit of 10MB – Shapefiles and KML files can have large compression ratios. While the registered file in Data.gov may be an under 10MB KMZ file, this can easily expand into a 100MB KML that then is streamed as JSON features to the client. This simply takes time.
• The information about the files is not enough to make an upfront assessment of whether the file is viewable or not. Almost every file in Data.gov is a .zip file. The GEO Viewer has to determine if it's dealing with an Esri Shapefile, OGC KML, Arc/Info Export (e00, remember these?), Microsoft Excel, CSV, or whatever format(s) until after it downloads the file. The metadata in neither raw data catalog nor geodata catalog includes this information. A result is that sometimes users will only be notified that the file type is not supported until after the viewer is launched.
• Registration of content is not readily usable by an application (James Fee found one of these...). There are several registrations of content that link to web pages or web applications, rather than the actual data. In this case, the content is however also available as an ArcGIS Server Map Service (although that's not in the registration in Data.gov).

 Are we there yet?

No. This GEO Viewer is not the end point. It's another step towards allowing users to interact and understand the data discoverable through Data.gov. The viewer illustrates the need to include more map services in Data.gov. Even registering map services alone may not be enough. In this world of service architectures, platforms, and such, we become more and more dependent on each other. Offering a service means signing up to a responsibility to keep this service running and available for an extended period of time.


Oh! And the proof of the pudding is in the eating! Here some samples:

1. My most favorite dataset on Data.gov: the locations and characteristices of world copper smelters.
2. For the patient folks: Active Mines and Mineral Plants in the US
3. My next house location: Geophysical Surveys of Bear Lake, Utah-Idaho
4. The dataset James was looking for: USGS Oil and Gas Assessment Database

Give it a whirl and provide your feedback to Data.gov.

Announcing the ArcGIS Editor for OpenStreetMap

For a while, ArcGIS users have been able to use the OpenStreetMap (OSM) content as a basemap in ArcGIS Desktop or in web applications thanks to a republishing of this content through ArcGIS Online. After the earthquakes, we have received many requests from users of ArcGIS who want to contribute to OSM, but who prefer to use the editing capabilities of ArcGIS Desktop.

For users of ArcGIS 10 this is now possible using the new free add-on ArcGIS Editor for OpenStreetMap.

The ArcGIS Editor for OpenStreetMap is designed to help ArcGIS desktop users to become an active member in the growing community of users building an open and freely available database of geographic data.

The provided tools allow the user to download data from the OSM servers and store it locally in a geodatabase. The user can then use the advanced editing environment of ArcGIS Desktop 10 to create, to modify, and to delete data. Once the edits are complete, the edit changes can be posted back to the OSM server and become available to all OSM users.

The interaction with the OSM server is accomplished using as set of geoprocessing tools to download, to manage, and to upload data.

A total of six tools support the a disconnected editing like workflow: download data from OSM, edit locally, and upload the result back to OSM.

OSM has a very flexible data model, to support some consistency in created feature types. However for more focused data capture activities, such as those that occurred after the Haiti and Chile earthquakes, a more focused data model approach is suggested. To use the new ArcGIS 10 template feature, we have mapped the common tags used in OSM to attributes and feature types, created templates for these, and implemented suggested symbols.

Editing is straightforward. After downloading your work area from OSM, you use the normal ArcGIS Desktop editing features. There are some things to keep in mind in this first release:

• Only simple and single part geometries are supported.
• You cannot create features with more than 5000 nodes. The OpenStreetMap server has a limit of accepting geometries with up to 2000 nodes.
• Deleting a point, line, or polygon can have an effect of changing the relation in which the feature participates.
• Editing of OSM relations or super-relations directly is not supported in this first release.
• Polyons generated from data downloaded from OSM may be corrupt. To be safe: run the repair geometry tool before starting to edit.

As with any multi-user editing environment, you may run into a situation where multiple users edit the same area. This results in conflicts when trying to upload your edits to OSM. In order to mitigate the conflict the ArcGIS Editor for OpenStreetMap offers a simple Conflict Editor to help resolve the situation. Best practice is to edit a relatively small area and to save back to OSM frequently.

We are releasing this first version of the ArcGIS Editor for OpenStreetMap. and are looking for feedback. More details on the tool will become available over time, including access to the source code, and enhanced documentation.

Enjoy!