Strange Connections

This past week was one of strange connections, when GI Science, friends and family, business and society all seemed intertwined...

It started off with AGILE, no, not this one, but the 14th edition of the get-together of the Association Geographic Information Laboratories Europe. This lead to some confusion, but perhaps we got some of the AGILE (the big one) to appreciate GIS a little bit...

We spoke about mixing and matching open source with closed source work with various samples shown coming from 52North, Research Studios Austria, and ourselves with respect to sensor observation, web processing, contributing to OpenStreetMap, and information discovery.

With the growing amount of information, the desire to filter out what's not relevant is growing as well. What I found one of the best paper presentations at the AGILE conference (the small one), was about Modeling Umwelten following the ideas of German scientist Jakob von Uexküll.

Jakob was no uilskuiken, no matter how similar the two terms may sound...

Jakob's theory states that an organism creates and reshapes its own Umwelt when it interacts with the world, creating a so-called 'functional circle'. This reminded by of Social Circles by @padday.

Filtering information becomes critical in disaster response situations. Some of the presentations at AGILE spoke of semantic filtering of information feeds using the context (dare I say Umwelt?) of the event and its participants. Surprisingly I found my self discussing emergency response using Eagle with my old colleagues at Esri NL.

During the week I crossed more of my Social Circles: meeting an old friend, spending time with Esri NL, and spending a day with my brother and our cousin-several-times-removed at DeLimes.

At DeLimes, we played urban golf with BuurtLab, studied how farms are managed differently, and what makes a local store survive, by actual visits and listening to the people themselves, following the 7S Framework.

When did you last ask a user how what they got actually was what they needed (will a new laptop really help you land a job?).

And so ended a week of interactions with different groups, discussions of software delivery models, management approaches, and catching up with family and friends. New links were established, old links where renewed and strengthened.

For those who haven't noticed yet: http://sourceforge.net/projects/geoportal/

Ah, thanks to Bill Thorp for reminding me to finish the trilogy...

Yes, Geoportal Server has been available on SourceForge since late October. Licensing, source code, proper credit to 3rd party components, etc are all in place. We migrated most of the Geoportal Extension documentation from the Esri Resource Center to the new Geoportal Server wiki.

Just before Xmas, we added the OpenLS connector for ArcGIS Server to the project! This is an update of an old connector that was available for ArcIMS several years ago. Configure it with ArcGIS Server services and provide an interface following the OGC specification of Open Location Service Implementation Standard. We're running a test instance that uses some ArcGIS Online services.

So now we're getting our plans together for 2011. Looking at:

• Complete wiki documentation
• GeoSPARQL interface
• Enhanced metadata editor
• Sitemap synchronizer
• WMTS client (JS API, Flex, Silverlight)
• 0-configuration database support
• Enhance OpenID/Oauth use
• ebRIM, Feature Data Dictionary, Feature Catalog support
• Upgrades to new underlying API (JS, Flex, Silverlight)

Some of these are just ongoing maintenance, ideas we've had for a while, or things already in the works (like the enhanced metadata editing). Others (like the GeoSPARQL interface) will take a while to develop and fully use. I'll put a brief description of these items on the feature list and would be curious to your responses or other features you would want to see in the Geoportal Server.

Are we there yet?

Is anybody home?
It's been three weeks since we announced that the Esri Geoportal extension would be released as open source. It may have seemed a bit quiet since, but rest assured, a lot has happened.

In my previous post I indicated we were looking at a Creative Commons-esk license model. Several people pointed out that this license is not recommended for source code. And yes we did see the FAQ on that topic.

And so started a quest for an appropriate license model that would give everyone: developers, implementers, and (sorry folks!) Esri what they need. If there only was a geek channel on TV. this would have made a great 'America's next top (license) model' show.

Along the way we revisited a great resource collected by the state of Massachusetts IT Division. It’s a bit dated, but the basics still apply.

While Esri has experience with open source from a licensing-in perspective, with the Geoportal extension going open source, we're flipping into a new role for the first time of licensing-out Esri software under an open source license.


From the 50+ (!) models discussed in that spreadsheet, it would have been great fun to use the Motosoto or Sleepycat license models, just because of their names. But no, we didn't select either.

Geoportal Extension will be released under the Apache 2.0 license!

Are we there yet?

Almost. We now have the task of moving source code, documentation, and such to a public source repository and find a proper way to integrate/link with the Esri websites (resource centers and such). But with this big step made, we're getting close.

Geoportal Extension to Become Open Source

Despite code documentation and samples as included on the Geoportal Resource Center, implementers of the Geoportal Extension have continued asking for source code access to support integration with content management systems, map viewers, desktop etc.

Esri listened to these requests and I am happy to be able to announce that:

the Geoportal Extension will enter a next phase in its evolution and become a Free and Open Source solution from Esri.

Seven years ago we created what was then called the GIS Portal Toolkit as a software and services solution, based on code from a number of earlier projects and prototypes for data discovery and map viewing. 

Since starting with GIS Portal Toolkit, those using it to create geoportals and clearinghouses have had access to its source code. Understandably, as people looked at creating websites that reflected the organization's identity.

At version 9.3 this resulted in Geoportal becoming a fully supported extension with a full maintenance program. We put a lot of effort in making Geoportal Extension configurable to a large extent with respect to authentication, metadata profile support, the index used for discovery, localization, skin development, and more.

The Geoportal Extension will be released under one of the variants of the Creative Commons open source license and will include elements like:

• Geoportal Web application
• OGC CSW 2.0.2 catalog service with OGCCORE and ISO Application Profile support
• INSPIRE compliant discovery service
• Extensible FGDC, ISO, DC metadata support
• Configurable search engine including spatial ranking algorithm
• Federated searches to standards-based (CS-W), Web 2.0 (OpenSearch), or other types of search providers (CMS, Document Management Systems, …)
• CSW clients (.NET + Java)
• Data Discovery Widget for Flex
• Data Discovery Widget for Silverlight
• Data Discovery Widget for HTML
• Ontology Service (java webapp)
• WMC clients (.NET)
• Publishing client (.NET)

We have some administration to do, but check back in later this month and next as we get the code out there. I'm looking forward to working with a larger community to further develop this product.

Data.gov Adds Geoviewer

Today, Data.gov added a new capability to its growing arsenal of tools that allow for using the data the website makes accessible. The so-called Data.gov GEO Viewer has some interesting capabilities:

• Data loaded into viewer in Real-time through web URLs – the viewer downloads data directly from the authoritative source. An ArcGIS Server Geoprocessing service uncompresses data if needed (.zip, .gz, .tar), transforms data to JSON, and streams this back to the flex viewer.
• The GEO Viewer loads data in Web Mercator (if data or service supports it).Otherwise the GEO Viewer changes its basemap projection to Geographic Coordinate System and loads the data.
• The viewer supports the following data types:
• Map Services: OGC WMS, ArcGIS
• Feeds: GeoRSS
• Files: KML/KMZ, Shapefile
• The GEO Viewer allows for mashing up multiple datasets, map services, and feeds in one view. It supports basic navigation using the keyboard (without the need to use the "shift+alt+F7+drag the mouse+release alt and mouse button at the same time"-like features...).
• Set a basic color for the added data layer, set transparency for the layers, and use a swipe/see-through feature.
• Basic identify operation on the added data.
• Switching the basemap.

There are some limitations with this viewer, most of which are due to the fact that it downloads data from the source every time someone wants to see it:

• File size limit of 10MB – Shapefiles and KML files can have large compression ratios. While the registered file in Data.gov may be an under 10MB KMZ file, this can easily expand into a 100MB KML that then is streamed as JSON features to the client. This simply takes time.
• The information about the files is not enough to make an upfront assessment of whether the file is viewable or not. Almost every file in Data.gov is a .zip file. The GEO Viewer has to determine if it's dealing with an Esri Shapefile, OGC KML, Arc/Info Export (e00, remember these?), Microsoft Excel, CSV, or whatever format(s) until after it downloads the file. The metadata in neither raw data catalog nor geodata catalog includes this information. A result is that sometimes users will only be notified that the file type is not supported until after the viewer is launched.
• Registration of content is not readily usable by an application (James Fee found one of these...). There are several registrations of content that link to web pages or web applications, rather than the actual data. In this case, the content is however also available as an ArcGIS Server Map Service (although that's not in the registration in Data.gov).

 Are we there yet?

No. This GEO Viewer is not the end point. It's another step towards allowing users to interact and understand the data discoverable through Data.gov. The viewer illustrates the need to include more map services in Data.gov. Even registering map services alone may not be enough. In this world of service architectures, platforms, and such, we become more and more dependent on each other. Offering a service means signing up to a responsibility to keep this service running and available for an extended period of time.


Oh! And the proof of the pudding is in the eating! Here some samples:

1. My most favorite dataset on Data.gov: the locations and characteristices of world copper smelters.
2. For the patient folks: Active Mines and Mineral Plants in the US
3. My next house location: Geophysical Surveys of Bear Lake, Utah-Idaho
4. The dataset James was looking for: USGS Oil and Gas Assessment Database

Give it a whirl and provide your feedback to Data.gov.

Announcing the ArcGIS Editor for OpenStreetMap

For a while, ArcGIS users have been able to use the OpenStreetMap (OSM) content as a basemap in ArcGIS Desktop or in web applications thanks to a republishing of this content through ArcGIS Online. After the earthquakes, we have received many requests from users of ArcGIS who want to contribute to OSM, but who prefer to use the editing capabilities of ArcGIS Desktop.

For users of ArcGIS 10 this is now possible using the new free add-on ArcGIS Editor for OpenStreetMap.

The ArcGIS Editor for OpenStreetMap is designed to help ArcGIS desktop users to become an active member in the growing community of users building an open and freely available database of geographic data.

The provided tools allow the user to download data from the OSM servers and store it locally in a geodatabase. The user can then use the advanced editing environment of ArcGIS Desktop 10 to create, to modify, and to delete data. Once the edits are complete, the edit changes can be posted back to the OSM server and become available to all OSM users.

The interaction with the OSM server is accomplished using as set of geoprocessing tools to download, to manage, and to upload data.

A total of six tools support the a disconnected editing like workflow: download data from OSM, edit locally, and upload the result back to OSM.

OSM has a very flexible data model, to support some consistency in created feature types. However for more focused data capture activities, such as those that occurred after the Haiti and Chile earthquakes, a more focused data model approach is suggested. To use the new ArcGIS 10 template feature, we have mapped the common tags used in OSM to attributes and feature types, created templates for these, and implemented suggested symbols.

Editing is straightforward. After downloading your work area from OSM, you use the normal ArcGIS Desktop editing features. There are some things to keep in mind in this first release:

• Only simple and single part geometries are supported.
• You cannot create features with more than 5000 nodes. The OpenStreetMap server has a limit of accepting geometries with up to 2000 nodes.
• Deleting a point, line, or polygon can have an effect of changing the relation in which the feature participates.
• Editing of OSM relations or super-relations directly is not supported in this first release.
• Polyons generated from data downloaded from OSM may be corrupt. To be safe: run the repair geometry tool before starting to edit.

As with any multi-user editing environment, you may run into a situation where multiple users edit the same area. This results in conflicts when trying to upload your edits to OSM. In order to mitigate the conflict the ArcGIS Editor for OpenStreetMap offers a simple Conflict Editor to help resolve the situation. Best practice is to edit a relatively small area and to save back to OSM frequently.

We are releasing this first version of the ArcGIS Editor for OpenStreetMap. and are looking for feedback. More details on the tool will become available over time, including access to the source code, and enhanced documentation.

Enjoy!

Can I Have One NSDI with Some Confusion on the Side Please?

In this age of publish first, then filter, and instant gratification, it is easy to loose some of the real questions out of sight. The merging of Data.gov and Geodata.gov (yes, that is the plan) raises some questions that have gotten lost in the excitement from the last week.

Here are a couple observations on the subject that could be made by anyone who has been following the two sites over the past year(s):

1. Geodata.gov harvests most of its content from over 300 other catalogs (visit the Geodata.gov Statistics tab and view the information on Partner Collections). Data.gov does not have this capability. These catalogs represent federal, state, and local government, academia, NGO, and commercial providers of geospatial resources (visit the same tab on Geodata.gov and view the information on Publisher Affiliations). Data.gov on the other hand focuses on content from the Executive Branch of the Federal Government. Where would the remaining content of Geodata.gov go? http://www.otherdata.gov?
2. Geodata.gov focuses on FGDC+ISO metadata with the industry looking at migrating to the new North American Profile of ISO 191xx metadata. Data.gov has developed its own metadata specification and vocabulary that is quite different from this. Just look at a details page on Data.gov to confirm this. What is the position on this subject of FGDC and other federal agencies who have created standards-based metadata for many years?
3. Geodata.gov has focused on the GIS analysts and first responders (check the original Statement of Work, I'm sure it's online somewhere). Data.gov seems to focus on a different audience (although honestly it's not entirely clear to me if that audience consists of developers or the general public. It’s a bit of both).
4. Geodata.gov has supported a number of user communities in two ways:
• by allowing them to create community pages with resources beyond structured metadata that are of interest to those communities. The content in these pages is managed by the communities themselves. How should Data.gov support these communities of interest?
• by supporting community-oriented collections that group metadata from multiple source catalogs. Examples are RAMONA (the states’ GIS inventory), the Oceans and Coast Working Group (interested in all content in the US coastal zone), and Data.gov (actually, this is also configured as a collection in geodata.gov). These collections are exposed on the Geodata.gov Search tab and in the CS-W and REST interfaces to the catalog.Where would these collections end up after a merger of Geodata.gov and Data.gov?
• Geodata.gov has created a Marketplace where those who are looking for data and those who have plans to acquire data can discovery each other and collaborate. A dating service of a different kind. While not specifically targeted at the masses, isn't one of the key principles of NSDI to collaborate to reduce redundant investments?
5. Geodata.gov has created a search widget that has been implemented by several agencies such as the State of Delaware that enables searching geodata.gov directly from the website and thus getting access to state and other geospatial resources covering the area of the state. This widget can mean significant cost savings for agencies as they don't have to create their own clearinghouses. Will Data.gov provide such a role as well?
6. Through FGDC CAP grants several tools were built that work against the Geodata.gov REST or CSW interfaces. I mentioned some of these capabilities and the links to these tools in my recent blog post. Merging Geodata.gov and data.gov would ideally not break these investments.

It would be nice to see the passion that was expressed over the last week be repeated, but now discussing some of these and other questions that affect the geospatial community at large.