ArcGIS Online REST API

And there it is! the ArcGIS Portal REST API documentation was made public today. The API was there all the time of course, but using it needed some detective work using FireBug or your favorite HTTP sniffer.

You can of course continue on that path. You never know what you find along the way. But for those who want to get going quickly developing custom apps on the ArcGIS Online platform, the documentation should come in handy.

We've been using the API in some of our projects (like the gallery in geo.data.gov and the open government data label in geoplatform.gov), Kentucky's KyGovMaps site, and even the integration of Geoportal Server with ArcGIS Online I described earlier is possible thanks to this API. But it doesn't end there. Want to build your site in Drupal or Microsoft SharePoint? Now there's an API for that.

Or if you want to develop a custom application template like this one:

This template takes a webmap with (in this case) the location of dams. One of the attributes is the age of those dams. Using the ArcGIS Portal REST API, this (JavaScript) app, fetches the data included in the webmap (uploaded from a CSV). It then calculates the average age of dams visible in the current map extent and uses that to drive the gauge.

Whether you want to access a group to build your gallery from or do something interesting with the content of individual webmaps, the ArcGIS Portal REST API documentation helps you build your apps.

Perhaps this will now push me to finish up this Drupal module I had started a while ago. OR perhaps someone out there is adventurous enough to help me with that...

I Want More Like This!

Last night I was reading Peter Morville's book on Information Architecture and apparently my brain did some batch processing overnight. Getting up before the rest of the house has the benefit of doodling a bit with some ideas I got delivered from my brain this morning.

I never knew there was a term for it, but apparently I was thinking of growing pearls. Every website has it. Even our Geoportal Server has it, but it's a bit underused at the moment and it's such a nice feature.

The Geoportal Server allows for searching for content as well as for browsing the catalog. The browse tree configuration essentially comes down to defining a number of searches (call them facets) that are listed. Typical examples are showing content by type or theme. This works well for those facets that have a limited domain.

The mechanism of browsing the entire catalog is also available for individual items. Open the Relationship page for an item and you'll notice a similar tree structure:

We searched (on the Geoportal sandbox site) for Redlands and then went to the details page for the map service returned. from there we see in the Relationships page that this map service 'operates on' a dataset for the ESRI Redlands Campus.

This relation is determined from the content metadata (see the full metadata for the items) using the ISO construction for this type of relation.

However, the Geoportal Server can do more than navigate relations included within the metadata. It can infer relations based on other things, like where did the metadata come from, who published it, and even show content with similar tags/keywords.

To enable this, you'll need to edit a few configuration files. No coding required.

Once you've setup your editing environment of choice, open .../geoportal/WEB-INF/classes/gpt/search/browse/browse-resource.xml. This file controls the relationships that will appear for individual resources (note: you'll see the structure for the overall Browse navigation in this same location).

The structure of this file is simple and reflects a basic tree of <item> elements:

The secret sauce is in the <query> element in this structure. In the overall Browse tree you'll find elements that essentially are basic 'fixed' searches like so:

<query>dataCategory=farming</query>

In the structure for the individual items, the element defines a search based on one of the indexed fields of the item at hand, like so:

urn:esri:geoportal:browse:resourceRelationship:keywords-keywords

In this example, the search generated will take the indexed keywords for the current item and then formulate a query based on that. This will be specific to every item and thus results in finding other related items.

The above example includes a couple similarity relations that will find other items with similar keywords/tags, similar content type, were harvested from the same repository (think other web services harvested from the same ArcGIS Server for example), and were published by the same publisher (111 items from NOAA's Coastal Services Center in the below example). From there you can filter and repeat your way through the catalog.

With the capability of adding your own fields to the Geoportal Server index, this allows you to build a rich network of related items that will give your users a different approach to finding interesting content from a single starting point: pearl-growing.

Next time I may write about 'berry-picking' or 'lobster trapping'. You never know what you end up finding in the field of information retrieval.

Tags used should be relevant to the video

Metadata. People expected me to talk about that when I got on stage during the plenary of the Esri Federal GIS Conference this past week. It can bite you where you don't want to be bitten. You either hate it or love it...

In my technical workshop about Spatial Data Infrastructures and Geoportals I typically do talk about metadata and the spectrum of metadata using this diagram:

Two seemingly separated worlds of metadata of brief descriptions focused on the end user vs verbose XML structures describing the content in detail. But they're actually not that different. Whether it's to ensure that people understand the content before they make (for example) building construction decisions or whether you want your item to be at the top of a search result, good metadata is key.

In our recent work we've been looking at how we can bridge that gap in between these views of metadata. One step in that direction is to enable editing standards-based metadata with the ArcGIS Online system.

Using the soon-to-be-published API we have connected the Geoportal Server metadata editor to ArcGIS Online to provide full FGDC/ISO/INSPIRE/GEMINI/ANZLIC/... metadata editing for the description of items.

 With this integration enabled, owners of items will get a new tool when editing the item:

This will open the metadata editor for the specification of choice:

This integration is not limited to the ArcGIS Online technology. We're also working on enabling this for ArcGIS Server 10.1 services that will also support full XML-based metadata in addition to the descriptions you can already provide.

SO, it doesn't matter if you use ISO 1911x or tag-style metadata, just don't forget to follow Albert's advice:

 

Strange Connections

This past week was one of strange connections, when GI Science, friends and family, business and society all seemed intertwined...

It started off with AGILE, no, not this one, but the 14th edition of the get-together of the Association Geographic Information Laboratories Europe. This lead to some confusion, but perhaps we got some of the AGILE (the big one) to appreciate GIS a little bit...

We spoke about mixing and matching open source with closed source work with various samples shown coming from 52North, Research Studios Austria, and ourselves with respect to sensor observation, web processing, contributing to OpenStreetMap, and information discovery.

With the growing amount of information, the desire to filter out what's not relevant is growing as well. What I found one of the best paper presentations at the AGILE conference (the small one), was about Modeling Umwelten following the ideas of German scientist Jakob von Uexküll.

Jakob was no uilskuiken, no matter how similar the two terms may sound...

Jakob's theory states that an organism creates and reshapes its own Umwelt when it interacts with the world, creating a so-called 'functional circle'. This reminded by of Social Circles by @padday.

Filtering information becomes critical in disaster response situations. Some of the presentations at AGILE spoke of semantic filtering of information feeds using the context (dare I say Umwelt?) of the event and its participants. Surprisingly I found my self discussing emergency response using Eagle with my old colleagues at Esri NL.

During the week I crossed more of my Social Circles: meeting an old friend, spending time with Esri NL, and spending a day with my brother and our cousin-several-times-removed at DeLimes.

At DeLimes, we played urban golf with BuurtLab, studied how farms are managed differently, and what makes a local store survive, by actual visits and listening to the people themselves, following the 7S Framework.

When did you last ask a user how what they got actually was what they needed (will a new laptop really help you land a job?).

And so ended a week of interactions with different groups, discussions of software delivery models, management approaches, and catching up with family and friends. New links were established, old links where renewed and strengthened.

For those who haven't noticed yet: http://sourceforge.net/projects/geoportal/

Ah, thanks to Bill Thorp for reminding me to finish the trilogy...

Yes, Geoportal Server has been available on SourceForge since late October. Licensing, source code, proper credit to 3rd party components, etc are all in place. We migrated most of the Geoportal Extension documentation from the Esri Resource Center to the new Geoportal Server wiki.

Just before Xmas, we added the OpenLS connector for ArcGIS Server to the project! This is an update of an old connector that was available for ArcIMS several years ago. Configure it with ArcGIS Server services and provide an interface following the OGC specification of Open Location Service Implementation Standard. We're running a test instance that uses some ArcGIS Online services.

So now we're getting our plans together for 2011. Looking at:

• Complete wiki documentation
• GeoSPARQL interface
• Enhanced metadata editor
• Sitemap synchronizer
• WMTS client (JS API, Flex, Silverlight)
• 0-configuration database support
• Enhance OpenID/Oauth use
• ebRIM, Feature Data Dictionary, Feature Catalog support
• Upgrades to new underlying API (JS, Flex, Silverlight)

Some of these are just ongoing maintenance, ideas we've had for a while, or things already in the works (like the enhanced metadata editing). Others (like the GeoSPARQL interface) will take a while to develop and fully use. I'll put a brief description of these items on the feature list and would be curious to your responses or other features you would want to see in the Geoportal Server.

Are we there yet?

Is anybody home?
It's been three weeks since we announced that the Esri Geoportal extension would be released as open source. It may have seemed a bit quiet since, but rest assured, a lot has happened.

In my previous post I indicated we were looking at a Creative Commons-esk license model. Several people pointed out that this license is not recommended for source code. And yes we did see the FAQ on that topic.

And so started a quest for an appropriate license model that would give everyone: developers, implementers, and (sorry folks!) Esri what they need. If there only was a geek channel on TV. this would have made a great 'America's next top (license) model' show.

Along the way we revisited a great resource collected by the state of Massachusetts IT Division. It’s a bit dated, but the basics still apply.

While Esri has experience with open source from a licensing-in perspective, with the Geoportal extension going open source, we're flipping into a new role for the first time of licensing-out Esri software under an open source license.


From the 50+ (!) models discussed in that spreadsheet, it would have been great fun to use the Motosoto or Sleepycat license models, just because of their names. But no, we didn't select either.

Geoportal Extension will be released under the Apache 2.0 license!

Are we there yet?

Almost. We now have the task of moving source code, documentation, and such to a public source repository and find a proper way to integrate/link with the Esri websites (resource centers and such). But with this big step made, we're getting close.

Geoportal Extension to Become Open Source

Despite code documentation and samples as included on the Geoportal Resource Center, implementers of the Geoportal Extension have continued asking for source code access to support integration with content management systems, map viewers, desktop etc.

Esri listened to these requests and I am happy to be able to announce that:

the Geoportal Extension will enter a next phase in its evolution and become a Free and Open Source solution from Esri.

Seven years ago we created what was then called the GIS Portal Toolkit as a software and services solution, based on code from a number of earlier projects and prototypes for data discovery and map viewing. 

Since starting with GIS Portal Toolkit, those using it to create geoportals and clearinghouses have had access to its source code. Understandably, as people looked at creating websites that reflected the organization's identity.

At version 9.3 this resulted in Geoportal becoming a fully supported extension with a full maintenance program. We put a lot of effort in making Geoportal Extension configurable to a large extent with respect to authentication, metadata profile support, the index used for discovery, localization, skin development, and more.

The Geoportal Extension will be released under one of the variants of the Creative Commons open source license and will include elements like:

• Geoportal Web application
• OGC CSW 2.0.2 catalog service with OGCCORE and ISO Application Profile support
• INSPIRE compliant discovery service
• Extensible FGDC, ISO, DC metadata support
• Configurable search engine including spatial ranking algorithm
• Federated searches to standards-based (CS-W), Web 2.0 (OpenSearch), or other types of search providers (CMS, Document Management Systems, …)
• CSW clients (.NET + Java)
• Data Discovery Widget for Flex
• Data Discovery Widget for Silverlight
• Data Discovery Widget for HTML
• Ontology Service (java webapp)
• WMC clients (.NET)
• Publishing client (.NET)

We have some administration to do, but check back in later this month and next as we get the code out there. I'm looking forward to working with a larger community to further develop this product.