Saturday, July 28, 2012

With a little help from a friend




Staying in the Olympic weekend theme... with a little help from a friend you can get the Census data from their recently released beta data API in a format more easily used in Excel.

I found out about the Census Data API and couldn't resist. Is there a map? What do you get? Quickly got an API key and off I went.

Oh... do you know the ANSI code for your state? Mine is 06. That's Ca/CA/California for most.
Wanted to get population by state? that's parameter code P0010001 in the 2010 Census Summary File 1 (SF1) dataset.

Enhancement Request #1 - provide more easily accessible lookup for the various parameters.
Enhancement Request #2 - allow state abbreviations in addition to ANSI codes.

Alright, once you have this plus the key, off you go:
http://api.census.gov/data/2010/sf1?key=______&get=P0010001,NAME&for=state:*

response: a JSON array like this:
[["P0010001","NAME","state"],
["4779736","Alabama","01"],
["710231","Alaska","02"],
["6392017","Arizona","04"],
["2915918","Arkansas","05"],
["37253956","California","06"], 
... 

that's ok if you're a web developer, but I had thought to use this in Excel!
 

Enhancement Request #3 - support additional common output formats like CSV.
 

Based on 'stackoverflow convert-json-format-to-csv-format-for-ms-excel' I created a simple 
wrapper for the requests I was sending. The resulting CSV opens nicely in Excel like so:
 

 
AHA! From here we're good with the new Esri Maps for Office add-in:
 
 
And then shared to ArcGIS Online: 

View Larger Map



Sunday, May 20, 2012

Using Walking Papers in ArcGIS

NOTE: This post is a shameless translation of Gaëtan Lavenu's post Utiliser OSM Walking-Papers dans ArcGIS over on his arcOrama blog. Somehow 'télécharger' sounds a lot better than 'download'...

Walking-Papers is a service that allows you to contribute to OpenStreetMap using a paper map. In summary, the user prints a paper version of an extent and then annotates it based on observations made in the field. By scanning the annotated map, it can then be uploaded to the website for Walking Papers, then enter its updates with an OpenStreetMap editor like Potlatch or ArcGIS Editor for OpenStreetMap. 


The traditional method to add your scanned map in ArcGIS Walking-Papers would be to download a version of the georeferenced TIFF image from the website. The user then adds it as a standalone raster. 


To simplify this procedure, Esri provides a new tool Walking Papers Requester for adding a scan directly into ArcMap Walking-paper and then use the tools of the ArcGIS Editor for OpenStreetMap to enter the updates. The approach is as follows:
 
1. Download a PDF of the work area from the website Walking-papers. 


2. After your fieldwork, you scan your annotated paper map. 


3. You download the scanned map from the Walking-Papers website noting well-ID assigned to it. For example, in the URL of this scan: http://www.walking-papers.org/scan.php?id=3gkn7wqm ID is 3gkn7wqm ) 


4. Open the tool Walking Paper Scan Request.

5. Enter the ID and the local directory that will store the georeferenced image.

6. Once the tool has run, the image is automatically added to the current map. Using the ArcGIS Editor for OpenStreetMap you can download the vector data on the area and then update and synchronize the OpenStreetMap database. 



This geoprocessing tool written entirely in Python is available on the ArcGIS Editor for OpenStreetMap website.

Tuesday, April 3, 2012

Wordle Doodle

Love playing with visualizations of catalogs. Whether using pivot viewer from Microsoft or just some stats, additional visualizations give a different perspective on a catalog. Then of course it's interesting to look at multiple catalogs and their visuals to compare content.

Here's a wordle from our Geoportal Server sandbox. Wordles can be generated from an ATOM feed and Geoportal Server support that in its REST interface:

Wordle: Geoportal Server

And here is one from Geo.data.gov using the same query:

Wordle: Geo.data.gov World

Difference is interesting. Geo.data.gov just had reharvested several thousand USGS quadsheets. the variety in words in significantly less compared to the Geoportal sandbox one and the words all seem to occur about the same number of times. This is because the metadata for the quadsheets is generated.

Take a different perspective on something once in a while. You might learn something.

Wednesday, March 21, 2012

ArcGIS Online REST API

And there it is! the ArcGIS Portal REST API documentation was made public today. The API was there all the time of course, but using it needed some detective work using FireBug or your favorite HTTP sniffer.

You can of course continue on that path. You never know what you find along the way. But for those who want to get going quickly developing custom apps on the ArcGIS Online platform, the documentation should come in handy.

We've been using the API in some of our projects (like the gallery in geo.data.gov and the open government data label in geoplatform.gov), Kentucky's KyGovMaps site, and even the integration of Geoportal Server with ArcGIS Online I described earlier is possible thanks to this API. But it doesn't end there. Want to build your site in Drupal or Microsoft SharePoint? Now there's an API for that.

Or if you want to develop a custom application template like this one:


This template takes a webmap with (in this case) the location of dams. One of the attributes is the age of those dams. Using the ArcGIS Portal REST API, this (JavaScript) app, fetches the data included in the webmap (uploaded from a CSV). It then calculates the average age of dams visible in the current map extent and uses that to drive the gauge.

Whether you want to access a group to build your gallery from or do something interesting with the content of individual webmaps, the ArcGIS Portal REST API documentation helps you build your apps.

Perhaps this will now push me to finish up this Drupal module I had started a while ago. OR perhaps someone out there is adventurous enough to help me with that...

Saturday, March 10, 2012

I Want More Like This!

Last night I was reading Peter Morville's book on Information Architecture and apparently my brain did some batch processing overnight. Getting up before the rest of the house has the benefit of doodling a bit with some ideas I got delivered from my brain this morning.

I never knew there was a term for it, but apparently I was thinking of growing pearls. Every website has it. Even our Geoportal Server has it, but it's a bit underused at the moment and it's such a nice feature.

The Geoportal Server allows for searching for content as well as for browsing the catalog. The browse tree configuration essentially comes down to defining a number of searches (call them facets) that are listed. Typical examples are showing content by type or theme. This works well for those facets that have a limited domain.

The mechanism of browsing the entire catalog is also available for individual items. Open the Relationship page for an item and you'll notice a similar tree structure:


We searched (on the Geoportal sandbox site) for Redlands and then went to the details page for the map service returned. from there we see in the Relationships page that this map service 'operates on' a dataset for the ESRI Redlands Campus.

This relation is determined from the content metadata (see the full metadata for the items) using the ISO construction for this type of relation.

However, the Geoportal Server can do more than navigate relations included within the metadata. It can infer relations based on other things, like where did the metadata come from, who published it, and even show content with similar tags/keywords.

To enable this, you'll need to edit a few configuration files. No coding required.

Once you've setup your editing environment of choice, open .../geoportal/WEB-INF/classes/gpt/search/browse/browse-resource.xml. This file controls the relationships that will appear for individual resources (note: you'll see the structure for the overall Browse navigation in this same location).

The structure of this file is simple and reflects a basic tree of <item> elements:


The secret sauce is in the <query> element in this structure. In the overall Browse tree you'll find elements that essentially are basic 'fixed' searches like so:

<query>dataCategory=farming</query>

In the structure for the individual items, the element defines a search based on one of the indexed fields of the item at hand, like so:

urn:esri:geoportal:browse:resourceRelationship:keywords-keywords

In this example, the search generated will take the indexed keywords for the current item and then formulate a query based on that. This will be specific to every item and thus results in finding other related items.

The above example includes a couple similarity relations that will find other items with similar keywords/tags, similar content type, were harvested from the same repository (think other web services harvested from the same ArcGIS Server for example), and were published by the same publisher (111 items from NOAA's Coastal Services Center in the below example). From there you can filter and repeat your way through the catalog.



With the capability of adding your own fields to the Geoportal Server index, this allows you to build a rich network of related items that will give your users a different approach to finding interesting content from a single starting point: pearl-growing.

Next time I may write about 'berry-picking' or 'lobster trapping'. You never know what you end up finding in the field of information retrieval.

Sunday, February 26, 2012

Tags used should be relevant to the video

Metadata. People expected me to talk about that when I got on stage during the plenary of the Esri Federal GIS Conference this past week. It can bite you where you don't want to be bitten. You either hate it or love it...



In my technical workshop about Spatial Data Infrastructures and Geoportals I typically do talk about metadata and the spectrum of metadata using this diagram:


Two seemingly separated worlds of metadata of brief descriptions focused on the end user vs verbose XML structures describing the content in detail. But they're actually not that different. Whether it's to ensure that people understand the content before they make (for example) building construction decisions or whether you want your item to be at the top of a search result, good metadata is key.

In our recent work we've been looking at how we can bridge that gap in between these views of metadata. One step in that direction is to enable editing standards-based metadata with the ArcGIS Online system.

Using the soon-to-be-published API we have connected the Geoportal Server metadata editor to ArcGIS Online to provide full FGDC/ISO/INSPIRE/GEMINI/ANZLIC/... metadata editing for the description of items.

 With this integration enabled, owners of items will get a new tool when editing the item:


This will open the metadata editor for the specification of choice:



This integration is not limited to the ArcGIS Online technology. We're also working on enabling this for ArcGIS Server 10.1 services that will also support full XML-based metadata in addition to the descriptions you can already provide.

SO, it doesn't matter if you use ISO 1911x or tag-style metadata, just don't forget to follow Albert's advice:


Sunday, April 24, 2011

Strange Connections

This past week was one of strange connections, when GI Science, friends and family, business and society all seemed intertwined...

It started off with AGILE, no, not this one, but the 14th edition of the get-together of the Association Geographic Information Laboratories Europe. This lead to some confusion, but perhaps we got some of the AGILE (the big one) to appreciate GIS a little bit...

We spoke about mixing and matching open source with closed source work with various samples shown coming from 52North, Research Studios Austria, and ourselves with respect to sensor observation, web processing, contributing to OpenStreetMap, and information discovery.

With the growing amount of information, the desire to filter out what's not relevant is growing as well. What I found one of the best paper presentations at the AGILE conference (the small one), was about Modeling Umwelten following the ideas of German scientist Jakob von Uexküll.

Jakob was no uilskuiken, no matter how similar the two terms may sound...

Jakob's theory states that an organism creates and reshapes its own Umwelt when it interacts with the world, creating a so-called 'functional circle'. This reminded by of Social Circles by @padday.

Filtering information becomes critical in disaster response situations. Some of the presentations at AGILE spoke of semantic filtering of information feeds using the context (dare I say Umwelt?) of the event and its participants. Surprisingly I found my self discussing emergency response using Eagle with my old colleagues at Esri NL.

During the week I crossed more of my Social Circles: meeting an old friend, spending time with Esri NL, and spending a day with my brother and our cousin-several-times-removed at DeLimes.

At DeLimes, we played urban golf with BuurtLab, studied how farms are managed differently, and what makes a local store survive, by actual visits and listening to the people themselves, following the 7S Framework.

When did you last ask a user how what they got actually was what they needed (will a new laptop really help you land a job?).

And so ended a week of interactions with different groups, discussions of software delivery models, management approaches, and catching up with family and friends. New links were established, old links where renewed and strengthened.