What is Pivot – from Microsoft Live Labs?

Pivot a research product currently available from GetPivot.com from Microsoft Live Labs. It uses technology from Seadragon – Deep Zoom technology also of MS Live Labs.

logo-pivot

A quote from the Pivot team:

“We tried to step back and design an interaction model that accommodates the complexity and scale of information rather than the traditional structure of the Web.”

Which translates into the ability to aggregate and interpolate large data sets.

If I’m losing you, this is a must see 5 minute TED 2010 video presented by Gary Flake.

Pivot User Interface

Pivot itself is an advanced type of web browser that facilitates this, but…

It is many things:

  • “The ability to slice and dice data.”
  • “Allows for the whole to be greater than the sum of the parts.”
  • “Lets you move from the tiniest detail, to the full scope of the entire set of the information.”
  • “Build a connection to the data, and be immersed.”
  • “Making large sets of data more approachable.”
  • “Empowering people to create new types of collections, and new types of experience from the interaction.”

That last point struck a chord with me, and I will attempt to create a simple collection. A collection in this case is a set of items with common attributes, ranging from hundreds to thousands to millions of items.

Collection Types

Pivot Collection Types

Pivot Collection Types

Steps to building a collection:

  1. Select your content; determine your linking level (diagram above).
  2. Expose your data in a way you can build the collection. OData service is an option.
  3. Build the cXML (Collection XML file).
  4. Navigate to the CXML file.
  5. Or navigate to the hosted version of the data set. (See Pivot API for .NET)

Navigate:
Pivot Navigation Bar

Resources and sources of quotes/images:

REMIX Melbourne 2010 Day 2

Kicking off Day 2. See the Day 1 post here.

Session 1
Web development with Visual Studio 2010 & ASP.NET 4
Alex Mackey

alex multi targetting

My day 2 began with Alex running through a cavalcade of improvements in Visual Studio 2010 and ASP.NET 4. These have been covered in great detail all over the place, but it is still nice to have them presented and demonstrated live allowing for feedback and questions.

Alex focussed on JavaScript and deployment general tips, along with touching on other areas.

I won’t cover off all the tips and tricks that Alex covered here, I’ll just link off to a good resource which is Scott Gu’s series on this.

Session 2
Riding the Geolocation Wave
Tatham Oddie

Tatham Geolocation

This was a great session, I hadn’t thought about the possibilities of geolocation to improve application experience. Tatham introduced the concepts and then demonstrated a simple pizza delivery application that tapped into the users current location (with their consent) to pre-seed a location aware list of options. The good news is that Windows 7 has geolocation support built in. We’ve already been using services on devices such as iPhone that use the assisted-gps (A-GPS) to tag things such as tweets with your location, drop a marker on Google maps with your current location and offer directions.

For resources and more information see Tatham’s blog post here.

Session 3
The future of exposing, visualising and interacting with data on the web.
Graham Elliot

Graham throwing punches

In this session OData was introduced in more detail to the REMIX audience. I’m covering off the basics of OData in a series myself so go check that out here.

The most audience pleasing concept demonstrated was the use of the awesome Microsoft Labs – Pivot available at GetPivot.com. It is a visual data interaction tool. To save me from failing to do it justice in a few lines of text, check out this 5 minute TED 2010 video presented by Gary Flake on Pivot.

Lunch & Live Frankly Speaking
Taping of Frankly Speaking
Michael Kordahi, Andrew Coates and guests.

Frankly Speaking

The REMIX audience during lunch got to be the live studio audience for a taping of Frankly Speaking. I’ll post a link to the episode here when it’s up.

Note the donkey is a reference to “taking the donkey work out of installation” – promoting the Web Platform Installer for Windows. It’s very handy check it out.

Labs
I didn’t end up going to any of the session in the slots 4 & 5. I joined a few colleagues in the labs.

remix labs

The first lab I attended was the XNA development introduction for Windows Phone 7. The lab was run by Luke Drumm (@lzcd) and Glenn Wilson (@myker). Glenn runs a blog focussed on this kind of development – virtualrealm.com.au

The second lab I attended was run by Steven Nagy, and was intended to get us configuring Azure AppFabric but some major hiccups like not having the lab PCs setup with internet access and some other mis-configured components prevented us from following along. None-the-less the lab ended up just being a discussion extension from Day 1s presentation on Azure. Several people had a lot of questions about actual deployment, from SLAs to locations to security.

Session 6
Pimp My App
Shane Morris

shanemo design tips

With the event coming to a close, it was nice to sit back and listen to the talented Shane Morris of Automatic Studio (former MS UX Guy) giving some basic design tips for developers to follow to ensure apps don’t suffer due to lack of professional designer input. There were 2 cool links to colour scheme assisting sites; kuler and Colour Lovers

There was a fair bit of discussion, analysis and reasoning provided by Shane, so I’ll just list out the conclusion slides.

Layout Steps:

  1. Map out the workflow.
  2. List your contents.
  3. Layout elements.
  4. Check grouping.

Presentation Steps:

  1. Remove unnecessary items.
  2. Minimise variation.
  3. Line stuff up.
  4. Space and size components evenly.
  5. Indicate grouping.
  6. Adjust visual weight.

Final Summary
All-in-all REMIX was a great 2 day event, and for the early bird price of < $200 was a bargain, when you factored in all you can drink coffee, buffet lunches and after party. Having already published my brief summary, all I can say here is that if you have the opportunity to attend REMIX, take it.

Working with the OData URI Conventions

Following on from my resource output format options post in my OData series. I thought I would briefly cover the topic of URI Conventions I had previously eluded to.

URI Conventions for OData allow for a simple standard way to control the resource that is returned. The documentation for these conventions can be found here on the OData.org site.

The first important choice being the result format. In my previous post I discussed the options of AtomPub and JSON, the way you go about making your format choice for the returned resources is via the $format option. If your choice is JSON then simply suffix query option $format=JSON, for example on the NetFlix OData service:

http://odata.netflix.com/Catalog/Titles?$format=JSON

As a quick interjection here for another convention; $callback is a handy feature to specify the name of a JavaScript function that will be called when the data returns, it’s even easy to use for an anonymous callback method by using “?” so $callback=?. I’ll go more into detail about this when covering my implementation in a future post.

In my experience with a few services the AtomPub choice is the default and the $format tag is not required, and in some cases (NetFlix in particular) $format=atom is not a valid input query. I’ll need to investigate this more, it seems like an oversight on at least the NetFlix service, I don’t see an issue with being more verbose even for a default value.

I also briefly mentioned deferred content and the use of $expand to force eager-loading of the result tree structure. This currently doesn’t seem to work on the NetFlix service, so this is yet another thing that will require some more investigation. It’s quite possible that the feature is disabled to prevent a type of attack (DoS) or just to prevent general abuse and waste of bandwidth/processing.

A simple convention is the $orderby option. It’s use is simple too and allows for chaining of various ascending and descending choices simply separated by commas.

http://odata.netflix.com/Catalog/Titles?$orderby=Runtime

and combined:

http://odata.netflix.com/Catalog/Titles?$orderby=ReleaseYear,Runtime desc,Name asc

I believe the browser will replace the spaces with the ‘%20‘ representation. If you make a mistake in the query (i.e. supplying an invalid order field) the error will look like:

No property ‘Title’ exists in type ‘System.Nullable`1

The last three I want to cover are; $filter, $top and $skip. They are related as they simply filter the returned resource. Skip and Top are quite simple and operate as you would expect (just like in LINQ).

Filter on the other hand is more advanced; it has all the logic operator options (==, !=, >=, and so on), all the arithmetic operators (add, sub, div, mult, mod) and precedence grouping (brackets). For the exact syntax and all the options see the specification.

I’ll just give a simple example that can be clicked through:

http://odata.netflix.com/Catalog/Titles?$filter=Runtime gt 3 and Runtime lt 90

I did say last, but as a bonus there’s a nice data reduction option to return only the selected called $select. Simply suffix an &$select=Name,Runtime set of params on any of the above queries and see the returned resource simplified to only the “data you want”.

For the lazy to modify here’s a click-able example:

http://odata.netflix.com/Catalog/Titles?$filter=Runtime gt 3 and Runtime lt 90&$select=Runtime,Name

NOTE: OData queries are case sensitive.

OData, AtomPub and JSON

Continuing my mini-series of looking into OData I thought I would cover off the basic structure of AtomPub and JSON. They are both formats that OData can deliver the requested resources (a collection of entities; e.g. products or customers).

For the most part there isn’t much difference in terms of data volume returned by AtomPub vs JSON, tho AtomPub being XML, is slightly more verbose (tags and closing tags) and referencing namespaces via xmlns. A plus for AtomPub for your OData service is ability to define the datatype as you’ll see below via m:type the example being an integer Edm.Int32. Whereas the lack of such features is a plus in a different way for JSON – it’s simpler, and a language such as JavaScript interprets the values of basic types (string, int, bool, array, etc).

I’m not attempting to promote one over the other, just saying that each can serve a purpose. If you’re after posts that discuss this is a more critical fashion, have a look at this post by Joe Gregorio.

What I do aim to show is that comparing the two side by side there’s only a slight difference, and based on what you’re intending to accomplish with processing said data the choice for format is up to you. If you’re just re-purposing some data on a web interface JSON would be a suitable choice. If you’re processing the data within another service first, making use of XDocument (C#.NET) would seem suitable.

There’s also a concept of ‘Deferred Content’ for both formats and it is achieved in a similar way through links. The objective being to conserve resources in processing and transmission by not transmitting the entire element tree on a request. In the comparisons below where there is a link to another URI that is content that has not been returned, the most obvious example is image data i.e. links to jpeg resrouces. OData has a URI command option called $expand that can force the inline return of the element data (this concept is called eager-loading). Have a look at my introductory post about the OData query options.

NOTE: In the examples that follow the returned result data is from the NetFlix OData service, I have stripped out some of the xmlns, and shortened/modified the urls in particular omitting http:// just so it fits better (less line wrapping).

So let us compare…

AtomPub
Yes that stuff that’s makes up web feeds.

Example from the NetFlix OData feed access via URL http://odata.netflix.com/Catalog/Titles

Atom Feed

<?xml version="1.0" encoding="iso-8859-1" standalone="yes"?>
<feed allThatOtherNameSpaceStuff="">
  <title type="text">Titles</title>
  <id>http://odata.netflix.com/Catalog/Titles/</id>
  <entry m:etag="abced">
    <id>http://odata.netflix.com/Catalog/Titles('movieName')</id>
    <title type="text">FullMovieTitle</title>
    <summary type="html">Your everyday regular movie</summary>
    <allTheOtherTags type="text">...</allTheOtherTags>
    <m:properties xmlns:m="severalNameSpaces">
      <d:Id>movieName</d:Id>
      <d:Synopsis>Your everyday regular movie</d:Synopsis>
      <d:Runtime m:type="Edm.Int32">3600</d:Runtime>
      <d:BoxArt m:type="NetflixModel.BoxArt">
          <d:SmallUrl>http://c.dn/boxshots/m1bx.jpg</d:SmallUrl>
      </d:BoxArt>
    </m:properties>
  </entry>
</feed>

JSON
Yes that simple text used in JavaScript.

Example from the NetFlix OData feed access via URL http://odata.netflix.com/Catalog/Titles?$format=JSON

Javascript Object Notation

{ 
  "d" : 
  { 
    "results": [ { 
      "__metadata": { 
        "uri": "o.ntf.lx/Ctlog/Titles('movieName')", 
        "etag": "abcdef", 
        "type": "NetflixModel.Title", 
        "edit_media": "o.ntf.lx/Ctlog/Titles('mvName')/$value", 
        "media_src": "c.dn/boxshots/large/mnbx.jpg", 
        "content_type": "image/jpeg", 
      },
      "Id": "movieName", 
      "Synopsis": "Your everyday regular movie"
      "Runtime": 3600
      "BoxArt": { 
        "__metadata": { 
            "type": "NetflixModel.BoxArt" }, 
            "SmallUrl": "http://c.dn/boxshots/m1bx.jpg"
        }
    } ]
  }
}

So What isĀ OData?

I started my blog with the intent of having in depth series on WCF, well my focus shifted as other interesting Microsoft technology came my way at work and out of work. It’s not feasible for me at the moment to be able to dedicate a substantial amount of time to a specific topic. So I won’t be committing too much on OData, but I will be presenting an introductory level session of OData at the beginning of May so would like to get my thoughts and notes out here on my blog.

So let’s get started with OData.

OData

Copy-pasting directly off the FAQ on the OData.org site is:

The Open Data Protocol (OData) is an open protocol for sharing data.

Hmm I’ve seen this kind, and this kind of idea before, but let’s keep reading…

It provides a way to break down data silos and increase the shared value of data by creating an ecosystem in which data consumers can interoperate with data producers in a way that is far more powerful than currently possible, enabling more applications to make sense of a broader set of data.

Ok, that sounds like a plan.

Every producer and consumer of data that participates in this ecosystem increases its overall value.

That last point has the marketing twist, which to me translates into; “unless there’s enough people using this protocol, it will only exist in obscurity.”

Only Joking!

Not to make this sound like an attack OData, it’s early days so let’s keep investigating.

One of the initial aspects of OData I find both interesting and commendable is the decision for the protocol to work through the standard HTTP request methods (verbs); GET, POST, PUT, DELETE. But my attention then quickly turned to access policies, security and authentication so I jumped ahead in my research and discovered the concept of Query Interceptors as part of implement an OData services that helps deal with this, I’ll go into detail on this in a future post.

Microsoft is putting some attention behind into OData, with it being featured in the day 2, MIX10 keynote presented by Doug Purdy. There’s also a mid May 2010 “roadshow” (awareness / day conference) happening across 8 locations (3 US, 2 EU, 2 Asia). I am looking forward to seeing the content that comes out of that.

Right now I’m in the process of dissecting the MIX10 Keynote OData segment, and the two other MIX10 sessions on OData. My objective is not only to learn a little bit about OData but also determine if it will assist in making some data as part of my day job more accessible.

open book

My attack plan for understanding the protocol concepts will be:

  • To first dig a little into the specification and the use of existing tech (ATOM and JSON), a summary post is here.
  • Create an OData service for part of complex system and large set of data.
  • Then hopefully being able to determine if it was worth it (so long as I get it working).

Along the way presenting findings here and at Melbourne user groups, so stay tuned.

Why WCF

UPDATE 9-July-2010:
This entry gets a lot of random hits, so I thought I would update the information as of July 2010, just to give links to newer areas relating to WCF.

WCF – Windows Communication Foundation, has become in fact more of a foundation for a larger set of abstractions on top it. Simply put WCF is a set of fundamental plumbing components with powerful extensibility features. But is not the easiest of things to use and configure.

The options now for more out of the box ready solutions that wrap and exist on top of WCF are:

These frameworks/abstractions/concepts are intended to simplify creating and deploying your web accessible service offerings.

Original post as it was 24-Feb-2009…

To quote the MSDN front page for WCF:

“It unifies the capabilities in ASMX, WSE, Remoting, COM+, and MSMQ; therefore developers need to learn only one programming model.”

Simply put “it’s where we are with Microsoft.NET framework”, if you’re going to build .NET applications it’s the right choice. Service Oriented Architecture is currently the most practical approach to build robust, distributed and maintainable applications. WCF allows you to simply and easily build such services according to an industry standard, with the intention to achieve a high level of interoperability. WCF takes care of: service interactions, marshalling, type conversions and other protocols allowing the developer to focus less on plumbing and more on delivering application value.

Key WCF notes:

  • Most messages are SOAP messages.
  • Not only for HTTP (can have: TCP, NamedPipes).
  • Can therefore communicate:
    • same machine
      • same app domain
      • cross process
    • cross machine
      • intranet
      • internet
  • Interoperable with other non-WCF services.

What I will attempt to demonstrate in subsequent posts is how simple it is to get the application plumbing working right and smoothly. Also down the track demonstrate some of the more complex scenarios you can setup with WCF especially the capabilities to further extend the framework.

Next up is “The WCF Run-Down“.