Friday, November 27, 2015

accessing the new web api service document ala odatav4 and scala with akka-http

If you need to access the root service document at the MS CRM Dynamics website, you can access it using standad OData techniques.
I typically use akka-http. Here's how you can obatin the service document at the start of your program. I'm not going to show all the related code but just enough to make it easier for you to access your service document.
...
val http = Http()
...
 val rootFuture = async {
      val response = await(http.singleRequest(HttpRequest(GET, "http://yourorg.crm.dynamics.com" + "/api/data", odataHeaders)))
      if(logger.underlying.isDebugEnabled()) {
        val body = Await.result(response.entity.toStrict(1 seconds).map(_.data.utf8String), 5 seconds)
        logger.debug(s"Service root document: $body")
      }
      await(Unmarshal(response).to[ServiceDocument])
    }
    val root = Await.result(rootFuture, 5 seconds)
I use async/await because any program should not proceed without first getting the root document then deriving all the URLs from it. Hence the use of awaits.
Here's I have defined a spray-json unmarshaller by defining a few case classes and then defining my own spray-json "formatter." Once you do this you can use the Unmarshal function (which is akka-http):
case class EntitySet(val name: String, val uri: Uri, val title: Option[String] = None, val kind: Option[String] = Some("EntitySet"))
case class ServiceDocument(val serviceRoot: Uri, val metadata: Uri, val entitySets: Seq[EntitySet]) {
  /**
   * Return the URI for obtaining an EntitySet.
   */
  def /(entitySetName: String): Option[Uri] = entitySets
    .find(_.name == entitySetName)
    .map(_.uri.resolvedAgainst(serviceRoot))
}

object OdataJsonProtocol extends SprayJsonSupport with DefaultJsonProtocol {

  implicit object UriReader extends JsonReader[Uri] {
    def read(value: JsValue): Uri = value match {
      case JsString(uri) => Uri(uri)
      case x@_ => throw new DeserializationException(s"Uri must be deserialized from a string not a $x")
    }
  }

  implicit object EntitySetFormat extends JsonReader[EntitySet] {
    def read(value: JsValue): EntitySet = value.asJsObject.getFields("name", "kind", "url", "title") match {
      case Seq(JsString(name), kind, JsString(url)) => EntitySet(name, uri = url, kind = kind.convertTo[Option[String]])
      case Seq(JsString(name), kind, JsString(url), JsString(title)) => EntitySet(name, uri=url, kind = kind.convertTo[Option[String]], title = Some(title))
      case x@_ => throw new DeserializationException(s"Unable to parse $x into an EntitySet")
    }
  }

  implicit object ServiceDocumentFormat extends RootJsonReader[ServiceDocument] {
    def read(value: JsValue): ServiceDocument = value.asJsObject.getFields("@odata.context", "value") match {
      case Seq(JsString(context), JsArray(values)) =>
        val entitySets = values.map { _.convertTo[EntitySet] }
        ServiceDocument(context.substring(0, context.indexOf("/$metadata")), context, entitySets)
      case _ => throw new DeserializationException("Error obtaining root document")
    }
  }
}
Notice the use of SprayJsonSupport. That allows normal spray-json to be used in the Marshal framework which is akka-http.
You'll also need some headers:
object OData {
  val odataHeaders = List(
    Accept(MediaTypes.`application/json`),
    RawHeader("OData-MaxVersion", "4.0"),
    RawHeader("OData-Version", "4.0"))
}
My odataHeaders are slightly modified to add the Bearer token (see my other blogs):
  val odataHeaders = List(Authorization(headers.OAuth2BearerToken(yourBearerToken))) ++ OData.odataHeaders

Wednesday, November 25, 2015

odata and scala - using odata java libraries in scala

Someone asked me about using odata java libraries with scala and other communication infrastructure such as dispatch, spray, akka-http or finangle.

You can use the Olingo (formerly SAP odata's library) or the SDL odata library with scala and use your own communication layer. Both libraries provide a communication layer if you want to use it.

However, both libraries have interconnecting sets of components to build up odata request content and parse response information. The communication layers in these two libraries use standard javax.net and java.net type mechanisms for communications.

Both libraries provide client side services:
  • A way to create queries. Queries in odata use query parameters such as $top or $select. The query objects in the libraries translate API into these query parameters on an HTTP request object.
  • A way to map java beans into odata entities. Typically this is done via annotations. You can think of this layer in the libraries similar to the way that an ORM needs to map jvm objects into a specific RDBMS repository put/get format.
  • A way to translate responses into java objects. For example, you can iterate over an entity set returned from an odata entity set request. The libraries provide API for the iteration that fits, somewhat, into standard java semantics.
Both libraries also allow you to create odata data sources, also known as the "server" side. Both libraries allow you to receive odata requests, for example for metadata, and create a metadata object that is automatically serialized as a response.

The real issue with the communication layer in these libraries is that they force a style of programming that is imperative and blocking. That's not an issue for many programs that you might write if they are simple and small enough (small in the sense of execution performance).

To a large degree, the libraries represent what is called "late bound" programming. Late bound programming means that you use annotations or other types of reflection programming to handle the java POJO translation process. Scala programmers would prefer something more typesafe and not use annotations since annotations in these libraries are not checked at compile time. You could take the CSDL description of the entities and generate "early bound" objects, but since the library only uses annotations and reflection, this does not buy much other than saving you some typing.

The libraries also have immature support for OAuth2. SDL odata seems to have some plumbing created but it does not support refresh tokens and timeout scenarios, so you'll have to roll your own. 

SDL odata does provide some scala support, but its mostly a map into the java side of the library. SDL includes an akka based server. It's not clear what state it is in, but it looks promising. The scala support in SDL appears to be for the server versus a focus on the client side.

Olingo is very heavy with a standard "factory-based" java API. Everything is a factory. The late binding approach means that entity "properties" are coded fairly laboriously e.g. you specify that an entity has a property that has a type that has a...and so on. Specific requests are typed so that they have a matching response object type. If you use these, then you can access the "members" of the response and it performs the underlying parsing of the REST odatav4 response body for you. That's all very nice. But it is very painful java syntax.

Tuesday, November 24, 2015

Accessing MS CRM Dynamics preview web api (CRM 2015 Update 1)...scala, dispatch, adal4j, etc.

I previously wrote about authenticating a native client app written in java against a Azure AD in order to access MS CRM Dynamics.
Once you authenticate, you can pass around your access token (using the OAuth2 model) and issues REST requests. The API uses the ODATAv4 protocol. ODATA is essentially what ODBC was to desktop computers in the 90s, a way to access data that is expressed in an object graph. The v4 specification finally gets things right by allowing navigation with links directly embedded in the returned data enabling a more robust hypermedia navigation style.
The actual entire protocol is fairly well documented here on MSDN. MS seems serious about allowing the polyglut programming world to finally access the CRM data.
We want to test our authentication. To do that, we'll first ask WhoAmI using the approach that Jason laid out in his java version located here.
We won't use the two leading ODATA libraries quite yet. The ODATA libraries (Apache Olingo and SDL ODATA) both assume an execution model in addition to a content composition model. We'll just use the scala library dispatch for execution. It uses a well proven and robust async HTTP library underneath called async-http-client.
Picking up from the last post on AD authentication for MS CRM. You can continue to use amm for your scala repl:
load.ivy("com.ning" % "async-http-client" % "latest.release")
load.ivy("net.databinder.dispatch" %% "dispatch-core" % "latest.release")
import dispatch._, Defaults._
If you need to refresh your authentication token, rerun two lines:
val resultFuture = context.acquireToken(resource, clientid, "username@yourdomain.onmicrosoft.com", "password", null)
val t = resultFuture.get
t.getExpiresOnDate
Now we can issue a request using the dispatch library:
val headers = Map("OData-MaxVersion" -> "4.0",
  "OData-Version" -> "4.0",
  "Accept" -> "application/json",
  "Authorization" -> ("Bearer " + t.getAccessToken))

val whoAmIRequest = (host("yourdomain.crm.dynamics.com").secure / "api" / "data" / "WhoAmI") <:< headers
val response = Http(whoAmIRequest OK as.String)
You can chain a handler onto the future so that when it completes, you print out the returned value. Or you could just wait a second (wall clock time) and then run
response.print
The userid, as a GUID, is in the response body.
If you want to get rid of having to add the headers to every request, you can define a filter:
// Setup the headers so they are automatically added
import com.ning.http.client.filter._
case class MSCRMFilter(val token: String) extends RequestFilter {
  def filter[T](ctx: FilterContext[T]): FilterContext[T] = {
    val newRequest = new RequestBuilder(ctx.getRequest)
    Map("OData-MaxVersion" -> "4.0",
      "OData-Version" -> "4.0",
      "Accept" -> "application/json",
      "Authorization" -> ("Bearer " + token)).foreach{ case (k,v) => newRequest.addHeader(k,v)}
    new FilterContext.FilterContextBuilder(ctx).request(newRequest.build).build
  }
}
// Setup a new AD aware HTTP client
val adhttp = Http.configure(_.addRequestFilter(new MSCRMFilter(t.getAccessToken)))
While this does not automatically handle the token timeout, it does automatically add the OAuth2 and OData headers to every request that goes out. Now we can issue the request without adding the headers:
val whoAmIRequest2 = host("yourdomain.crm.dynamics.com").secure / "api" / "data" / "WhoAmI"
val response2 = adhttp(whoAmIRequest2 OK as.String)
// Be lazy and check response2.isCompleted to ensure it completes :-) then print the result
response2.print
You will get the same response as before.
To get a list of accounts, you can issue another OData like request:
val crmhost = host("yourdomain.crm.dynamics.com").secure // setup a reusable host value
val accountRequest = (crmhost / "api" / "data" / "accounts") <<? Map("$select" -> "name", "$top" -> "20")
val accounts = adhttp(accountRequest OK as.String)
accounts.print
Or obtain the contacts:
val contactsRequest = (crmhost / "api" / "data" / "contacts") <<? Map("$top" -> "20")
val contacts = adhttp(contactsRequest OK as.String)
accounts.print
But you'll notice that you will want to specify more query parameters in order to customize the ODATA response and not just obtain foreign keys as values for some af the properties, but the actual object content. You'll neeed to look at the ODATA documentation for details on how to pull more values from related entities in the same query.
In the above URLs we are violating good design. Because ODATAv4 follows a much more adherent design to REST, you can navigate from the initial metadata and entity list to any object without ever having to construct the actual URL path. You will need to add query parameters and more headers to cusomize the ODATA request, but you should never really have to spell out the paths like we did above.

Quick Take On MS CRM Dynamics, java, scala and AD Authentication

I needed to connect to MS CRM Dynamics using a simple java. The main instructions are on MSDN.
First, I had to register my application in the Azure AD using a "native" client app and ensure that it was given permission to access my MS CRM instance.
Then I followed the instructions. I found a few blogs including one that seems to hit on my wanting to test my connection. Jason Lattimer had already done the hard work. But I found that I just wanted to test this out quickly without creating a java program.
So I fired up scala's amm. amm is a new scala repl. It allows you to do things on the fly. I also looked at the example code from the java library that microsoft created.
load.ivy("com.microsoft.azure" % "adal4j" % "latest.release")
val authority = "https://login.windows.net/common"
val clientid = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
import java.util.concurrent._
val eservice = Executors.newFixedThreadPool(1)
val resource="https://yourdomain.crm.dynamics.com"
val resultFuture = context.acquireToken(resource, clientid, "username@yourdomain.onmicrosoft.com", "password", null)
resultFuture.get
which results in:
res15: AuthenticationResult = com.microsoft.aad.adal4j.AuthenticationResult@40d8ea60
Unfortunately, this is not very scala, and java futures are not great but this does illustrate that everything worked as planned.
amm has a robust tab completer so you can hit tab on the result:
val t = resultFuture.get
t.<hit tab>
!=                                  getAccessToken                      getExpiresOnDate                    hashCode                            |>
==                                  getAccessTokenType                  getIdToken                          isInstanceOf
asInstanceOf                        getClass                            getRefreshToken                     isMultipleResourceRefreshToken
equals                              getExpiresAfter                     getUserInfo                         toString
and navigate through the result.
Jason worked up a nice, more complete example of using the new web api here.
You will note that the adal4j library has many assumptions and dependencies that it pulls in and assumes an execution model underneath. Yo ucould of course program your own using something more pluggable, but unfortunately, that's alot of work if you just need to connect quickly. Underneath it uses:
  • nimbus oauth2
  • bouncycastle (java crypto api)
  • ...a few others...
Underneath, adal4j uses javax.net standard java classes for transport. The nimbus classes are used to construct an OAuth2 request and process the return headers, etc.
If you get a strange error, e.g. something about the request body needing some specific content, you may have created a web app OAuth2 application grant instead of a "client" application. See here for details.