Saturday, December 31, 2011

Scala closures vs Guava collection functions

Let me show you the same method using list transformations (aka "map" in functional programming speak), one in Java programming language:

    @GET @Path("user/{userId}/likes") @Produces(MediaType.APPLICATION_JSON)
    public Payload<Iterable<Map<String, Object>>> getUserLikes(@PathParam("userId") long userId) {
        User user = neo4j.findOne(userId, User.class);
        Iterable<Interest> likeInterests = user.getLikeInterests();
        Iterable<Map<String, Object>> likes = Iterables.transform(likeInterests, new Function<Interest, Map<String, Object>>() {
            @Override
            public Map<String, Object> apply(Interest interest) {
                HashMap<String, Object> row = new HashMap<String, Object>();
                row.put("id", interest.getNodeId());
                row.put("name", interest.getName());
                return  row;
            }
        });
        return new Payload<Iterable<Map<String, Object>>>(likes);
    }


And one in Scala programming language :

    @GET @Path("user/{userId}/likes") @Produces(Array(MediaType.APPLICATION_JSON))
    def getUserLikes(@PathParam("userId") userId: Long): Payload[Iterable[Map[String, Object]]] = {
        val user = neo4j.findOne(userId, classOf[User])
        val likeInterests = user.getLikeInterests
        val likes = likeInterests.map(interest =>
          Map("id"->interest.getNodeId, "name"->interest.getName) )
        new Payload(likes)
    }

Is Scala hard to read? I'll leave it to you to decide. :-)

To learn more about Scala programming, I recommend Programming in Scala: A Comprehensive Step-by-Step Guide, 2nd Edition.

Coding JSON REST JAX-RS Service Application to Access Neo4j Database in Scala

Spring Data Neo4j makes it easy to accessing graph data in Neo4j graph database.

After separating the AspectJ-enhanced Spring Data Neo4j node entities, relationship entities, and repositories, it's very fun to access the entities from a Scala web application. In this case, I'll show you a JAX-RS application written in Scala programming language, consuming and producing JSON in REST-style. The application exposes Spring Data Neo4j through HTTP, which can be accessed via curl or any other web client.

Java programming language version:

@Path("node") @Stateless
public class NodeResource {

    private transient Logger logger = LoggerFactory.getLogger(NodeResource.class);
    @Inject Neo4jTemplate neo4j;
    @Inject InterestRepository interestRepo;
   
    @GET @Path("interest") @Produces(MediaType.APPLICATION_JSON)
    public Iterable<Interest> interest() {
        ClosableIterable<Interest> records = neo4j.findAll(Interest.class);
        return records;
    }

    @GET @Path("user") @Produces(MediaType.APPLICATION_JSON)
    public Payload<User> getUser() {
        return new Payload<User>(neo4j.findOne(5L, User.class));
    }

    @POST @Path("interest")
    @Consumes(MediaType.APPLICATION_JSON)
    @Produces(MediaType.APPLICATION_JSON)
    public Response createInterest(Payload<Interest> payload) {
        logger.debug("createInterest {}", payload.data);
        Interest interest = payload.data;
        interest.persist();
        return Response.created(URI.create(String.format("interest/%d", interest.getNodeId())))
                .entity(new Payload<Interest>(interest)).build();
    }

    @DELETE @Path("interest/{id}")
    public Response deleteInterest(@PathParam("id") long id) {
        try {
            Interest node = neo4j.findOne(id,  Interest.class);
            node.remove();
            return Response.ok("deleted").build();
        } catch (DataRetrievalFailureException e) {
            return Response.status(Status.NOT_FOUND).entity(e.getMessage()).build();
        }
    }

    @GET @Path("interest/by/facebookId/{facebookId}")
    @Produces(MediaType.APPLICATION_JSON)
    public Payload<Interest> findInterestByFacebookId(@PathParam("facebookId") long facebookId) {
        logger.debug("findInterestByFacebookId {}", facebookId);
        Interest interest = interestRepo.findByFacebookId(facebookId);
        if (interest == null)
            throw new WebApplicationException(Response.status(Status.NOT_FOUND).entity("Interest with facebookId="+ facebookId +" not found").build());
        return new Payload<Interest>(interest);
    }

}


Scala programming language version:

@Path("node") @Stateless
class NodeResource {

    private lazy val logger = LoggerFactory.getLogger(classOf[NodeResource])
    @Inject var neo4j: Neo4jTemplate = _
    @Inject var interestRepo: InterestRepository = _
    
    @GET @Path("interest") @Produces(Array(MediaType.APPLICATION_JSON))
    def interest: Iterable[Interest] = neo4j.findAll(classOf[Interest])

    @GET @Path("user") @Produces(Array(MediaType.APPLICATION_JSON))
    def getUser: Payload[User] = new Payload[User](neo4j.findOne(5L, classOf[User]))

    @POST @Path("interest")
    @Consumes(Array(MediaType.APPLICATION_JSON))
    @Produces(Array(MediaType.APPLICATION_JSON))
    def createInterest(payload: Payload[Interest]): Response = {
        logger.debug("createInterest {}", payload.data)
        val interest = payload.data
        interest.persist
        Response.created(URI.create(String.format("interest/%d", interest.getNodeId)))
                .entity(new Payload[Interest](interest)).build
    }

    @DELETE @Path("interest/{id}")
    def deleteInterest(@PathParam("id") id: Long): Response = {
        try {
            val node = neo4j.findOne(id, classOf[Interest])
            node.remove
            Response.ok("deleted").build
        } catch {
          case e: DataRetrievalFailureException =>
            Response.status(Status.NOT_FOUND).entity(e.getMessage).build
        }
    }

    @GET @Path("interest/by/facebookId/{facebookId}")
    @Produces(Array(MediaType.APPLICATION_JSON))
    def findInterestByFacebookId(@PathParam("facebookId") facebookId: Long): Payload[Interest] = {
        logger.debug("findInterestByFacebookId {}", facebookId)
        val interest = interestRepo.findByFacebookId(facebookId)
        if (interest == null)
            throw new WebApplicationException(Response.status(Status.NOT_FOUND).entity("Interest with facebookId="+ facebookId +" not found").build
        new Payload[Interest](interest)
    }

}


Apart from less code and cruft, Scala code is not much different in structure.

If there are list processing functions or closures, then Scala code will read much easier, while the Java code will use Guava library and clunky syntax (at least until Java 8 arrives).

To learn more about Scala programming, I recommend Programming in Scala: A Comprehensive Step-by-Step Guide, 2nd Edition.

Graph Analysis with Scala and Spring Data Neo4j

Neo4j graph database is the an awesome technology for graph data persistence. With Spring Data Neo4j accessing graph data becomes even easier.

The easiest way to use Spring Data Neo4j is by enabling its AspectJ weaving. Because I wanted to use both Scala and Spring Data Neo4j in my web application, using both languages (AspectJ and Scala) in the same project isn't possible for now. A tip for you, separate the AspectJ-enhanced classes (@NodeEntity, @RelationshipEntity, and repositories) into a separate AspectJ project, then depend on the AspectJ from the Scala application.

The result is I'm able to fully utilize the versatile Scala and the excellent Spring Data libraries with Neo4j.

Now for mandatory code comparison between Java and Scala :-)

Java programming language version:

package com.satukancinta.web;

import java.util.List;
import java.util.Map;

import javax.enterprise.context.ApplicationScoped;
import javax.inject.Inject;
import javax.inject.Named;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.data.neo4j.conversion.Result;
import org.springframework.data.neo4j.support.Neo4jTemplate;

import com.google.common.base.Function;
import com.google.common.collect.Iterables;
import com.google.common.collect.Lists;
import com.satukancinta.domain.User;

/**
 * @author ceefour
 * Various analysis of the social graph.
 */
@Named @ApplicationScoped
public class GraphAnalysis {
   
    private transient Logger logger = LoggerFactory.getLogger(GraphAnalysis.class);
    @Inject Neo4jTemplate neo4j;
   
    public List<MutualLikeRow> getMutualLikesLookup(User user) {
        logger.debug("getMutualLikesLookup {}/{}", user.getNodeId(), user);
        Result<Map<String, Object>> rows = neo4j.query("START x=node("+ user.getNodeId() +") MATCH x-[:LIKE]->i<-[:LIKE]-y RETURN id(y) AS id, y.name AS name, COUNT(*) AS mutualLikeCount", null);
        Iterable<MutualLikeRow> result = Iterables.transform(rows, new Function<Map<String, Object>, MutualLikeRow>() {
            @Override
            public MutualLikeRow apply(Map<String, Object> arg) {
                return new MutualLikeRow((Long)arg.get("id"), (Integer)arg.get("mutualLikeCount"),
                        (String)arg.get("name"));
            }
        });
        return Lists.newArrayList(result);
    }

}

Scala programming language version:

package com.satukancinta.web
import collection.JavaConversions._
import org.slf4j._
import javax.inject.Inject
import org.springframework.data.neo4j.support.Neo4jTemplate
import com.satukancinta.domain.User
import javax.enterprise.context.ApplicationScoped
import javax.inject.Named

/**
 * @author ceefour
 * Analysis functions of friend network graph.
 */
@Named @ApplicationScoped
class GraphAnalysis {

  private lazy val logger = LoggerFactory.getLogger(classOf[GraphAnalysis])
  @Inject private var neo4j: Neo4jTemplate = _
 
  def getMutualLikesLookup(user: User): java.util.List[MutualLikeRow] = {
    logger.debug("getMutualLikesLookup {}/{}", user.getNodeId, user)
    val rows = neo4j.query(
        "START x=node("+ user.getNodeId() +") MATCH x-[:LIKE]->i<-[:LIKE]-y RETURN id(y) AS id, y.name AS name, COUNT(*) AS mutualLikeCount", null)
    val result = rows.map( r =>
          new MutualLikeRow(r("id").asInstanceOf[Long],
            r("mutualLikeCount").asInstanceOf[Integer].longValue,
            r("name").asInstanceOf[String]) )
        .toList
    result.sortBy(-_.mutualLikeCount)
  }
 
}


As you can see, the Scala version is not only much more concise, easier to understand, but actually has added functionality (sorted using .sortBy) with less code. Thanks to collection functions and closure support.

To learn more about Scala programming, I recommend Programming in Scala: A Comprehensive Step-by-Step Guide, 2nd Edition.

Wednesday, October 12, 2011

How to Exclude Scala/Java Source Files by Name in sbt

I thought this small tip would be useful to someone as I recently needed to do this and didn't know how to.

The project base directory is by default a source directory in addition to src/main/scala. You can exclude source files by name (butler.scala in the example below) like:

excludeFilter in unmanagedSources := "butler.scala"

Read more on How to exclude .scala source file in project folder - sbt Google Groups, also checkout Classpaths - sbt Wiki.

To learn more about Scala programming, I recommend Programming in Scala: A Comprehensive Step-by-Step Guide, 2nd Edition.

Tuesday, June 21, 2011

Using UnboundID LDAP SDK Directory API from Scala

I'm very new in the world of LDAP Directory Services. After installing ApacheDS Server, Apache Directory Studio LDAP GUI Client, and trying the plain old ldapsearch and ldapcompare command-line utilities, it was time for me to do some integration with Java applications (I'm actually trying to query LDAP directory from Ant build script task).

The most popular and programmer-friendly LDAP API at the moment is UnboundID LDAP SDK. To get a feel of how it works, I decided to use Scala REPL interpreter. I really love this tool (and the Scala programming language itself) more and more every time I use it!

The screen captures below doesn't do justice to the Scala interpreter. The tab-completion works really well!
Indeed the UnboundID LDAP SDK is really easy to use, and it's better with powerful Scala & its versatile interpreter for a quick ride. :-)

Learn Scala programming language quicker! Get Programming in Scala: A Comprehensive Step-by-Step Guide, 2nd Edition.

Connect & Bind using UnboundID LDAP SDK and Scala

ceefour@annafi:~/vendor/unboundid-ldapsdk-2.2.0-se$ scala -cp unboundid-ldapsdk-se.jar  Welcome to Scala version 2.9.0.1 (Java HotSpot(TM) 64-Bit Server VM, Java 1.6.0_25). Type in expressions to have them evaluated. Type :help for more information. scala> import com.unboundid.ldap.sdk._ import com.unboundid.ldap.sdk._ scala> val con = new LDAPConnection con: com.unboundid.ldap.sdk.LDAPConnection = LDAPConnection(not connected) scala> con.connect("localhost", 10389) scala> con.bind("uid=admin,ou=system","password") res5: com.unboundid.ldap.sdk.BindResult = LDAPResult(resultCode=0 (success), messageID=1)

Connect & Bind via SSL

scala> import com.unboundid.util.ssl._ import com.unboundid.util.ssl._ scala> var sslUtil = new SSLUtil( new TrustAllTrustManager() ) sslUtil: com.unboundid.util.ssl.SSLUtil = com.unboundid.util.ssl.SSLUtil@43763e0b scala> var socketFactory = sslUtil.createSSL createSSLContext createSSLServerSocketFactory createSSLSocketFactory scala> var socketFactory = sslUtil.createSSLSocketFactory socketFactory: javax.net.ssl.SSLSocketFactory = com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl@43d7a5a scala> val con = new LDAPConnection LDAPConnection LDAPConnectionInternals LDAPConnectionOptions LDAPConnectionPool LDAPConnectionPoolHealthCheck LDAPConnectionPoolHealthCheckThread LDAPConnectionPoolStatistics LDAPConnectionReader LDAPConnectionStatistics scala> val con = new LDAPConnection(socketFactory, "localhost", 10636) con: com.unboundid.ldap.sdk.LDAPConnection = LDAPConnection(connected to localhost:10636) scala> con.bind("uid=admin,ou=system","password") res18: com.unboundid.ldap.sdk.BindResult = LDAPResult(resultCode=0 (success), messageID=1)

Search

scala> val results=con.search("ou=system",SearchScope.SUB,"(uid=admin)") results: com.unboundid.ldap.sdk.SearchResult = SearchResult(resultCode=0 (success), messageID=3, entriesReturned=1, referencesReturned=0) scala> results.getSearchEntries res8: java.util.List[com.unboundid.ldap.sdk.SearchResultEntry] = [SearchResultEntry(dn='uid=admin,ou=system', messageID=3, attributes={Attribute(name=uid, values={'admin'}), Attribute(name=keyAlgorithm, values={'RSA'}), Attribute(name=sn, values={'administrator'}), Attribute(name=objectClass, values={'person', 'organizationalPerson', 'inetOrgPerson', 'tlsKeyInfo', 'top'}), Attribute(name=displayName, values={'Directory Superuser'}), Attribute(name=cn, values={'system administrator'}), Attribute(name=publicKey, base64Values={'MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJjX2EbUDyBZYFPfsSNMZRsW3mLc/KiKRO6pck3J3eFMKErXA7jxpiho/Eoc6RWgxut2K3aITnQZIl1hHF6Hv30CAwEAAQ=='}), Attribute(name=privateKeyFormat, values={'PKCS#8'}), Attribute(name=publicKeyFormat, values={'X.509'}), Attribute(name=privateKey, ba... scala> results.getSearchEntries.get(0) res11: com.unboundid.ldap.sdk.SearchResultEntry = SearchResultEntry(dn='uid=admin,ou=system', messageID=3, attributes={Attribute(name=uid, values={'admin'}), Attribute(name=keyAlgorithm, values={'RSA'}), Attribute(name=sn, values={'administrator'}), Attribute(name=objectClass, values={'person', 'organizationalPerson', 'inetOrgPerson', 'tlsKeyInfo', 'top'}), Attribute(name=displayName, values={'Directory Superuser'}), Attribute(name=cn, values={'system administrator'}), Attribute(name=publicKey, base64Values={'MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJjX2EbUDyBZYFPfsSNMZRsW3mLc/KiKRO6pck3J3eFMKErXA7jxpiho/Eoc6RWgxut2K3aITnQZIl1hHF6Hv30CAwEAAQ=='}), Attribute(name=privateKeyFormat, values={'PKCS#8'}), Attribute(name=publicKeyFormat, values={'X.509'}), Attribute(name=privateKey, base64Values={'MII...

Compare

scala> con.compare("uid=admin,ou=system", "userPassword", "password") res12: com.unboundid.ldap.sdk.CompareResult = LDAPResult(resultCode=5 (compare false), messageID=4, opType='compare', matchedDN='uid=admin,ou=system') scala> con.compare("uid=admin,ou=system", "uid", "admin") res13: com.unboundid.ldap.sdk.CompareResult = LDAPResult(resultCode=6 (compare true), messageID=5, opType='compare', matchedDN='uid=admin,ou=system')

Get Entry & Attribute Value

scala> val entry = con.getEntry("uid=admin,ou=system") entry: com.unboundid.ldap.sdk.SearchResultEntry = SearchResultEntry(dn='uid=admin,ou=system', messageID=7, attributes={Attribute(name=uid, values={'admin'}), Attribute(name=keyAlgorithm, values={'RSA'}), Attribute(name=sn, values={'administrator'}), Attribute(name=objectClass, values={'person', 'organizationalPerson', 'inetOrgPerson', 'tlsKeyInfo', 'top'}), Attribute(name=displayName, values={'Directory Superuser'}), Attribute(name=cn, values={'system administrator'}), Attribute(name=publicKey, base64Values={'MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAJjX2EbUDyBZYFPfsSNMZRsW3mLc/KiKRO6pck3J3eFMKErXA7jxpiho/Eoc6RWgxut2K3aITnQZIl1hHF6Hv30CAwEAAQ=='}), Attribute(name=privateKeyFormat, values={'PKCS#8'}), Attribute(name=publicKeyFormat, values={'X.509'}), Attribute(name=privateKey, base64Values={'MII... scala> entry.getAttribute getAttribute getAttributeValue getAttributeValueAsBoolean getAttributeValueAsDN getAttributeValueAsDate getAttributeValueAsInteger getAttributeValueAsLong getAttributeValueByteArrays getAttributeValueBytes getAttributeValues getAttributes getAttributesWithOptions scala> entry.getAttribute def getAttribute(String): Attribute def getAttribute(String, schema.Schema): Attribute scala> entry.getAttribute("userPassword") res16: com.unboundid.ldap.sdk.Attribute = Attribute(name=userPassword, values={'{SHA}W6ph5Mm5Pz8GgiULbPgzG37mj9g='}) scala> entry.getAttributeValue("userPassword") res17: java.lang.String = {SHA}W6ph5Mm5Pz8GgiULbPgzG37mj9g=

Scala 2.9.0.1 final version released: programming language with parallel features for Java VM

UPDATE: Scala 2.9.0.1 has replaced 2.9.0 as it hot-fixed several important bugs.

Scala 2.9.0 final:

We are happy to announce the release of the new stable release of the Scala distribution. The new Scala 2.9.0 final is available from our Download Page. The Scala 2.9.0 codebase includes several additions, notably the new Parallel Collections, but it also introduces improvements on many existing features, and contains many bug fixes.

Scala 2.9.0 binaries are available for the following libraries:

Of course.

Here are some recommended learning resources on Scala programming language:

Also, the Typesafe Stack is also out, which brings together Scala, Akka, and a few other things to get one up-and-running quickly. Much fun.

On the collection side of things, one of the first questions I saw was: do parallel collections share a common interface with standard collections. The answer is yes, they do, but not one that existed in 2.8.1.

You see, a trouble with parallel collections is that, now that they are available, people will probably be passing them around. If they could be passed to old code -- as it was briefly contemplated -- that old code could crash in mysterious ways. In fact, it happens with REPL itself.

For that reason, ALL of your code comes with a guarantee that it will only accept sequential collections. In other words, Iterable, Seq, Set, etc, they all now share a guarantee to be sequential, which means you cannot pass a parallel sequence to a method expecting Seq.

The parallel collections start with Par: ParIterable, ParSeq, ParSet and ParMap. No ParTraversable for now. These are guaranteed to be parallel. They can be found inside scala.collection.parallel, scala.collection.parallel.immutable, etc.

You can also get a parallel collection just by calling the ".par" method on it, and, similarly, the ".seq" method will return a sequential collection.

Now, if you want your code to not care whether it receives a parallel or sequential collection, you should prefix it with Gen: GenTraversable, GenIterable, GenSeq, etc. These can be either parallel or sequential.

And, now, something fun to try out:

def p[T](coll: collection.GenIterable[T]) = coll foreach println; p(1 to 20); p((1 to 20).par)

 

I highly recommend the following books for more information about Scala programming language:

 

 

Copied mostly verbatim from this announcement and that announcement.