Monday, February 27, 2012

Scala Source Code and Density

I love Scala. It enables me to write concise / dense and clear code at the same time.

Take for example the piece of code below.
Had I written it in Java, it would be much longer, while at the same time less dense, and with much less clarity of what it is doing.

val site = config.sites.find(_.name == siteName).getOrElse(
    throw new IllegalArgumentException("Cannot find site " + siteName))
val group = config.groups.find(_.name == groupName).getOrElse(
    throw new IllegalArgumentException("Cannot find group " + groupName))
group.repositories.foreach { repo =>
  val repoDir = new File(group.path, repo.name)
  println("Backing up to repository %s at %s starting..." format (repo.name, repoDir))
  if (repoDir.mkdir())
    println("Created directory %s" format repoDir)
  repo.facets.foreach { facet =>
    println("Exporting facet %s..." format facet.name)
    val source = site.databases.find(_.name == facet.source).getOrElse(
        throw new IllegalArgumentException(
          "Facet %s references non-existing database %s" format (facet.name, facet.source)))
    println("Source database is %s:%s at %s" format (source.kind, source.name, source.url))
    (source.kind, facet) match {
      case ("neo4j", Facet(name, sourceName, "node", typeName, "json")) =>
        val outFileName = new File(name + ".json")
        println("Exporting %s nodes to %s" format (typeName, outFileName))
        val dbDir = new File(new URI(source.url))
        println("Neo4j database directory: %s" format dbDir)
        val graphDb = new EmbeddedReadOnlyGraphDatabase(dbDir.getPath)
        val (rowMeta, rows) = try {
          val exporter = new Neo4jNodeExporter(graphDb, typeName)
          println("Fetching meta...")
          val rowMeta = exporter.fetchMeta()
          println("Columns are: " + rowMeta.columns.mkString(", "))
          val rows = exporter.export(rowMeta)
          (rowMeta, rows)
        } finally {
          graphDb.shutdown()
        }
        
        println("Reading rows...")
        val mapper = new ObjectMapper
        mapper.getSerializationConfig.set(Feature.INDENT_OUTPUT, true)
        val jsonRows = rows.map { row =>
          val obj = mapper.createObjectNode()
          for ((value, i) <- row.values.view.zipWithIndex) {
            if (value != null)
              obj.put(rowMeta.columns(i), value)
          }
          obj
        }
        val jsonArray = mapper.createArrayNode().addAll(jsonRows.toList)
        val jsonData = mapper.createObjectNode()
        jsonData.put("data", jsonArray)
        val outFile = new File(repoDir, outFileName.toString)
        
        print("Writing %s..." format outFile)
        mapper.writeValue(outFile, jsonData)
        println(" [OK]")
        
      case x: Any => println("Skipping unrecognized facet: " + x)
    }
  }
  println("Backing up to repository %s finished." format repo.name)
}

Scala is also fantastic at storing ad-hoc object graph / trees, check a look a this :

  sites = List(
    Site(name = "dev", databases = List(
      Database(name = "graph", kind = "neo4j", url = "file:///together/project/SatukanCinta/satukancinta-neo4j-db_dev_1.6/"))),
    Site(name = "test",
      databases = List(Database(name = "graph", kind = "neo4j", url = "file:///together/project/SatukanCinta/dumptest/graph")))),
  groups = List(Group(name = "main", path = "/together/project/SatukanCinta/dump_main",
    repositories = List(
      Repository(name = "like", kind = "*", facets = List(
        Facet(name = "user", source = "graph", primitive = "node", typeName = "com.satukancinta.domain.User", format = "json"),
        Facet(name = "topic", source = "graph", primitive = "node", typeName = "com.satukancinta.domain.Interest", format = "json"),
        Facet(name = "like", source = "graph", primitive = "relationship", typeName = "LIKE", format = "graphml"))))))) {

Beat that, Java!

Still don't believe me? How about processing a bunch of collections, and sprinkle built-in parallel capability :

    val indexHits = graphDb.getAllNodes.par.filter(_.getProperty("__type__") == typeName)
    log.info("Index for {} returned {} nodes", typeName, indexHits.size)
    val columnNames = indexHits.par.flatMap( node =>
      node.getPropertyKeys.filter( _ != "__type__" ) ).toSet
    val sortedColumns = columnNames.toList.sorted
    log.info("Columns for {}: {}", typeName, sortedColumns.mkString(", "))
    RowMeta(columns = sortedColumns)

I really can't imagine doing that (including the concurrency) in Java. Phew.

Verbose code == easy to read ? Not always. This one is much easier on the eyes.

Tip: To learn more about Scala programming, I recommend Programming in Scala: A Comprehensive Step-by-Step Guide, 2nd Edition.

Wednesday, February 15, 2012

UnproxyableResolutionException Workaround when using Scala Closures and javax.inject CDI Beans Together

Some powerful Scala programming language features like closures, pattern matching, and lazy vals don't work well with dependency injection frameworks like javax.inject aka CDI. This is due to stricter class structure requirements to enable proxying.

For example, this "innocent" code will not work:

  @Inject private var fbPhotoImporterFactory: Instance[FacebookPhotoImporter] = _
  @Produces @Named("facebookPhotoImporter") private var fbPhotoImporter: ActorRef = _

  @PostConstruct
  def init() {
    logger.debug("Starting FacebookPhotoImporter actor")
    fbPhotoImporter = Actor.actorOf(fbPhotoImporterFactory.get())
    fbPhotoImporter.start()
  }

It will throw:

Caused by: org.jboss.weld.exceptions.UnproxyableResolutionException: WELD-001437 Normal scoped bean class com.satukancinta.web.Persistence is not proxyable because the type is final or it contains a final method public final javax.enterprise.inject.Instance com.satukancinta.web.Persistence.com$satukancinta$web$Persistence$$fbPhotoImporterFactory() - Managed Bean [class com.satukancinta.web.Persistence] with qualifiers [@Any @Default].
    at org.jboss.weld.util.Proxies.getUnproxyableClassException(Proxies.java:225)
    at org.jboss.weld.util.Proxies.getUnproxyableTypeException(Proxies.java:178)
    at org.jboss.weld.util.Proxies.getUnproxyableTypesExceptionInt(Proxies.java:193)
    at org.jboss.weld.util.Proxies.getUnproxyableTypesException(Proxies.java:167)
    at org.jboss.weld.bootstrap.Validator.validateBean(Validator.java:110)
    at org.jboss.weld.bootstrap.Validator.validateRIBean(Validator.java:126)
    at org.jboss.weld.bootstrap.Validator.validateBeans(Validator.java:345)
    at org.jboss.weld.bootstrap.Validator.validateDeployment(Validator.java:330)
    at org.jboss.weld.bootstrap.WeldBootstrap.validateBeans(WeldBootstrap.java:366)
    at org.jboss.as.weld.WeldContainer.start(WeldContainer.java:82)
    at org.jboss.as.weld.services.WeldService.start(WeldService.java:89)
    ... 5 more

As you can see, my use code isn't exactly "edge cases".

It's actually a pretty common use case: create a Akka actor and pass a factory function to it, as a closure.
The code above doesn't look like it's using a closure, but it actually is when written like this: (same functionality, but still breaks CDI)

    fbPhotoImporter = Actor.actorOf { fbPhotoImporterFactory.get() }

I can see why CDI has a strict requirement, and I can also understand why Scala implements it the way it is (Scala developers definitely already has a lot of problems working around powerful Scala features into a very restrictive JVM bytecode requirements). This is the price we pay for having a somewhat inferior language (Java, please don't get offended) in the first place.

But I as an application developer want to have a quick fix for this issue. Re-coding the class in plain Java is one option, but it turns I don't need to. There is a workaround, by creating a helper method then using it:

  @Inject private var fbPhotoImporterFactory: Instance[FacebookPhotoImporter] = _
  @Produces @Named("facebookPhotoImporter") private var fbPhotoImporter: ActorRef = _
 
  def createFbPhotoImporter() = fbPhotoImporterFactory.get()
 
  @PostConstruct
  def init() {
    logger.debug("Starting FacebookPhotoImporter actor")
    fbPhotoImporter = Actor.actorOf(createFbPhotoImporter)
    fbPhotoImporter.start()
  }

Now Scala is happy and CDI is also happy. Yes it's a bit more verbose but not too bad. And I guess the code is now somewhat more understandable for Java guys. :)

Tip: To learn more about Scala programming, I recommend Programming in Scala: A Comprehensive Step-by-Step Guide, 2nd Edition.

Sunday, January 22, 2012

Scala "Bug" with CDI Dependency Injection

“Sometimes”, Scala creates public final methods, although the .scala source defines
no public final method at all.
There are two things that CDI doesn’t like (which Scala “sometimes” generates):
1. public final methods
2. public fields
To investigate and reproduce these problems I created a scala-cdi project at GitHub.

public final method: The Bug

Referencing a parent field from a closure / inner class triggers this behavior:
@RequestScoped @Named class IndexBean { private lazy val log = LoggerFactory.getLogger(classOf[IndexBean]) def testExecutor() = { val executor = Executors.newFixedThreadPool(4); executor.submit(new Runnable() { override def run(): Unit = log.debug("Executor is running") }) } }
Compiles to:
$ javap -p IndexBean Compiled from "IndexBean.scala" public class com.soluvas.scalacdi.IndexBean extends java.lang.Object implements scala.ScalaObject{ private org.slf4j.Logger com$soluvas$scalacdi$IndexBean$$log; ... public final org.slf4j.Logger com$soluvas$scalacdi$IndexBean$$log();
Deploying this app in Weld will throw:
org.jboss.weld.exceptions.UnproxyableResolutionException: WELD-001437 Normal scoped bean class com.soluvas.scalacdi.IndexBean is not proxyable because the type is final or it contains a final method public final org.slf4j.Logger com.soluvas.scalacdi.IndexBean.com$soluvas$scalacdi$IndexBean$$log() - Managed Bean [class com.soluvas.scalacdi.IndexBean] with qualifiers [@Any @Default @Named].         at org.jboss.weld.util.Proxies.getUnproxyableClassException(Proxies.java:225)         at org.jboss.weld.util.Proxies.getUnproxyableTypeException(Proxies.java:178)         at org.jboss.weld.util.Proxies.getUnproxyableTypesExceptionInt(Proxies.java:193)         at org.jboss.weld.util.Proxies.getUnproxyableTypesException(Proxies.java:167)         at org.jboss.weld.bootstrap.Validator.validateBean(Validator.java:110)

public final method: Workaround

Create a final local variable to hold the parent instance’s value:
def testExecutor() = { val executor = Executors.newFixedThreadPool(4); // this avoids 'log' becoming 'final' like: // private org.slf4j.Logger com$soluvas$scalacdi$IndexBean$$log; // public final org.slf4j.Logger com$soluvas$scalacdi$IndexBean$$log(); val log = this.log; executor.submit(new Runnable() { override def run(): Unit = log.debug("Executor is running") }) }
Which now compiles to:
$ javap -p IndexBean Compiled from "IndexBean.scala" public class com.soluvas.scalacdi.IndexBean extends java.lang.Object implements scala.ScalaObject{ private org.slf4j.Logger log; private org.slf4j.Logger log();

public field

I’m not able to reproduce this yet...
Tip: To learn more about Scala programming, I recommend Programming in Scala: A Comprehensive Step-by-Step Guide, 2nd Edition.

Sunday, January 1, 2012

Implementing RichFaces ExtendedDataModel for JSF Paging with Spring Data Neo4j and Scala

Satukancinta-datatable

JSF 2.1 and JBoss RichFaces 4.1.0 make it easy to do server-side paging and sorting, which improves performance of Java EE web applications significantly compared to returning list of all rows from the database and filtering it later in the application.

By implementing ExtendedDataModel, the data model can be used directly by rich:dataTable and rich:dataScroller JSF components.

ExtendedDataModel for Spring Data Neo4j Finder/Query Methods

To implement this on Neo4j graph database, by using Spring Data Neo4j finder methods we can implement ExtendedDataModel like below: (in Scala programming language)

package com.satukancinta.web

import collection.JavaConversions._
import org.ajax4jsf.model.ExtendedDataModel
import javax.faces.context.FacesContext
import org.springframework.data.domain.Page
import org.ajax4jsf.model.DataVisitor
import org.springframework.data.neo4j.repository.GraphRepository
import org.springframework.data.domain.PageRequest
import org.springframework.data.neo4j.aspects.core.NodeBacked
import org.ajax4jsf.model.Range
import org.ajax4jsf.model.SequenceRange
import org.slf4j.LoggerFactory
import org.springframework.data.domain.Sort
import org.springframework.data.domain.Sort.Direction
import org.springframework.data.domain.Pageable

abstract class FinderModel[E]() extends ExtendedDataModel[E] {
  private lazy val log = LoggerFactory.getLogger(classOf[FinderModel[E]])
 
  private lazy val rowCount: Int = {
    val result= getRowCountLazy
    log.debug("Total rows: {}", result)
    result
  }
 
  private var rowIndex: Int = _
  private var page: Page[E] = _
  private var pageData: List[E] = _
  private var lastRange: (Int, Int) = _

  log.trace("Created {}", this.getClass)
 
  def getRowCountLazy: Int
  def find(pageable: Pageable): Page[E]

  def setRowKey(key: Object): Unit = setRowIndex(key.asInstanceOf[Int])
  def getRowKey: Object = getRowIndex: java.lang.Integer
 
  private def loadData(range: Range): Unit = {
    val seqRange = range.asInstanceOf[SequenceRange]
    val curRange = (seqRange.getFirstRow, seqRange.getRows)
    if (lastRange == curRange) {
      log.debug("loadData returning cached")
      return
    }
    lastRange = curRange
   
    val pageNum = seqRange.getFirstRow / seqRange.getRows
    // ORDER BY name is painfully slow: https://groups.google.com/group/neo4j/t/f2219df41f5500a9
    val pageReq = new PageRequest(pageNum, seqRange.getRows/*, Direction.ASC, "y.name"*/)
    log.debug("loadData({}, {}) -> PageRequest({}, {})",
        Array[Object](seqRange.getFirstRow: java.lang.Long, seqRange.getRows: java.lang.Long,
            pageNum: java.lang.Long, seqRange.getRows: java.lang.Long))
    val startTime = System.currentTimeMillis
    page = find(pageReq)
    val findTime = System.currentTimeMillis - startTime
    log.debug("Page has {} rows of {} total in {} pages, took {}ms",
        Array[Object](page.getSize: java.lang.Long, page.getTotalElements: java.lang.Long,
            page.getTotalPages: java.lang.Long, findTime: java.lang.Long))
    pageData = page.toList
//    val pageIds = pageData.map( _.asInstanceOf[NodeBacked].getNodeId )
//    log.debug("Node IDs: {}", pageIds);
  }

  def walk(context: FacesContext, visitor: DataVisitor, range: Range, argument: Object): Unit = {
    loadData(range)
    for (val index <- 0 to pageData.size - 1) {
      visitor.process(context, index, argument)
    }
  }

  def isRowAvailable: Boolean = rowIndex < pageData.length

  def getRowCount: Int = rowCount

  def getRowData: E = {
    val result = pageData(rowIndex) // repository.findOne(rowKey.asInstanceOf[Long])
    val node = result.asInstanceOf[NodeBacked]
    log.trace("getRowData({}) = #{}: {}",
        Array[Object](rowIndex: java.lang.Long, node.getNodeId: java.lang.Long, node))
    result
  }

  def getRowIndex: Int = rowIndex
  def setRowIndex(index: Int): Unit = rowIndex = index

  def getWrappedData: Object = { null }
  def setWrappedData(wrappedData: Object): Unit = { /* dummy */ }

}

To use the FinderModel, it's much easier if we create a repository first and add some finder/query methods returning count and Page:

public interface InterestRepository extends GraphRepository<Interest> {

    @Query("START u=node({userId}) MATCH u-[:LIKE]->y RETURN COUNT(y)")
    public Long findUserLikeCount(@Param("userId") long userId);

    // ORDER BY name is still slow: https://groups.google.com/group/neo4j/t/f2219df41f5500a9
//    @Query("START u=node({userId}) MATCH u-[:LIKE]->y RETURN y ORDER BY y.name")
    @Query("START u=node({userId}) MATCH u-[:LIKE]->y RETURN y")
    public Page<Interest> findUserLikes(@Param("userId") long userId, Pageable pageable);

}

How to create a FinderModel instance from Java :

    @Inject InterestRepository interestRepo;
    private FinderModel<Interest> userLikesModel;

    @PostConstruct public void init() {
        userLikesModel = new FinderModel<Interest>() {

            @Override
            public int getRowCountLazy() {
                return interestRepo.findUserLikeCount(user.getNodeId()).intValue();
            }

            @Override
            public Page<Interest> find(Pageable pageable) {
                return interestRepo.findUserLikes(user.getNodeId(), pageable);
            }
        };
    }

And how to use this data model from a JSF Template .xhtml file:

<rich:dataTable id="interestTable" var="interest" value="#{userLikes.userLikesModel}" rows="20">
    <rich:column>
        <f:facet name="header">Name</f:facet>
        <h:link outcome="/interests/show?id=#{interest.nodeId}" value="#{interest.name}"/>
    </rich:column>
    <f:facet name="footer"><rich:dataScroller/></f:facet>
</rich:dataTable>

Quite practical, isn't it?


ExtendedDataModel for Spring Data Neo4j Repository

FinderModel can then be further subclassed to handle any Spring Data Neo4j repository:

class GraphRepositoryModel[E]() extends FinderModel[E] {
  private var repository: GraphRepository[E] = _
 
  def getRowCountLazy: Int = repository.count.toInt
  def find(pageable: Pageable): Page[E] = repository.findAll(pageable)

  override def getWrappedData: Object = repository
  override def setWrappedData(wrappedData: Object): Unit =
    repository = wrappedData.asInstanceOf[GraphRepository[E]]

}

And use it like this:

@Inject InterestRepository interestRepo;
private GraphRepositoryModel<Interest> interestModel;
   
@PostConstruct public void init() {
    interestModel = new GraphRepositoryModel<Interest>();
    interestModel.setWrappedData(interestRepo);
}

Hope this helps.

To learn more about Scala programming, I recommend Programming in Scala: A Comprehensive Step-by-Step Guide, 2nd Edition.