Simplicity, Speed and Instant gratification with Play/Slick

About 4 months back when I started development for my startup, I had 2 choices
1. Spring MVC / JPA/ Hibernate/ MySQL
2. Play / Slick/ Akka / MySQL

I was very comfortable with Spring and had of tons examples/templates. It would provide IOC, Aspects, Security etc.

It would have been a safe bet considering fact that we had to iterate quickly , at least for 3-4 months
I had worked on Scala prior but not with Play. Heard lot about it, especially the Productivity aspect

After contemplating/ considering options, I thought Simplicity and Productivity was key. Spring did provide lots of features, but it also came at increased layers/components/complexity. Also we didn’t need all those features
So I decided use  Play (2.2.3).

In hindsight it turned out be very good decision. I Was able to pick up/ develop/iterate quickly.
At this point we have 3 apps running Play Stack and working like charm
In this post I want to share important/necessary pieces needed to get your application up and running quickly using Play

1. Simple and Intuitive Configuration

  • application.conf – All application related configuration like database, email etc can be specified here
  • routes.conf – Mapping for urls can be specified here
  • build.sbt – Dependencies for your app can be specified here. Simple and smooth. No complex heirarchies, no urls etc……
  • play.plugins – External plugins go here

2. DB Integration with slick

I used slick for DB integration. When using Slick you need to provide mapping between Scala Classes and Database Tables.

Below is example code to generate mappings.

package com.abstractlayers
import scala.slick.driver.MySQLDriver.simple._
import scala.slick.model.codegen.SourceCodeGenerator
import scala.slick.driver.JdbcProfile
import scala.reflect.runtime.currentMirror

object sourceCodeGenerator extends App{
  
  val driver: JdbcProfile = currentMirror.reflectModule(
  currentMirror.staticModule("scala.slick.driver.MySQLDriver")
).instance.asInstanceOf[JdbcProfile]
  
 Database.forURL("jdbc:mysql://$host/$db?characterEncoding=UTF-8", 
                  "$username", "$password", null, "com.mysql.jdbc.Driver").
       withSession { implicit session =>
                         new SourceCodeGenerator(driver.createModel).writeToFile("scala.slick.driver.MySQLDriver", 
                                                    "/com/abstractlayers",  "com.abstractlayers")
                     }
  println("completed")
 
}

When starting off with Slick I used code generator,  However over period of time I did some improvisations

  • Instead of mapping entire table just mapped columns which are necessary , especially in case of retrievals.
  • When only few columns are mapped of a particular table , used plain sql for insertions.
  • Used plain sql for complex joins etc.
  • When working with DateTime used slick-joda-mapper

3. Dynamic Reloading

In case of Play,  application reloads dynamically.In combination with light weight editor like sublime text , you can save good amount of time during development.

4. JSON support.

Standard types like Int, String etc is supported by default.Below is example

      val someVal = ////////// get it from DB
      Ok( Json.toJson(someVal))
 

Below is example of Custom Types

      case class Item(name:String, category:String, price:Double) 
      case class ItemList ( items:List[Item] , size :Int)

        implicit val itemListWrites: Writes[ItemtList] = (
            (JsPath \ "items").write[List[Item]] and
           (JsPath \ "size").write[Int]
       )(unlift(ProductList.unapply))

    implicit val itemWrites: Writes[Item] = (
          (JsPath \ "name").write[String] and
        (JsPath \ "category").write[String] and
        (JsPath \ "price").write[Double] 
     )(unlift(Item.unapply))

    def getItemLists () =  =  DBAction { implicit dbSession =>
          val itemLists = ItemsDAO.getItems()
          val json = Json.toJson(itemLists)
    }
  

Notice size parameter in case class ItemList and also in itemListWrites. If you don’t include it , you run into compilation Issue. Not sure whats going on there. Seems like in case of custom types , you need to specify at least 2 parameters . Below is error

[error] ……………………. overloaded method value write with alternatives:
[error]   (t: List[models.Item])(implicit w: play.api.libs.json.Writes[List[models.Item]])play.api.libs.json.OWrites[play.api.libs.json.JsValue] 
[error]   (implicit w: play.api.libs.json.Writes[List[models.Item]])play.api.libs.json.OWrites[List[models.Item]]
[error]  cannot be applied to (models.ItemList => (List[models.Item], Int))
[error]         (JsPath \ "results").write[List[Item]] 
[error]                                   ^
[error] one error found
[error] (compile:compile) Compilation failed

Other than above issue i never had any major problem

5. Asynchronous and non blocking

Easy to implement asynchronous and non-blocking code.

 def getMonthlyCount() = Action.async { implicit request =>
    var monthlyCount = 0
     Future  {
    	        DB.withSession {   implicit s:Session =>   
                          monthlyCount = Monthly.getMonthlyCount()
    	     }
            Ok( Json.toJson(monthlyCount))
    }
  } 

7. Webservices

Integration with webservices is easy. Below is snippet of code which integrates with remote service asynchronously

       val holder : WSRequestHolder = WS.url("http:// someurl.com")
        val complexHolder : WSRequestHolder = holder.withQueryString("param1" ->param1)
                                                    .withQueryString("param2" -> param2)

        val futureResponse : Future[Response] = complexHolder.get()

        futureResponse.map ( response => {
             Logger.info("response:"+response.body.trim)
                   ///////// do something with response
            }
        )

8. Logging

Logging works with just simple import statement

     
         import play.api.Logger    

         Logger.info("Some Logs for application")
      

Gone are days where you need to configure log4j.properties, logback.xml etc. I still need to figure out how to separate error logs and rotation etc , but that can happen at later time

9. Security Filters are easy to Implement

Example below demonstrates how you can restrict access to certain actions based on business requirements.

      object Global extends WithFilters(AuthorizedFilter("secure_action1",
                                                         "secure_action2"
                                      )) with GlobalSettings{}                                      
                                                            
      object AuthorizedFilter {
           def apply(actionNames: String*) = new AuthorizedFilter(actionNames)
       }

      class AuthorizedFilter(actionNames: Seq[String]) extends Filter {
          override def apply(next: (RequestHeader) => Future[SimpleResult]) (request: RequestHeader): Future[SimpleResult] = {
  	  if(authorizationRequired(request)) { 
                         if (isValidUser(request)) next(request)
    		   else Future.successful(Unauthorized("Access Denied"))
                  }
                else next(request)
            }

          private def isValidUser(request:RequestHeader) = {
                                      /////// validate user per business requirements
          }

       }

Below are key-points

  • Create Global Object with trait AuthorizedFilter
  • Define actions to be secured in Authorized Filter
  • Override apply method in AuthorizedFilter

10. Mix and Match Scala and Java.

One of benefits of Play Framework is you can always mix-match Java and Scala based on comfort level

11. Vertical separation by having seperate containers

Often times you need to have multiple applications connecting to single DB. You can have customer facing website and admin portion of it. One way to design it is to have separate contexts in single tomcat instance. However this introduces dependencies as life cycles of 2 apps could be different. With Play you can run each in separate container. So need not worry about dependencies.

Common trend in all of above being, choice between Speed of Development/Simplicity Vs Perfection. Of course if Speed is of essence then Play helps you very very much. Not to say its not perfect. Its just that to attain perfection you need spend more time. But Speed is available out of box.
In case of start-ups time is of of essence as you need to validate concepts and iterate quickly. In such case Play very handy tool to have in your arsenal

Converting huge pdf to images using apache pdfbox

Recently I ran into issue where we wanted to convert pdf documents to series of images.

Ran into this stackoverflow article. This uses pdf-render to accomplish the task. However there were two issues with program

  • Code is little verbose
  • Code works well for small sized pdf. However once you have large pdfs ( more than 100 pages / 30-40 MB) you start running into Out of Memory error

So in search of better solution , I stumbled upon Apache PDFBox and it works beautifully addressing both the shortcomings listed above.

Below is snippet of code (with relevant imports)


import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import javax.imageio.ImageIO;
import org.apache.pdfbox.pdmodel.PDDocument;
import org.apache.pdfbox.pdmodel.PDPage;
import org.apache.pdfbox.pdmodel.PDPageNode;


public List<String> generateImages(String pdfFile) throws IOException  {
          
     String imagePath = "/Users/$user/pdfimages/";
     List <String> fileNames = new ArrayList<String>();
	  document = PDDocument.load(pdfFile);	//// load pdf	 
	  node = document.getDocumentCatalog().getPages(); ///// get pages
	  List<PDPage> kids = node.getKids();
	  int count=0;
	  for(PDPage page : kids) {   ///// iterate
	       BufferedImage img = page.convertToImage(BufferedImage.TYPE_INT_RGB,128);
	       File imageFile = new File(imagePath+ count++ + ".jpg");
               ImageIO.write(img, "jpg", imageFile);
               fileNames.add(imageFile.getName());      
           }
           return fileNames;	
    }

Fairly Simple, Straightforward and has low memory foot print

Covariance and Contravariance in scala

In this post I want to give an overview of Covariance and Contravariance in scala

Covariance

Consider diagram below which conveys the concept
Covariance

If you have 2 classes A,B such that A is subtype of B, and
if you have 2 parameterized types C[A],C[B] such that C[A] is subtype of C[B] then C is said to be covariant

Examples
Immutable List and Queue are examples of covariant Types in scala libraries.

 sealed abstract class List[+A] extends .........

 class Queue[+A] extends ........

Covariance is indicated by notation + on type parameter passed

Immutablity and Covariance

If you notice in api-docs covariance is generally associated with immutable types , eg scala.collection.immutable.Queue, scala.collection.immutable.List

There is a very good reason for this. From designers perspective, there were 2 reasons where covariance relation could fail

  • Re-Assignment of values. Classic example being java array reassignment problem, resulting in ArrayStore exception
  • Passing generic type as a parameter in method.

Thus to avoid above pitfalls , covariant types are immutable

If we try second scenario above ( passing type as parameter to method), we get compilation error.

scala> class MyGenericExample[+T] {
     |
     |  def doSomeThing(t:T) = {}
     |
     | }

:10: error: covariant type T occurs in contravariant position in type T of value t
        def doSomeThing(t:T) = {}

However there is a work around


scala> class MyGenericExample[+T] {
     |    def doSomeThing[U >:T] (t:U) = {}
     | }
defined class MyGenericExample

Notice how we are passing in [U >:T], where >: signifies lower bound. This declaration lets you pass anything that is supertype of T.

Below are some api’s from List collection where lower bounds is specified


   ////// cons method. expanded form
  def ::[B >: A](x: B): List[B]
  def reduce[A1 >: A](op: (A1, A1) ⇒ A1): A1

Also in case of Immutable collections,  the type is bumped up when you pass in a super type  as parameter to existing list . Consider below


scala> val list= List("Java","Scala","Ruby")
list: List[String] = List(Java, Scala, Ruby)

scala> 1::list
res1: List[Any] = List(1, Java, Scala, Ruby)

Things to note

  • Notice type of list in line 2 – List[String]
  • We appended 1 (type Int) to list in line 3
  • We got a new List of Type – List[Any] .
    • If you use mutable List ( which is invariant) , this wont work.
    • return type is bumped up to List[Any] which is common super type of Int and String

Contravariance

Consider diagram below which conveys the concept
Contravariance

If you have 2 classes A, B such that A is subtype of B, and
if you have 2 parametrized Types C[A], C[B] such that C[B] is subtype of C[A] then C is said to be contravariant

Examples
OuputChannel trait from scala api is example of contravariant type

 trait OutputChannel[-Msg] extends AnyRef

Contravariance is indicated by notation – on type parameter passed. 

Example below illustrates contravariance and covariance
Variance-Example
Key points to note

  • We have simple class hierarchy represented by Data, Xml Json, Html
  • Request and Response are higher types which accept other types as parameter. ( In this example Data or one of its subtype)
  • Request is Contravariant . Any method that accepts Request[Xml] can accept Request[Data] as well. ( left hand side in picture)
  • Response is Covariant . Any method that returns Response[Data] can return any sub-types of Response[Data] as well
  • Contravariant types are used as arguments and covariant types are used as return types

Real- world example of Contravariance is scala’s Function api.

   trait Function1[-T1, +R] extends AnyRef

where

  • T1 – arguments to function. (contravariant)
  • R – return type from function. (covariant)

That completes quick overview of covariance and contravariance. Great reference for above topic is Programming In Scala, chapter 19.

scala concepts through api

In this post I would like discuss some api’s from scala docs and highlight important concepts associated with them.

Below are apis (from List Collection ) I would like to discuss.

  • map
  • collect
  • count
  • collectFirst

Below are the concepts which will be illustrated

  • Higher Order Functions , Anonymous Functions
  • Currying
  • Implicit Parameters
  • BuilderFactory ( CanBuildFrom)
  • Partial Functions, Predicate Functions
  • Option Type

map

Map is a transformation function. When applied on a collection transforms elements of collection.
Below is the signature of method

     def map[B](f: (A) ⇒ B): List [B]    //////// Default

Understanding Parameters

  • [B] : Generic type parameters – Int, String etc…./li>
  • f: (A) ⇒ B : function to be applied. Input to function is A and output is B
  • List [B] : return type

Below is an example


   scala> val myList = List(1,2,3,4)
   myList: List[Int] = List(1, 2, 3, 4)

   scala> def f(x:Int):String = x.toString
   f: (x: Int)String

   scala> myList.map(f(_))
   res0: List[String] = List(1, 2, 3, 4)

Mapping parameters in example to api-docs

  • [B] : [String] Inferred by compiler, line 8
  • f: (A) ⇒ B : f , function converting Int to String, line 4
  • List [B] : List[String] , line 8

Below is expanded form of api

def map[B, That](f: (A) ⇒ B)(implicit bf: CanBuildFrom [List[A], B, That]): That   ////// Expanded form

Understanding parameters

  • [B, That] : That represents type collection returned (Set,List… etc) , B represents Type of elements in collection ( Int, String….)
  • (f: (A) ⇒ B) : Transformation function
  • implicit bf: CanBuildFrom [List[A], B, That] : Implicit BuilderFactory. This facilitates creation of Collection That, with elements B, when input is of Type List[A]
  • That: Type of collection returned

Lets look at example


scala> import scala.collection.mutable.{Builder, ListBuffer}
import scala.collection.mutable.{Builder, ListBuffer}

scala> import scala.collection.generic.{GenericSetTemplate, GenericCompanion, CanBuildFrom}
import scala.collection.generic.{GenericSetTemplate, GenericCompanion, CanBuildFrom}

scala> def newListBuilder[A]: Builder[A, List[A]] = new ListBuffer[A]
newListBuilder: [A]=> scala.collection.mutable.Builder[A,List[A]]

scala> def canBuildFrom = new CanBuildFrom[List[Int] , String , List[String]] {
     | def apply (from:List[Int]) = newListBuilder
     | def apply() = newListBuilder
     | }
canBuildFrom: scala.collection.generic.CanBuildFrom[List[Int],String,List[String]]

scala> myList.map[String,List[String]](_.toString)(canBuildFrom)
res1: List[String] = List(1, 2, 3, 4)

Mapping parameters in example to api-docs

  • [B, That] : [String,List[String]] , line 16
  • (f: (A) ⇒ B) : _toString , line 16
  • implicit bf : CanBuildFrom [List[A], B, That] – canBuildFrom[List[Int],String,List[String]], line 14
    • Our custom Builderfactory – To create List[String] from List[Int]
  • That : List[String] , line 17

Comparing 2 apis, whats the difference

  • In case of Example 1, we don’t specify BuilderFactory. Since parameter is marked Implicit, compiler will select most appropriate one based on scope. In this case it will look for same in companion object associated with List Type.
  • In case of Example 2, we pass all the parameters explicitly.( Builder Factory and return Types).
  • For most cases default (Example 1) will serve the purpose. You don’t need to pass Builderfactory or specify Types.

Scala Concepts from above api

  • Higher Order Functions – These functions can be passed around as parameters. In this example our function is f:Int=>String . You can read about them here
  • Anonymous Functions – In Example 2 we passed .toString function to map function directly without declaring it. This is example of anonymous function
  • Currying – Concept in functional programming where you can pass multiple parameters list to a function. In Example 2 , parameters f and bf are curried. Look at example below
    • Regular function call – f(a,b)
    • Curried Function – f(a)(b)

    You can read more about them here

  • Implicit Parameters and Builder Factory- Builder Factory is Generic factory class which facilitates creation of one type of collections from another. In scala this represented by trait CanBuildFrom. You can read more about Implicit Parameters here and BuilderFactory here.( Look at section Factoring out common operations)

collect

When applied on Collection, this api returns a new collection by applying partial function passed

Below is the signature from docs ( for brevity sake I shall consider default signature)

   def collect[B](pf: PartialFunction[A, B]): List[B]

Understanding parameters

  • [B] – Type of elements in List to be returned
  • pf: PartialFunction[A, B] – PartialFunction to be applied. This function takes A and returns B
  • List[B] : return type

Lets look at example.
In this example we have Customer with some attributes (totalPurchases,avgPurchaseCost,etc…) and we want to able to collect all customers based below criteria

  • all customers who have made atleast 10 purchases or
  • if their average purchasePrice is greater than 100

scala> case class Customer(id:Int, totalPurchases:Int, avgPurchasePrice:Double)
defined class Customer

scala> val customerList = List(Customer(1,20,1000.0), Customer(2,10,100.0), Customer(3,0,0.0), Customer(4,1,20.0), Customer(5,4,50.0))
customerList: List[Customer] = List(Customer(1,20,1000.0), Customer(2,10,100.0), Customer(3,0,0.0), Customer(4,1,20.0), Customer(5,4,50.0))

scala> def valuedCustomers:PartialFunction[Customer,Int] =  {
     | case Customer(id,_,avgPurchasePrice) if avgPurchasePrice >=100.0 => id
     | case Customer(id,totalPurchases,_) if totalPurchases >=10 => id
     | }
valuedCustomers: PartialFunction[Customer,Int]

scala> customerList.collect(valuedCustomers)
res12: List[Int] = List(1, 2)

Mapping parameters in example to api-docs

  • pf: PartialFunction[A, B] – valuedCustomers: PartialFunction[Customer,Int] -line 11
  • List[B] – List[Int] – line 14

Scala Concepts from above api

  • Partial Functions – These functions are used along with case classes to facilitate pattern matching. You can read about case classes here and partial functions here

count

This function returns count based on Predicate function passed.

Below is signature

  def count(p: (A) ⇒ Boolean): Int

Understanding parameters

  • p: (A) ⇒ Boolean – Predicate function with A as input and Boolean return type
  • Int : Return type , Int

Lets look at example.

In this case we are filtering customers who have made at-least 1 purchase. We are passing predicate function to filter customers


scala> def purchasePredicate(customer:Customer):Boolean = if (customer.totalPurchases > 0) true else false
purchasePredicate: (customer: Customer)Boolean

scala> customerList.count(purchasePredicate)
res16: Int = 4

Mapping parameters in example to api-docs

  • p: (A) ⇒ Boolean – purchasePredicate: (customer: Customer)Boolean -line 2
  • Int : Return type , Int

Scala Concepts from above api

  • Predicate Functions – Functions which return true or false

collectFirst

This api can be used to collect first value which matches the Partial Function that is passed

Below is signature

    def collectFirst[B](pf: PartialFunction[A, B]): Option[B]

Understanding parameters

  • (pf: PartialFunction[A, B]) : PartialFunction to be applied. This function takes A and returns B
  • Option[B] : Return type – Option[B] , Option Container of type B

Lets look at example.
In this scenario we want to able to get first customer which matches the partial function we are passing.There could be scenario where there is no match. Hence in this case we are using Option to store values


scala> customerList.collectFirst(valuedCustomers)
res18: Option[Int] = Some(1)

Mapping parameters in example to api-docs

  • pf: PartialFunction[A, B] : valuedCustomers: PartialFunction[Customer,Int]
  • Option[B] : Return Type : Some[1] , Some Container with value of ID 1 , first match to our Partial Function

Scala Concepts from above api

  • Option Type – Container to handle values that could be potentially null. It uses Some ( if value is present ) or None is no value is present. You can read about them here

Summary

We looked at some api’s and underlying concepts associated with them. In next post I shall cover some more apis,
Idea being to understand api’s better which in turn will facilitate writing better scala code.

Actor Based system with Akka

This post gives an overview on how to build simple Actor Based system using Akka.

To demonstrate use case , I develop a application which facilitates streaming of Logs

I had developed similar prototype using Java some-time back. You look it up here. However since am learning Scala, decided to port the code to scala

In this POC I use Akka . Akka provides high level abstractions to build asynchoronous/reactive/multi threaded systems without getting to low-level semantics like threads, locks etc. Conceptually it consists of Actors and these Actors communicate by passing Messages.

If you are not familiar with Akka take a look at this example.That should get you started.

Overview of Application

Concept can be depicted in pictorically below

Scala Actors

Below are key components

  • StreamingLogs – Scala Object to Bootstrap application
  • Logwatcher – Actor which watches logs and sends updates to Coordinator
  • TCPServer – Actor which accepts http connections and streams updates to all its clients
  • Coordinator- Actor (parent) which co-ordinates messaging with 2 actors – LogWatcher and TCPServer
    • Starts individual actors
    • Receives updates from LogWatcher when logs are updated
    • Sends mesage to TCPServer regards updates to Logs
  • LogWriter  – Actor which simulates real-world logging

Lets look at important aspects of each component

StreamingLogs

This class facilitates boot-strap of application.


  val coordinator = system.actorOf(Props(new Coordinator(logFileName)), name = "co-ordinator")
  val logWriter = system.actorOf(Props(new LogWriter(logFileName)), name = "log-writer")

  logWriter ! StartWriting()
  coordinator ! StartCoordinator()

  case class StartWriting()
  case class StartCoordinator()
  case class StopCoordinator()
  case class LogMessages(ls: List[String])
  case class Watch(msg: String)
  case class StartServer(port: Int)
  case class StopServer()

Key points to note

  • Co-Ordinator is initialized – line 1
  • Case classes ( StartWriting, StartCoordinator, LogMessages etc….)are defined. These represent messages which are passed around actors
  • Message sent to LogWriter to get started. This actor simulates logging – line 4.
  • Message sent to Co-ordinator to get started – line 5

Coordinator

This actor performs task of co-ordination, by passing messages across various actors.

class Coordinator(logFileName: String) extends Actor {
    val tcpServer = system.actorOf(Props(new TCPServer()), name = "tcpServer")
    val logWatcher = system.actorOf(Props(new LogWatcher(logFileName)), name = "logWatcher")

    def receive = {

      case StartCoordinator() => {
        println("Starting the coordinator.....")
        tcpServer ! StartServer(8080)
        logWatcher ! Watch("start watching")
      }

      case StopCoordinator() => {
        println("Stopping the coordinator...")
      }

      case LogMessages(ls: List[String]) => {
        tcpServer ! new LogMessages(ls)
      }

    }
  }

Key points to note

  • Actors are initialized – line 2, line 3
  • When this actor recieves message (StartCoordinator), it in turns sends message to LogWatcher, TCPServer to get started- line 9 , line 10
  • Message received from LogWatcher is sent to TCPServer – line 18

TCPServer

This represents simple HttpServer which accepts connections from clients and streams log messages.

class TCPServer() extends Actor {

  val socketConnections = new ArrayList[ReadHandle]()

  def receive = {
    case IO.NewClient(server) => {
      server.accept()
    }

    case IO.Read(rHandle, bytes) => {
      socketConnections.add(rHandle)
      println("Added new client.....")
    }

    case StartServer(port: Int) => {
      IOManager(context.system).listen(new InetSocketAddress(port))
      println("Server started on port " + port)
    }

    case LogMessages(ls) => {

      for (i <- 0 until socketConnections.size()) {     
        val handle = socketConnections.get(i)
        println("sending streams to client.....")
        for (line <- ls) {       
          val byteString = ByteString(line)
          handle.asSocket.write(byteString)
        }
      }
    }

  }
}

Key points to note

  • Server started – line 16
  • New clients are added – line 11
  • LogMessages sent to client – line 22

LogWatcher

This Actor keeps watching log files and sends messages to co-ordinator as file gets updated
Below is section of code associated with receive method of this actor

 def receive = {
    case Watch(msg) => {
      println("Msg recieved to watch.....")
      var bytesToSkip = 0
      while (keepRunning) {
        //println("original length=" + originalLength + "current length" + fileName.length())
        if (fileName.length() > originalLength) {
          val lines = getNewLinesFromFile(bytesToSkip)
          sender ! new LogMessages(lines)
          bytesToSkip = bytesToSkip + getBytesToSkip(lines)
          originalLength = fileName.length()
        }
        Thread.sleep(5000)
      }
    }

Key Points to note

  • This actor gets started when it receives Watch message – line 2
  • If contents of file is changed sends message back to Coordinator – line 9

Complete code base can be downloaded at github

Running The Application

  • Checkout the code and import it in eclipse
  • Ensure scala,akka libraries are present in buildpath
  • Start app by running ScalaStreamingLogs ( update location of logfile in line 3)
  • Open browser and type – http://localhost:8080. After about 30 secs you should see log messages

Overall Actor based model is much simpler,modular and easier to understand than the original Java version.
Further you don’t have to worry about Threads, Locks, Queues etc. Akka takes of low-level plumbing details

Implicits in scala

In this post I want to give an overview of Implicit Conversions in Scala.
These can be used in 3 places

  • Convert from one Type to another
  • Conversions of receivers – to facilitate calling of methods which are not defined for particular object
  • Conversions of Parameters

Lets look at each of them in detail

Convert from one Type to another

This technique can be used when you want to convert from one type to another. Consider a scenario where you have a function which does some processing on List. At some point later you need to be able to apply same function on Map, then you could use this technique. Below example illustrates the point

scala> val list = List(1,2,3,4,5)
list: List[Int] = List(1, 2, 3, 4, 5)

scala> def process(mylist:List[Int]):List[Int] = mylist.filter( _ <  5)
process: (mylist: List[Int])List[Int]

scala> process(list)
res15: List[Int] = List(1, 2, 3, 4)

scala> val myMap = Map ("key1"->1,"key2"->2,"key3"->3,"key4"->4,"key5"->5)
myMap: scala.collection.immutable.Map[String,Int] = Map(key4 -> 4, key5 -> 5, key1 -> 1, key2 -> 2, key3 -> 3)

scala> process(myMap)
<console>:25: error: type mismatch;
 found   : scala.collection.immutable.Map[String,Int]
 required: List[Int]
              process(myMap)
                      ^
scala> import scala.language.implicitConversions
import scala.language.implicitConversions

scala> implicit def mapToList(myMap:Map[String,Int]):List[Int]= myMap.valuesIterator.toList
mapToList: (myMap: Map[String,Int])List[Int]

scala> process(myMap)
res17: List[Int] = List(4, 1, 2, 3)

scala> implicit def mapToList(myMap:Map[String,Int]):List[Int]= myMap.valuesIterator.toList.sorted
mapToList: (myMap: Map[String,Int])List[Int]

scala> process(myMap)
res18: List[Int] = List(1, 2, 3, 4)
                             

In above example, we are

  • creating a list — line 1
  • applying function (process) on it. This returns a new list — line 4
  • creating a map and applying function (process) on it. This results in error — line 13
  • import implicitConversions — line 19
  • created implicit def mapToList which converts Map to List — line 22
  • function (process) is applied on map and this time it works fine –line 25

Conversions of receivers

This type of conversion can be used when you want to apply new behavior on existing objects.
Lets say you have Phone Object which supports following methods

  • makeCall
  • sendSMS

and a Tablet Object which supports following method

  • browseInternet

then by using this conversion – you can enable browseInternet method on Phone and makeCall,sendSMS method on Tablet
Below code illustrates the same.


scala > class Phone (n:String) {
         var name:String = n
         def makeCall()  = println("Making call with "+name)
         def sendSMS() = println("Sending sms with"+name)
   }

scala >defined class Phone

scala > class Tablet (n:String) {
 var name :String = n;
 def browse() = println("Browsing with "+name);
}

scala > defined class Tablet

scala> var phone =  new Phone("iPhone-5c")
phone: Phone = Phone@70d18a80

scala> phone.makeCall();
Making call with iPhone-5c

scala> phone.browse
<console>:11: error: value browse is not a member of Phone
              phone.browse

scala> var tablet = new Tablet("iPad");
tablet: Tablet = Tablet@4efade06

scala> tablet.browse
Browsing with iPad

scala> tablet.makeCall
<console>:11: error: value makeCall is not a member of Tablet
              tablet.makeCall

scala> implicit def phoneToTablet(x:Phone) = new Tablet (x.name)
phoneToTablet: (x: Phone)Tablet

scala> implicit def tabletToPhone(x:Tablet)   = new Phone (x.name)
tabletToPhone: (x: Tablet)Phone

scala> tablet.makeCall
Making call with iPad

scala> phone.browse
Browsing with iPhone-5c

Things to note

  • we defined classes Phone and Tablet — line 1, line 9
  • we get error message when we do phone.browse — line 22
  • we get error message when we do tablet.makeCall — line 32
  • we added implicit conversions — line 36, line 39
  • phone.browse and tablet.makeCall works fine — line 42, line 46

A wonderful explanation of these type of conversions ( View As Conversions) can be read here

Conversions for parameters

This type of conversion is used when you want to pass in some default parameters to your functions.
Consider a scenario where you want to be able to display Home Page for user based on particular Theme selected by him/her. However you also want to provide sensible defaults. By using this particular technique you can achieve the same


scala> class Theme(val name:String)
defined class Theme

scala> object Controller {
  def displayHomePage(name:String)(implicit theme:Theme) {
    println("Displaying home page for = "+name);
    println("Using theme = "+theme.name);
 }
}

scala> defined module Controller

scala> val simpleTheme = new Theme("simple");
simpleTheme: Theme = Theme@218f5a04

scala> Controller.displayHomePage("user1")(simpleTheme)
Displaying home page for = user1
Using theme = simple

scala> Controller.displayHomePage("user1")
<console>:20: error: could not find implicit value for parameter theme: Theme
              Controller.displayHomePage("user1")
                                        ^
scala> object ThemePrefs {
     |    implicit val theme = new Theme("default")
     | }
defined module ThemePrefs

scala> import ThemePrefs._
import ThemePrefs._

scala> Controller.displayHomePage("user1")
Displaying home page for = user1
Using theme = default

Things to note

  • Theme and Controller is defined — line 1, line 4
  • displayHomePage is called with Theme — line 16
  • displayHomePage being called without theme results in error — line 20
  • defaultTheme is defined and imported — line 24 , line 29
  • displayHomePage being called without theme works fine — line 32

That completes a quick overview on Implicits in Scala. For more thorough treatment refer to links below

Java Convert Date to a specific Timezone

As many would agree Java dates are messy. In this post I want to give overview on how you can convert dates in java from one Timezone to another using Joda Time library

Lets say you want to display dates to users based on their timezone. Below can be a static method to do the same

 public static Date getDateInTimeZone(Date currentDate, String timeZoneId) {
       
        TimeZone tz = TimeZone.getTimeZone(timeZoneId);
        Calendar mbCal = new GregorianCalendar(TimeZone.getTimeZone(timeZoneId));
        mbCal.setTimeInMillis(currentDate.getTime());
        Calendar cal = Calendar.getInstance();
        cal.set(Calendar.YEAR, mbCal.get(Calendar.YEAR));
        cal.set(Calendar.MONTH, mbCal.get(Calendar.MONTH));
        cal.set(Calendar.DAY_OF_MONTH, mbCal.get(Calendar.DAY_OF_MONTH));
        cal.set(Calendar.HOUR_OF_DAY, mbCal.get(Calendar.HOUR_OF_DAY));
        cal.set(Calendar.MINUTE, mbCal.get(Calendar.MINUTE));
        cal.set(Calendar.SECOND, mbCal.get(Calendar.SECOND));
        cal.set(Calendar.MILLISECOND, mbCal.get(Calendar.MILLISECOND));
        return cal.getTime();
    }

Now lets call above method and print the dates which we get

     Date currDate = new Date();
     Date dateIndia = DateUtil.getDateInTimeZone(currDate,"Asia/Kolkata");
     System.out.println("India date      :"+dateIndia);
     System.out.println("India time unix :"+dateIndia.getTime());

     Date dateUtc = DateUtil.getDateInTimeZone(currDate,"UTC");
     System.out.println("Utc date :"+dateUtc)
     System.out.println("Utc time unix :"+dateUtc.getTime());

Below will be output when you run it

     India date       :Sun Dec 15 12:09:06 IST 2013
     India time unix  :1387089546262

     Utc  date      :Sun Dec 15 6:39:06 IST 2013
     UTC time unix  :1387089546262

Notice how time and date is correct but timezone is messed up. Always shows IST (in my computer)

Lets try to do the same using Timezone and offset. Below is piece of code

public static Date getDateInTimeZone(Date currentDate, String timeZoneId) {
        TimeZone tz = TimeZone.getTimeZone(timeZoneId);
        Date convDate = new Date(currentDate.getTime()+ tz.getOffset(currentDate.getTime()));
        return convDate;
    }

Now lets run it using below snippet

     
     Date dateIndia = DateUtil.getDateInTimeZone(new Date(1387089546262l),"Asia/Kolkata");
     System.out.println("India date      :"+dateIndia);
     System.out.println("India time unix :"+dateIndia.getTime());

     Date dateUtc = DateUtil.getDateInTimeZone(new Date(1387089546262l),"UTC");
     System.out.println("Utc date :"+dateUtc)
     System.out.println("Utc time unix :"+dateUtc.getTime());
  

Below is the output

     India date       :Sun Dec 15 12:09:06 IST 2013
     India time unix  :1387089546262

     Utc  date      :Sun Dec 15 6:39:06 IST 2013
     UTC time unix  :1387089546262

Unfortunately same thing. It prints date and time correctly but timezone is not correct. (Internally it uses you settings associated with time zone computer you are running the code).

After looking at above i resorted to Joda Time (DateTime object) and things worked like charm. Take a look at example below

     
    public static String getDateForTimeZoneAsString(long dbDate,String timezoneId){
        DateTimeFormatter fmt = DateTimeFormat.forPattern("E,MMM d yyyy HH:mm:ss zz");
        DateTime origDate = new DateTime(dbDate);
        DateTime dtTz = origDate.withZone(DateTimeZone.forID(timezoneId));
        String dateForTz = dtTz.toString(fmt);
        return dateForTz;
    }

Below is snippet of code to run it

     String dateIndia = DateUtil.getDateInTimeZone(new Date(1387089546262l),"Asia/Kolkata");
     System.out.println("india date :"+dateIndia)
     String dateUtc = DateUtil.getDateInTimeZone(new Date(1387089546262l),"UTC");
      System.out.println("Utc date :"+dateUtc)

and below is the output

     India date       :Sun Dec 15 2013 12:09:06 IST
     Utc  date        :Sun Dec 15 2013 6:39:06 UTC

Things work like charm. Look at timezone. It prints correctly .Indeed very simple and elegant.

Functions in Scala

In this post I want to briefly talk about functions in scala.

In scala functions are first class citizens. They can be assigned to variables and passed around. Look at example below.

scala> (x:Int) => x * 2
res0: Int => Int = <function1>
scala> res0(2)
res1: Int = 4

Things to note

  • Line 1 illustrates example of declaring a simple function.
  • Line 2 shows that function is of name res0, parameter is Int  and return type is Int
  • Line 3 shows how function is called

Now lets assign function to a variable. Once we do that function can be used in expression. Consider example below

scala> var multiply = (x:Int) => x * 2
multiply: Int => Int = <function1>
scala> multiply(2)
res2: Int = 4
scala> multiply(multiply(2))
res3: Int = 8

Things to note

  • we assigned function to variable multiply  – Line 1
  • we can use multiply in any expression – Line 3, Line 5

This is the fundamental premise of functional programming. Functions can be passed around as variables in expression and each function performs some action and returns value. It does not have any side-effect

Consider example below. Further illustrates concept of passing functions as variables

scala> val myList = List(1,2,3,4,5)
myList: List[Int] = List(1, 2, 3, 4, 5)
scala> for(x:Int <- myList) yield multiply (x)
res6: List[Int] = List(2, 4, 6, 8, 10)

Things to note

  • In line 3, we are passing multiply function to for loop and using it with yield to return

In scala functions which take other functions are argument are called Higher Order Functions.
Consider example below.


scala> var multiply = (x:Int) => x * 2
scala> var add  = (x:Int) => x +2
scala> val myList = List(1,2,3,4,5)

scala> def transFormList(list:List[Int], myfunc:Int=>Int ) =  for(x:Int <- list) yield myfunc (x) 
transFormList: (list: List[Int], myfunc: Int => Int)List[Int]

scala> transFormList(myList,add)
res13: List[Int] = List(3, 4, 5, 6, 7)

scala> transFormList(myList,multiply)
res14: List[Int] = List(2, 4, 6, 8, 10)
  •  In line 1 we define function and assign it to multiply
  • In line 2 we define function and assign it to add
  • In line 5 we define a transformation function ( This is Higher order function)
    • parameters to function are – list, myfunc( a function).
    • It applies myfunc on list and returns new list
  • Lines 9 and 12 illustrates the result

Hope these examples provide an overview of how functions work in scala

Scala concepts for Beginners

In this post I would like to discuss about important concepts in scala when starting out. These were questions I asked when I started working on scala but had to dig around to find answers. My hope is this article can be help and shorten the learning curve.
Below are topics I would talk about

  • Type Inference
  • Immutability
  • Arrays,Collections and Tuples
  • Loops and Expressions
  • Functions
  • Classes and Companion Object

Type Inference

Scala as language does type inferencing. So you don’t need to specify the type when you declare any variables. Lets look at example below.   Open scala REPL and type below ( All examples mentioned in this article are tried in REPL)

scala> val number1:Int =5
number1: Int = 5

scala> val number2=5
number2: Int = 5

In example above in first case we declared variable number1 and defined its Type as Int ( after colon)
In second case we didnt specify Type and scala inferenced it as Int.
Also in Scala you dont need to specify semi-colon. It is inferenced . Thus you need to type less

Immutability

Scala as language promotes immutability, aligned with principles of functional programming. So if you don’t specify any modifier, by default val will be used. If you use var then value associated with variable can be changed . An exception to rule is while using REPL, where if you declare variable as val, then you can change the value

Arrays , Collections and Tuples

Arrays
Below are couple of examples of Arrays in Scala. They are self explanatory. Take a look at type inferencing( String for first case and Int for second)

scala> val names = Array("John","Ron","Tom")
names: Array[String] = Array(John, Ron, Tom)

scala> val numbers = Array(1,2,3)
numbers: Array[Int] = Array(1, 2, 3)

To access specific element of array just specify the index in parenthesis.

scala> numbers(1)
res0: Int = 2

Another interesting function is mkString which you can use to concatenate elements of array or collections for that matter

scala> names.mkString(",")
res6: String = John,Ron,Tom

Collections
Scala supports both mutable and immutable collections. Default is immutable.  Take a look at below diagram which shows set hierarchy. Similar hierarchy exists for Map

ref: Talks by Bill Venners and Dick Wall at Parleys

ref: Talk by Bill Venners and Dick Wall at Parleys

Set
By default when you create a set, immutable set is created. However if you want to create mutable set, you can use import scala.collection.mutable.Set.   Look at example below

scala> val citySet = Set("LA","NY","MUM")
citySet: scala.collection.immutable.Set[String] = Set(LA, NY, MUM)

scala> citySet+="SFO"
:9: error: value += is not a member of scala.collection.immutable.Set[String]
              citySet+="SFO"
                     ^

scala> import scala.collection.mutable.Set
import scala.collection.mutable.Set

scala> val citySetNew=Set("LA","NY","MUM")
citySetNew: scala.collection.mutable.Set[String] = Set(MUM, NY, LA)

scala> citySetNew+="SFO"
res3: citySetNew.type = Set(MUM, SFO, NY, LA)

scala> citySetNew
res4: scala.collection.mutable.Set[String] = Set(MUM, SFO, NY, LA)

Map
Rules for Map is similar to set above. Look at examples below

scala> val cityMap = Map(1->"Mum",2->"NY",3->"LA")
cityMap: scala.collection.immutable.Map[Int,String] = Map(1 -> Mum, 2 -> NY, 3 -> LA)

scala> cityMap += (4->"SFO")
:10: error: value += is not a member of scala.collection.immutable.Map[Int,String]
              cityMap += (4->"SFO")
                      ^

scala> import scala.collection.mutable.Map
import scala.collection.mutable.Map

scala> val cityMap2 =  Map(1->"Mum",2->"NY",3->"LA")
cityMap2: scala.collection.mutable.Map[Int,String] = Map(2 -> NY, 1 -> Mum, 3 -> LA)

scala> cityMap2 +=(4->"SFO")
res6: cityMap2.type = Map(2 -> NY, 4 -> SFO, 1 -> Mum, 3 -> LA)

List
Below are examples for list. Note – If list is declared as val , you cannot add new element to it, however if you declare it as var , you can add element

scala> val list1 = List(1,2,3,4)
list1: List[Int] = List(1, 2, 3, 4)

scala> list1 = 5::list1
:8: error: reassignment to val
       list1 = 5::list1
             ^

scala> var list2 = List(1,2,3,4)
list2: List[Int] = List(1, 2, 3, 4)

scala> list2 = 5::list2
list2: List[Int] = List(5, 1, 2, 3, 4)

scala> list2.size
res0: Int = 5

Tuples
You can use tuples when you want to return multiple values/types from a function.

scala> val tup1 = (4,"SFO")
tup1: (Int, String) = (4,SFO)

Expressions  and Loops

In scala every statement can be an expression (Any statement that returns a value is considered an expression).
If Loop

Below is example of if loop in scala


scala> val num1 = 4
num1: Int = 4

scala> val num2 =5
num2: Int = 5

scala> val finalNum = if(num1 > num2) num1 else num2
finalNum: Int = 5

scala> println(finalNum)
5

Result of if/else expression is assigned to finalNum. This is very much similar to Ternary Operator in Java

For Loop
Below is example of simple for loop in scala.

scala> val array1 = Array(1,2,3)
array1: Array[Int] = Array(1, 2, 3)
scala> for(x:Int <- array1) print (x)
123

Above for loop prints value not idiomatic scala way to do it as it has a side-effect.
Ideally in purely functional terms loop should return some value which can be used futher down the line. Look at example below. It uses yield to return value .

scala> array1
res10: Array[Int] = Array(1, 2, 3)
scala> for(x:Int<-array1) | yield(x*2)  
res11: Array[Int] = Array(2, 4, 6)  

In above case a new array (res11) is returned

Functions

Below is an example of declaring a function in scala

 
scala> def max(a:Int,b:Int):Int ={
          if (a > b) a
          else b
      }
max: (a: Int, b: Int)Int

scala> max(3,4)
res12: Int = 4

Things to note

  •  function definition should begin with def keyword
  • parameters are specified along with their types( as always type should be specified after name)
  • Return type of function specified after parameters . In above case it is Int
  • Function definition is specified after equals (=) inside braces

A streamlined version of above can be specified as below, which is more idiomatic scala

scala> def maxoneline(a:Int,b:Int):Int = if(a > b) a else b
maxoneline: (a: Int, b: Int)Int

Classes and Companion Object

Below is example of  class in Scala.

scala> class Product(n:String, p:Int) {
      var name :String= n
      var price:Int=p
  }
defined class Product
scala> val p = new Product("Product1",100)
p: Product = Product@17d9d565

scala> print(p.name)
Product1
scala> print(p.price)
100

Rather very concise and simple. The variables are public by default hence you can change them. checkout this article for example of private variables and getters/setters in scala

Above also illustrates example of Primary constructor (where values to variable name and price is assigned)

In scala class anything which is not included inside def will be executed as part of constructor.

Below is example of Auxiliary constructor


scala> class Product(n:String, p:Int) {
        var name :String= n
        var price:Int=p
        def this(n:String) = this(n , 200)
        println("name="+name+" , price="+price)
      }
defined class Product
scala> val p2 = new Product("product2")
name=product2 , price=200
p2: Product = Product@7a64f150

scala> p2.name
res4: String = product2

scala> p2.price
res5: Int = 200

As you can see from above Auxilary constructor uses Primary constructor internally. Below example illustrates use to overriding toString in scala


scala> class Product(n:String, p:Int) {
       var name :String= n
       var price:Int=p
       def this(n:String) = this(n , 200)
       override def toString = ( "name ="+name+", price="+price)
 }
defined class Product
scala> val p7 = new Product("product7",700)
p7: Product = name =product7, price=700

Companion Object

Scala doesnt have concept of static. It however has something called Companion object. A companion Object has same name as class name. You can invoke methods on them directly without creating instance


class Product(n:String, p:Int) {
   var name :String= n
   var price:Int=p
   def this(n:String) = this(n , 200)
   override def toString = ( "name ="+name+", price="+price)
}
object Product {
    def computeTax(price:Int):Double = {
    price+price*10
}
}
defined class Product
defined module Product

scala> val p8 = new Product("product8",800)
p8: Product = name =product8, price=800

scala> val tax = Product.computeTax(p8.price)
tax: Double = 8800.0

In above example computeTax can be called on product class without creating an instance.
As you have public static void main in Java, you need to define main method inside Companion Object in Scala

That completes some basic concepts in scala which can help you get started. I would recommend to look at below fantastic videos which covers above in more details

  1. First steps with Scala – part 1
  2. First steps with Scala – part 2

Nginx configurations

In this article I would like to write about some basic Nginx configurations.   These configurations can be made in default file which is usually located in /etc/nginx/sites-available folder in nginx. I would talk about below topics

  • Creating a proxy pass
  • Url re-write
  • Setting up ssl
  • Configuring php

Creating a Proxy Pass

Often when developing applications , UI layer would need access to api layer which could be deployed in another domain. To avoid cross-domain issue you need to set up proxy pass in Nginx. Below is the example configuration

 upstream _tomcat {
    server beta.somedomain.com :8080;
 }
 ##### urls which you want request to be sent to app server
 location ~ /url1* {
     proxy_pass http://_tomcat;
 }
 location ~ /url2* {
     proxy_pass http://_tomcat;
 }

 Url re-write

At times we run into issue where we want to be able to do redirect old urls to new urls ( without changing url in browser bar). Consider a case where you have old url of form

/products which needs to be redirected to  /products/listNew.html
/products/productName   which needs to be redirected to /products/listProduct.php?name=productName. Below is example configuration for same

  rewrite ^/products/$    /pr/listNew.html redirect;
  rewrite ^/products/(.*)  /pr/listProduct.php?slug=$1 break;

In above configs pr  is folder in nginx where all the static/phps are located

Setting Up SSL

Configuring nginx with ssl is super simple. First thing you need to do is get certificate for your domain from some certificate providers. You would need following to enable ssl

  • key  : key with which you will sign your certficate , ex yourdomain.key
  • crt   : certficiates which you can download from your provider after you sign up , ex yourdomain.com.chained.crt

You can place both of these in ssl folder in nginx.  After that to enable ssl for you domain add below configuration

   listen 80;
   listen 443 default_server ssl;
   ssl_certificate      /etc/nginx/ssl/yourdomain.com.chained.crt;
   ssl_certificate_key  /etc/nginx/ssl/yourdomain.key;

In case of above both http and https will work. If however you want all traffic to be served via https, add below configuration

server {
   listen 80;
    return 301 https://$host$request_uri;
}

Configuring Php

To be able to configure php in nginx, you need to install below packages

  • php5-common
  • php5-curl
  • php5-fpm

Below is nginx config

 location ~ .php$ {
   try_files $uri =404;
   fastcgi_split_path_info ^(.+\.php)(/.+)$;
   fastcgi_pass unix:/var/run/php5-fpm.sock;
   fastcgi_index index.php;
   include fastcgi_params;
 }

In addition make sure below line is un-commented /etc/php5/fpm/pool.d/www.conf

  listen = /var/run/php5-fpm.sock

That completes some basic nginx configurations which can come in handy. Good reference about same can be found here

Follow

Get every new post delivered to your Inbox.