Monthly Archives: August 2015

Play Framework (5) – performance optimization (1)

It is a long way on this road. When you finish one feature, it is just a start. The rest thing is to make it better and better. You will need to care more about performance. Servals things which I experience here I write down.

  1. Deployment method

    1. Don’t use start or run to deploy your production. It is totally wrong. You need to use:
      activator clean; activator dist

      to package your project and then unzip it which is under

      unzip #your_project_path/target/universal/  

      and then cd to this folder. you will find under /bin/ there is script which can run. My normal jvm parmaters like this: (Please note here my project uses Nginx as web server and New Relic to collect application performance info. You need to modify jvm parameters according to your project. )

      nohup bin/#YourApp -Dhttp.port=9081 -Dhttp.address=#MyPrivateServerIP -J-Xms4096m -J-Xmx4096m -J-Xmn2048m -J-javaagent:../../../newrelic/newrelic.jar &
  2. Third-party Tool

    1. check cpu and memory usage by New Relic (I have another post which tell you how to use free New Relic)
  3. Command Tool

    When you really meet problem, New Relic just tells you that your app is not normal. It is not enough to solve the problem. So the rest thing for you is to dig out the real cause. Here are some useful commands which will help you a lot.

    1. htop     This is more detailed info than New Relic. You can do some actions on your application and monitory by htop to see whether these actions will cause performance changes.
    2. jcmd     This will list all jvm applications and you will get each PID. Or jps -l 
    3. jcmd <PID> help  This will list commands to tell you how to use it to get which info you want.
      • The following commands are available:
        1. VM.native_memory
        2. VM.commercial_features
        3. GC.rotate_log
        4. ManagementAgent.stop
        5. ManagementAgent.start_local
        6. ManagementAgent.start
        7. Thread.print
        8. GC.class_histogram
        9. GC.heap_dump
        10. GC.run_finalization
        11. GC.runVM.uptime
        12. VM.flags
        13. VM.system_properties
        14. VM.command_line
        15. VM.version
        16. GC.class_stats    : print the name of the ClassLoader
    4. jcmd <PID> ###     Here ### is getting from the list in above command’s result. I care about cpu usage, so normally I use Thread.print to check which threads are using.
    5. jstack This directly outputs running threads. You also can output to local file by jstack > #YourFile
  4. Problems which I met

    When you already find root cause, the last thing is to solve it.

    1. Today my problem is that I use Akka to do some tasks, but each time scheduler finishes task and doesn’t shutdown its ActorSytem. The more tasks it does, the more threads it opens. Finally my cpu usage reaches 100% and then the project doesn’t have any response.(All of them are analyzed by above commands) I use above commands to find which actors are running and shutdown them when they finish. Now my project’s cpu usage is only 1.0% or so.
    2. Everything seems perfect. I also though my code would not meet performance in future. But nothing is perfect enough. So I come back to update this post to add more problems which I met. Today problem appears when I migrate application from old single-cpu server to new 8-cores server. The cpu usage for my application in old single-cpu server is only 0.5% every day. But for new high configuration cpu it is almost 400% (because it is 8 cores, it is equal to use up half of cpu). I use the method in above: and find root cause is HashMap.
      The easy method is to change hashmap to ConcurrentHashMap. After this modification, cpu usage goes to normal, 0.5%.

      var schedulerIDs = new ConcurrentHashMap[String, Cancellable]().asScala

      ConcurrentHashMap can solve the HashMap’s thread 100% cpu issue. HashMap doesn’t support multiple threads. Detailed info please read here: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6423457
      ConcurrentHashMap concurrency is the size of segment, by default it is 16. This means at most there are 16 threads operates ConcurrentHashMap. This is great benefit for ConcurrentHashMap, not for HashTable.

    3. My performance issue continues, now it is hard to find the root cause. I try to read more to understand them better. But it is really hard. This time, my memory is always increasing triggering by some actions which are hard to find role. To be honest, I find some hit in Thread.print which mentions lots of time of “ForkJoinPool”. So I go back to my code to check which thread will be created by forkjoinpool and how does forkjoinpool manage memory. Here I read these following words. In fact, in my old code, i used scala standard library, it caused increasing memory problem. When i change it to play.api.libs, thing becomes better.
    4. In Play 2.3.x and prior, play.core.Execution.Implicits.internalContext is a ForkJoinPool with fixed constraints on size, used internally by Play. You should never use it for your application code. From the docs: 
      Play Interval Thread Pool - This is used internally by Play. No application code should ever be executed by a thread in this thread pool, and no blocking should ever be done in this thread pool. Its size can be configured by setting internal-threadpool-size in application.conf, and it defaults to the number of available processors.
      Instead, you would use play.api.libs.concurrent.Execution.Implicits.defaultContext , which uses an ActorSytem.
      In 2.4.x, they both use the same ActorSystem. This means that Akka will distribute work among its own pool of threads, but in a way this is invisible to you (other than configuration). Several Akka actors can share the same thread.
      scala.concurrent.ExecutionContext.Implicits.global is an ExecutionContext defined in the Scala standard library. It is a special ForkJoinPool that using the blocking method to handle potentially blocking code in order to spawn new threads in the pool. You really shouldn't use this in Play application, as Play will have no control over it. It also has the poential to spawn a lot of threads and use a ton of memory, if you're not careful.

To be honest, there is no general method to optimize performance. The more experiences you have, the less errors you create. When you meet performance issue, don’t be afraid and pay attention on finding root cause step by step. Only when you know what’s the cause, you can find right solution to fix it. Before that, any guess or others’ method is only useless.

Advertisements

Docker Configuration for development env on Mac

Recently, my computer meets a problem; it already shuts down automatically. I already influences my normal development work. I went to apple store and then they checked everything. They said nothing wrong with hardware, maybe something wrong with software. They suggested me to refresh everything and recover back to empty operating system. It is really pain thing for me. Backup all documents, images and videos cost me 5 hours. This is ok, because I only need copy them back. But all applications and development env are gone. In fact, applications are also ok, since I can re-install them back. The worst thing is for the development env. I need to re-install and re-configure. So in order to avoid this worst thing happens again, I decide to use Docker to re-config my development env. If in the future, the problem happens again or I need to switch to another new device to work, it is more easily for me and helps me to shorten the transformation time. So let’s do it!

My final target development env

Nginx: load balance, file system, web server
Play framework: web architecture
Git : version control
Mysql: data storage
Gulp and sass: front-end css development

How to touch Docker

Step1: Download and Install

Go to here https://github.com/boot2docker/osx-installer/releases/tag/v1.7.1 to download and install. Now there is another method which also can help you install docker, named Docker Toolbox. https://www.docker.com/docker-toolbox

Step2: Open your terminal

$ boot2docker init // it will create a Docker VM
$ boot2docker start // It will start Docker VM
$ boot2docker start -m=512 //If your memory size is less than 4G, you'd better manually figure out VM size
$ boot2docker ip
$ boot2docker ssh (Default password: tcuser) // it will enter Docker VM
$ exit // it will quit Docker VM
$ boot2docker stop // it will stop Docker

Step3: Some useful Docker commands in VM

$ docker search centos // It will help you to search one image
$ docker pull centos // It will help you to download one image
$ docker images // it will list all images which you have
$ docker ps // it will show all running containers
$ docker ps -a // it will show all containers
$ docker rm // it will remove container
$ docker rmi // it will remove image
$ docker rm `docker ps -a -q` // it will remove all containers
$ docker history image_name // list image's history
$ docker info // get docker info

Step4: Start your own image

Step4.1 Enter into a new container

$ docker search ubuntu
$ docker pull ubuntu
$ docker run -i -t ubuntu // -i: sync container's stdin, -t: sync container's output, -d: deamon and uses backend to start this container

4.2 Start to install everything which you need

$ apt-get install nginx
$ apt-get install git
$ apt-get install vim
$ apt-get update
$ apt-get install software-properties-common python-software-properties // for JAVA
$ add-apt-repository ppa:webupd8team/java // for JAVA
$ apt-get update // for JAVA
$ apt-get install oracle-java8-installer // for JAVA
$ java -version // check java
$ wget http://downloads.typesafe.com/typesafe-activator/1.2.12/typesafe-activator-1.2.12.zip // for Typesafe
$ apt-get install unzip // for unzip
$ unzip typesafe-activator-1.2.12.zip // for Typesafe (I have problem on the newest version 1.3.5 when unzip; but it is ok for 1.2.12. So here I still use 1.2.12)
$ rm typesafe-activator-1.2.12.zip // for Typesafe
$ vi ~/.bashrc // add PATH=$PATH:/activator-1.2.12
$ source ~/.bashrc // activate bashrc
$ echo $PATH // check it
$ activator -help // check typesafe
$ apt-get install mysql-server mysql-client // for MySQL (during this process, you need to set your MySQL password)
$ mysql -V // check mysql

4.3 Submit your container

$ docker ps -l
$ docker commit -m "add nginx from ubuntu:latest" -a "haimei" 98710810b120 haimei/ubuntu-nginx:v1 // -m: add description; -a: figure author; 98710810b120: container's ID; haimei/ubuntu-nginx:v1 figures imageName,repositoryName and tagName

4.4 Connect between different containers

Step5: Publish to Docker Hub

$ docker login // input your info when you already signup on docker hub
$ docker push haimeili/ubuntu-nginx:v1

Step5: Fix bugs

“no space left on device”

If you’re using Boot2Docker with a large number of images, or the images you’re working with are very large, your pulls might start falling with “no space left on device” errors when the Boot2Docker volume fills up.

Here are some steps you need to do to fix it.

$ boot2docker config
$ boot2docker config > ~/.boot2docker/profile
$ vi ~/.boot2docker/profile // modify: DiskSize = 50000
$ boot2docker poweroff
$ boot2docker destroy
$ boot2docker init
$ boot2docker up

Scala Basic Data Type

The more you learn, the more you might be lost. So going back to original basic part is a good choice to find way back. So here I list basic data type and how to use it in Scala. Very simple, but very useful.

1. Array

val greetStrings = new Array[String](3)

for (i <- 0 to 2)
  print(greetStrings(i))

greetStrings(0) = "hello"

greetStrings.update(0, "hello")

val numNames= Array("zero", "one", "two")
val numNames1 = Array.apply("zero", "one", "two")

2. Lists

val one = List(1,2,3)
val two = List(4,5)
val three = one ::: two // List(1,2,3,4,5)
val three1 = 1 :: two // List(1,1,2,3)
val three2 = 1 :: 2 :: 3 :: Nil // List(1,2,3)

3. Tuples

val pair = (1, "2")
println(pair._1)

4. Sets and Maps

var jetSet = Set("boeing", "airbus")
jetSet += "Lear"
println(jetSet.contains("abc"))
import scala.collection.immutable.HashSet
val hashSet = HashSet("a", "b")
println(hashSet + "c")
import scala.collection.mutable.Map
val treasureMap = Map[Int, String]()
treasureMap += (1 -> "a")
println(treasureMap(1))

5. Learn to recognize the functional style

def printArgs(args: Array[String]): Unit = {
  args.foreach(println)
}
def printArgs(args: Array[String]): Unit = {
  for (arg <- args) 
    println(arg)
}
def formatArgs(args: Array[String]) = args.mkString("\n")

6. Read lines from a file

import scala.io.Source
if (args.length > 0 ) {
  for (line <- Source.fromFile(args(0)).getLines())
    println(line.length + " " + line)
} else 
  Console.err.println("Please enter filename")

val lines = Source.fromFile(args(0)).getLines().toList

7. File Operation

val filesHere = (new java.io.File(".")).listFiles
for (
  file <- filesHere 
  if file.isFile
  if file.getName.endsWith(".scala"))
  println(file)
def fileLines(file: java.io.File) = 
  scala.io.Source.fromFile(file).getLines().toList

def grep(pattern: String) =
  for {
    file <- filesHere
    if file.getName.endsWith(".scala")
    line <- fileLines(file)
    trimmed = line.trim
    if trimmed.matches(pattern)
  } println(file + ": " + trimmed)
val forLineLengths = 
  for {
    file <- filesHere
    if file.getName.startWith("hello")
    if file.getName.endsWith(".scala")
    line <- fileLines(file)
    trimmed = line.trim
    if trimmed.matches(".*for.*")
  } yield trimmed.length

8. String

println("""Welcome to Ultamix 3000. 
           Type "Help" for Help.""")
println("""|Welcome to Ultamix 3000.
           |Type "Help" for help.""".stripMargin)

9. Operators are methods

val s = "hello"
s indexOf 'o' 
s.indexOf('o')
s toLowerCase
s.toLowerCase

10. Exception handling with try expressions

import java.io.FileReader
import java.io.FileNotFoundException
import java.io.IOException
try {
  val f = new FileReader("input.txt")
} catch {
  case ex: FileNotFoundException =>
  case ex: IOException =>
}
import java.io.FileReader
val file = new FileReader("input.txt")
try {
} finally {
  file.close()
}

11. Placeholder syntax

someNumbers.filter(_ > 0)
val f = (_: Int) + (_: Int)

12. Simplify client code

Bad code:

def containsOdd(nums: List[Int]): Boolean = {
  var exists = false
  for (num <- nums)
    if (num % 2 == 1)
      exists = true
  exists
}

Good code:

def containsOdd(nums: List[Int]) = nums.exists(_ % 2 == 1)