Improve your Java builds with Buck

31 October 2013

Shawn Pearce

Maintainer, Gerrit Code Review

Build Languages Suck

My Application

package org.eclipsecon.europe2013;

import com.google.common.base.Joiner;

public class Printy {
  public static void main(String[] argv) {
    System.out.println(Joiner.on(' ').join(argv));
  }
}

Prints command line arguments.
Uses Guava (an external library) to join strings.

Simple.

Make

SRC=src/main/java
OUT=build
DST=dist
LIB=guava.jar
SRCS := $(shell find $(SRC) -name '*.java')

all: $(DST)/printy_lib.jar

$(DST)/printy_lib.jar: compile
    mkdir -p $(DST)
    jar cf $@ -C $(DST) .

compile: $(SRCS)
    mkdir -p $(OUT)
    javac -d $(OUT) -classpath $(LIB) $<

clean:
    rm -rf $(OUT) $(DST)

Thank $DIETY we have Ant.

Ant

<project name="Printy" basedir=".">
  <property name="src" location="src/main/java"/>
  <property name="out" location="build"/>
  <property name="dst" location="dist"/>

  <target name="init">
    <mkdir dir="${out}"/>
    <mkdir dir="${dst}"/>
  </target>

  <target name="package" depends="init">
    <javac srcdir="${src}" destdir="${out}">
      <classpath>
        <pathelement location="guava.jar"/>
      </classpath>
    </javac>
    <jar jarfile="${dst}/printy_lib.jar" basedir="${out}"/>
  </target>

  <target name="clean">
    <delete dir="${out}"/>
    <delete dir="${dst}"/>
  </target>
</project>

Maven

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>org.eclipsecon.europe2013</groupId>
  <artifactId>Printy</artifactId>
  <version>1.0-SNAPSHOT</version>

  <dependencies>
    <dependency>
      <groupId>com.google.guava</groupId>
      <artifactId>guava</artifactId>
      <version>15.0</version>
    </dependency>
  </dependencies>
</project>

Thank $DIETY we no longer have Ant.

Buck

java_library(
  name = 'printy_lib',
  srcs = glob(['src/main/java/**/*.java']),
  deps = [':guava'],
)

prebuilt_jar(
  name = 'guava',
  binary_jar = 'guava.jar',
)

Thank $DIETY we no longer need to use Maven.

Buck - Self Contained Application

java_library(
  name = 'printy_lib',
  srcs = glob(['src/main/java/**/*.java']),
  deps = [':guava'],
)

prebuilt_jar(
  name = 'guava',
  binary_jar = 'guava.jar',
)

java_binary(
  name = 'printy',
  main_class = 'org.eclipsecon.europe2013.Printy',
  deps = [':printy_lib'],
)

No contest.
Sorry Maven Shade Plugin, you lose.

Buck?

What is Buck?

java_library(
  name = 'printy_lib',
  srcs = glob(['src/main/java/**/*.java']),
  deps = [':guava'],
)

prebuilt_jar(
  name = 'guava',
  binary_jar = 'guava.jar',
)

Builds Are Slow

Maven vs. Buck on Gerrit Code Review

Clean build 

mvn package -Dmaven.{javadoc,test}.skip=true  ... 6m50s
buck build :gerrit                            ... 2m 3s
buck test --all                               ... 2m 5s

No-op incremental rebuild 

mvn package -Dmaven.{javadoc,test}.skip=true  ... 4m44s
buck build :gerrit                            ...    2s

Buck is FAST

Even faster with buckd running in the background 

~/buck/bin/buckd
time buck :gerrit                             ...    0.5s

Build System Implementations Suck

Multiple CPUs

make

Ant, Maven

Buck

Up-to-Date Check

make, Ant, Maven 

IF mtime(src) > mtime(out) THEN
  REBUILD

Buck 

IF sha1(src.content) != sha1(out.src.content) THEN
  REBUILD

Buck is significantly faster while providing perfect accuracy.

Deleted Classes and Resources

make, Ant

Maven

Buck

Artifact Caching

make, Ant

Maven

Buck

Resuable Build Fragments

make

Ant

Maven

Buck

Aside: Extending Buck

JUnit Tests in Buck

java_test(
  name = 'tests',
  srcs = ['ClientTest.java', 'WindowTest.java'],
  deps = [':client', '//lib:junit'],
  source_under_test = [':client'],
)
java_test(name = 'ClientTest', srcs = ['ClientTest.java'], deps = [':client', '//lib:junit'])
java_test(name = 'WindowTest', srcs = ['WindowTest.java'], deps = [':client', '//lib:junit'])

Macro Rules

Python functions to shorten build files 

def parallel_test(srcs, deps = []):
  """Defines one java_test rule per source file."""
  for s in srcs:
    n = s[:s.index('.java')]
    java_test(name = n, srcs = [s], deps = deps)

Use macros in BUCK files 

include_defs('//tools/parallel_test.defs')

parallel_test(
  srcs = glob(['*Test.java']),
  deps = [':client', ':junit'],
)

Reducing Rebuilds with Buck

Buck is Java Language Aware

java_library(name='window', srcs=['Window.java'])
java_library(name='client', srcs=['Client.java'], deps=[':window'])

public class Window {
  public int getHeight() { return 500; }
}

public class Client {
  public void display(Window w) {
    System.out.println(w.getHeight());
  }
}

Change getHeight to return 800 instead of 500?

Buck is Java Language Aware

java_library(name='window', srcs=['Window.java'])
java_library(name='client', srcs=['Client.java'], deps=[':window'])

public class Window {
  public int getHeight() { return 500; }
}

public class Client {
  public void display(Window w) {
    System.out.println(w.getHeight());
  }
}

Change getHeight to return 800 instead of 500?
client does not rebuild

No API changed, only implementation.
Implementation changes do not impact the compliation of client.

Buck is Java Language Aware

java_library(name='window', srcs=['Window.java'])
java_library(name='client', srcs=['Client.java'], deps=[':window'])

public interface Window {
  public int getHeight();
}

public class Client {
  public void display(Window w) {
    System.out.println(w.getHeight());
  }
}

Change Window from class to interface?

Buck is Java Language Aware

java_library(name='window', srcs=['Window.java'])
java_library(name='client', srcs=['Client.java'], deps=[':window'])

public interface Window {
  public int getHeight();
}

public class Client {
  public void display(Window w) {
    System.out.println(w.getHeight());
  }
}

Change Window from class to interface?
client rebuilds (Java bytecode is different to call an interface.)

Buck rebuilds modules only when API changes.

Implicit Java Compliation

class Main { public static void main(String[] a) { new App(); } }
class App {}

$ javac Main.java

Java compiler implicitly builds App.java from same directory.

Implicit Compliation in Buck

java_library(name = 'app', srcs = ['App.java'])
java_library(name = 'main', srcs = ['Main.java'])

class Main { public static void main(String[] a) { new App(); } }
class App {}

Attempting to build main fails: 

$ buck build :main
[-] PARSING BUILD FILES...FINISHED 0.2s
[+] BUILDING...0.7s
 |=> //:main...  0.7s (running javac[0.6s])
/Users/sop/git/talks/2013/ec-buck/h2h/fail/Main.java:1: error: cannot find symbol
class Main { public static void main(String[] a) { new App(); } }
                                                       ^
  symbol:   class App
  location: class Main
BUILD FAILED: //:main failed on step "javac" with exit code 1

Buck isolates library builds, even from the same source directory.
Permits fine grained segmentation within a package.

How Gerrit Code Review escaped Maven

What is Gerrit Code Review?

Maven build process

Migration to Buck - Parallel Builds

java_library(
  name = 'this-module',
  srcs = glob(['src/main/java/**/*.java']),
  resources = glob(['src/main/resources/**/*']),
  deps = [
    '//other-module:other-module',

    '//lib:gson',
    '//lib:gauva',
  ],
  visibility = ['PUBLIC'],
)

maven_jar Macro

lib/BUCK 

include_defs('//lib/maven.defs')

maven_jar(
  name = 'gson',
  id = 'com.google.code.gson:gson:2.1',
  sha1 = '2e66da15851f9f5b5079228f856c2f090ba98c38',
  license = 'Apache2.0',
)

maven_jar(
  name = 'guava',
  id = 'com.google.guava:guava:14.0',
  sha1 = '67b7be4ee7ba48e4828a42d6d5069761186d4a53',
  license = 'Apache2.0',
)

Thank you

Shawn Pearce

Maintainer, Gerrit Code Review

http://facebook.github.io/buck/