Introduction to Buck

Maven vs. Buck

mvn package -Dmaven.{javadoc,test}.skip=true  ... 6m50s
buck build :gerrit                            ... 2m 3s
buck test --all                               ... 2m 5s

mvn package -Dmaven.{javadoc,test}.skip=true  ... 4m44s
buck build :gerrit                            ...    2s

• Buck is FAST
• Automatically exploits parallelism in builds and tests
• Incrementals are reliable (no more mvn clean;mvn package)
• Partial builds are simple (buck build gerrit-server:constants)
• Implemention changes don't require API consumers to rebuild
• Local and remote artifact caching

Introduction to Buck

Installation

Buck comes in source form as a Git repository

git clone https://github.com/facebook/buck
cd buck
ant                # 8.736s

PATH=$(pwd)/bin:$PATH

Alternative is to symlink to buck

ln -s $(pwd)/bin/buck  ~/bin/buck
ln -s $(pwd)/bin/buckd ~/bin/buckd       # optional

facebook.github.io/buck/setup/install.html

Android SDK configuration is only necessary to build Android applications

Build Rules

Written in files named BUCK

java_library(
  name = 'client',
  srcs = ['Client.java', 'Window.java'],
)

• A rule identifies one build artifact to be created
• Syntax is similar to a Python function call
• Arguments are passed by name

Build Rules - Naming

Targets (rules) are identified by filesystem path and name

plugins/replication/BUCK:
  java_library(name = 'replication', ...)
  java_library(name = 'utils', ...)

//plugins/replication:replication

• Rule names must be unique within a BUCK file
• Common to limit to [a-zA-Z_][a-zA-Z_0-9.-]*

Building Java Stuff

Example Java Project

lib/BUCK

prebuilt_jar(
  name = 'guava',
  binary_jar = 'guava.jar', 
)

src/BUCK

java_library(
  name = 'client',
  srcs = ['Client.java', 'Window.java'],
  resources = ['Client_en.properties', 'icon.gif'],
  deps = ['//lib:guava'],
)

java_binary(
  name = 'my_app',
  main_class = 'Client',
  deps = [':client'],
)

Java Rules - Libraries

prebuilt_jar(
  name = 'guava',
  binary_jar = 'guava.jar', 
)

• Identifies an existing JAR file to later use as a dependency (dep)
• Common for external libraries not built from source

java_library(
  name = 'client',
  srcs = ['Client.java', 'Window.java'],
  resources = ['Client_en.properties', 'icon.gif'],
  deps = ['//lib:guava'],
)

• Compiles source files in isolation, linked against deps
• Packs *.class and resource files into output JAR

Java Rules - Application Binary

java_binary(
  name = 'my_app',
  main_class = 'Client',
  deps = [':client'],
)

• Combines transitive deps into a single self-contained JAR
• Similar to Maven shade plugin, without class renaming
• Writes META-INF/MANIFEST.MF with Main-Class

Java Rules - Testing

java_test(
  name = 'tests',
  srcs = ['ClientTest.java', 'WindowTest.java'],
  deps = [':client', '//lib:junit'],
  source_under_test = [':client'],
)

• Compiles source files for JUnit test cases
• deps must include your own JUnit library
• Runs tests single threaded in a new JVM

• Parallelism is at the rule level

java_test(name = 'ClientTest', srcs = ['ClientTest.java'], deps = [':client', '//lib:junit'])
java_test(name = 'WindowTest', srcs = ['WindowTest.java'], deps = [':client', '//lib:junit'])

Macro Rules

Python functions to shorten build files

def parallel_test(srcs, deps = []):
  """Defines one java_test rule per source file."""
  for s in srcs:
    n = s[:s.index('.java')]
    java_test(name = n, srcs = [s], deps = deps)

Use macros in BUCK files

include_defs('//tools/parallel_test.defs')

parallel_test(
  srcs = glob(['*Test.java']),
  deps = [':client', ':junit'],
)

• Functions may declare arguments, have defaults, and invoke rules.
• Rule invocation inside macros behaves as if inlined into BUCK file.

Special Function glob()

• Expands file pattern into a file list
• Lists can be concatenated using +
• '*' matches anything in a directory, '/**/' matches recursively

TEST = glob(['src/main/java/**/*Test.java'])
API = ['src/main/java/com/mypkg/Greeter.java',
       'src/main/java/com/mypkg/Group.java']

java_library(
  name = 'hello_world',
  srcs = glob(['src/main/java/**/*.java'], excludes = API + TEST),
  deps = [':api'],
)

java_library(name = 'api', srcs = API)

java_test(
  name = 'tests',
  srcs = TEST,
  deps = [':api', ':hello_world'],
)

Not Compiling Java? - genrule()

Automatically download from Maven Central

prebuilt_jar(
  name = 'some_lib',
  binary_jar = genfile('some.jar'),
  deps = [':download'],
)

genrule(
  name = 'download',
  cmd = 'curl http://repo1.maven.org/maven2/.../some.jar >$OUT',
  out = 'some.jar',
)

• relies on /bin/bash -ec for execution of cmd
• $OUT replaced with location of out
• other rules may reference the output file with genfile()

Custom Logic? - genrule()

Concatenate 3 files together

genrule(
  name = 'concat',
  srcs = ['content.txt', 'header.html', 'footer.html'],
  cmd = 'cat $SRCDIR/header.html $SRCDIR/content.txt $SRCDIR/footer.html >$OUT',
  out = 'content.html',
)

• $SRCDIR is a flat directory tree of all files named as srcs
• rule must produce a single output file

ANTLR Support? - genrule()

genrule(
  name = 'gen_antlr',
  srcs = ['Query.g'],
  cmd = '$(exe :antlr) -o $TMP $SRCS; jar cf $OUT -C $TMP .',
  deps = [':antlr'],
  out = 'query.src.zip',
)

java_library(
  name = 'query_parser',
  srcs = [genfile('query.src.zip')],
  deps = [':gen_antlr'],
)

prebuilt_jar(name = 'antlr_jar', binary_jar = 'antlr.jar')
java_binary(name = 'antlr', main_class = 'org.antlr.Tool', deps = [':antlr_jar'])

• java_binary can be referenced using ($exe target)
• $TMP is an empty directory private to the genrule
• java_library can compile Java sources files from a .src.zip

Gerrit Code Review's Maven Build

What is Gerrit Code Review?

• Web interface for peer code review (talk today at 13:30)

• Java servlet application, Google Web Toolkit UI
• WAR file, deployable to Tomcat, Jetty, etc.
• Self contained executable using Jetty

Maven build process

• 31 pom.xml files
• 2,998 lines of XML (excluding copyright notices)
• Reactor build includes 30 modules
• 2 profiles

Migration to Buck - Parallel Builds

• Created a ~6 line BUCK file alongside each pom.xml

java_library(
  name = 'this-module',
  srcs = glob(['src/main/java/**/*.java']),
  resources = glob(['src/main/resources/**/*']),
  deps = [
    '//other-module:other-module',

    '//lib:gson',
    '//lib:gauva',
  ],
  visibility = ['PUBLIC'],
)

• lib/BUCK enumerated all prebuilt_jars() from Maven Central
• ANTLR, GWT, PrologCafe, AsciiDoc quickly supported with genrules
• 1,719 lines of BUCK vs. 2,998 lines of pom.xml (42% shorter)

maven_jar Macro

lib/BUCK

include_defs('//lib/maven.defs')

maven_jar(
  name = 'gson',
  id = 'com.google.code.gson:gson:2.1',
  sha1 = '2e66da15851f9f5b5079228f856c2f090ba98c38',
  license = 'Apache2.0',
)

maven_jar(
  name = 'guava',
  id = 'com.google.guava:guava:14.0',
  sha1 = '67b7be4ee7ba48e4828a42d6d5069761186d4a53',
  license = 'Apache2.0',
)

• maven_jar downloads artifacts on demand and verifies SHA-1
• license annotation is used later to build documentation
• versions bound in a single location, neatly aliased for use

Getting Along with Buck

Successful Buck Tips #1 - Buck Version

Create .buckversion with Git SHA-1 of Buck:

(cd ~/buck; git rev-parse HEAD) >.buckversion
git add .buckversion

• Ensure repeatability by pinning to a specific version of Buck.
• Buck automatically upgrades (or downgrades) as necessary.
• Accept language updates on your schedule.

Temporarily disable version enforcement using .nobuckcheck

touch .nobuckcheck

Successful Buck Tips #2 - Project Defaults

Add aliases for common targets to .buckconfig

[alias]
  docs = //Documentation:html
  server = //gerrit-server:server

buck build docs          # instead of buck build Documentation:html

Configure project-wide macro rules in .buckconfig

[buildfile]
  includes = //tools/default.rules

Successful Buck Tips #3 - Even Faster Builds

Use buckd for faster incremental builds:

~/buck/bin/buckd
time buck :gerrit      ...  0.5s        # without buckd, 2s

If using git and buckd, ignore .git in .buckconfig

[project]
  ignore = .git

Enable local artifact caching in .buckconfig.local

[cache]
  mode = dir
  dir = buck-cache

• Avoids expensive rebuilding when switching branches
• Per-developer configuration using .buckconfig.local
• Shared team cache available backed by Apache Cassandra

Successful Buck Tips #4 - Maven Converts

Configure source roots in .buckconfig

[java]
  src_roots = java, resources

• Ensures resource files are correctly packaged in JARs

Borrow Gerrit Code Review's maven_jar() rule

maven_jar(
  name = 'guava',
  id = 'com.google.guava:guava:15.0',
  sha1 = 'ed727a8d9f247e2050281cb083f1c77b09dcb5cd',
  license = 'Apache2.0',
)

• Downloads on demand from Maven Central (faster than mvn!)
• Verifies content using known good signature stored in your project

Thank you