Automatic Builds at Your Fingertips With GCP Cloud Build

You are looking for an easy way to automatically build your application in the Cloud? Then maybe Google Cloud Platform (GCP) Cloud Build is something for you. In this post, we will build a Spring Boot Maven project with Cloud Build, create a Docker image for it and push it to GCP Container Registry.

1. Introduction

Cloud Build is the build server tooling of GCP, something similar as Jenkins. But, Cloud Build is available out-of-the-box in your GCP account and that is a major advantage. The only thing you will need, is a build configuration file in your git repository containing the build steps and that is about it. Each build step is running in its own Docker container. Several cloud builders which can be used as a build step are generally available. You can read more about Cloud Build on the overview and concepts website of GCP. There are three categories of build steps:

  • Official cloud builders provided by GCP, e.g. a Maven builder is available here;
  • Community cloud builders, e.g. a SonarQube builder is available here;
  • You can create your own Docker image which can be used in a build step. We will create a custom Docker image in this post later on.

Before getting started, you will need to create a GCP account if you don’t already have one. Check out the first paragraph of a previous post how to do so.

First, we need to create a new project mygcpcloudbuild.

Create GCP project

Go to the left menu to Tools – Cloud Build and click e.g. the History item.

GCP tools menu

A message is shown that we first need to enable the Cloud Build API.

Enable Cloud Build API

After enabling the Cloud Build API, we are ready to go!

2. Create the Example Application

We are going to create a Spring Boot application, therefore we go to the SpringInitializr website. We select Web and Java 11 and we add a HelloController to the project which returns a simple welcome message. Sources can be found at GitHub (use the tag withoutcustommavendocker first, the master branch contains changes needed at the end of this post). Before we pushed everything to GitHub, we verified whether the project builds locally:

$ mvn clean install -DskipTests

3. Create the Build File

As mentioned in the introduction, we need to create a build configuration file cloudbuild.yaml in the root of our git repository. First, we will only add a Maven build step and verify whether this builds successfully on Cloud Build. It would be logical to choose for the official Maven Cloud builder, but, at the time of writing, only JDK 8 and Maven version 3.3.9 and 3.5.0 are supported. We are using JDK 11 in the application and therefore we will use the official Maven Docker image with Maven 3.6.0. We add the following cloudbuild.yaml file to the root of the git repository:

steps:
  - name: maven:3.6.0-jdk-11-slim
    entrypoint: 'mvn'
    args: ['clean', 'install', '-DskipTests']

4. Build on Cloud Build

Now that our git repository is ready, we return to GCP. We want a build to be triggered on every commit to our repository. Navigate to the menu and click Cloud Build – Triggers. The triggers are still a beta feature, but it worked fine during our experiment with Cloud Build. Click the Create Trigger button.

Cloud Build create trigger 1

We need to select a repository. We choose GitHub and click the Continue button.

Cloud Build create trigger 2

In the next step, we need to authorize GCP to access our repository. Enter your GitHub credentials and continue.

Cloud Build create trigger 3

We select the repository we want to build and click Continue.

Cloud Build create trigger 4

And finally, we need to configure some settings. In the Build Configuration section, we choose for Cloud Build configuration file (yaml or json) and click the Create Trigger button. You can specify more things over here, e.g. which branches you want to build.

Cloud Build create trigger 5

The build triggers overview now contains the trigger we just created. We start our first build by clicking the Run Trigger button.

Cloud Build - first build

The build history now contains an entry for our successful build.

Cloud Build - successful build

5. Add Docker Build Step

Now that our Maven build is successful, it is time to add a build step for creating a Docker image for our application. We therefore create the following Dockerfile and add it to the root of our git repository:

FROM openjdk:11-jdk
VOLUME /tmp
ADD target/mygcpcloudbuildplanet-0.0.1-SNAPSHOT.jar app.jar
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]

We adapt the cloudbuild.yaml file with an extra build step for creating the Docker image and for pushing it to the GCP Container Registry.

  - name: 'gcr.io/cloud-builders/docker'
    args: ['build', '-t', 'gcr.io/$PROJECT_ID/mygcpcloudbuildplanet', '.']
images: ['gcr.io/$PROJECT_ID/mygcpcloudbuildplanet']

After pushing these changes to the git repository, a build is automatically triggered at Cloud Build. The Docker image is available in the Container Registry after the build has finished.

Cloud Build - docker registry

The Maven build is successful, the Docker build is successful, the Docker image is available in the Docker repository. But does the application run successfully? Let’s find out. Open the Cloud Shell in GCP and run the Docker image in detached mode.

$ docker run -p 127.0.0.1:8080:8080/tcp -d gcr.io/mygcpcloudbuild/mygcpcloudbuildplanet

Verify whether the Docker container is running with the docker ps command:

$ docker ps
CONTAINER ID    IMAGE                                            COMMAND                  CREATED           STATUS           PORTS                       NAMES
f17203279c03    gcr.io/mygcpcloudbuild/mygcpcloudbuildplanet    "java -Djava.securit…"    18 seconds ago    Up 17 seconds    127.0.0.1:8080->8080/tcp    jolly_mcclintock

The container is up-and-running, we verify the response of our welcome message:

$ curl http://localhost:8080/hello
Hello Google Cloud Build! From host: f17203279c03/172.18.0.2

The URL can also be accessed via the Web Preview feature of Cloud Shell, you only need to add hello to the URL otherwise a 404 HTTP error is presented:

https://8080-dot-6340638-dot-devshell.appspot.com/hello/?authuser=0

Pretty cool, isn’t it? We just created in a few minutes a basic build pipeline with Cloud Build.

6. Something About Maven Dependencies

We are using a Docker image for building the application. This means that for each build, all the Maven dependencies are downloaded again. This isn’t a major problem for a very small application, but the number of Maven dependencies can grow significantly over time. If this happens, it will contribute for a specific amount of your build time. Three options are available for resolving this issue:

  1. Do nothing, we can accept the extra build time and it doesn’t bother us;
  2. We can try to cache the Maven repository between builds, a solution which is provided at the Optimizing build speed page;
  3. We can create a custom Docker Maven image which contains the Maven dependencies for our build as described here.

We already tried solution 1, let’s take a closer look at the two other solutions.

6.1 Cache Maven Repository Between Builds

The solution in paragraph Caching directories with Google Cloud Storage at the Optimizing build speed page suggests to copy the results of a previous build to a Google Cloud Storage bucket and then copy them again before a next build. The changes to the git repository for this experiment can be found in branch feature/mavencache at GitHub.

Via the menu in GCP, go to Storage – Browser and create a bucket com-mydeveloperplanet-mavenrepo. We will use this storage for our Maven repository.

Change the cloudbuild.yaml file. We have 3 build steps:

  1. Copy the Maven dependencies from the bucket to the Docker volume mavenrepo;
  2. Run the Maven build where we also mapped the Docker volume mavenrepo in order that the Maven dependencies don’t need to be downloaded;
  3. Copy the Maven dependencies back to the bucket. New downloaded Maven dependencies will always be available in the bucket for the next build.
steps:
  # Copy Maven dependencies to volume mavenrepo
  - name: gcr.io/cloud-builders/gsutil
    volumes:
      - name: 'mavenrepo'
        path: '/root/.m2'
    args: ['cp', '-r', 'gs://com-mydeveloperplanet-mavenrepo', '/root/.m2']
  # Run the Maven build
  - name: maven:3.6.0-jdk-11-slim
    volumes:
      - name: 'mavenrepo'
        path: '/root/.m2'
    entrypoint: 'mvn'
    args: ['clean', 'install', '-DskipTests']
  # Preserve the Maven dependencies into the bucket
  - name: gcr.io/cloud-builders/gsutil
    volumes:
      - name: 'mavenrepo'
        path: '/root/.m2'
    args: ['cp', '-r', '/root/.m2', 'gs://com-mydeveloperplanet-mavenrepo']

Running this build the first time returns the following error:

Starting Step #0
Step #0: Already have image (with digest): gcr.io/cloud-builders/gsutil
Step #0: CommandException: No URLs matched: gs://com-mydeveloperplanet-mavenrepo
Finished Step #0
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/gsutil" failed: exit status 1

This seems to be an existing issue with the gsutil command. Adding a random file into the bucket solves the problem. After the build, the complete Maven repository needed for our build is located in the bucket. Initially, the first build step did not need to copy anything. The next builds we executed always failed due to a build timeout. When analyzing the build step time a bit closer, we notice that copying the Maven repository to the bucket and vice versa takes more than 6 minutes. To conclude with, the build with downloading the Maven dependencies takes approximately 54 seconds, the build with caching the Maven repository between builds will take approximately 13 minutes. Not really an increase in speed. This isn’t a good solution.

6.2 Custom Docker Maven Image

Another solution for caching Maven dependencies is to create a custom Docker image based on the Maven Docker image but extended with the Maven dependencies. We create a dockerfile named Dockerfile-maven in order to create the custom Docker image.

FROM maven:3.6.0-jdk-11-slim as target
WORKDIR /build
COPY pom.xml .
RUN mvn dependency:go-offline

Go to Cloud Shell and clone the git repository:

$ git clone https://github.com/mydeveloperplanet/mygcpcloudbuildplanet.git

Navigate to the directory mygcpcloudbuildplanet and build the Docker image:

$ docker build -f Dockerfile-maven -t gcr.io/mygcpcloudbuild/mymavengcpcloudbuild .

List the Docker images and verify the presence of the Docker image:

$ docker images
REPOSITORY                                    TAG                  IMAGE ID        CREATED           SIZE
gcr.io/mygcpcloudbuild/mymavengcpcloudbuild   latest               8371e285038d    17 seconds ago    572MB
maven                                         3.6.0-jdk-11-slim    c7428be691f8    3 weeks ago       489MB

Now push the Docker image to the Container Registry:

$ docker push gcr.io/mygcpcloudbuild/mymavengcpcloudbuild

Verify the presence of the Docker image in the Container Registry.

Cloud build - custom maven docker

Change the cloudbuild.yaml file in order that the Maven build step uses the custom Docker image instead of the official Maven Docker image:

steps:
  # Run the Maven build
  - name: gcr.io/mygcpcloudbuild/mymavengcpcloudbuild
    entrypoint: 'mvn'
    args: ['clean', 'install', '-DskipTests']
  - name: 'gcr.io/cloud-builders/docker'
    args: ['build', '-t', 'gcr.io/$PROJECT_ID/mygcpcloudbuildplanet', '.']
images: ['gcr.io/$PROJECT_ID/mygcpcloudbuildplanet']

It is important that you adapt the cloudbuild.yaml in the master branch, because this is the build configuration which is being used.

Let’s take a look at the results. A build with the official Maven Docker image takes 1 minute and 8 seconds including downloading the Maven dependencies and creating and pushing the Docker image to the Container Registry. A build with the custom Maven Docker image takes 47 seconds. A difference of approximately 15 to 20 seconds for a simple project like ours. It is definitely beneficial to use a custom Maven Docker image for your builds.

7. Conclusion

GCP Cloud Build is a very easy way to set up a build pipeline in just a few minutes. You can make use of provided cloud builders, but it is also easy to create custom ones. It is advised to use a custom cloud builder for Maven builds in order to speed up build time.

4 thoughts on “Automatic Builds at Your Fingertips With GCP Cloud Build”

  1. Hi,

    I am trying to setup trigger for my Gradle build for Java project. Problem I have is our .git is a level higher than my gradle project folder. e.g. ~/projects/mytest/mytest all the src, build other directories are in ~/projects/mytest/mytest and .git for Google cloud repository mytest are in ~/projects/mytest. I am trying to use nebula release plugin 11.1.0 which allows you to set git.root if different from gradle project directory. I am trying to set up in my cloudbuild.yaml

    name: ‘gcr.io/cloud-builders/gradle’
    args: [‘build -Pgit.goot=/workspace/$REPO_NAME’]
    dir: ‘$REPO_NAME’

    I am assuming that .git is copied in /workspace a level higher than gradle project name which mytest set in $REPO_NAME. It doesn’t work. I see on Cloudbuild documentation that it make a shallow copy of the project being build. Is there a way I can resolve this issue?

    Like

    1. I cannot give you a direct answer. I would start with analyzing the problem. You are assuming quite some things. Let’s get rid of the assumptions. Run the Docker image separately and verify by means of CLI what the contents of the Docker container is. From there on, check the cloned git repository, log files, etc

      Like

  2. My Spring Boot project has a reference of a JAR created from another Spring Boot Project. In Local Development machine, it is able to resolve the dependency defined in POM.xml but results in a compilation error while trying to deploy using cloud builds in GCP. Any clue?

    Like

    1. It depends on what the exact compilation error is. Is the jar of the other Spring Boot Project also build on GCP? If not, then probably the jar is not present in the Maven repository on GCP. In your local development machine, it will be available because you probably built it also on your local machine and therefore the jar will be available in your local Maven repository. Hope this helps.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.