Liz Douglass

Posts Tagged ‘Maven

Migrating from Maven to Simple Build Tool

leave a comment »

A while ago I moved our Scala project build from Maven to Simple Built Tool (sbt).

Why sbt?

  • sbt is made for Scala projects. The buildfile is written in Scala and is as concise as the Buildr ones that I have worked with previously.
  • sbt has several project types including the basic and web project types. Each has multiple build tasks/actions defined and all of these can be customised.
  • sbt has support for dependencies to be declared in either Ivy or Maven configuration files. All our dependencies were already specified in pom files, so not having to migrate these (at least straight away) made the transition to sbt easier.
  • sbt compiles fast. Robert first added sbt into our project so that he could compile the code more quickly than is possible in Intellij.
  • sbt has support for ScalaTest – the framework that we use for all our unit and integration tests. When we were running our ScalaTest tests as part of our Maven build we found that we needed to include the word ‘Test’ somewhere in the class name. Forgetting the requirement had cost one of our project team members several hours on one occasion.
  • We can now run both our webapp and integration tests at the same time. We’d found that it wasn’t possible to configure Maven to do this. We had instead started up a Jetty server from within our integration test project in order to run the web project.
  • sbt promotes faster development. Continuous compilation, testing and redeployment means that our work cycle is faster. This is particularly noticeable when we are working on a feature that requires us to make changes in both our Django front end and Scala backend. We can make changes in the source code in both and have both projects automatically redeploy.

Creating the sbt buildfile:

At the time of the migration to sbt our Scala backend was divided into 3 subprojects:

  • Core : A Scala project with ScalaTest unit tests
  • IntegrationTests – ScalaTest integration tests
  • Webapp : Basic webapp

Part 1: Declare all the sub-projects in the buildfile:

The first step in creating our build file was to declare all three subprojects. The core and integration subprojects are sbt DefaultProjects and have tasks such as compile and test defined. The webapp is an sbt WebappProject and has additional tasks such as jetty-run. Both the webapp and integration projects depend on the core project:

lazy val core = project("my-api-core", "Core", info => new DefaultProject(info))
lazy val webapp = project(webappProjectPath, "Webapp", info => new WebappProject(info), core)
lazy val integration = project("integration", "IntegrationTests", info => new DefaultProject(info), core)

Part 2: Getting the webapp running from sbt:

Although the webapp was declared as a Webapp project, it wasn’t possible to run it without declaring some additional Jetty dependencies. These were specified as inline sbt dependencies in the WebappProject class. This class extends from the sbt DefaultWebProject (please see below). Note that the port and context path can also be specified.

class WebappProject(info: ProjectInfo, port: Int) extends DefaultWebProject(info) {

    val jetty7Webapp = "org.eclipse.jetty" % "jetty-webapp" % "7.0.2.RC0" % "test"
    val jetty7Server = "org.eclipse.jetty" % "jetty-server" % "7.0.2.RC0" % "test"

    override def jettyPort = 8069
    override def jettyContextPath = "/my-api"
}

Part 3: Getting the integration tests running from sbt:

As mentioned above we need to run our webapp project at the same time as our integration test project. This is so that our integration tests can make calls to the webapp project endpoints. In addition to this our integration tests need to have a MongoDB database populated with some test data.

1) Populating the Mongo database for testing:

All our integration test classes extend from a common trait (below). This trait populates the MongoDB database at the start of each test:

trait JsonIntegrationTest extends FeatureSpec with BeforeAndAfterEach with ShouldMatchers {
    implicit val formats = new Formats {
        val dateFormat = DefaultFormats.lossless.dateFormat
    }

    override def beforeEach = {
        IntegrationDB.init
        IntegrationDB.eval(suiteData)
    }
}

The init method populates the database. The name of the MongoDB database is taken from a configuration file:

object IntegrationDB extends Assertions {
    private val config = new ConfigurationFactory getConfiguration("my-api")
    val dbName = config.getStringProperty("mongo.db")
    val db = new Mongo().getDB(dbName)

    def init = {
        val testDataFile: java.lang.String = "mongodb/IntegrationBaseData.js"
        val testDataFileInputStream = getClass.getClassLoader.getResourceAsStream(testDataFile)
        if (testDataFileInputStream != null) {
            eval(Source.fromInputStream(testDataFileInputStream).mkString)
        } else {
        fail("a message")
        }
    }

    def eval(js: String) = {
        db.eval(js)
    }
}

We are using The Guardian Configuration project to manage the config of the all of our Scala sub-projects. One of the features of this library is that it enables you to read properties from a number of sources. All of the properties for our projects are Service Domain Properties, meaning that they will be loaded from files on the classpath. The Configuration project library loads the properties from whichever correctly named file it encounters first on the classpath.

As mentioned above, our IntegrationTests project has a dependency on the Core project and therefore the config files of both projects will appear in the classpath. Both projects had a properties file with the same name that specified the mongo.db property. The intention was for the integration test properties to override those in the Core project. This did not work as planned because the ordering of the two config files in the classpath could not be guaranteed. Sbt does allow you to add items additional items to the classpath of your project using the +++ method. I did try to promote the integration test properties using this method (below). Unfortunately this did not guarantee classpath order either.

class IntegrationProject(info: ProjectInfo) extends DefaultProject(info) {
    val pathFinder: PathFinder = Path.lazyPathFinder(mainResourcesPath :: Nil)
    override def testClasspath = pathFinder +++ super.testClasspath
}

The work around was to set a system property called int.service.domain. This is used by the Guardian Configuration project to define the name of the properties file that that should be loaded from the classpath. Our integration test project now has a properties file with a different name to the one in the core project. The test action in the IntegrationTests project calls a method to switch to the integration test properties before the tests are run:

Now in the integration tests:

val useIntegrationConfig = true;
override def testAction = super.testAction dependsOn (task {setSystemProperty(useIntegrationConfig); None})

private def setSystemProperty(integrationConfiguration: Boolean) = {
    if (integrationConfiguration)System.setProperty("int.service.domain", integrationTestConfigDomain)
    else System.setProperty("int.service.domain", defaultConfigDomain)
}

2) Starting the main web application from the Integration test project:

As I mentioned above we’d found it hadn’t been possible to start up the webapp as well as run the integration tests using Maven. With sbt it is possible to do this. Another webapp project has was declared inside the definition of the IntegrationTests project. This sbt project has a separate output path and jetty port to the main webapp. This enables us to keep the main webapp running and run the integration tests at the same time.

class IntegrationProject(info: ProjectInfo) extends DefaultProject(info) {
    val useIntegrationConfig = true;

    lazy val localJettyPort = 8071
    lazy val localWebappOutputPath: Path = "target" / "localWebappTarget"
    lazy val localWebapp = project(webappProjectPath, "IntegrationTestWebapp",
        info => new WebappProject(info, localJettyPort, localWebappOutputPath, useIntegrationConfig), core)

    override def testAction = super.testAction dependsOn (task {setSystemProperty(useIntegrationConfig); None}, localWebapp.jettyRestart)
    lazy val startLocalWebapp = localWebapp.jettyRestart
    lazy val stopLocalWebapp = localWebapp.jettyStop
}

class WebappProject(info: ProjectInfo, port: Int, targetOutputDir: Path, useIntegrationTestConfiguration: Boolean) extends DefaultWebProject(info) {
    override def outputPath = if (targetOutputDir != "default") targetOutputDir else super.outputPath

    val jetty7Webapp = "org.eclipse.jetty" % "jetty-webapp" % "7.0.2.RC0" % "test"
    val jetty7Server = "org.eclipse.jetty" % "jetty-server" % "7.0.2.RC0" % "test"

    override def jettyPort = port
    override def jettyContextPath = "/my-api"
    override def jettyRunAction = super.jettyRunAction dependsOn (task {setSystemProperty(useIntegrationTestConfiguration); None})
}

Note that the testAction for the IntegrationTests project starts up the webapp but does not shut it down after the tests have finished. I did try several techniques for getting this to happen including the one below.  This attempt started jetty, stopped jetty then tried to run the tests. Please let me know if you have any better ideas 🙂

</span></span>
<pre>def startJettyAndTestAction = super.testAction dependsOn (localWebapp.jettyRun)
override def testAction = task{None} dependsOn (startJettyAndTestAction, localWebapp.jettyStop)

Written by lizdouglass

November 1, 2010 at 9:47 pm

Posted in Uncategorized

Tagged with , ,