In the first part of this series we discussed one of the toughest challenges we faced when developing software for an orchestration toolkit inside the Thirty Meter Telescope: developing a domain specific language that could handle the necessary complexity of the system without pushing it onto the scientists who would actually be using the toolkit.
We looked at our attempt to develop a DSL using Scala and why it caused us to run into some problems. In this post we’ll explain how we solved it with the help of Kotlin.
The second attempt: simplifying things with Kotlin
We had heard Kotlin was a language tailored for embedded DSLs. It had implicit receivers, top level statements in scripts (.kts file), DSL markers, and a default synchronous (awaiting) approach for programming using suspending functions and coroutines.
Coroutines are lightweight threads with suspending functions; they are mechanisms for async programming in Kotlin. Kotlin allows you to create lots of coroutines while simultaneously making programs look synchronous with the help of suspending functions, but execute this asynchronously on JVMThreads. This can be simply described as write sync, execute async.
Kotlin simplified scripts by eliminating the need for async and await statements, without losing the benefits of async programming. It also allowed users to explicitly code asynchronous behavior wherever required.
We migrated our DSL because we were impressed by Kotlin. But we migrated only enough to ensure our DSL would work while still using the script engine we had already implemented in Scala.
The script in the example above translated to the example below when using DSL implemented in Kotlin:
// TcsSync.kts
script {
val motorAssembly = Assembly(TCS, "motor1", 5.seconds)
val timeKey = taiTimeKey("time")
onSetup("observationPrep") { command ->
val executeAt = command(timeKey).head()
scheduleOnce(executeAt) {
val moveCommand = Setup(prefix, "move30Degrees", command.obsId)
motorAssembly.submit(moveCommand)
}
}
}
The DSL used the Kotlin coroutines and suspending functions to execute async operations. We used a coroutine scope with a single threaded executor and DSL support to ensure all the coroutines in the script were executed on the same coroutine scope. This approach made concurrent operations simpler and eliminated problems like race conditions and state corruption due to parallel programming.
In the example below each line is an async call, but during execution each call will be completed before executing the next line, giving it synchronous semantics.
motorAssembly1.submit(moveCommand)
publish(currentState())
motorAssembly2.submit(moveCommand)
publish(currentState())
Getting Scala and Kotlin to work together
We chose Kotlin with Scala to implement the DSL because both languages run on the JVM platform, which offers full interoperability with Java. We used Java as a bridge between Kotlin and Scala.
We had a CommandHandler trait (Interface of Java) in Scala, where all exposed types were Java types — for example, the CompletionStage and Void. CommandHandler encapsulates the logic to execute a specific command:
trait CommandHandler {
def handler(command: SetupCommand): CompletionStage[Void]
}
To execute these command handlers in Scala, we converted Java types into Scala types. So, CompletionFuture was mapped to Future, using functions provided by the Scala standard library `asScala,` and Void was mapped to unit type as shown in the handleSetupCommand method below:
class Script {
var setupHandler: CommandHandler = _
def addSetupHandler(handler: CommandHandler): Unit ={
setupHandler = handler
}
def handleSetupCommand(command: SetupCommand): Future[Unit] {
setupHandler.handler(command).asScala.map(_ => ())
}
}
On the Kotlin side, these Java types were converted to Kotlin types. Extending the example above, Kotlin’s `Deferred` was mapped to Java CompletableFuture using Kotlin utilities and passed to the addSetupHandler method:
//DslSupport.kt
class DslSupport {
val script = Script()
fun addSetupHandler(handler suspend Setup-> Void) = {
script.addSetupHandler(CommandHandler { command ->
val deferred: Deferred<Void> = CoroutineScope.async {handler(command)}
deffered.asCompletableFuture()
})
}
}
So, the Scala interface exposing the Java type was instantiated in Kotlin and passed back to Scala.
Implementation details
The diagram represents a simplified view of our dependency tree. The top module has the CommandHandler trait and Script class declared in Scala.
The DSL support module (on the left) instantiates the interface in Kotlin to create DSL.
The Execution Engine & App module (on the right) is an implementation of the Execution Engine, which executes scripts. The Execution Engine assumes script binaries will be available on the classpath while initiating execution. It loads the requested script using a Java reflection, as a Script interface instance and performs the necessary executions on it. The main class to run the Sequencer (Scripts and Execution Engine) is implemented in this module.
The catch here is the Scripts module (bottom most) depends on both modules. The left one provides DSL support to write scripts and the right one has the Execution Engine & App.
This Script module is where script writers can write scripts. It’s located in a separate repo, which ensures script writers won’t be exposed to complicated code. It gives them an isolated environment to write scripts.
The Sequencer application launches from the Script module, using the main class declared in the Execution Engine & App module, to make scripts available on the classpath while running the application.
A unique experiment
This was a unique experiment in which we were able to get Scala and Kotlin to interact with one another. It was a challenge because we weren’t able to find any other documented examples on the Internet — we had no point of reference or guide for what we were trying to do.
To take the project to its final stage, we developed our own models. For example, we wanted to build Scala and Kotlin modules in a shared monorepo build setup to simplify code modifications. This led us to develop a unique SBT build setup to support Scala and Kotlin in a monorepo setup, which is a story for another blog post! If you’re interested, you can read a little bit more about it here.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.
