Imagine a rectangular grid of cells, in which each cell has a value – Either black (dead) or white (alive). And imagine that:
- Any live cell with two or three live neighbors survives for the next generation.
- Any cell with four or more neighbors dies from overpopulation.
- Any cell with one or no neighbors dies from isolation.
- Any dead cell with exactly three neighbors comes to life.
These are the four simple rules of Conway’s Game of Life . You could hardly imagine a simpler set of rules to code on your computer and you wouldn’t expect any interesting result at all, but…
Behold the wonders of its hidden might!
The Conway’s Game of Life is an example of how a simple set of rules can produce amazing and totally unexpected results.
In the previous post, we agreed that the rules governing Scala’s implicit context weren’t complex at all, all of it can be summed up as: Mark some parameters as special, let the compiler know there are some values and some functions it can use when it is struggling with your code compilation… and hope it can manage to fill the gaps.
No matter how simple the implicit context rules are, they are definitely more complex than Conway’s. And, as in his little game of life, they bring complex results that, sometimes, make Scala newcomers sweat. However, under the developer’s control, they can become the glue that holds the most important Scala patterns together – the dark matter behind its visible universe. In this article we’ll walk through some examples of the marvels implicits can lead to.
Scala’s Dark Matter
Whereas it is theorised that some non-visible matter is responsible for helping maintain the galaxies together, we know for sure that some non-straightforwardly-visible values and functions hold many Scala constructions together. Let’s see some of these constructions.
Implicits application 101: The hidden context
Imagine you’re enjoying a session of peer programming. Your mate has the keyboard while you keep your eyes wide open, eager to suggest changes and solutions. Wouldn’t you find it stupid to repeat your friend’s name every time you address her/him? It is implicit that you are talking to her/him and not to another person who may be walking around!
Why should it be any different in a block of code which repeatedly interacts with a service?!
If you come from Java you might be thinking: “What are you, dude, some kind of slacker?”. Well, you also keep writing “;”s which, let’s be honest, are ignored by everybody’s brain… Lets give a concrete example from Oracle’s documentation on Java Futures:
ExecutorService executor = ...
Future<String> futureA = executor.submit(new Callable<String>() {
public String call() {
...
}});
Future<String> futureB = executor.submit(new Callable<String>() {
public String call() {
...
}});
It’s like saying: “Hey you! Executor, do this“, “Hey you! Executor, do this other thing”. It would be more comfortable, by far, to just agree to use a concrete executor at the beginning of a code block, thus avoiding writing its identifier over and over again. Take a look at fairly equivalent code written in Scala:
implicit val executor: ExecutionContextExecutor = ...
val futureA = Future[String] { ... }
val futureB = Future[String] { ... }
Now take a closer (unfolded) look at the creation of the first future: val futureA = Future[String] { … } . This is calling for Future objects to apply factory method…
val futureA = Future.apply[String] { ... }
…with the following method signature indicating it can get its executor parameter from an implicit value:
def apply[T](body: ⇒ T)(implicit executor: ExecutionContext): Future[T]
Sound familiar? apply’s body parameter is to Java’s Executor#execute command parameter what the executor implicit parameter is to the Executor instance repeatedly and explicitly referred to in Java.
This pattern easily comes up when developing a context dependent library and it isn’t difficult at all to develop a quick example.
You are developing a platform game in which your open source hero, Tux, moves around a stage, dodging and killing enemies, collecting rewards…
Your task is to write a comfortable collision detection module whereby its users can easily determine how the game reacts to different positions of our hero, at concrete times given a scenario. A first approach could be something like…
case class Stage(...) case class TuxState(pos: (Int, Int), points: Int, lives: Int) ... def newTuxState(tuxState: TuxState, t: Time, stage: Stage): TuxState = ???
and it could be used to run the game loop step by step as follows …
def calculateNextStage(implicit stage: Stage): Stage = ???
def runGameFrame(t: Time, prevTuxSt: TuxState, stage: Stage): GameStats = {
doPaintStage(t, stage) //Some side...
doPaintTuxSprite(prevTuxSt) //... effects.
...
tuxSt = newTuxState(prevTuxSt, t, stage)
nextStage = if(goalReached(tuxSt, stage)) calculateNextStage(stage) else stage
...
if(gameOver) stats
else runGameFrame(t + (1 turn), tuxSt, nextStage)
}
Isn’t stage a context in the game solution? It doesn’t vary over a game-loop step (runGameFrame) and is constantly and explicitly used by many of the operations performed. By noticing this and making it an implicit parameter for these operations we get a much clearer and concise code:
...
def newTuxState(tuxState: TuxState, t: Time)(implicit stage: Stage): TuxState = ???
def calculateNextStage(implicit stage: Stage): Stage
...
def runGameFrame(t: Time, prevTuxSt: TuxState)(implicit stage: Stage): (TuxState, Stage) = {
doPaintStage(t) //Some side...
doPaintTuxSprite(prevTuxSt) //... effects.
...
tuxSt = newTuxState(prevTuxSt, t)
...
if(gameOver) stats
else runGameFrame(t + (1 turn), tuxSt) {
if(goalReached(tuxSt)) calculateNextStage else stage
}
}
Types, tourists at run-time
Well, gluing contexts to methods, functions or constructor calls is cool, saves time and keystrokes and those are great advantages of a language for which developers aren’t paid by the written byte.
Now is time to meet a pattern providing something you wouldn’t get by other means: Run-time types awareness.
Types are citizens of the compilation time, they determine how – or if – your program will compile. However, without the help of the goddess of reflection, the only way to make decisions upon value types at run-time is through pattern matching:
val obj: Any = 1
obj match {
case _: Int => println("It's an integer!");
case _ => println("I don't know what is it, but is not an Integer.")
}
However, the underlying mechanism behind type matching is nothing else than JVM’s instanceof operator which is constrained by JVM’s type erasure. This means that all types generated by a type generator (a genetic type) are classified as instances of the same class. Have you ever tried to do something like the following?
val la = List(1,2,3)
val lb = List("Daisy", "Daisy", "give", "me", "your", "answer", "do")
val l = lb
l match {
case _: List[Int] => println("It's a list of integers")
case _: List[String] => println("It's a list of strings")
}
If you type the above code into Scala’s REPL it’ll yield: “It’s a list of integers” Ups!
The matter is that the above code is, in some way, like writing the following Java snippet where the parametric type has completely disappeared from the list type:
if(lb instanceof List) System.out.println("It's a list of integers")
else if(lb instanceof List) System.out.println("It's a list of strings")
As the first condition is true, we can understand why the pattern match isn’t working.
What a shame of a language if we are not able to make such a decision at run-time! … But don’t run from Scala just yet! It provides a way. It always does…
The way to do it, is by leveraging our dark matter: Implicit values. The Scala compiler automatically provides implicit values for any of the types generated by TypeTag[T] . If you don’t believe me, check for yourself:
import scala.reflect.runtime.universe._
implicitly[TypeTag[Int]]
// res0: reflect.runtime.universe.TypeTag[Int] = TypeTag[Int]
implicitly[TypeTag[List[Int]]]
// res1: reflect.runtime.universe.TypeTag[List[Int]] = TypeTag[scala.List[Int]]
implicitly[TypeTag[List[String]]]
// res2: reflect.runtime.universe.TypeTag[List[String]] = TypeTag[scala.List[String]]
// It's noticing the nuance behind the parametric type! Not as that silly "instanceof"...
case class RuntimeTourist[T]()
implicitly[TypeTag[RuntimeTourist[Int]]]
// res3: reflect.runtime.universe.TypeTag[RuntimeTourist[Int]] = TypeTag[RuntimeTourist[Int]]
// You know-it-all compiler! TAKE THIS!!
implicitly[TypeTag[RuntimeTourist[RuntimeTourist[RuntimeTourist[Int]]]]]
/* res4: reflect.runtime.universe.TypeTag[RuntimeTourist[RuntimeTourist[RuntimeTourist[Int]]]] =
TypeTag[RuntimeTourist[RuntimeTourist[RuntimeTourist[Int]]]]
This thing goes deep, and keeps track of nested parametric types.
*/
//... GOTCHA!!
implicitly[TypeTag[List[Map[List[Int], RuntimeTourist[List[String]]]]]]
/* res5: reflect.runtime.universe.TypeTag[List[Map[List[Int],RuntimeTourist[List[String]]]]] =
TypeTag[scala.List[Map[scala.List[Int],RuntimeTourist[scala.List[String]]]]]
*/
// It will work, no matter how complex the type parametrization is.
The point now is to use these labels to make decisions based on the types at run time. Remember the unsatisfactory attempt with pattern matching? Let’s try again. This time using the new cutting edge dark matter drive:
import scala.reflect.runtime.universe.{TypeTag, typeOf}
def whatIsIt[T](l: T)(implicit tag: TypeTag[T]): Unit = {
if(tag.tpe =:= typeOf[List[Int]]) println("It's a list of integers")
else if(tag.tpe =:= typeOf[List[String]]) println("It's a list of strings")
}
whatIsIt(la)
// It's a list of integers
scala> whatIsIt(lb)
// It's a list of strings
Yeah! OK, so far so good – but don’t think this is the only way, or even the best way to defeat type erasure. But this post is about implicit things.
I’d be more than happy to show you more about types at run time, but this is like a Star Wars film (the oldies) – if you want to know how the Death Star plans were stolen – chapter and verse – you’ll have to wait for Rogue One™…
However, there is something related to implicits and to implicits only, which make the above code a lot less verbose…
Implicit type bounds
Coming back to whatIsIt[T](l: T)(implicit tag: TypeTag[T]): Unit . Look at the second parameter list. Yes, the one for implicit parameters. It is worth noting that the code pattern for methods like someMethod[T](…)(implicit SomeParametricType[T]) is so common that the language provides a shortcut for it: someMethod(…)[T : SomeParametricType] and our whatIsIt could make good use of it:
def whatIsIt[T : TypeTag](l: T): Unit = {
val tag = implicitly[TypeTag[T]]
if(tag.tpe =:= typeOf[List[Int]]) println("It's a list of integers")
else if(tag.tpe =:= typeOf[List[String]]) println("It's a list of strings")
}
Don’t you love this language?!
To conclude this interlude, a last piece of advice: Don’t think that implicit type bounds are incompatible with other types of bounds. The following piece of code is not only correct, but also represents a common use of type bounds:
//(for a class C) def f[T <: C : TypeTag, S](x: T): S = ???
By the way, the implicit parameter for the TypeTag is known as the implicit evidence for T.
The magnetic attraction of the “classy type”
Have you ever tried to add functionality to a case class? Have you ever wanted to work with a third party class as if it was a subclass of one of yours?
Farm Management Ltd. has given us a great library with classes to represent animals in a farm management system. Among others, it contains the classes: Cat, Cow, Dog and Unicorn. All these classes implement the interface Animal…
Animal establishes that all its implementations should offer the operation takeCareOf(caretaker: PersonId, date: Date): ScheduledActivity which returns a ScheduledActivity value representing a programmed activity in the managed farm operations.
Our goal is to expand the farm management system so it can also manage activities related to equipment maintenance – such as fencing, fixing machinery, … we then provide classes representing “things to be maintained” which are not animals:
Equipment establishes the operation fix(worker: PersonId, date: Date): ScheduledActivity . So far so good… but now a problem arises: How will our management system be able to integrate animals to be taken care of with machines to be fixed in a common scheduler for all the farm activities? Remember, we CAN NOT modify the class Animal, that is: We CAN NOT create a common ancestor for Equipment and Animal to polimorphically work with their implementations’ instances.
An experienced Java developer would probably say something like “What’s the matter bro?! Wrap it up and carry on!” meaning: Just create a wrapper for the class provided by Farm Management Ltd.
trait Maintainable {
def scheduleMaintenance(worker: PersonId, date: Date): ScheduledActivity
}
trait Equipment extends Maintainable
case class Car() extends Equipment {
def scheduleMaintenance(worker: PersonId, date: Date): ScheduledActivity = ???
}
case class Fence() extends Equipment {
def scheduleMaintenance(worker: PersonId, date: Date): ScheduledActivity = ???
}
case class Tractor() extends Equipment {
def scheduleMaintenance(worker: PersonId, date: Date): ScheduledActivity = ???
}
case class AnimalWrapper(animal: Animal) extends Maintainable {
def scheduleMaintenance(worker: PersonId, date: Date): ScheduledActivity = animal.takeCareOf(worker, date)
}
This is great, but not so comfortable. All Animal instances need to be manually boxed within AnimalWrapper instances. That is, we need a kind of force to glue the farm element families together. e.g:
val animals = Cat()::Dog()::Unicorn()::Nil
val vehicles = Car()::Tractor()::Nil
val butchTheFarmer: PersonId = { ... }
val sarahTheMechanic: PersonId = { ... }
val scheduledTasks = (
animals map (animal => AnimalWrapper(animal).scheduleMaintainance(butchTheFarmer, tomorrow))
) ++ (
vehicles map (_.scheduleMaintainance(sarahTheMechanic, tomorrow))
)
In the above example, that force is the explicit boxing of animals (AnimalWrapper(animal) ). What an inconvenience to manually box every single Animal instance you want to integrate in your management system!
No! We are well into the 21st century. Besides, we agree that we have been kind of slack, don’t we? There must be a better way!
What we are actually searching for is a way to categorise types, or classes, after their declaration, some kind of delayed type classification. By using old school OOP inheritance we are classifying our types when they are defined, therefore the integration problems with third party entities. To overcome this kind of situation, it would be cool to have a mechanism which allows us to set classes of entities with a defined set of operations and to link types of instances to some classes whenever needed, not only at definition time. This idea is not new and such a mechanism exists for Haskell since 1990: Type Classes.
This is the way you could solve the problem in Haskell – Don’t run away just yet! I promise it will be quick and painless:
import AnimalFarmLtd.AnimalManagement -- Will bring animal types from third party library
data PersonId = ... -- This is PersonId type definition
data ScheduledActivity = ... -- And ScheduledActivity...
data Date = ...
data Car = Car { brand :: String, model :: String } -- This is the type of car values
data Tractor = Tractor { brand :: String, model :: String, horsePower: Int } -- likewise for tractors.
-- At this point we have already defined the types for cars, tractos and...
-- we've brought the types for each kind of animal. You could create values of those types:
let babeThePig = Pig "Babe" -- A pig called Babe
let fordMustang = Car "Ford" "Mustang"
-- We've values (like instances of animals an cars, but have not classified them)
class Maintainable a where
scheduleMaintainance :: a -> PersonId -> Date -> ScheduledActivity
-- Now, lets classify Car type as a Maintainable thing by providing an implementation for `scheduleMaintainance`
instance Maintainable Car where -- Here, the type Car gets classified as Maintainable
scheduleMaintainance carValue worker dateWhen = ...
instance Maintainable Animal where -- as Car, type Animal gets the `Maintenable` interface.
-- `scheduleMaintainance` is implemented as Animal's `takeCareOf` function
scheduleMaintainance = takeCareOf
-- Henceforth, you'll be able to use pass `babeThePig` to `scheduleMaintainance`.
You don’t need to understand the above code beyond the idea of having a way to say that some types can be associated with a certain interface even after definition. Remember:
Type classes are classes of types (classes of classes in Scala), not classes of instances
Despite Scala’s type system being closer to Java’s than to Haskell’s, we can use our magical dark matter – yes, our implicit values and conversions – to build Type Classes.
- Step 1, Untangling the classes hierarchy: The above classes graph is simplified since we will break the hard-coded composition relation with Animal interface.
trait Equipment { def fix(worker: PersonId, date: Date): ScheduledActivity } case class Car() extends Equipment { def fix(worker: PersonId, date: Date): ScheduledActivity = ??? } case class Fence() extends Equipment { def fix(worker: PersonId, date: Date): ScheduledActivity = ??? } case class Tractor() extends Equipment { def fix(worker: PersonId, date: Date): ScheduledActivity = ??? } - Step 2, Declaring the type class (Maintainance) and its interface as in
class Maintainable a where scheduleMaintainance :: a -> PersonId -> Date -> ScheduledActivity
Which, in Scala, is a trait:
trait Maintainable[T] { def scheduleMaintainance(x: T, worker: PersonId, date: Date): ScheduledActivity } - Step 3, Making classes part of the type class: In a delayed fashion of course…
implicit object InstanceMaintainableEquipment extends Maintainable[Equipment] { def scheduleMaintainance(x: Equipment, worker: PersonId, date: Date): ScheduledActivity = x.fix(worker, date) } //Henceforth, all Equipment instances are part of Maintainable type class implicit object InstanceMaintainableAnimal extends Maintainable[Animal] { def scheduleMaintainance(x: Animal, worker: PersonId, date: Date): ScheduledActivity = x.takeCareOf(worker, date) } //Henceforth, all Animal instances are part of Maintainable type class - Step 4, Making use of type class interface throught the implicit context:
def manageMaintenanceTasks[T : Maintainable](pending: T): ScheduledActivity = { ... ... implicitly[Maintenable[T]].scheduleMaintainance(pending, ...) }
And BOOM! There it is, a kind of Haskell type system embedded in Scala and you know… classes – of types – have always existed in Haskell.
However, there is something not so cool about these kind of type classes in which you need to pass the instance to the operations. It is possible to push the limits of OOP into this realm thus enabling the lovely infix notation we always feel comfortable with. Thus applying what has been called …
… which was first described on Spray’s development blog.
The latter article is so well written and so clearly explains the concept, that copying its ideas would be not only useless, but most probably misleading.
There is little differences between the magnet pattern and the type classes pattern as described above: Both rely on implicit values to link types to their class, that is to a set of operations. There is, however, a main difference which clearly sets both patterns apart:
- Type classes rely on implicit objects, that is: values of the type TypeClassClass[T] – Where T is the type to be associated to the type class – thus enabling the type class of frozen operations through an implicit evidence within the implicit context.Another example of type class is Scala is the Ordering[T] trait:
/** * This example os a modification of the one found at * Scala's API reference: * http://www.scala-lang.org/api/current/#scala.math.Ordering * **/ import scala.util.Sorting case class Person(name:String, age:Int) val people = Array(Person("bob", 30), Person("ann", 32), Person("carl", 19)) // sort by age: This'll be the actual implicit evidence for quickSort implicit object AgeOrdering extends Ordering[Person] { def compare(a:Person, b:Person) = a.age compare b.age } Sorting.quickSort(people) - With magnets, on the other hand, the operations are not fixed as they depend on value parametrization which leads to an object containing the implementation of the operation for that parametrization. This implies that magnets are not provided by implicit values accessed through implicit evidence but, otherwise, are implicit conversions from the operation parameters to the implementation. e.g:
// Magnet trait, base of all implementation objects trait MaintenableMagnet { def apply(worker: PersonId, date: Date): ScheduledActivity } // This is thre association of the type Equipment to the // magnet (or "cool type class") object MaintenableEquipementOps { implicit def scheduleMaintainance(x: Equipment): MaintenableMagnet = new MaintenableMagnet { def apply(worker: PersonId, date: Date): ScheduledActivity = x.fix(worker, date) } }An example of use:
import MaintenableEquipementOps._ val myCar = Car() def manageFarmElement(magnet: MaintenableMagnet): ScheduledActivity = magnet("Tom", "2016-10-16") manageFarmElement(c)
It is possible to fiddle with Haskell-like type classes and Magnet patterns in order to explore a vastly wide space of variations between them.
Other notorious Implicit Dimension citizens
This article is already “TLDR” so let’s briefly visit a couple of interesting implicit value/conversion cases.
Implicit classes & Implicit object
When we were getting to know the Type Classes nuts and bolts we saw that a singleton object can be tagged as implicit:
trait T
implicit object ObjectEvidence extends T {
...
}
It seems pretty obvious that it is just a regular singleton object tagged as implicit and, therefore, becomes also an implicit value to be used as an actual value of any call to a function with an implicit formal parameter of type T :
def f(implicit v: T): Unit = ??? f //Will receive `ObjectEvidence` as an actual parameter for `v`
The reasoning applied to understand what implicit objects are, can be extended to implicit classes. Whereas singleton object constructors do not require any parameter and can be built straightaway – without providing any parameter. There is a special type of class, the type of classes whose constructor receives exactly one parameter, than can be regarded as an implicit conversion from that parameter to instances of themselves (of the classes):
implicit class SpinSpinSugar(x: String) {
def doSomething(times: Int): Unit =
println("call on my...\n"*times)
}
"underground" doSomething 10
Convenient to write DSL, isn’t it?
Dangers and pitfalls
A regular matter universe is full of perils, it seems reasonable to think that playing with the mysteries of dark matter will bring problems of its own…
Multiple completions
When I read the words “error: ambiguous implicit values” I feel like Han Solo entering an asteroid field when trying to escape from the empire…
But there is not so much to fear. To solve this problem, at least in a quick and dirty way, is a piece of cake. It just means that a method is implicitly expecting a parameter of type, let’s say, T and the implicit context has more than one value of that type to fill the gap which leads to ambiguity… The compiler is politely telling us: Dear developer, please, can you be a little more specific?
The easiest solution to this problem is to explicitly provide an actual parameter of that type to the function call:
object ImplicitsA {
implicit val i: Int = 1
}
object ImplicitsB {
implicit val j: Int = 2
}
def f(implicit x: Int): Unit = ???
def main: Unit = {
import ImplicitsA._
import ImplicitsB._
//f Problematic call
f(ImplicitsA.i) // Dirty solution
}
But we are not despicable bounty bug hunters, are we? As Scala developers & software engineers we should strive for clean solutions and to find the root of the problem. In this case, as in most similar cases, the implicit context gets populated with implicit values of the same type from different imports. Just removing one of them would be the wiser approach.
By the way, tools and IDEs – such us IntelliJ (with its Scala Plugin) – can help locate were the implicit values come from. In the case of the mentioned IDE, selecting the calling expression, the one receiving ambiguous implicit actual parameters, and using the key shortcut CONTROL+SHIFT+P will show a floating window showing the part of the implicit context to be used by the expression and the source of its values.
It may happen that we can’t get rid of one of the imports because it may contain other useful implicit values, in such cases, the problematic values can be avoided by diverting their symbol import:
object ImplicitsA {
implicit val i: Int = 1
implicit val s: String = "HELLO DOLLY"
}
object ImplicitsB {
implicit val j: Int = 2
}
def f(implicit x: Int): Unit = ???
def g(implicit x: Int, y: String): Unit = println(x, y)
def main: Unit = {
import ImplicitsA.{i => _, _} //This will import everything from A except 'i'
import ImplicitsB._
g // j, from ImplicitsB and s from ImplicitsA will be passed
}
Recursive completions: What can be cooler than a run-time recursion Stack Overflow Error?
A recursion stack overflow error which you can’t see, of course!
To wrap this post up, let’s take a look at my favorite error in the Scala ecosystem. We have all come across the run-time stack overflow errors when defining wrong stop conditions – or none at all – def soFarSoGood: Long = 1 + soFarSoGood .
Well, there is a misuse of implicit completions which can lead to unexpected crashes under similar circumstances: Implicit auto-completion.
case class A() case class B() implicit def a2b(a: A): B = a val b: B = A() //BOOM!
Be aware of cross implicit conversions and…
Enjoy
