nosewheelie

Technology, mountain biking, politics & music.

Archive for the ‘Scala’ Category

Why Scala?

without comments

Fellow Graceless Failures poster Alex has posted his slides from C4 on why Twitter has chosen Scala to build their backend systems. Scalaz even got a look in! Nice one Alex.

Written by Tom Adams

September 8th, 2008 at 9:30 am

Posted in Scala

Tagged with

Handling nulls

without comments

I got some feedback to my Error handling with Either, or, Why Either rocks! post, basically saying it was incomprehensible.

So, here’s a nice introductory post on Option: Handling nulls. Should be a gentler introduction.

Written by Tom Adams

August 21st, 2008 at 11:42 am

Instinct 0.1.9 Release

without comments

I’m happy to announce the release of Instinct 0.1.9. This is a maintenance release that also allowed me to introduce some new features such as custom lifecycles. Other tidbits include multiple specification failures reported, Scala examples, additional Functional Java matchers (List, Either, Option), Maven bundle and a seperate source JAR.

Downloads are available from the project site.

Here’s the full list of changes:

  • Core Features
    • Custom specification lifecycles can now be plugged in using the @Context annotation. This would allow for example, a Spring application context to be initialised and wired into each specification method allowing integration-style specifications using a Spring beans.
    • Multiple errors are now displayed when a specification fails with more than one exception.
  • Expectation API
    • Added fj.data.List matchers: expect.that(List.nil().cons(1)).isEqualTo(List.nil().cons(1))
    • Added fj.data.Option matchers: expect.that(Some(1)).isSome(1), expect.that(foo.bar()).isNone().
    • Added fj.data.Either matchers: expect.that(error()).isLeft(runtimeException), expect.that(foo.bar()).isRight(42).
  • Scala integration
    • Added Scala stack example, showing the use of Instinct under Scala to spec out Scala code.
  • Maven integration
    • Added Maven bundle to allow uploading to central Maven repository.
  • Packaging
    • Moved source code out of main jar into instinct-<version>-sources.jar (for Maven).
    • Moved javadoc into jar instead of zip instinct-<version>-javadoc.jar (for Maven).
  • Infrastructure
    • Upgraded to Functional Java 2.10.
    • Downgraded (again) to CGLib 2.1.3 (for Maven integration).
  • Bugs
    • (Issue 21) Expected exception try-catch should only wrap specs, not before and after.
    • (Issue 19) Throwing exception out of mocked code confused expected exception handling.
    • (Issue 40) Formatting of multiple exception report could be nicer.

Written by Tom Adams

August 8th, 2008 at 4:10 pm

Posted in BDD,Instinct,Java,Scala

Error handling with Either, or, Why Either rocks!

with 3 comments

Instinct, like most xUnit-like frameworks provides the ability to run methods, and have the status of those methods reported. In xUnit frameworks these methods are called tests, in Instinct they’re called specifications.

Specifications are ordinary instance methods that are marked in a way (naming convention or annotation) that tells Instinct to run them. Each specification has a lifecycle associated with it, where both the creator of the method (the developer specifying code) and the framework itself performs pre- and post-specification steps (Instinct tries to take away and simplify a lot of the drudgery involved in traditional testing).

Specifications have the following lifecycle (the default implementation can be overridden):

  1. The context class for the specification is created.
  2. The mockery is reset, all mocks now contain no expectations.
  3. Specification actors are auto-wired.
  4. Before specification methods are run.
  5. The specification is run.
  6. After specification methods are run.
  7. Mock expectations are verified.

Any step of this lifecycle can throw exceptions causing the specification to fail. For example a before specification method may throw a NullPointerException, or a specification may pass while a mock used in it may not have an expectation met.

The framework needs flexibility in choosing which parts of the lifecycle to run, which parts are important when executing the specification, what failures constitute stopping the run of a specification, etc.

Here’s the code we’re starting with:

private SpecificationResult runSpecification(final SpecificationMethod specificationMethod) {
  final long startTime = clock.getCurrentTime();
  try {
    final Class<?> contextClass = specificationMethod.getContextClass();
    final Object instance = invokeConstructor(contextClass);
    runSpecificationLifecycle(instance, specificationMethod);
    return createSpecResult(specificationMethod, SPECIFICATION_SUCCESS, startTime);
  } catch (Throwable exceptionThrown) {
    final SpecificationRunStatus status = new SpecificationRunFailureStatus(exceptionSanitiser.sanitise(exceptionThrown));
    return createSpecResult(specificationMethod, status, startTime);
  }
}

private void run(final Object contextInstance, final SpecificationMethod specificationMethod) {
  Mocker.reset();
  actorAutoWirer.autoWireFields(contextInstance);
  try {
    runMethods(contextInstance, specificationMethod.getBeforeSpecificationMethods());
    runSpecificationMethod(contextInstance, specificationMethod);
  } finally {
    try {
      runMethods(contextInstance, specificationMethod.getAfterSpecificationMethods());
    } finally {
      Mocker.verify();
    }
  }
}

This implementation of of the specification runner is overly simplistic. It runs everything within a large try-catch block, which means there’s no way to tell which part of the specification failed (before, spec, after, etc.). It also cannot collect up errors, so if an error occurs in a specification and a mock fails to verify, only the verification error is propagated. These are currently two of the highest priority user reported issues on Instinct.

Here’s my first attempt at isolating which part of the specification failed, each of the constants passed to fail define the location of the failure.

public SpecificationResult run(final SpecificationMethod specificationMethod) {
  try {
    final Class<?> contextClass = specificationMethod.getContextClass();
    final Object instance = invokeConstructor(contextClass);
    Mocker.reset();
    try {
      actorAutoWirer.autoWireFields(instance);
      try {
        runMethods(instance, specificationMethod.getBeforeSpecificationMethods());
        try {
          runSpecificationMethod(instance, specificationMethod);
          return result(specificationMethod, SPECIFICATION_SUCCESS);
        } catch (Throwable t) {
          return fail(specificationMethod, t, SPECIFICATION);
        } finally {
          try {
            try {
              runMethods(instance, specificationMethod.getAfterSpecificationMethods());
            } catch (Throwable t) {
              return fail(specificationMethod, t, AFTER_SPECIFICATION);
            }
          } finally {
            try {
              Mocker.verify();
            } catch (Throwable t) {
              return fail(specificationMethod, t, MOCK_VERIFICATION);
            }
          }
        }
      } catch (Throwable t) {
        return fail(specificationMethod, t, BEFORE_SPECIFICATION);
      }
    } catch (Throwable t) {
      return fail(specificationMethod, t, AUTO_WIRING);
    }
  } catch (Throwable t) {
      return fail(specificationMethod, t, CLASS_INITIALISATION);
  }
}

Obviously, this is very ugly, it’s also hard to reason about. But, as we now have the location of the failure we can make decisions as to whether we fail the specification, or not, so we’ve solved our first issue. But we haven’t made our second task any easier, we aren’t generally able to keep processing (we still validate mocks in the above code upon specification failure) and we don’t collect all the errors that occur.

And at about this time enters Either (in Scala):

The Either type represents a value of one of two possible types (a disjoint union). The data constructors; Left and Right represent the two possible values. The Either type is often used as an alternative to Option where Left represents failure (by convention) and Right is akin to Some.

Either can be used in place of conventional exception handling in Java, or, to wrap APIs that use conventional exception handling (a more thorough treatment of this issue is given in Lazy Error Handling in Java, Part 3: Throwing Away Throws). Here’s an example of the latter, using both Either and Option (discussed later).

public Either<Throwable, List<Field>> wireActors(final Object contextInstance) {
  try {
    return right(actorAutoWirer.autoWireFields(contextInstance));
  } catch (Throwable t) {
    return left(t);
  }
}

...

public Option<Throwable> verifyMocks() {
  try {
    Mocker.verify();
    return none();
  } catch (Throwable t) {
    return some(t);
  }
}

At a high level, the good thing about using Either is that your methods no longer lie; they don’t declare that they’ll return an Int, or, maybe, they’ll throw an exception, they come right out and say it: I’ll return either an exception or an Int. This is akin to conventional checked exceptions in Java (which Scala does away with), where a checked exception is used to represent a recoverable failure (enforced by the compiler) and an unchecked exception to represent an unrecoverable failure (not compiler enforced). Scala takes the correct approach here, it uses unchecked exceptions to represent the bottom value in non-terminating functions, and Either to represent recoverable failure.

Either is also much more flexible than exceptions, you can map across it, convert it into an option, add them into a container, and generally treat them like any other data structure [1].

So armed with this new knowledge, here’s the new specification lifecycle broken out from the runner itself (note, there are eleven steps in the lifecycle, including validation, however only these are exposed).

interface SpecificationLifecycle {
  <T> Either<Throwable, ContextClass> createContext(Class<T> contextClass);
  Option<Throwable> resetMockery();
  Either<Throwable, List<Field>> wireActors(Object contextInstance);
  Option<Throwable> runBeforeSpecificationMethods(
      Object contextInstance, List<LifecycleMethod> beforeSpecificationMethods);
  Option<Throwable> runSpecification(
      Object contextInstance, SpecificationMethod specificationMethod);
  Option<Throwable> runAfterSpecificationMethods(
      Object contextInstance, List<LifecycleMethod> afterSpecificationMethods);
  Option<Throwable> verifyMocks();
}

Now we need to make use of this in the specification runner, one step of which is determining the overall result, from the sequence of steps. Here’s my first attempt at this, using Functional Java’s Either to represent the result of each of the steps.

public <T extends Throwable> Either<List<T>, SpecificationResult> determineLifecycleResult(
    final Either<T, Unit> createContextResult,
    final Either<T, Unit> restMockeryResult,
    final Either<T, Unit> wireActorsResult,
    final Either<T, Unit> runBeforeSpecificationMethodsResult,
    final Either<T, SpecificationResult> runSpecificationResult,
    final Either<T, Unit> runAfterSpecificationMethodsResult,
    final Either<T, Unit> verifyMocksResult) {
  List<T> errors = List.nil();
  if (createContextResult.isLeft()) {
    errors = errors.cons(createContextResult.left().value());
  }
  if (restMockeryResult.isLeft()) {
    errors = errors.cons(restMockeryResult.left().value());
  }
  if (wireActorsResult.isLeft()) {
    errors = errors.cons(wireActorsResult.left().value());
  }
  if (runBeforeSpecificationMethodsResult.isLeft()) {
    errors = errors.cons(runBeforeSpecificationMethodsResult.left().value());
  }
  if (runSpecificationResult.isLeft()) {
    errors = errors.cons(runSpecificationResult.left().value());
  }
  if (runAfterSpecificationMethodsResult.isLeft()) {
    errors = errors.cons(runAfterSpecificationMethodsResult.left().value());
  }
  if (verifyMocksResult.isLeft()) {
    errors = errors.cons(verifyMocksResult.left().value());
  }
  return errors.isNotEmpty() ? Either.<List<T>, SpecificationResult>left(errors)
      : Either.<List<T>, SpecificationResult>right(runSpecificationResult.right().value());
}

All those ifs are a bit ugly (what happens when we have more?), and we’ve got a mutable list, surely we can do better? We’ve spotted a pattern here, and we could clean this up by folding across a list of results, pulling out the left of each Either, however Either does this for us, using Either.lefts() (it performs the fold for you).

Here’s the next cut, making use of a list of results and Either.left():

public <T extends Throwable> Either<List<Unit>, Unit> determineLifecycleResult(
    final List<Either<T, Unit>> allResults, final Either<T, Unit> specificationResult) {
  final List<T> errors = lefts(allResults);
  return errors.isEmpty() ?
      Either.<List<T>, Unit>right(specificationResult.right().value()) :
      Either.<List<T>, Unit>left(errors);
}

So what’s this doing? It takes a list of results and goes through each of the lefts (the errors) returning them as a list. As Either is a disjunction (we’ll have an error or a result, but not both), if any of the results contain an error on the left, our list will be non-empty, meaning our specification failed to run. In this case we return the errors on the left. If we have no errors (i.e. the list is empty) we return the real result on the right.

This code can be simplified further by using Option instead of Either. Option would allow us to place any exception into the some data constructor, the Unit we’re placing into Either becomes the none (we’re used to thinking of void as nothing in Java anyway). The only hassle comes if we want to treat the Option as an Either (say in the lefts call above), in that case we’d need to lift the Option into an Either.

Option<Throwable> option = ...
Either<Throwable, Unit> either = option.toEither(unit()).swap();

Option also allows use to pull each some out of a list of Options, in a similar way to how we pulled the lefts out of a list of Eithers.

List<Option<Throwable>> results = ...
List<Throwable> errors = somes(results);
Option<Throwable> overall = errors.isEmpty() ?
    Option.<Throwable>none() :
    some((Throwable) new AggregatingException(errors));

Given that we’ve now decoupled the lifecycle from the runner and we know have a better way of handling errors, here’s the pattern of the new runner code:

private SpecificationResult runLifecycle(final long startTime,
    final SpecificationLifecycle lifecycle, final SpecificationMethod specificationMethod) {
  ...
  List<Option<Throwable>> lifecycleStepErrors = nil();
  final Either<Throwable, ContextClass> createContextResult =
      lifecycle.createContext(specificationMethod.getContextClass());
  lifecycleStepErrors = lifecycleStepErrors.cons(createContextResult.left().toOption());
  if (createContextResult.isLeft()) {
    return fail(...);
  } else {
    final ContextClass contextClass = createContextResult.right().value();
    ...
    lifecycleStepErrors = lifecycleStepErrors.cons(contextValidationResult.left().toOption());
    if (contextValidationResult.isSome()) {
      return fail(...);
    } else {
      ...
      lifecycleStepErrors = lifecycleStepErrors.cons(...left().toOption());
      if (...isSome()) {
        return fail(...);
      } else {
        ...
        if (...isSome()) {
          return fail(...);
        } else {
          ...
          if (...isSome()) {
            return fail(...);
          } else {
            ...
            if (...isSome()) {
              return fail(...);
            } else {
              ...
              return determineResult(..., lifecycleStepErrors);
            }
          }
        }
      }
    }
  }
}

See the pattern there? Let’s see it in slow motion. Assume each of the lifecycle results is called a, b, c, etc.

if (a.isLeft()) {
  return fail()
} else {
  if (b.isLeft()) {
    return fail()
  } else {
    if (c.isLeft() {
    } else {
      ...
    }
  }
}

What we’re doing is binding through each lifecycle result, if we get an error, we fail fast, if we don’t, we execute the next step. There’s some other muck going on here too, we’re destructively updating the list of errors (lifecycleStepErrors), and the last few steps (run the specification, run after methods, verify mocks) are always executed, regardless of whether any fail. So how do we clean the code up? We anonymously bind through Either on the right, and sequence through the rest accumulating errors. What???

Here’s a simple example that contains eleven steps representative of running a specification. For the first eight (a through h), each step’s predecessor must succeed (i.e. we have at most one error). For the last three (i through k), we execute all of them regardless of whether they fail and accumulate the errors. We make use of the new Validation class in Functional Java (in version 2.9) to perform the last three steps (full source; this example has been further refined in the trunk).

class X {
  // The first sequence of steps...
  Either<Throwable, Unit> a;
  Either<Throwable, Unit> b;
  Either<Throwable, Unit> c;
  Either<Throwable, Unit> d;
  Either<Throwable, Unit> e;
  Either<Throwable, Unit> f;
  Either<Throwable, Unit> g;
  Either<Throwable, Unit> h;
  // The second sequence of steps...
  Either<Throwable, Unit> i;
  Either<Throwable, Unit> j;
  Either<Throwable, Unit> k;

  // Execute the first sequence of steps, fail on the first error.
  Either<Throwable, Unit> t1() {
    return a.left()
        .sequence(b).right()
        .sequence(c).right()
        .sequence(d).right()
        .sequence(e).right()
        .sequence(f).right()
        .sequence(g).right()
        .sequence(h);
  }

  // Execute the second sequence of steps, accumulate the errors.
  Option<NonEmptyList<Throwable>> t2() {
    return validation(t1()).nel().accumulate(
        Semigroup.<Throwable>nonEmptyListSemigroup(),
        Validation.<Throwable, Unit>validation(g).nel(),
        Validation.<Throwable, Unit>validation(h).nel(),
        Validation.<Throwable, Unit>validation(i).nel());
  }
}

Each of the fields in the above represents the result of executing a step in the specification lifecycle (including validation, which is beyond the SpecificationLifecycle itself), t1 represents the first eight steps, t2 the last three steps. t1 sequences through (anonymous bind) the result of each step, failing if any individual step fails. t2 executes [2] each step, continuing execution of the remaining steps if any step fails, and accumulates the errors.

Remember that this is what t1 looked like originally:

if (a.isLeft()) {
  return fail()
} else {
  if (b.isLeft()) {
    return fail()
  } else {
    if (c.isLeft() {
    } else {
      ...
    }
  }
}

Some simpler examples may make the binding clearer; consider Scala’s Option (used here for brevity). We can bind through Option using orElse:

scala> Some(7).orElse(Some(8))
res0: Option[Int] = Some(7)

Here we execute Some(7), if that fails (i.e. returns none), we execute Some(8). As we see, the result is Some(7). Let’s take a failure case:

scala> None.orElse(Some(8))
res1: Option[Int] = Some(8)

We execute None, if that fails (i.e. returns none), which it does, we execute Some(8). As we see, the result is Some(8).

Taking it back to our simple Java example, we evaluate the result of step a [2], if it fails, we return the failure, if it succeeds, we evaluate step b, and so on. This is the same logic we saw in the nested if-else blocks earlier. If any of the first eight steps fail, we get back either one error (from t1), if any of the last three steps fail, we get back at most 3 errors (from t2)

If we apply this pattern to our specification runner code, we get the following:

private SpecificationResult runLifecycle(final long startTime, final SpecificationLifecycle lifecycle,
    final SpecificationMethod specificationMethod) {
  final Either<Throwable, ContextClass> createContext = lifecycle.createContext(specificationMethod.getContextClass());
  if (createContext.isLeft()) {
    return fail(startTime, specificationMethod, createContext.left().value(), false);
  } else {
    final ContextClass contextClass = createContext.right().value();
    final Either<Throwable, Unit> validation = validateSpecification(contextClass, specificationMethod);
    if (validation.isLeft()) {
      return fail(startTime, specificationMethod, validation.left().value(), false);
    } else {
      return runSpecification(startTime, lifecycle, contextClass, specificationMethod);
    }
  }
}

That looks bit better, but where’s the complexity gone? OK, here it is…

private SpecificationResult runSpecification(final long startTime, final SpecificationLifecycle lifecycle, final ContextClass contextClass,
    final SpecificationMethod specificationMethod) {
  final Object contextInstance = constructorInvoker.invokeNullaryConstructor(contextClass.getType());
  final Validation<Throwable, Unit> preSpecificationSteps =
      validate(resetMocks().f(lifecycle)).sequence(validation(wireActors().f(lifecycle, contextInstance)))
          .sequence(validate(befores().f(lifecycle, contextInstance, contextClass.getBeforeSpecificationMethods())));
  if (preSpecificationSteps.isFail()) {
    return fail(startTime, specificationMethod, preSpecificationSteps.fail(), Option.<Throwable>none());
  } else {
    final Option<Throwable> specification = specification().f(lifecycle, contextInstance, specificationMethod);
    final Option<NonEmptyList<Throwable>> result = preSpecificationSteps.nel().accumulate(throwables(), validate(specification).nel(),
        validate(afters().f(lifecycle, contextInstance, contextClass.getAfterSpecificationMethods())).nel(),
        validate(verifyMocks().f(lifecycle)).nel());
    if (result.isSome()) {
      return fail(startTime, specificationMethod, result.some().toList(), specification);
    } else {
      return success(startTime, specificationMethod);
    }
  }
}

Here’s the complete old and new versions of the code if you’re so inclined…

This code combined with the extracted lifecycle class is functionally equivalent to the first snippet of code I presented above. It may look verbose (it would be much simpler in Scala for example), but an interesting thing came out of it; it made explicit a bunch of places where I wasn’t handling exceptions correctly. It forced me to make a decision as to what to do in each case, so I got a much finer grained exception handling mechanism. Of course, I could get the same using try-catch (arguably more verbose), and I can choose to ignore left results (errors) if I want. The other thing it highlights is Java’s woeful generics implementation [3].

When I started down this path to error handling I had two objectives (two reported issues to resolve); to allow the runner of a specification to know which parts failed (this gives the flexibility to allow before specs to error and not be reported as expected exceptions) and to return all the errors resulting from running a specification. I didn’t initially intend to go down this path, however after talking in the office, decided that there was a better way to handle this than nested try-catch blocks. The resulting code is smaller (even using Java), simpler and much more flexible than the traditional Java method of exception handling. A win all round. There are some downsides however, firstly, the verbosity of Java’s typing leads to a mess of Either<Throwable, Unit> and this method of error handling will be foreign to a lot of Java developers today.

Epilogue

The concept of sequencing while accumulating errors has been generalised in Functional Java (from version 2.9) as validation, here is an example of it in action. Scalaz contains a similar concept, though this uses applicative functors over higher kinds (something which Java’s type system does not support), here’s a small example of its use.

Footnotes

  1. There’s no reason why you couldn’t catch an exception and wrap it up in a list, or your own data structure instead of using Either, but most of the work is already done for us (i.e. useful functions across Either have been defined) and it’s a useful convention.
  2. Java is strict, so we don’t get the benefit of lazy evaluation in this case, but could emulate it with a function.
  3. The problems I’ve encountered mainly have to do with covariance in type parameters and differences between the way methods returning types are treated vs. local variables (see the bottom of StandardSpecificationLifecycle for details).

Written by Tom Adams

August 6th, 2008 at 2:27 pm

Testing in Scala using a Java tool

without comments

My first post on Graceless Failures, a blog about adventures on the path to learning Scala:

Scala, like a lot of other languages these days, ships with a unit testing framework – SUnit – built in. Many other Scala specific “testing” frameworks have sprung up in recent times that contain similar or vastly different feature sets to the traditional xUnit tools. These include Reductio, ScalaCheck, Specs, ScalaTest, and SUnit (built into the Scala distribution).

And as Scala is “just Java” you can also use Java frameworks such as JUnit and TestNG. Having only used Reductio, I can’t vouch for any others, though ScalaTest is getting good airplay on Artima and Specs seems to have the Scala BDD mindshare.

These tools can be loosely categorised as traditional unit testing tools, ala xUnit, or automated specification testing tools, ala QuickCheck. Reductio and ScalaCheck are incarnations of automated specification testing, while Specs, ScalaTest and SUnit are more your traditional xUnit frameworks.

However, I’m not to write about any of these frameworks, instead, I’m going to write about Instinct, a Java BDD framework that I’ve been developing for around 18 months, and for which I’ve recently started to add specific support for Scala into the codebase. Good fodder for blog posts!

Continue reading Testing in Scala using a Java tool.

Written by Tom Adams

July 29th, 2008 at 9:43 am

Rails 10x more productive, Scala 2x. Really?

with 18 comments

Via Coderspiel comes a Quick RailsConf Update by Josh Susser.

I’m going to bite and take issue with this point that Josh makes:

I think this is the most likely threat to the Rails surplus, that C# or Scala or something can do a good enough job that people can double their productivity with far less of a change in mindset or tools, and eventually no one will care about the ten times (or whatever) productivity of Rails. “Good enough is good enough.”

Now I’ve not used C# much in anger (I have however seen some of the cool ideas pulled in from F#) so won’t comment on it, but I’ve now developed one large Rails app (multi-year, multi-person), one small Rails app and been peripherally involved in one medium & one small Scala projects (in addition to three Scala OSS projects), preceded by about 10 years of Java/C++/PHP projects.

So, I feel quite qualified to offer an opinion on the contention that Scala’s doubling of productivity is “good enough”. Too summarise this post early, this is just plain wrong [1], the productivity boost that is, not the “good enough” bit. Learning functional techniques will not only make you more productive, but at worst they’ll make you a better “Rails programmer”.

Note that I’m not going to address the “Rails vs. Scala” debate, which would be better rephrased as “Ruby vs. Scala”, or “Rails vs. Lift or a hypothetical Scala web app framework”.

Rails

I’m quite happy for people to claim the now standard “10x” productivity boost for new Rails apps, having been through a couple now, I can see how it gets you up and running quickly, mainly by taking a bunch of decision points away from you (that you’d have to make in a Java project for example, what O/R tool, what directories do I need, etc.) and how it simplifies a lot of the drudgery found in most web projects. Throw in some clunky higher order functions for good measure and you’re away!

Actually, I’m not entirely happy with accepting this, I’ve found that in my experience you save about 25% of total development time for small projects, but the curve flattens out the longer the project goes on (say, after a few months), if you add more people, if the project is complex or it’s not a webapp. If any of these get large, you loose the advantage of Rails. Ruby’s lack of a static type system also leads to issues, however I don’t want to address these in this post.

So I have no real problem with Rails for simple web projects; Rails is optimised for the general case, and when you stick to what it’s good for Rails is fine (ignoring the thorny issues of side-effects & composability). What I do have a problem with is claims that developers would want to settle for Scala, or, any (pure or not) functional language.

Scala

I’m not a functional programming guru, however I know enough to be dangerous and I like to improve the way I develop software. There are many compelling reasons why people love functional languages. They are succinct, elegant and composable. I’ve heard “functional programmers” [2] make claims that they can employ high-level abstractions (with funky names like monad & functor) to get massive increases in productivity (usually one to two orders of magnitude). What’s more, they can back up these claims also!

I can only speak first hand about the tools that I’ve used, but I can certainly see how people can make these claims. In what little Scala I’ve written, I’ve seen how you can drastically reduce the size of your code, increase its readability and improve your ability to reason about what’s going on [3]. And this has helped me a lot in my current Rails project!

I’ll say it in different terms, knowing functional programming techniques means you can write Rails apps faster (yes, better than 10x!) [4]. I’ve experienced this first hand on my current project. I’ve written code that would have taken me half a day in Java in 10 minutes, made possibly by only a miniscule knowledge of functional techniques (e.g. map & fold). There’s another post on countering claims that no-one ever uses these abstractions… I’ve used two folds and a map already this morning.

Also, once you learn the basics, with functional languages you also get a better handle on what it is you are actually doing as the language of the problem is the language you’re coding in. You stop thinking in terms of the machine, and more in terms of the problem you’re trying to solve. For example consider this piece of code from Furnace:

removeHeader(stream).filter(!newLines.contains(_)).take(40)

What does it do? Well it removes the header from a stream of bytes, filters out newlines and takes the first 40 bytes. Does the code look similar to the way I’ve described it?

When I first tried to write this code, I thought about the underlying stream of bytes, pulling them off one by one, what about buffering, what happens if I read too many and need to skip back, etc. My thoughts had been warped by years of imperative thinking. When I discussed this problem in the office, I realised that this can be represented simply, if I simply though about what it was I wanted to do, and the code came very easily after that [5].

Summary

Of course there are tradeoffs here, and learning function techniques is mind-bending (in a good way), you’ll also realise how little you actually know. By using a language like Scala you get to have your cake (e.g. great APIs, type inferencing, DSLs, etc.) and eat it too (static typing). You also get great productivity improvements. Am I willing to put a number on it, no, I don’t have enough experience yet, but I can see it has the potential to be large.

This has been a bit of a rambling post, but my central premise is these two things; 1) Using functional techniques will let you be more productive in the general case (i.e. not just web apps) than Rails makes you for writing web apps; and 2) learning functional techniques will make you a better “Rails programmer”.

Footnotes

[1] To be fair, Josh is probably just throwing out the names of the latest languages that are causing a buzz, so Scala gets lumped into that. Still, this doesn’t make the statement true.
[2] Functional programmers usually don’t like to use this term, but I’ll use it here for quick categorisation.
[3] You also get other nice things like improved testability, increased maintainability, flexibility, etc.
[4] I have come up against some of the barriers to making things even easier, for example using APIs that rely on side-effects and are not referentially transparent breaks your ability to compose functions, which at its least is really annoying, and its at worst has a huge impact on a project.
[5] This code is not without its faults. It builds the stack, but is composable. A solution using iterators won’t build the stack and is more efficient, but is not composable, leading to code that is harder to reason about.

Written by Tom Adams

June 27th, 2008 at 3:46 pm

Posted in Ruby,Scala

Easy DSLs in Scala

without comments

Debasish Ghosh has another excellent Scala post, this one on External DSLs made easy with Scala Parser Combinators:

External DSLs are hard since implementing them involves reinventing most of the mechanisms found in a general purpose language. Designing internal DSLs are equally hard, more so in a statically typed language. Dynamically typed languages like Ruby offer strong meta-programming facilities, which help in implementing internal DSLs. But metaprogramming in Ruby is still considered elitist by many, and is not an art mastered by programmers at large.

Here is a sample DSL (simplified for brevity) for accepting client orders to buy/sell equities ..

(buy 100 IBM shares at max USD 45, sell 50 CISCO shares at min USD 25, buy 100 Google shares at max USD 800) for trading_account "SSS1234"

The equivalent XML will be too verbose, too painful for the eyes, and will definitely need more extraneous infrastructure over native language support for meaningful processing.

Written by Tom Adams

April 22nd, 2008 at 3:30 pm

Posted in Scala

Tagged with , , , ,

Pimp my (Java) methods

with 7 comments

A friend of mine has been playing around with Scala and asked about its meta-programming ability. In particular, he’s a Ruby fiend, so is used to its nice reflection abilities. Specifically he wanted to be able to explore an API interactively from the console, like the following for an integer in Ruby:

>> 1.class
=> Fixnum
>> 1.methods
=> ["%", "inspect", "<<", "singleton_method_added", "&", "clone", ">>", "method", "round", "public_methods", "instance_variable_defined?", "divmod", "equal?", "freeze", "integer?", "chr", "*", "+", "to_i", "methods", "respond_to?", "-", "upto", "between?", "prec", "truncate", "/", "dup", "instance_variables", "__id__", "modulo", "object_id", "succ", "|", "eql?", "zero?", "require", "~", "id", "to_f", "singleton_methods", "send", "prec_i", "taint", "step", "to_int", "frozen?", "instance_variable_get", "__send__", "^", "instance_of?", "remainder", "to_a", "+@", "nonzero?", "-@", "type", "**", "floor", "<", "protected_methods", "<=>", "instance_eval", "==", "prec_f", "quo", ">", "display", "===", "downto", "id2name", "size", "instance_variable_set", "kind_of?", "abs", "extend", ">=", "next", "to_s", "<=", "coerce", "hash", "ceil", "class", "tainted?", "=~", "private_methods", "gem", "div", "nil?", "untaint", "times", "to_sym", "[]", "is_a?"]
>> Fixnum.methods
=> ["inspect", "private_class_method", "const_missing", "clone", "method", "public_methods", "public_instance_methods", "instance_variable_defined?", "method_defined?", "superclass", "equal?", "freeze", "included_modules", "const_get", "methods", "respond_to?", "module_eval", "class_variables", "dup", "protected_instance_methods", "instance_variables", "public_method_defined?", "__id__", "object_id", "eql?", "const_set", "require", "id", "singleton_methods", "send", "class_eval", "taint", "frozen?", "instance_variable_get", "include?", "private_instance_methods", "__send__", "instance_of?", "private_method_defined?", "to_a", "name", "autoload", "type", "<", "protected_methods", "instance_eval", "<=>", "==", ">", "display", "===", "instance_method", "instance_variable_set", "kind_of?", "extend", "protected_method_defined?", "const_defined?", ">=", "ancestors", "to_s", "<=", "public_class_method", "allocate", "hash", "class", "instance_methods", "tainted?", "=~", "private_methods", "gem", "class_variable_defined?", "induced_from", "nil?", "untaint", "constants", "autoload?", "is_a?"]

Unfortunately Scala’s meta-programming abilities are currently limited to Java’s, which is to say not very good. As I’m in Rails-land at the moment, I asked Tony to have a crack at it, to which, he’s obliged:

class Classs[T](c: Class[T]) {
  lazy val methods = scala.collection.immutable.HashSet(c.getDeclaredMethods: _*) ++ c.getMethods
}

object Classs {
  implicit def c[T](c: Class[T]) = new Classs(c)
}

Here’s how to use it, we ask for all of Integer‘s methods that start with a “t”:

scala> classOf[Integer].methods.filter(_.getName.startsWith("t")).foreach(println(_))
public static java.lang.String java.lang.Integer.toBinaryString(int)
public static java.lang.String java.lang.Integer.toHexString(int)
public static java.lang.String java.lang.Integer.toOctalString(int)
public java.lang.String java.lang.Integer.toString()
public static java.lang.String java.lang.Integer.toString(int,int)
public static java.lang.String java.lang.Integer.toString(int)
private static java.lang.String java.lang.Integer.toUnsignedString(int,int)

Tony’s original source: http://fprogramming.org/paste/viewpaste.php?id=107.

Written by Tom Adams

April 21st, 2008 at 12:40 pm

IntelliJ IDEA 8 Roadmap

with 4 comments

Via Talios, the IntelliJ 8 roadmap has been refined a little, notable inclusions are:

  • Scala (finally!)
  • Python
  • git???

I’d be nice to see a preview drop of the Scala plugin, it currently doesn’t build from source.

Written by Tom Adams

April 17th, 2008 at 8:36 am

Recent Scala links

without comments

Some recent Scala links that may be of interest to people:

Written by Tom Adams

March 25th, 2008 at 3:06 pm

Posted in Scala