name: image layout: true class: center, middle, image --- name: image-white layout: true class: center, middle, image, image-white --- name: image-last layout: true class: center, middle, image, image-last --- name: ambiata layout: true class: ambiata --- name: ambiata-full layout: true class: center, middle, ambiata-full --- name: code-small layout: true class: code-small --- name: question layout: true class: center, middle, question --- name: rule layout: true class: center, middle, rule --- name: boom layout: true class: boom --- name: inverse layout: true class: center, middle, inverse --- name: starwars layout: true class: center, middle, starwars --- class: center, middle template: image <img src="images/love-the-bomb.jpg" height="100%" /> # How I learned to stop unit testing and love property-based testing --- layout: false template: image <img src="images/chopsticks.jpg" /> # Charles O'Farrell --- template: ambiata ## Ambiata - Property-based testing is our default - We have a majority of property-based tests - Need a compelling argument to write unit tests ??? - "Hardline" --- template: starwars ## One year ago - Had heard good things about property-based testing - Had written one or two properties - But had to brace myself before writing one --- template: image <img src="images/jedi-master.jpg" height="100%" /> # Today ??? - Have found it to live up the hype - Surprising thing is that I now find it _easier_ to write properties - Counterintuitive --- template: ambiata ## https://github.com/ambiata/ivory - ~600 properties - ~30 non-property tests - Mostly error/sanity tests --- template: inverse ## What Have I Learned? --- layout: false ## Overview <ul> <li> Properties <ul> <li>Patterns</li> <li>Laws</li> </ul> </li> <li>Generators</li> <li>Bug hunting</li> </ul> --- <ul class="toc"> <li> Properties <ul> <li class="toc-selected">Patterns <ul> <li>Symmetry</li> <li>Model-based</li> <li>Invariants</li> </ul> </li> <li>Laws</li> </ul> </li> <li>Generators</li> <li>Bug hunting</li> </ul> <img src="images/patterns.jpg" style="position: absolute; width: 50%; top: 30px; right: 30px" /> --- <ul class="toc"> <li> Properties <ul> <li>Patterns <ul> <li class="toc-selected">Symmetry</li> <li>Model-based</li> <li>Invariants</li> </ul> </li> <li>Laws</li> </ul> </li> <li>Generators</li> <li>Bug hunting</li> </ul> <img src="images/property-inverse.png" style="position: absolute; top: 180px; right: 0px; width: 600px;" /> --- ## Show me the code ```scala def toBytes(s: String, c: Charset): Array[Byte] def fromBytes(b: Array[Bytes], c: Charset): String ``` --- ## Show me the code ```scala def toBytes(s: String, c: Charset): Array[Byte] def fromBytes(b: Array[Bytes], c: Charset): String forAll { (s: String, c: Charset) => val b = toBytes(s, c) fromBytes(b, c) == s } ``` --- template: boom ## Show me the code ```scala def toBytes(s: String, c: Charset): Array[Byte] def fromBytes(b: Array[Bytes], c: Charset): String forAll { (s: String, c: Charset) => val b = toBytes(s, c) fromBytes(b, c) == s } ``` <pre class="error"> "돪" != "?" > ARG_0: "돪" > ARG_1: windows-1252 </pre> --- ## JSON ```scala def toJson(user: User): Json def fromJson(json: Json): Either[Exception, User] ``` --- ## JSON ```scala def toJson(user: User): Json def fromJson(json: Json): Either[Exception, User] forAll { user: User => fromJson(toJson(user)) == Right(user) } ``` --- ## Database ```scala def insert(u: User): UserId def get(u: UserId): Option[User] ``` --- ## Database ```scala def insert(u: User): UserId def get(u: UserId): Option[User] forAll { u: User => val id = userDb.insert(u) userDb.get(id) == Some(u) } ``` --- template: boom ## Database ```scala def insert(u: User): UserId def get(u: UserId): Option[User] forAll { u: User => val id = userDb.insert(u) userDb.get(id) == Some(u) } ``` <pre class="error"> Some(User(\NULL)) != Some(User()) </pre> --- template: image <img src="images/golden-ticket.jpg" /> ## Symmetry is your golden ticket ??? - Actually a bad analogy - _Lots_ of examples/uses --- <ul class="toc"> <li> Properties <ul> <li>Patterns <ul> <li>Symmetry</li> <li class="toc-selected">Model-based</li> <li>Invariants</li> </ul> </li> <li>Laws</li> </ul> </li> <li>Generators</li> <li>Bug hunting</li> </ul> <img src="images/model-based.png" style="position: absolute; width: 550px; top: 200px; right: 0px;"/> --- ## Performance ```scala def sum(l: List[Int]): Int = { var v = 0 l.foreach ( i => v += i ) v } ``` --- ## Performance ```scala def sum(l: List[Int]): Int = { var v = 0 l.foreach ( i => v += i ) v } forAll { l: List[Int] => sum(l) == l.foldLeft(0)(_ + _) } ``` --- ## Sorting ```scala def timSort(l: List[Int]): List[Int] ``` --- ## Sorting ```scala def timSort(l: List[Int]): List[Int] forAll { l: List[Int] => timSort(l) == bubbleSort(l) } ``` --- template: ambiata ## Date ```scala case class Date(value: Int) def toJoda(date: Date): JodaDate def fromJoda(date: JodaDate): Date ``` --- template: ambiata ## Date ```scala case class Date(value: Int) def toJoda(date: Date): JodaDate def fromJoda(date: JodaDate): Date forAll { d: Date => d.toJoda.fromJoda == d } ``` --- ## Test Oracle ```scala def dayPlus(d: Date, i: Int): Date = { ... } ``` --- ## Test Oracle ```scala def dayPlus(d: Date, i: Int): Date = { ... } forAll { (d: Date, i: Int) => dayPlus(d, i) == d.toJoda.plusDays(i).fromJoda } ``` --- template: boom ## Test Oracle ```scala def dayPlus(d: Date, i: Int): Date = { ... } forAll { (d: Date, i: Int) => dayPlus(d, i) == d.toJoda.plusDays(i).fromJoda } ``` <pre class="error"> Date(2004,2,29) != Date(2004,3,1) ARG_0: Date(2004,2,28) ARG_1: 1 </pre> --- ## Database ```scala def listSortByName: List[User] = "SELECT * FROM user ORDER BY name ASC" ``` --- ## Database ```scala def listSortByName: List[User] forAll { users: List[User] => users.foreach(u => userDb.insert(u)) val l = userDb.listSortByName l == users.sortBy(_.name.toLowerCase) } ``` --- ## Database 2 ```scala def findByPostCode(postcode: Int): List[User] = "SELECT * FROM user WHERE postcode = ?" ``` --- ## Database 2 ```scala def findByPostCode(postcode: Int): List[User] forAll { (postcode: Int, users: List[User]) => users.foreach(u => userDb.insert(u)) val has = users.filter(_.postcode == postcode) userDb.findByPostCode(postcode) == has } ``` ??? - God forbid you're not using prepared statements this would find a bug pretty quick --- ## Database 2 ```scala def findByPostCode(postcode: Int): List[User] forAll { (postcode: Int, u1: List[User], u2: List[User]) => val has = u1.map(_.copy(postcode = postcode)) val without = u2.filter(_.postcode != postcode) (has ++ without).foreach(u => userDb.insert(u)) userDb.findByPostCode(postcode) == has } ``` --- template: ambiata ## Hadoop ```scala forAll { users: List[Users] => val out = hadoopJob(users) val expected = users.groupBy(_.name) .mapValues(_.sortBy(_.postcode)) out == expected } ``` ??? - Horrible Java APIs - Byte manipulation --- template: ambiata ## Hadoop ```scala forAll { users: List[Users] => val out = hadoopJob(users) val expected = users.groupBy(_.name) .mapValues(_.sortBy(_.postcode)) out == expected } ``` <h3 style="color: green"> Ivory: ~40 Properties </h3> --- <ul class="toc"> <li> Properties <ul> <li>Patterns <ul> <li>Symmetry</li> <li>Model-based</li> <li class="toc-selected">Invariants</li> </ul> </li> <li>Laws</li> </ul> </li> <li>Generators</li> <li>Bug hunting</li> </ul> --- ## Invariants ```scala forAll { l: List[Int] => l.map(a => a) == l } forAll { l: List[Int] => l.filter(_ => true) == l } forAll { l: List[Int] => l.filter(_ => false) == Nil } ``` ??? - Need to be combined with multiple properties to test the entire function --- ## User Invariants ```scala def tax(u: User): Tax ``` --- ## User Invariants ```scala def tax(u: User): Tax forAll { u: User => tax(u) == ??? } ``` ??? - This is _hard_ to model without rewriting --- ## User Invariants ```scala def tax(u: User): Tax forAll { (u: User, wage: Int) => wage < 10000 ==> tax(u.copy(wage = wage)) .bracket == 0% } ``` --- ## User Invariants ```scala def tax(u: User): Tax forAll { (u: User, wage: Int) => wage > 1000000 ==> tax(u.copy(wage = wage)) .bracket == 45% } ``` --- ## User Invariants ```scala def tax(u: User): Tax forAll { (u: User, h: PrivateHealth) => tax(u.copy(privateHealth = Some(h))) .surcharge == false } ``` --- template: ambiata ## Ivory <a href="https://github.com/ambiata/ivory/blob/master/ivory-storage/src/test/scala/com/ambiata/ivory/storage/plan/SnapshotPlanSpec.scala"> <img src="images/invariants.png" width="100%" /> </a> --- ## Idempotence ```scala forAll { s: List[Int] => l.distinct.distinct == l.distinct } forAll { l: List[Int] => l.sorted.sorted == l.sorted } ``` --- ## Database Idempotence ```scala forAll { u: User => val r1 = userDb.createOrUpdate(u) val r2 = userDb.createOrUpdate(u) r1 == r2 } ``` --- ## Database Idempotence ```scala forAll { m: Migration => db.migrate(m) val s1 = db.schema() db.migrate(m) val s2 = db.schema() s1 == s2 } ``` --- template: image <img src="images/everywhere2.jpg" height="120%" style="margin-top: -60px" /> ## The same patterns everywhere --- template: inverse ## Is it too slow to run 100 times? --- ## Configuration - Can increase/toggle runs - Separate build with different sizes - Without changing a single line of code --- template: inverse-toc <ul class="toc"> <li> Properties <ul> <li>Patterns</li> <li class="toc-selected">Laws</li> </ul> </li> <li>Generators</li> <li>Bug hunting</li> </ul> --- ## [Scalaz](https://github.com/scalaz/scalaz/blob/master/scalacheck-binding/src/main/scala/scalaz/scalacheck/ScalazProperties.scala) ```scala libraryDependencies += "org.scalaz" %% "scalaz-binding" % "7.1.2" property("user equal") = scalaz.scalacheck.equal.laws[User] property("set functor") scalaz.scalacheck.functor.laws[Set] property("db monad") scalaz.scalacheck.monad.laws[DB] ``` ??? - Most people have seen a Monad tutorial - Mentioned the laws at some point --- ## JSON ```scala def toJson(user: User): Json def fromJson(json: Json): Either[Exception, User] forAll { a: User => fromJson(toJson(a)) == Right(a) } ``` --- ## JSON ```scala def toJson[A: EncodeJson](user: A): Json def fromJson[A: DecodeJson](json: Json) forAll { a: User => fromJson(toJson(a)) == Right(a) } ``` --- ## JSON ```scala def toFromJSON [A: EncodeJson: DecodeJson: Arbitrary]: Prop = forAll { a: A => fromJson(toJson(a)) == Right(a) } ``` --- ## JSON ```scala def toFromJSON [A: EncodeJson: DecodeJson: Arbitrary]: Prop = forAll { a: A => fromJson(toJson(a)) == Right(a) } property("user") = toFromJSON[User] property("job") = toFromJSON[Job] ``` --- ## "Abstraction" ```scala trait DB { def create(user: User): UserId def read(id: UserId): Option[UserId] ... } ``` ??? - Good abstractions are likely to have some set of "laws" between functions --- ## "Abstraction" ```scala trait DB { def create(user: User): UserId def read(id: UserId): Option[UserId] ... } object PostgresDB extends DB object MemoryDB extends DB ``` ??? - For the "cool kids" in the room - Could be your free interpreters --- ## Database Laws ```scala def dbLaws(db: DB): Prop = new Properties("DB Laws") { ``` --- ## Database Laws ```scala def dbLaws(db: DB): Prop = new Properties("DB Laws") { property("symmetry") = forAll { (u: User) => db.read(db.create(u)) == Some(u) } ``` --- ## Database Laws ```scala def dbLaws(db: DB): Prop = new Properties("DB Laws") { property("symmetry") = forAll { (u: User) => db.read(db.create(u)) == Some(u) } property("empty") = forAll { (id: UserId) => db.read(id) == None } ``` --- ## Database Laws ```scala def dbLaws(db: DB): Prop = new Properties("DB Laws") { property("symmetry") = forAll { (u: User) => db.read(db.create(u)) == Some(u) } property("empty") = forAll { (id: UserId) => db.read(id) == None } } property("postgres DB laws") = dbLaws(PostgresDB) property("memory DB laws") = dbLaws(MemoryDB) ``` --- template: inverse-toc <ul class="toc"> <li> Properties <ul> <li>Patterns</li> <li>Laws</li> </ul> </li> <li class="toc-selected">Generators</li> <li>Bug hunting</li> </ul> --- ## Free Variables ```scala def testUser = { val user = User("bob", 2000) val id = db.insert(user) db.load(id) == Some(user) } ``` --- ## Just Add Properties ```scala forAll { (name: String, postcode: Int) => val user = User(name, postcode) val id = db.insert(user) db.load(id) == Some(user) } ``` ??? - A property just waiting to get out --- template: boom ## Just Add Properties ```scala forAll { (name: String, postcode: Int) => val user = User(name, postcode) val id = db.insert(user) db.load(id) == Some(user) } ``` <pre class="error"> Invalid username: "" Invalid postcode: -1 </pre> --- ## Documentation is awesome ```scala trait UserDb { /** * Inserts a user. * * The username must not be blank * Postcode must be >= 1000 or < 10000 */ def insert(user: User): Int } ``` --- ## Be Precise - Types! ```scala class Postcode private(val value: Int) extends AnyVal ``` ??? - Preaching to the choir - Not strictly related to property based testing - But something that makes it easier - TDD - Deserves it's own talk - Mention Eric? --- ## Be Precise - Types! ```scala class Postcode private(val value: Int) extends AnyVal object Postcode { def fromInt(i: Int): Option[Postcode] = if (i >= 1000 || i < 10000) Some(new Postcode(i)) else None } ``` --- ## Invest in Generators ```scala def genUsername: Gen[Username] = Gen.nonEmptyListOf(Gen.alphaChar) .map(Username.fromString) def genPostcode: Gen[Postcode] = Gen.choose(1000, 9999) .map(Postcode.fromInt) ``` ??? - Yes it doesn't compile - Just ignore --- ## Invest in Generators ```scala def genUsername: Gen[Username] = Gen.nonEmptyListOf(Gen.alphaChar) .map(Username.fromString) def genPostcode: Gen[Postcode] = Gen.choose(1000, 9999) .map(Postcode.fromInt) forAll { (u: Username, p: Postcode) => ... } forAll { (u1: Username, u2: Username) => ... } forAll { p: List[Postcode] => ... } ``` ??? - We can add more properties using the _same_ generator --- template: ambiata ## Ivory Generators - Dictionary: ~100 properties - Date: ~110 properties - Fact: ~140 properties --- template: question ## True Story - "null" username Broke when migrating databases <img src="images/exploits_of_a_mom.png" /> --- ## Previously ```scala def testNullUsername = { dbMigration.migrate("null") } ``` ??? - Only tests one code-path --- ## Generators FTW ```scala def genUsername: Gen[Username] = Gen.frequency( 19 -> Gen.alphaStr , 1 -> Gen.const("null") ).map(Username.fromString) ``` ??? - Ideally you don't have to add any more tests --- template: image <img src="images/everywhere2.jpg" height="120%" style="margin-top: -60px" /> ## Generators for everything (with types) --- <ul class="toc"> <li> Properties <ul> <li>Patterns</li> <li>Laws</li> </ul> </li> <li>Generators</li> <li class="toc-selected">Bug hunting</li> </ul> --- ## [@markhibberd](https://twitter.com/markhibberd) <i style="font-size: 26px"> "Maybe you could test a relatively well known open source library and find a bug for something they have unit tests for" </i> --- ## [@markhibberd](https://twitter.com/markhibberd) <i style="font-size: 26px"> "Maybe you could test a relatively well known open source library and find a bug for something they have unit tests for" </i> <img src="images/challenge.jpg" style="position: absolute; width: 600px; right: 0px; bottom: 0px;" /> --- ## Joda ```scala import org.joda.time._ forAll { dt: DateTime => val formatter = DateTimeFormat.fullDateTime() val s = formatter.print(dt) formatter.parseDateTime(s) == dt } ``` ??? - Symmetrical --- template: boom ## Joda ```scala import org.joda.time._ forAll { dt: DateTime => val formatter = DateTimeFormat.fullDateTime() val s = formatter.print(dt) formatter.parseDateTime(s) == dt } ``` <pre class="error"> Invalid format: "Sunday, September 22, 2148 9:08:08 PM ART" is malformed at "ART" </pre> --- ## Bug or Feature? - http://stackoverflow.com/questions/15642053/joda-time-parsing-string-throws-java-lang-illegalargumentexception - http://comments.gmane.org/gmane.comp.java.joda-time.user/1385 - https://github.com/JodaOrg/joda-time/commit/14863a51230b3d44201646dbc1ce5d7f6bb97a33 --- template: image-white <a href="https://twitter.com/da_terry/status/587602658011189252"> <img src="images/alex.png" > </a> --- ```scala import org.rauschig.jarchivelib._ def archiveProp(archiver: ArchiverFactory): Prop = forAll { files: Map[FilePath, Array[Byte]] => val archive = archiver.create(files) archive.loadAll() == files } ``` ??? - Symmetrical --- template: boom ```scala import org.rauschig.jarchivelib._ def archiveProp(archiver: ArchiverFactory): Prop = forAll { files: Map[FilePath, Array[Byte]] => val archive = archiver.create(files) archive.loadAll() == files } archiveProp(createArchiver(Jar, PACK200)) ``` <pre class="error"> Expected "" but got "PKPACK2000" </pre> --- template: boom ```scala import org.rauschig.jarchivelib._ def archiveProp(archiver: ArchiverFactory): Prop = forAll { files: Map[FilePath, Array[Byte]] => val archive = archiver.create(files) archive.loadAll() == files } archiveProp(createArchiver(Tar)) ``` <pre class="error"> file name '...' is too long (> 100 bytes) </pre> --- template: boom ```scala import org.rauschig.jarchivelib._ def archiveProp(archiver: ArchiverFactory): Prop = forAll { files: Map[FilePath, Array[Byte]] => val archive = archiver.create(files) archive.loadAll() == files } archiveProp(createArchiver(Ar)) ``` <pre class="error"> Expected "aⲐ" "a?" </pre> --- ## Haskell Too ```haskell import Data.Aeson toFromJSON :: (ToJSON a, FromJSON a, Eq a, Show a) => a -> Property prop_time :: UTCTime -> Property prop_time = toFromJSON ``` ??? - Symmetrical --- template: boom ## Haskell Too ```haskell import Data.Aeson toFromJSON :: (ToJSON a, FromJSON a, Eq a, Show a) => a -> Property prop_time :: UTCTime -> Property prop_time = toFromJSON ``` <pre class="error"> Falsifiable (after 1 test and 54 shrinks): Right 1858-11-17 00:00:00 UTC /= Right 1858-11-17 00:00:00.000000000001 UTC </pre> --- ## Bug - https://github.com/bos/aeson/issues/240 - https://github.com/bos/aeson/pull/241 - "Updates UTCTime ToJSON instance to output full picosecond resolution" --- # Haskell Again ```haskell import Aws.S3 prop_s3 :: String -> ByteString -> Property prop_s3 key body = do putObject bucket key body r <- getObject bucket key assert $ r === body ``` ??? - Symmetrical --- template: boom # Haskell Again ```haskell import Aws.S3 prop_s3 :: String -> ByteString -> Property prop_s3 key body = do putObject bucket key body r <- getObject bucket key assert $ r === body ``` <pre class="error"> The request signature we calculated does not match the signature you provided. ARG_0: 檤翂 <pre> --- ## Bug - https://github.com/aristidb/aws/pull/154 - "Use urlEncode on object keys" - Also causes problems for the Java CLI client --- template: inverse # Conclusion --- template: ambiata ## Shameless Plug - https://github.com/ambiata/disorder.scala - DistinctPair - PositiveInt - https://github.com/ambiata/disorder.hs - DistinctPair - OrdPair - Aeson/Lens laws ??? - Under the disorder "umbrella" project --- <ul class="toc toc-selected"> <li> Properties <ul> <li>Patterns <ul> <li>Symmetry</li> <li>Model-based</li> <li>Invariants</li> </ul> </li> <li>Laws</li> </ul> </li> <li>Generators</li> <li>Bug hunting</li> </ul> ??? - Easy once you get started - Look for patterns - Test more with less --- template: inverse # "Real World" <img src="images/all-the-things.png" height="60%" style="position: absolute; bottom: 0; right: -100px;" /> ??? - Property-test all the things --- layout: false ## Links - "Choosing properties for property-based testing" - http://fsharpforfunandprofit.com/posts/property-based-testing-2/ - John Hughes - "Testing the Hard Stuff and Staying Sane" - https://www.youtube.com/watch?v=zi0rHwfiX1Q - [@charlesofarrell](https://twitter.com/charlesofarrell) - https://github.com/charleso/property-testing-preso .