00:00:02.760
right um so hello welcome to my talk uh it's going to be a live coding talk I'm
00:00:08.599
going to try to Live code a testing library in Ruby starting from scratch
00:00:15.360
I've set up Vim to with a shortcut every time I press enter I execute the current
00:00:22.480
file which is a ruby script and we get the output at the
00:00:27.560
bottom um so that we can iterate quickly uh I'm going to leave something like
00:00:34.920
done at the bottom so that we make sure that we have some output that everything is working correctly uh so the first
00:00:42.399
thing the the basis of the basis of of of any testing framework is uh some way
00:00:50.719
to assert something uh so we want an assert
00:00:55.760
function right some something that that can do this well now it's not defined
00:01:01.840
let's define assert
00:01:07.479
oops um wrong number of arguments takes one argument let's call it the condition
00:01:13.040
because yeah okay uh it does nothing um
00:01:18.159
let's try to force the case where it should fail um it doesn't fail either
00:01:25.439
um so this this is interesting how how to to check that it fails
00:01:30.880
um you wrap the thing in a block you rescue
00:01:37.479
an exception so um I'm going to call this setion failure
00:01:44.000
and Define an error class so assertion failure is a standard
00:01:54.119
error um and so we say that by default we
00:02:01.759
haven't raised um if we catch this assertion failure we have
00:02:08.080
raised and we raise unless we we have not raised if if that makes sense
00:02:15.560
um rais nothing raised
00:02:20.640
unless raised and we need to Define this this thing
00:02:26.599
here um right so it says nothing rais because we haven't raised anything by
00:02:31.800
the way uh feel free to interrupt me at at any time if if something is not in clear or slight weird the full on test first
00:02:41.319
approach yeah from scratch it doesn't have to be
00:02:46.440
but you know the smaller iterations the the more confidence we get in theories
00:02:52.760
because if if you're basing other well of course this is not going into production but if you're you're going to
00:03:00.480
to use this for testing other code you need to have absolute reliability and
00:03:07.080
one way is to really manually test and then automate the manual testing of I mean automate the the testing of of the
00:03:14.599
library itself um right so here we'll raise assertion failure unless
00:03:22.519
condition uh and we made it pass so to give another example um
00:03:31.519
yeah no um should be good like if here we
00:03:37.159
say two assertion failure um right and and and now that we have
00:03:44.400
assert uh we can Define assert equal for instance um
00:03:51.920
that's because instead of sorry um instead of
00:03:57.239
having like uh this it's much nicer to
00:04:02.760
to have a specific function that says that well this will fail but let's say
00:04:09.799
start with fals f um it doesn't exist
00:04:14.959
we Define
00:04:20.919
it okay um it takes two arguments the expected value and the
00:04:28.120
actual value and then we implement it in terms of uh
00:04:33.240
assert itself so we say
00:04:40.360
expected right um oops so let's see it
00:04:45.479
fail again to verify that that we are doing everything correctly okay um one thing that's not
00:04:53.840
cool is here we have the generic message uh that you know there's some exception
00:04:59.800
somewhere in the code um so we can we can add that uh we can do this here as a
00:05:07.440
message which by default can be um assertion failed and I'm adding a
00:05:14.639
default because I don't want to specify a message for the for the
00:05:20.080
failure case every time and what we do
00:05:25.160
is you can you can print the me you can have the
00:05:32.479
message to go along with your exception right uh I'm going to rewrite this a
00:05:39.080
bit nicer okay and then we're good and then
00:05:44.600
for this one we can define a message saying something like um
00:05:52.160
expected expected uh but got this one
00:05:59.840
and we pass it in here and yeah expected Fu got bar um
00:06:06.160
something that is not nice is uh it's hard to distinguish between
00:06:11.919
the um you know literal values and uh error text so actually one thing we can
00:06:19.560
do is we can have uh we can have a human
00:06:26.639
representation of values right and now we have with
00:06:37.560
points right um let me quickly check my
00:06:45.919
notes um right and since again we want to see
00:06:52.639
that this fails well we have two cases we have a case that doesn't fail and a
00:06:58.720
case that fails but instead of uh implementing this this thing again this this logic I'm going to extract it in a
00:07:06.240
in a method of itself which I'll call assert to many assert error uh which
00:07:14.599
will take a block and
00:07:22.280
well so what do we do
00:07:28.039
oops uh we yield here meaning that now we can replace
00:07:37.280
this thing by just assert
00:07:45.759
error and here again assert
00:07:52.560
error and we're good all right so far
00:08:01.199
um okay um one thing we have so far is that by default we have no no output we
00:08:08.759
have this done here but we we're especially for those since
00:08:14.319
since the what makes noise is the failure but not the success we have no idea if this
00:08:21.360
test is still executing um so one thing we might want to do is
00:08:30.720
add some add some output but before we get
00:08:36.039
there I want to group The assertions into test cases because maybe you want
00:08:42.000
several assertions pertaining to the same same concept uh so I'm just going to Define functions so U let's call them
00:08:50.320
test assert for now because one tests the assert function okay and test assert e
00:09:01.200
equal oops yeah this this talk has plenty of
00:09:06.279
me um using VI in correct law right so print uh I'll just print a
00:09:14.120
DOT so if we see a DOT it's been
00:09:19.160
executed here it's not being executed because I haven't called them yet so um
00:09:25.720
oops assert test assert equal and test
00:09:30.959
assert okay two dots and we're good yeah cool um this of
00:09:39.279
course is a bit tedious uh the whole point of of the the framework is that it
00:09:44.800
it executes all your test you don't want to uh to write all these Bo boilet plates so what we're going to do is try
00:09:51.399
to to group test in a test suite and there's several methods we can we can do
00:09:57.640
that but the the most straightforward I would say in in Ruby is just to put them in a class because a class is a concept
00:10:04.760
that already exist it's like a a natural part of Ruby everyone knows about it um
00:10:10.839
and you get some nice properties for instance if here since those are now uh
00:10:17.839
functions or methods uh in a class so let's say assertion no um ass
00:10:26.200
tests um since those are in a class um we get some benefits
00:10:36.240
like not sharing State between the test cases um so for instance if here I have
00:10:43.000
uh a um
00:10:48.959
well let me do this first no I have a right but
00:10:54.639
if here I accidentally use a it says it's not there um meaning that
00:11:02.279
if um yeah stuff like if here a is another value or whatever there's no
00:11:08.040
leakage between the tests uh so this is kind of important so anyway we group them in a
00:11:14.240
class and then what we do is we instantiate of course and run them
00:11:31.839
uh we can do something here um there's a way in Ruby to
00:11:39.600
get the methods defined in a class I'll demonstrate this one
00:11:45.680
quickly so let's say I have a class Fu
00:11:51.200
which defines uh me bar and okay so let's let's create a new
00:12:02.360
okay um there's a uh thing called public
00:12:08.920
method it lists all the symbols so all the all the messages that this this F
00:12:14.800
responds to and um since it responds to way too much stuff because uh some of it
00:12:21.440
comes from object or other um so we know it's the whole hierarchy you can get
00:12:27.399
just um the methods defined um in the class itself there's this argument in
00:12:35.160
the docs it's just called all and it says when all is false you get just the
00:12:40.399
methods from the class itself so here we have bar we Define bar we don't get any of the other stuff
00:12:47.720
um right so what we do is um we enumerate
00:12:56.160
those methods and and for um each of them so the suit the
00:13:04.320
suite has tests I would say and we just
00:13:11.839
um send it test um does everyone understand what this
00:13:18.519
does um this is basically say sweet.
00:13:24.040
test but test is uh can be any of those values this is an array of of names and
00:13:31.199
so we use meta programming here to just execute each of
00:13:36.360
them good so far right um but obviously we don't want to be
00:13:43.800
doing this either um so we're going to first
00:13:50.959
Define a a method here no um let's define a a
00:13:58.360
generic class called um test suite and a test Suite will be able to
00:14:05.880
no sorry before doing that maybe we can just extract this in a in a function so
00:14:11.440
we Define a function called run that takes a suite does this um so we just
00:14:19.440
have uh run. s still
00:14:24.800
there um now I want to move this into a a class so I have this test Suite
00:14:33.320
class um okay and we test suite. new. run then
00:14:42.199
we pass our suite of tests which is this other class uh this is a bit cumbersome
00:14:47.600
so we'll do a refactoring to um to not have
00:14:54.160
to um to instantiate the classes that contain the tests
00:14:59.360
I'm just going to move this over here okay um so the first thing we do
00:15:11.000
is we will change um this thing to
00:15:16.880
only run methods that start by test underscore uh we can do this well since
00:15:23.519
this is an array of of um names we can use a regular expression to match
00:15:30.800
everything that starts with test underscore um and we're still we're still good um so the next thing we do is
00:15:42.319
we make this Suite an in a subass of the the suite and we can
00:15:50.800
do this directly no okay but it's a bit awkward to have to to pass itself um um
00:15:59.639
in the method here so we we'll have a step-by-step refactoring
00:16:07.040
um we pass in so good uh here we did
00:16:14.319
oops so now it's it's still still running um next thing
00:16:20.720
is well since nobody is using this argument we just um we just re uh we do
00:16:30.040
this oh right no the next part is we just use self implicitly still going and
00:16:37.560
then we remove the argument all together and we're still
00:16:43.199
good
00:16:48.360
questions um okay and um but so imagine if we had a class of
00:16:57.560
other test which also inherited from this so we
00:17:04.240
have F don't care about this maybe we just want to make sure we're running it so we
00:17:11.319
have so now it's not running and we need to do this again uh
00:17:20.240
run yeah now we have three test runs but this again we we've just transformed from running meth uh different functions
00:17:26.839
to running to inan class and and running methods on them um so we want we would
00:17:34.039
like to have a single entry point uh and this entry point can just be a method a
00:17:39.280
class method on on test Suite called run um and to get there
00:17:46.760
we um we'll use a again
00:17:52.799
some some some Ruby features we can uh keep
00:17:59.440
an array of of sub classes
00:18:07.039
so there's this call back called um
00:18:12.080
inherited so whenever a class inherits from this test Suite um this this
00:18:19.799
inherited um class method got gets called and so what we can do is we can
00:18:26.480
remember um this remember the class and then we
00:18:34.120
Define another method that just um iterates over the registered
00:18:41.360
classes uh each here we need to instantiate because
00:18:51.600
um I mean yeah inan shate because we just have the the class itself we don't have an an object an instance of that
00:18:58.400
class and then we call run and so still good now we have them
00:19:05.600
running twice and we can just get rid of that and we
00:19:13.080
good um any questions
00:19:19.280
here that was a bit fast so okay what what have why don't I don't have to run so um
00:19:29.240
um let's see here I have this right um I have assertion tests inheriting from
00:19:36.919
test suite and when this
00:19:42.880
happens this code get gets executed and so we just keep track of
00:19:48.480
all the sub classes of this one and then we run you know the one
00:19:55.880
method that that um which is this one which just traverses
00:20:01.280
all the classes and executes the run method on these classes so this code is executed on definition time the classes
00:20:07.679
in a sense when you define the class when you run this one is executed when
00:20:12.880
you define yes yeah that's step yeah it's a call back it's a call
00:20:20.159
back mentally kind of weird think of compiling classes right cuz that doesn't exist in other
00:20:27.320
language this way you can at run time Define more classes and the gets added
00:20:32.360
to this thing and go there's inherited
00:20:37.799
and call back and vote whenever a subass of the current class is created can't you just have the subass
00:20:46.280
another maybe I don't know this one is pretty
00:20:51.480
cool I to find some kind of sub classes have public
00:20:56.640
methods yeah super class kind of yeah not sure if you have sub
00:21:04.080
classes you don't need it but but actually you don't have to inherit track
00:21:09.760
yeah yeah I don't know and you have something similar for modules you have um included
00:21:17.080
when when it's included you can execute some code or when it's extended or when a class get extended by
00:21:27.279
module yeah um let me check my notes again if we have some other cool
00:21:34.159
stuff um
00:21:41.520
right okay oh yeah one one thing we get for free is since now we have this
00:21:47.279
structure where here we execute each test we
00:21:52.440
can have a setup and a tear down step um
00:21:57.840
so this is the cure in mini test but in in our spec it's before and after um and
00:22:05.320
right now it will fail because they don't exist so we can just provide defaults um and since all these
00:22:14.360
um since every class containing tests inherits from this super class uh they
00:22:21.080
can just have the default and if they want to do some set setup or tear down they can um override them
00:22:29.919
and and then another thing uh we can do
00:22:35.799
is remove those pesy uh print statements from the from the individual test so um
00:22:44.279
we we'll do a similar thing uh as before we rescue um rescue assertion failure
00:22:53.360
and we print uh an f as in failed
00:22:59.080
and otherwise we just print a DOT so if this fails here get the F
00:23:06.120
instead um okay now we have too many
00:23:12.039
dots so we
00:23:17.600
just this one okay
00:23:24.640
cool right so this is this is the basis um
00:23:30.520
and I have much more to show but but since the actual programming takes a while I suggest I show you the the
00:23:39.720
result kind of um I I wrote a library like this last summer
00:23:46.120
and have the same the same structure in a way
00:24:12.559
to right so we have a test directory which so this tests the actual Library
00:24:18.880
so it's testing itself um and this is n trick here with the auto run um so I
00:24:26.080
don't know if you have if if you use mini test but in mini
00:24:32.159
test so you can have basically a ruby file uh so mini test
00:24:37.640
demo um if you require um mini test auto
00:24:50.880
have thanks and have this um now if I run this it runs the it it's assuming
00:24:59.000
it's a test file and of course I have no test Define in there but um that's
00:25:04.320
that's what it does and this is not that hard to implement um so I have it in a
00:25:11.240
separate file just to have this this one include so if you include the the
00:25:17.279
default Library by default you don't get the auto run um Behavior but with this you do get it um and this requires
00:25:26.559
the the main thing uh and the magic is kind of here so we have this
00:25:33.440
um um this class level variable at Exit
00:25:38.760
registered um and this makees sure makes sure that
00:25:44.000
we only register test for execution once so the magic is all here this exit hook
00:25:51.679
um will be that exit
00:26:02.799
um right so at Exit executes something be before the program
00:26:10.720
closes so what what miniest does in probably rspc and others what what what you do is you register all your tests
00:26:17.559
and you you make sure that all of your production code is required before the tests are and everything like this and
00:26:24.720
then once you access the program you're sure that everything was defined and then you can run the tests and you get the same thing with
00:26:31.520
you know uh registering all the test Suites and like the subass and all this stuff uh so it's pretty neat so um
00:26:44.240
well right so here I just called this this one
00:26:49.279
uh method and it just does add exit run
00:26:54.600
this and run does something else it it pares some some some command line arguments and basically does what I did
00:27:02.080
before but instead of doing test site suite. run at the end of the file I can
00:27:07.760
just require this auto run uh module or well um library and then it does it
00:27:16.159
automatically another interesting thing is um stack traces um in my example
00:27:23.960
before if I Ran So
00:27:29.279
uh let's say this
00:27:34.679
fails so if this oh but now I added the the catch behavior um but the point is
00:27:41.919
if it fails you get the the whole stack phrase and you don't care about the library internals you don't care about
00:27:47.159
the testing Library internals unless you're testing the library itself but us
00:27:52.360
just don't care about that um so what you can do is add a place where you actually output
00:28:00.760
the message you can filter the back trace and uh remove all the lines that
00:28:07.279
are in your library um there's a similar thing that happens in Min
00:28:13.640
test it's a bit more complicated I didn't understand all of the all of the
00:28:19.840
lines there because I don't know there's probably some much cases probably if you use this introduction you you get to see
00:28:25.799
them and why you need to ex exclude more stuff but basically what happens here is that if you get a failure so I'm pretty
00:28:33.679
sure I have some test cases so if
00:28:41.120
um if I have
00:28:50.120
this um now I just get this one line instead of the whole stack Trace
00:28:58.799
uh and as you can see I'm testing the library using itself
00:29:04.039
um so testing assertions uh that's the same kind of stuff I showed
00:29:09.840
before I am doing a thing where I'm testing the
00:29:15.159
output um so we want to see for instance a summary in the end that uh we have run
00:29:21.960
that many assertions and we have that many failures and stuff like this we
00:29:33.640
uh yeah order for instance um so I
00:29:38.840
implemented a thing where you can pass uh command line flag to
00:29:44.120
the uh to the file and you define a seed and then you can reproduce the the test
00:29:50.200
run and I'm making sure that that the the ordering is correct so how do you
00:29:56.799
did the random order you
00:30:03.240
just is it you just Shuffle uh the things
00:30:13.080
and I pass in either a new seed or um or
00:30:18.320
you know you already get the existing seed that you passed in um another interesting thing is I talked
00:30:25.399
about let me let me close this one I talked about how I'm testing well this
00:30:32.360
thing is testing itself right but there's a problem and a problem I didn't realize would occur
00:30:40.600
until I I ran it uh we did this thing where you
00:30:46.200
would um register a subass when you inherit uh from the super class
00:30:53.039
right but my test Suite contains test
00:30:59.080
twet so here I have this defined Suite which is just
00:31:04.639
um uh which is just creating a subass from this thing and and I need to to add
00:31:11.559
this thing and register what happens is that um this is a top level this is a
00:31:18.360
test Suite right that's that's using this but if inside there's a test Suite that's only for test purposes you get
00:31:24.279
into an infinite Loop situation where you're registering and registering and registering and like since you register
00:31:32.240
a thing inside then it runs all the and then yeah and so I had to add this
00:31:38.000
unregister thing which just
00:31:43.519
uh yeah which would just remove the entry so for those temporary Suites that
00:31:49.000
are Anonymous basically we just want to register them once um and register
00:31:56.320
them uh you using The Primitives that are exposed in the internals but we
00:32:01.639
don't want them to be run by this thing so once it's here you
00:32:08.279
know we we have it and it it gets run but this thing doesn't see it as a as a
00:32:14.840
sweet to be it's a bit
00:32:20.760
yeah uh another interesting thing is uh this here with uh string.io so
00:32:33.559
sum report at the end and so on and how do you
00:32:38.639
test conso output well you can
00:32:45.200
have uh this iio object so by default it's standard out and whenever you need
00:32:52.639
to for instance do print or put or whatever you
00:32:58.159
use this iio object
00:33:12.279
and well it doesn't matter so also
00:33:17.760
rues uh it's s i right but I believe that so it implements all the same
00:33:24.360
things that that um like the standard output or or files uh Implement so you
00:33:31.960
can just use this pass this in in the tests um and then make assertions on it
00:33:40.919
so for instance here I'm asserting that um so first I can I convert the the
00:33:48.000
output to to string because this this output is a string string I object
00:34:00.080
right Returns the underlying string object and then I run the assertion on
00:34:05.919
it and here I'm using the assertion that I wrote myself um it also gets tricky here what
00:34:13.919
what what you do is um since I'm I'm testing the output like exactly what
00:34:19.480
what I show in the end the report and how many test run and so on at first I can just count them and then run the
00:34:28.159
test that that actually asserts that those um things are accounted for and
00:34:34.760
then as I add new tests I can okay so I have this output from my own test Suite
00:34:40.800
uh so that means it's fine and then I've tested the basics of you know assertions and so on so that's fine and then you
00:34:47.440
build upon it uh and
00:34:54.480
it's super fast because I mean first of all I'm not
00:35:02.119
requiring anything uh external and then yeah so I can reproduce this this
00:35:09.119
one um yeah
00:35:14.440
so yeah that's about it i' I've skipped for instance one thing I skipped is um
00:35:20.960
well test Devils you can just do yourselves um stubs you can do yourself
00:35:26.920
kind of uh like one manual way of of doing stabs
00:35:33.119
is if you want to stop an instance so you have your object which is object new
00:35:41.440
you can Define the method on this object right to to have something and this is a
00:35:47.000
St um Mo I don't know how to well I
00:35:52.480
haven't thought about how to implement them but um yeah not not that that
00:35:57.839
complicated in a in a simple case and then depending on what your needs are
00:36:03.119
can get vary here but we Mo you need to pay attention to verify that the the the function was
00:36:09.960
called so there's some meta programming there um
00:36:17.200
right that's it for me may have questions
00:36:28.599
do you care about the exit code like when a test fails you um return it back
00:36:43.079
uh do you know how it's Dollar
00:36:49.079
Question yeah so but it was zero so here
00:36:55.920
it's okay and and if we make something
00:37:09.240
fail no okay so that's a that's a
00:37:17.040
two how big is the risk that you break the part in your testing Library which
00:37:23.920
identifies failures and then you wouldn't see that fact
00:37:32.520
anymore because you're blind to failures now and so it
00:37:37.640
coulded I would say um this you can track historically so if let let's say I
00:37:43.839
was use I wanted to use this I would have it on continuous integration and I
00:37:50.640
would have I would run each thing right and so for instance in the last run I
00:38:06.319
if if for some reason the thing that fails doesn't report
00:38:11.520
anymore it probably means that that it doesn't get output right there's no F if
00:38:17.480
if something fails or since no since the test that the test that that the output
00:38:23.480
is correct is is still there and I haven't modified it if the function it fails well
00:38:29.760
either I see the failure or the output
00:38:35.000
is not there anymore and then sudden H last time I had 35 tests why do I only
00:38:40.400
have 20 tests this time I I for me this is one way but I haven't thought about
00:38:47.839
it in more detail it's kind of
00:38:54.800
mind um by the way I do recommend a reading mini test if you ever want to
00:39:02.240
it's about a 1,500 lines I think but uh
00:39:08.240
with all of the all of its things so miniest has spec uh syntax and all of
00:39:14.400
the other stuff but the basics are very very small as well and this one
00:39:27.520
yeah it's just 60 lines of code and yeah and then
00:39:33.839
some test code so it's not that hard to implement you so I would never use this in
00:39:39.280
production but uh I kind of did it to prove myself that it's not not that hard
00:39:45.160
and also it it was
00:39:54.800
fun you have an opinion on this whole think about using more integration tests
00:40:00.560
versus using more unit tests um this
00:40:05.599
thing is definitely made more for well actually I don't
00:40:16.800
know uh but to answer your question it depends on what you're
00:40:22.720
testing I think um
00:40:29.119
here in the thing in the library itself I wanted to do integration tests
00:40:36.319
for the output but it was a bit hard I saw how
00:40:41.359
miniest does it what they do is since you can have different failure messages
00:40:47.000
or different file names what they do is they capture the output same way with string. string
00:40:53.119
iio and then they change some characters and make
00:40:58.359
them X's or something and then you really get
00:41:04.640
the whole outputs without caring about any of the internals and then in the test itself there's a temp there's a
00:41:10.800
string that says this must look like this and this is integration testing for me because you know uh you don't have to
00:41:19.599
do any of this um thing I do here with setting up you
00:41:26.160
know the reports and and so on like this is this is too much too close
00:41:34.240
to the internals I I would like to test this but if I was writing another kind
00:41:40.359
of reporter then maybe I would unit test this this thing
00:41:48.240
itself I I do both I I think it's important to do both um in larger
00:41:53.880
projects I tend to only do integration tests for important parts because
00:42:00.280
integration tests tend to Break
00:42:05.319
um for unrelated reasons and so if you have too many of
00:42:10.520
them then you're trying to fix your test all the time um I I think it's the metaphor
00:42:17.880
there is the the pyramid so base layer of lots of unit tests and
00:42:23.160
then um as you go up in abstraction in in the test um um you get fewer and
00:42:36.800
fewer
00:42:42.280
cool so thanks
00:42:47.880
again so as long
