00:00:16.400
thanks everyone for coming out really appreciate it you guys are looking great
00:00:22.640
working out looking good so i just introduced myself my name is jim
00:00:28.880
jones a ruby on rails engineer i work as a consultant in san francisco area have
00:00:34.719
14 years of experience working with rails on and off the past
00:00:41.040
six years so i like probably many of you
00:00:47.360
have a lot of side projects you know to the point where most aren't completed you
00:00:52.399
haven't forgotten about some and there are times when i'm experimenting with certain images or
00:00:59.520
trying to write some copy text and i feel stuck hey
00:01:04.879
i stuck to a point where colors start looking awkward or my writing sounds strange or it's 2 am and
00:01:10.720
i'm trying to be way too funny than i probably am and i really can't tell what the issue
00:01:16.400
is and so i sought out to find usability solutions and while
00:01:23.759
there's many usability services out there there aren't very many that allow me to
00:01:28.960
evaluate certain bits of the site most of these are very formalized
00:01:34.079
services where as i wanted a quick point of feedback without the need of
00:01:39.439
like formalizing my requirements and such and so i had some prior experience with
00:01:44.880
mechanical turk with data gathering so i started to kind of go down this route with mechanical chirp
00:01:52.000
and so for those who don't have a background in this i'm going to give you a little rundown of the service
00:01:57.280
um mechanical turk is a crowd sourcing marketplace and it allows individuals to
00:02:04.159
coordinate just like small tasks to be performed by people so
00:02:09.200
imagine someone proposes the problem says you have an army of 1 million people
00:02:15.440
that will each do a 10 second 20 second task for you like what would you use that for so
00:02:22.800
these are the sorts of kind of micro tasks but type of problems that
00:02:28.000
mechanical turk is really good at so these are generally tasks that computers
00:02:34.160
are unable to do or don't do very well and so
00:02:39.200
when we think about mechanical chart most people their experience is with
00:02:45.440
very straightforward tasks as far as the tagging of images scientific surveys
00:02:50.560
which usually extend to like demographic data um gathering uh
00:02:56.080
um analyses in that regards so most people have a very straightforward thinking of the service
00:03:03.200
so hopefully we'll kind of expand that and get you guys thinking a little bit differently
00:03:08.480
towards the end of this but just to make sure that we're on the same
00:03:13.680
page i'm going to do a little bit of terminology here when i refer to
00:03:20.239
a hit it's that's a human intelligence task so this is the task that you're
00:03:25.519
posting mechanical turn this is the task that instructs the worker to do something on your behalf
00:03:33.120
the checkers are the workers that are performing the task yes they do
00:03:38.480
call themselves turkers and an assignment is something within that particular hit
00:03:46.000
so one tricker will complete an assignment within the hit so if you had a
00:03:51.920
i want to classify this image and you want say 100 responses on that
00:03:57.760
and you're going to take the most common responses classification for that image you would have one hit and say 100
00:04:03.840
assignments
00:04:09.040
with the mechanical turret service you have uh the estimates are as many as a couple
00:04:16.079
million you have lots of different motivations here and uh um
00:04:22.880
there was a there was a survey done this would have been back in 2008 but a lot
00:04:28.160
of people responding with um tons of different motives here you know most is like monetary compensation most
00:04:34.720
are probably like trying to supplement their income and such but you have also people like one guy at the top it's kind
00:04:40.400
of cut off here he says he's battling insomnia and some guy needs drug money
00:04:46.160
apparently and so it's not all monetary compensation i mean when you start to think about
00:04:52.880
people past the time playing farmville or something like that
00:04:58.240
next time you get a little bored you should actually explore the service because there's just a multitude there's just the the types of hits that are out
00:05:06.000
there to span so many different domains such that it could be rather entertaining and they would obviously be
00:05:12.400
making money so some of the demographics for this is
00:05:18.800
the united states dominates the service and then postal following is india and
00:05:24.080
then you definitely have other countries but majority is definitely the united
00:05:29.759
states and india for completing these tasks and as you can see the gender breakdown
00:05:36.160
is primarily female in the united states whereas in india is actually primarily
00:05:41.280
male with the ages of workers we've got the 24 to
00:05:47.199
33 year old range is most dominant for both of the countries
00:05:52.320
so it's definitely uh younger generations that are utilizing your service and such and
00:05:57.919
are aware that it exists and here's probably the most uh surprising fact the people
00:06:05.039
are taken back by is that the education level majority of the users actually have bachelor's degree and when we start
00:06:11.759
going over some feedback that i've received i think you guys will be really impressed the level of quality that's
00:06:17.440
given um and it'll be quite apparent that these people are very educated and such
00:06:24.400
so um it may not may not be a shock towards the end of the presentation
00:06:31.440
and then average income at least for like the u.s it tends to be primarily middle-class people
00:06:36.560
supplementing their work or supplementing their income so we start kind of with the basics here as
00:06:43.680
far as working with mechanical chert there's three different ways that you
00:06:50.639
can interface with the service you have their very basic web
00:06:56.160
web interface which is just this very simple form builder allows you to put text fields text areas and that sort of
00:07:02.400
thing they also have an api for the ruby community we have
00:07:08.000
our chart mark percival gym ruby aws and then you have my gems
00:07:13.440
turkey which is built on top of our tert which provides some convenience methods
00:07:18.720
for integrating within rails and there there also is a command line tool
00:07:25.039
which is kind of api coupled with some xml files that drive interfacing with it
00:07:32.000
most people's experience is probably with the basic web interface which is just very basic form fields and
00:07:38.160
so i think that's where you get the perception of just very straightforward linear surveys and data
00:07:44.960
collection so that's kind of where that perception has stayed because that's the most
00:07:50.000
people's first exposure to the service so there's actually
00:07:56.080
two different ways that uh a hit be displayed
00:08:01.120
if you use their simple interface that's going to be an html form that's
00:08:06.160
generated html page is generated and just served up from s3 um but there's also this concept of what
00:08:13.840
we call an external hit and this is something that most people don't take advantage of but provides the most the
00:08:20.720
most power here with my gem i turkey supports
00:08:27.280
an external hit and what this does is uh the trigger is able to interact with
00:08:33.360
your website directly and then your website is actually displayed within an iframe
00:08:39.360
so and i'll give an example of this here in a little bit but uh given that that means that you have
00:08:46.240
full control over the form what is displayed and uh how it's input from the
00:08:51.519
trigger the turkey gem does a couple of things
00:08:57.839
um allows the developer to to easily integrate with mechanical turkey rails
00:09:03.760
using it's it's a very convenient form four light helper
00:09:09.200
where you can see we've got the turkey forum core just takes an instantiated object you
00:09:14.880
can pass some params and it just takes your your form field data the big difference here is that
00:09:21.839
instead of the data being posted back to your server the data is actually posted back to
00:09:27.920
mechanical term and then the second portion that turkey allows that helps
00:09:34.640
development is it allows you to easily retrieve that data that's been posted to mechanical turk
00:09:39.839
and import it right back into your models and so the workflow generally is is that
00:09:45.440
a trigger completes your assignment data gets posted to mechanical tert
00:09:51.279
you run a process of in this case it'll be a rake task that retrieves the data
00:09:56.480
it's going to import it into your models and you're going to programmatically approve or reject it
00:10:01.680
and so this this basically determines whether that trigger gets paid or not and they can obviously contest whether
00:10:08.160
it should have been rejected or not but that that's the general workflow so
00:10:14.320
okay so that thing all things aside i want to show you here
00:10:20.880
um a very basic app that i've written with turkey we're going to
00:10:26.000
go on the limb here do something in real time and break that rule which is probably a terrible idea
00:10:32.399
but we're gonna go ahead and do it anyways so let's see
00:10:38.640
if we can okay let me let me swing around here this is really
00:10:44.240
awkward my neck didn't turn that far
00:10:49.360
up here so
00:10:56.800
we're gonna say please
00:11:04.720
please say hello all right one second let me type this
00:11:19.120
so
00:11:29.519
okay
00:12:06.880
i disconnected
00:12:18.720
let's try this one more time
00:12:33.200
uh
00:12:45.519
oh man oh well so what's up
00:12:53.279
um
00:13:19.120
um what the turker is going to see so
00:13:26.000
so that you guys can get a feel for what an external hit looks like because it's kind of hard to conceptualize and such
00:13:31.920
so they're going to encounter that that says say hello here's the directions
00:13:37.279
this is just an iframe right here this is wrapping the application
00:13:42.639
and this is the form that's been created using the turkey 4 core method that's going to be posted back to the
00:13:47.839
mechanical church so i control this entire form i control this entire page it's just a rails out
00:13:53.360
running on a heroku instance and so this is what they're one day interacting with
00:13:58.800
so we'll come back and revisit it and see if anyone said hello hopefully
00:14:04.639
you don't say anything about my family or anything embarrassing so we'll get back to that so
00:14:12.240
remind me we'll jump back there we'll see if anything okay
00:14:17.279
so now we have kind of the base cases set up and we're going to go ahead and
00:14:23.760
get into the more advanced cases what the top what this talk is actually about
00:14:28.839
so we're going to talk about soliciting feedback
00:14:34.639
my goal was how could i gather meaningful feedback quickly and i wanted to make
00:14:40.800
better decisions within my development and i wanted to do this
00:14:47.279
extremely quickly i wanted to skip deployment entirely this is about
00:14:54.160
trying to get feedback as early as possible with your particular project
00:15:00.160
it's like like i was saying there's times in the 2am in the morning i'm writing something
00:15:05.680
i know it sounds awkward and if this is a project that i'm the only person working on i have no one else to bounce
00:15:11.440
things off of so i want to be able to gather feedback but i don't necessarily
00:15:16.480
want to say merge into master if you are on a team you may not want to merge it in you
00:15:22.079
may not want to deploy to a staging server because you may be deploying over another branch or something so sometimes
00:15:27.760
you just want to hit that local dev instance really try on try out some girly ideas and get feedback on it
00:15:35.199
so after some exploration i came with the first tool choice would
00:15:40.720
be the forward gem um and what this gem allows you to do is to easily expose your local dev instance
00:15:47.760
to the outside world and it gives you a nice convenient domain that others can use to access
00:15:54.320
your server now there is this one is a paid service there is another one called local tunnel
00:16:00.720
that is free the reason i prefer forward over local tunnel is that local tunnel will actually give you an https
00:16:07.279
interface and actually gives you the domain that they give you um remains static versus level tunnels into change
00:16:13.759
which was that would just be a big change so uh i think ford was kind of superior in
00:16:19.759
that and so second as you probably already guessed
00:16:24.800
we're going to use something called turkish list of feedback so we're going to need a gem to interact with the servants
00:16:30.000
since i wrote turkey i know turkey will use turkey so
00:16:36.560
with the latest turkey uh revision i'm introducing this concept to what i
00:16:41.920
call a study and given this code you can see that i called it this helpful method called
00:16:47.600
turkey study and what this does is displays a nice forum on your website
00:16:54.320
that gives the trigger the instructions that you define when you're posting your tasks and it gives them a nice text area where
00:17:00.720
they can enter in feedback so when you have once you have this
00:17:06.720
method inserted within your your application template you can go ahead and you can run the rape task which is just
00:17:13.120
right turkey create study um you're going to want to pass in the url which is the url
00:17:20.079
that forward has given you so this is the important part right that forward is exposing that local dev
00:17:26.799
instance so imagine you're getting ready you've just made this big kind of experimental branch you're not sure if
00:17:32.880
it's going to work you want some feedback on it imagine you're going to lunch and you're going to be eating a big
00:17:38.960
burger like super duper burger or something and it's going to be a long lunch and you're like i'd really like to get
00:17:44.400
some feedback on this so you open up your dev incidents the outside world and we're going to be posting
00:17:50.160
our hit to hit that particular dev instant so this is where we get around
00:17:55.360
the whole deployment this is where we we get to have the trickers give us feedback on the experimental portion of
00:18:02.080
our development and then lastly uh retrieving the
00:18:07.200
feedback is just a simple break task and this is for
00:18:12.840
turkey uh here we go so just to give you kind of a brief
00:18:18.960
background i've been working on this project in the evening it's called 5s5 and basically
00:18:24.799
it's a website that allows fans to bid on the spot to meet their favorite
00:18:31.120
celebrity in a video chat and so it's this auction platform and
00:18:36.880
um been working on it for a while now and this is one of those particular
00:18:43.120
projects that i kind of reached the dead ends at the nut at the end of the night and so
00:18:48.400
here you can see here's the home page we've got
00:18:53.799
this photo the directions overlay that's part of
00:18:58.960
the turkey study form helper so this is what's going to be overlaid on every page
00:19:04.320
um and this is the one that allows the triggers to enter in their feedback and submit so
00:19:10.960
since we're putting in the application helper application template that it's going to be displayed on every page
00:19:21.039
so given that page here's some sample feedback that i got from the turkers so
00:19:26.080
you can kind of get a sense for what the quality is first one says you should make the logo
00:19:32.799
bolder than the menu they blend in too much the image screen could be lightened up because the saturation is too much
00:19:38.720
for my eyes the gray box over the image could contain more information or better
00:19:44.000
call to action and the second one said revero i had problems with your site
00:19:50.080
with the sign in when i clicked in with facebook it gave me an error page cannot
00:19:55.280
be displayed other than that the site is doing great loads fast and it's easy to understand
00:20:01.520
you should have more content find more fans willing to do the online web chat
00:20:06.880
great point turker and so as you can see though
00:20:12.640
this happens a lot too that there's a lot of people with very technical backgrounds
00:20:18.000
for whatever reason or playing around with the service um obviously this guy knows about
00:20:23.039
saturation you obviously can see that my photoshop skills suck and
00:20:28.960
but he's he's willing he points that particular uh that he points that out and that
00:20:35.520
actually became a common thread throughout the feedback that um i never would have thought i'm like for some
00:20:40.640
reason i'm programming me i love saturation and but there's a lot of people that are like man this is driving
00:20:46.400
my eyes crazy and i probably wouldn't have known that unless i solicited feedback and so i actually will have to
00:20:52.480
hire a designer so um but it's good the quality of the feedback you're most
00:20:59.360
the time you're you're kind of looking for these threats these common threads that people are pointing out and such
00:21:04.480
and so um a few pro tips when you're soliciting feedback
00:21:10.159
it's always better that you make your task as a personal appeal
00:21:15.679
don't come in there and say yeah i'm i'm making this website for att big
00:21:23.520
time client i'm making a lot of money if you help me out it'd be great
00:21:28.559
they're not going to appeal that you're making a personal appeal you make it like hey this is something i'm doing on my own i really would like it if people
00:21:34.480
could help me out and most of the time they're very happy to oblige in fact i would say when you
00:21:40.240
make it personal appeals they actually sometimes overextend themselves i've gotten four or five paragraph
00:21:46.240
feedbacks before that were posted for like a quarter and it's incredibly helpful the research
00:21:52.559
paper will call those people eager be verbs you'll actually see that in a lot of different mechanical turret research
00:21:57.919
so it's an interesting term i thought and you also want to ensure
00:22:04.159
that you're inviting the checkers to give you negative feedback so the turkers they obviously get
00:22:09.919
compensated when they complete a task but there's a lot of them that are fearful that if they provide negative
00:22:15.520
feedback then you're going to go ahead and reject it and so you actually kind of have to overcompensate for that and you have to
00:22:22.559
invite them and say like hey what don't you like about this site because otherwise like oh super duper
00:22:28.799
great it's looking good keep your head up you know and it's just like it doesn't really give you anything so
00:22:34.799
you really have to make it a safe place to take place for some negative feedback
00:22:42.559
okay so now we're going to get into a little more advanced case of a b testing
00:22:51.600
so i'm going to use the vanity gem this is just an a b testing framework
00:22:57.840
has nice integration with rails relies on redis and active record pretty easy
00:23:03.280
to get set up you can see my definition of my test
00:23:08.640
here is i want to i'm going to be experimenting with the home page photos and which one i think is more effective on
00:23:15.600
getting people to interact and as you can see within my events
00:23:21.039
controller what i'm considering a successful interaction is if they actually reach the events controller the
00:23:27.120
show page for the events one one note here is that
00:23:34.559
intent is really hard to test with mechanical chirp these are paid workers they're not your
00:23:40.880
customers and they're not interested in your product so
00:23:45.919
the terrible way to say is someone's gonna really buy this at least from an inter interface perspective right you may be
00:23:52.320
able to ask would you buy this or not and maybe that's a good process maybe not but they're still being paid so there
00:23:57.679
could be ulterior motives to their response so we have to keep that in mind
00:24:04.559
but interaction can still be measured so when given a goal uh which length does the turkey click
00:24:11.840
first if they're asked to sign up what's their progression within the site that they actually take
00:24:18.480
um to reach that particular bulb that you define here
00:24:24.080
so let's go through our my example test was like given two large images on the home
00:24:31.360
page under which image are the trickers more successful
00:24:36.559
in reaching the event page so i kind of have some built-in
00:24:42.080
assumptions with this particular test here i'm going to say the
00:24:49.039
image where the triggers reach that event page is the image that was more compelling that that led them to actually want to
00:24:55.200
interact because that's the image that not only a lot of that took her to say yes i'm
00:25:01.440
going to accept that particular hit but actually wasn't compelling enough where they were going to hunt around and reach
00:25:06.720
that particular goal that i could find and so they have a lot of options there they have a lot of times they can abort
00:25:13.039
right when they first see the particular hit they're going to see my webpage just loaded there and they don't and that
00:25:19.440
counts as a page impression you know vanity is is looking at that as an impression and a lot of times as you'll
00:25:26.080
see in the results here a lot of times they just skip by so
00:25:31.200
it's fairly open-ended as to why that is maybe it's the image maybe it's not but we're just trying to generate some
00:25:36.880
hypotheses here not necessarily reaching any conclusions
00:25:42.400
but within this particular task i had option a which is this
00:25:49.120
really cool guy i think that's overly saturated now that i know that's crowd
00:25:54.559
surfing and he's holding his hands in the air and it's like oh my gosh
00:26:00.400
he's obviously a really cool fan right wow i can't wait to go to 5s5 it's probably
00:26:06.559
wasting he doesn't match i can't wait to get home so you can see the directions on the right
00:26:13.279
that pump for feedback and then option b um these are just a couple girls waiting
00:26:19.520
outside of concert they're obviously having a great time they
00:26:25.039
are also probably thinking about 5-5 and how they can meet
00:26:30.720
the band that they're there for i mean very exciting right
00:26:36.840
so here's the interesting thing so the trigger instructions are looked at my site
00:26:42.480
find a way to the band's event page then get feedback below what you don't
00:26:47.919
like about my site and tell them to submit and the instructions are fairly
00:26:53.600
open-ended and that kind of implies that there's multiple ways to get to the event page
00:26:58.720
and since we don't have intent for a user that's really interested in this product
00:27:03.919
we'll just give them an open-ended objective and see how they interact with the site and see how they go about achieving this
00:27:11.120
this open-ended goal that i've defined um the other thing is the tricker doesn't
00:27:16.559
really know that we're measuring which image right that's our underlying test
00:27:21.919
but we're just giving them a very open-ended goal for which they're to achieve and when they achieve
00:27:27.679
that goal then we're going to go ahead and confer a few things from this underlying test that we've set up so we
00:27:33.600
have something very specific that we're testing the turkey doesn't know that trigger just has a very open knitting
00:27:41.919
so i'll show you the results and the rather boring because it ends up being inconclusive
00:27:48.640
um you have option a the crowd trooper guy actually converted a little bit better but
00:27:56.559
it's really not statistically significant and so it's kind of sad for me because i would
00:28:03.039
like a direction not b direction but a direction and so i
00:28:08.159
will have to formulate more tests and such but you can see it'll show like the number
00:28:13.440
of participants obviously the number of participants total are much higher than those who
00:28:18.559
actually participated and those are trickers that just actually looked at that particular task but didn't choose
00:28:24.159
to accept it but you could still use that as a data point right they elected not to even participate in your task so
00:28:30.880
was it the image you know it's the most prominent thing on the page it could be copy text there's a lot of hypotheses
00:28:36.880
still to explore but it's there
00:28:42.840
uh lastly we're going to talk about dealing with spam data okay
00:28:50.720
and mechanical turk has definitely gotten a bad rap about this in the past few years
00:28:55.760
in terms of a lot of garbage that flows through the system it's as high as 40 on
00:29:01.600
all submissions i actually think that's lower they got um a new
00:29:07.360
uh cto i believe students and i don't remember what her title is but she seems to be adding a lot more
00:29:13.120
features and they seem to be getting better reputations reputation systems and such so there's a
00:29:18.640
few more things in place now to mitigate against this but it's still definitely a problem
00:29:24.799
and so what you have is you have a lot of people that have just set up bots who will arbitrarily
00:29:30.960
accept tasks they'll look for form fields they'll input junk data and they're hoping that you're just
00:29:37.039
passively approving the tasks and so programmatically just passively approving it so they're hoping to just
00:29:44.080
collapse on some of those so it really is a problem you're going to encounter it
00:29:49.520
probably in your first pass of things but there's a few there's a few ways you can fight against it
00:29:56.399
so kind of the first naive solution it's just a gold standard question kind of capture
00:30:02.240
where you have i i like to say like four plus five or something i will tend to expand those
00:30:07.600
the english words and maybe even remove a letter or something like that so yeah four plus five and
00:30:15.039
that that sometimes is good enough but there's actually the better way that you can do
00:30:21.760
a better way you can do this that will improve quality of your feedback as well so
00:30:28.720
you ask them a validating question your gold standard question but that particular validating gold standard
00:30:34.640
question is actually based on the very feedback that they entered and so
00:30:41.039
what this does is not only validates whether they're human or not but then they actually have to go back and reread
00:30:47.279
what they've entered so they end up checking their work and i
00:30:53.360
don't remember the exact paper but when they surveyed afterwards um the turkers said yeah i actually ended up rewriting
00:31:00.080
a lot of things that i didn't like so it tends to improve the quality so you get a double bonus there
00:31:11.360
so i have uh as you can see along with the feedback form on the turkey study
00:31:17.679
i've actually instituted uh a goal question form field as well
00:31:24.080
and i do that check for you so uh that's a bank drive in i haven't released it as a gym but it's
00:31:30.640
part of the the master of merging the master now so if you check out the turkey master branch you can definitely
00:31:36.640
start to utilize that and uh so hopefully that will get some better responses
00:31:43.840
so that kind of covers the usability testing portion i kind of like to give
00:31:48.960
some more general research regarding the pinnacle church that some findings that you may find surprising and such and
00:31:55.840
then we'll probably open it up for some questions q a
00:32:05.440
so the point of this study was they showed that increasing compensation does not
00:32:11.440
necessarily increase accuracy so if your particular hit study requires some sort of data
00:32:18.159
data gathering or something um bumping up that particular number the
00:32:23.360
compensation number might not necessarily give you a better more accurate results and if i recall from
00:32:29.200
this study it ended up being to get better and more accurate results ended up they had institute redundancy so they
00:32:34.960
did triple check quadruple checks and such looking for the commonalities between it but they found there was
00:32:40.640
definitely a sweet spot and just by you're not necessarily going to get more accurate results just by arbitrarily
00:32:46.640
increasing the amount you're paying those workers
00:32:52.799
this particular study showed that most of their tasks were completed
00:32:58.559
between 6 am gmt and 3 pm gmt which actually translates to between 2 a 2 a.m
00:33:04.640
and 11 a.m in the united states and 11 30 a.m to 8 30 p.m in india
00:33:11.440
you know those are the two countries that provide most of the the work for the clinical chair
00:33:16.720
and since these attended their particular task was tagging images
00:33:22.399
and if for some reason for whatever reason uh it seemed to be the it was workers in india that like that
00:33:28.559
particular past more often and here's a really really important
00:33:34.080
point here um that applies to all mechanical turf uh hits that
00:33:40.240
when you're posting your hits trickers tend to work the queue sorted by the default which is the
00:33:46.399
number of hits you have posted and so if you have very large tasks you can
00:33:52.080
actually exploit that in fact i know of companies who will arbitrarily inflate their number of hits that they have
00:33:58.399
posted to the service later on um later on pull some of those back and actually
00:34:05.200
just to have their hit numbers um inflated so that trickers are more out to see their particular tasks posted
00:34:11.679
on the service and such and so that's really important if you can get
00:34:16.720
that number higher then you have a better chance of being visible without without not without
00:34:23.839
having to increase the compensation so that's really important
00:34:29.280
and then lastly i just want to leave you some other creative uses so you can kind of start to see that mechanical turk
00:34:34.960
isn't just a survey platform which is not a scientific gathering platform these are actual humans and i don't
00:34:40.159
think i think people are just starting to touch the surface as far as like content creation and such and especially when
00:34:46.079
you're writing new applications this is definitely a service that you can start to use
00:34:51.599
you can be very resourceful you can start to envision with external forms you can start to envision having
00:34:56.639
triggers record user videos right like was a service that was recently launched
00:35:02.880
hacker news and you could have them on an external form start capturing video you can do content creation if anyone
00:35:09.280
wants to talk about content creation afterwards uh we can talk about iterative content building where you
00:35:14.640
have turkers like submit first sentence and then you have turkish build off that sentence and you have another set of
00:35:19.920
turkers build off that sentence start forming these very long paragraphs
00:35:25.280
some of the some of the creative projects you can check this out on my github page
00:35:31.280
i call it add vote and so if you have an adwords campaign and you would like to test that say like
00:35:38.079
you're doing this example is grasshead b you can envision something like mortgage refinance where you're talking like 50
00:35:44.640
per click what it does is allows you to upload your ads all your variations it's going to
00:35:50.640
download your competitors for that particular set of keywords it's going to download the organic results it's going
00:35:56.720
to mix them all in into a google-like search page result and it asks the trickers which would you be more likely
00:36:02.079
to click on if you were searching for say grass-fed beef and so it kind of provides a
00:36:07.760
google adwords simulation um without having to pay google adwords
00:36:16.800
we've got 10 000 sheet this is really cool um as far as this particular artist had a
00:36:23.599
flash based application that allowed um trickers to draw a picture of a sheep
00:36:28.640
and he would record it as their as they're drawn and you can go ahead and visit the site and it'll show
00:36:34.240
all these different sheets that are drawn and it just shows the exact same manner that the tricker
00:36:39.359
developed it quite cool dr seussify the news so there was just a
00:36:44.880
research group that wanted to see how creative trickers would actually get and so
00:36:50.560
uh some of they were having the trickers rewrite headlines so i like the the last one says beat the
00:36:56.720
headline was bp tries to cap the well protest set to start and so turkey writes bp tries to cap what they have
00:37:03.680
while protesters start to race how about the well
00:37:09.359
but that's it open it up to some questions and such
00:37:15.359
thanks everyone for coming out
00:37:27.440
so
00:37:51.599
you