00:00:18.400
Um, okay, so how many of you would call yourselves search engine experts?
00:00:24.160
Okay, that's good! I'm talking to the right crowd. In this talk, I'm not going to give you any silver bullets at all because I basically don't have any. Search engine optimization is a bit of a black magic, and I'm going to talk a lot about my mistakes, probably a little bit more frankly than I should.
00:00:34.559
I promise that you'll have a lot of fun and that I'll help you see the mistakes that I made, so maybe you won't make them yourself. But first, let's talk a little bit about search engine 101. So when you type something into Google and wonder what's going to come out on top, there's a simple formula: importance times relevance.
00:00:54.559
Importance is all about something called PageRank. PageRank is not named after the web page; it's named after a guy named Larry Page, who built an algorithm that Google now uses. It basically means it's going to take the number of links from other important sites, and it's going to pass a share of that link, that PageRank, to you.
00:01:07.600
Pages with higher PageRank will appear higher, all other things being equal. You'll hear this talked about as SPR. If you've got the Google toolbar, there's a little green bar. If it's all white, that means you don't have any PageRank at all. That's the importance, but we're not going to talk much about that in this talk because this is primarily a public relations topic.
00:01:31.920
If you get a couple of links from The New York Times, then you know you're probably good. Or you can get like 150,000 PageRank one links. What we're really going to focus on is relevance, and that's actually what's on the page and how that affects your search engine optimization.
00:01:52.159
This is really the framework side. The other side is really the public relations side. There's a little bit of crossover, but that's basically how it goes. How many of you were in my talk last year?
00:02:06.080
Okay, great! So this is a lemming. We talked a lot about the idea that some frameworks develop these fanatical followers. You know, we just got a dog, and another example is that there's a guy up on our street who walks his three dogs and a goat, and all the neighborhood kids would feed the goat anything, right?
00:02:29.360
They'd be like, 'Dad, this goat will eat anything!' But then I'd ask, 'Yes, but should that goat be eating anything?
00:02:54.959
And I mean, you've known Java developers like that who would try anything from XML to configuration to JSPs to EJBs. And we're much better than that, right? Well, no! We've got our fanatical followers too! In fact, any commercially successful framework has to have a core of leaders and a broad base of fanatical followers. That's what makes the ecosystem work.
00:03:06.959
That's how we get jobs that pay and how we can have a critical mass to have a conference. Some of those things that we follow make a lot of sense, and some of them don't. I heard that Dave talked a lot about that this morning.
00:03:34.799
But some of these things can have a profound impact on places where the framework doesn't express a strong opinion. Let me give you an example, and everyone will know it when they see it. This is actual scaffolding, and it's actually being used from what I understand.
00:04:10.959
If you've seen my talks before, you know that I think that scaffolding is of critical importance in the Rails community because that's often the first example of Rails application code that anybody ever sees. Other frameworks have their examples too. Java had what? Anyone know? Yeah, Pet Store! We've got our scaffolding.
00:04:41.760
And this is going to have a substantial effect on your search engine optimization. In fact, in the first couple of years of Rails, this is what your URLs looked like, right? Show 9923. That's your dog's name to the search engine. And now it looks much better with the restful routes and everything.
00:05:10.960
But if you think about it, this ID that's on the end of all our URLs is baked into the default migrations, baked into the default routes, and even the redirects. This means that every link in the system that uses those routes by default has this behavior.
00:05:34.160
I know that a lot of Rails people, myself included, back as early as a couple of years ago, believed that if you built this kick-ass application, then smart people would just work hand over fist to find your application and use it. But that's not the typical user. This is the typical user: either my mom, the driver, or some random picture from the internet.
00:06:04.240
You can decide which one. My mom is an interesting driver because she's had pretty bad neck problems, so she's got this massive mirror in the middle of the car so that she can see what's going on behind her. She's also got a mirror that's almost as big on the other side, like driving with a billboard beside you, so she can tell what's going on behind her.
00:06:35.360
But if you look to the right, there's no mirror at all. That's because she can't turn her neck to the right. She has to turn her whole body. So she tries to devise a route from place to place where she can only make left-hand turns.
00:07:03.760
The thing is that my mom doesn't see herself like this. She sees herself like this: there's this time when she got her little Corolla bumped by a beer truck. She got mad and tore off after this beer truck after he left the scene of the accident.
00:07:29.760
So she's whipping through Memphis, right, on a stale, and she kind of corners him at the 7-eleven where he's dropping off his beer. He probably just didn't know she was back there. And she got in his face, and this big burly guy was so flabbergasted that he pulled out his driver's license, registration, and insurance.
00:07:52.880
So she doesn't see herself as someone with limitations, but she doesn't always see all the signs, and she doesn't always make the prudent choice. To bring this back to an SEO talk, you need lots of obvious signs. So when I'm building a page, when I want to crank up relevance, I use the tools that are most important to me. The URL is probably the most important thing you could do; it should have the keyword that you're linking to.
00:08:31.040
Then the page title should contain it. You should have a single H1 tag with the words that you're trying to crank up the relevance for. There are a couple of other things, but this is the way that you ought to view the Google bots, you know, as my mom the driver, not some super intelligent thing that's going to divide the purpose of your site.
00:09:03.600
You shouldn't view your users as people who will go out of their way to find your blog and read what you're posting, and then go find your site. You need lots of just brain-dead obvious signs—usually the more, the better. But if you're trying to game the search engine, you might get yourself in trouble.
00:09:32.640
Going back to the default Rails implementation, you know we've all got this map resources. For us, we've got a plant application; it's a gardening application. In the views, there's an H2 tag that links to a title, which is good, right? It shows the relevance.
00:09:49.600
But I completely missed the boat in terms of the URL that I'm pointing to. Another thing I could do here is drop in a title tag, which can increase my relevance. But what I really need is a smarter URL, and there are a couple of ways to solve this problem that are interesting to me.
00:10:10.800
One way is the idea that a lot of blogs use, especially in the Rails space: they use something called a slug or a permalink, which is nothing more than a smart title that's only created once the first time you create the object. Maybe it'll grab it from the title of the blog and drop that slug in there.
00:10:31.920
But the problem with this kind of approach is that you have to let go of some of the conventions of Rails, and every time you let go of the conventions, it takes you a little bit closer to hell, doesn't it?
00:10:54.720
Another problem with this approach is that I might misspell the name of my dog Spot, right? Instead of that, I might accidentally type Lou. I might not notice that until I see the page rank of my dog, and so when it's time to change, when it's time to improve the SEO, or when I notice a mistake, I can't have a commercial site with a dog named Lou.
00:11:18.440
So I need to correct that permalink. What happens to my PageRank? Well, I lose it all. Here, I don't really have enough information to maintain all those incoming links.
00:11:38.720
One thing I can do is use a fuller path and actually include some redundancy. In this case, I think the redundancy is okay. So I have the ID and then I tack on a title at the end of that. I actually do the lookup based on the ID, and the title is for Google, and that's not a bad solution.
00:12:04.080
I've actually seen this applied a couple of times. The solution that I like is to actually change the way that the object itself is parameterized because the Rails URL by default is the site name plus the pluralized form of the controller.
00:12:29.760
For my application, it's plants, and then there's a two-param call to that object. That two-param call, by default, returns an ID. Well, Ob Fernandez posted a really cool blog post on one way to handle this problem where you override that two-param. You have the ID, which is an integer, and then a hyphen, followed by whatever you want to put there.
00:13:02.880
In this case, I put either the common name or the botanical name. The cool thing about this is that when you call two-I on this, you're just going to get the ID, right? And that happens by default on the finders since we're coming from a string anyway.
00:13:44.080
This is going to work by default with just about everything you do, except for find by ID. If I'm doing something like find by ID, I have to tack the two-I on the end.
00:14:02.880
So this gets you most of the way there from a URL perspective. Now let's switch gears to the page title, not because it's a hard problem to solve, but because I think this can become too dogmatic, and we can overthink this.
00:14:21.440
What's interesting is that a page title really looks and smells like a view-type problem, but a lot of us would prefer to solve this problem in the controller. The reason is that there's this rule that's been hammered into our heads: you set instance variables in the controller for the consumption of the view.
00:14:54.400
To me, that doesn't smell quite right, so I'll break that rule. In the layout itself, which is really the problem, you want to set this data in the metadata that appears in the layout, not the typical body of the view.
00:15:18.560
In the layout, I just grab the instance variable, and if nothing's set there, I tack on "Dig the Dirt," which is the name of the site. In the view, I call a helper method called page title, and I can pass into that whatever I want it to.
00:15:49.680
The layout itself is where I set an instance variable. Again, if you're thinking about the standard Rails conventions, this might smell kind of wrong, but if you look at the usage pattern in the view, it looks pretty cool.
00:16:11.680
So the last thing I want to talk about here is the idea that images are a rich source for SEO because you can really crank up your keyword density this way. This is a good example of an early application where I really wasn't paying attention to SEO at all.
00:16:40.880
I have the picture model ID, and then I have something that I basically replaced the inbound user file name, which would have been richer for SEO. In order to keep from converting— to keep it normalized— I replaced it with user underscore and then whatever thumbnail size gets passed.
00:16:57.600
The Rails image tag picks up a default alternate. What I should be doing is using everything that I can to load that image tag and point it to something important.
00:17:18.560
So the alt tag should actually be the person's name, and it would probably be better for me to preserve the user's name. This way, I can get some keyword richness that I hadn't really anticipated before. The SRC tag is if I'm using a smart file name; that's going to work in my favor. I can pick up a title attribute and make that go away with CSS if I want to.
00:17:41.040
But that's also going to add to the keyword density and, if I want to, I can multiply the effects by adding a link to that. I would get the SRC, the alt tag, or the title tag on the link as well. So images can be an incredibly effective kicker for your SEO, and this is probably a mistake that I made way more often than I should have.
00:18:09.360
I would use AJAX pretty liberally in our user interface and completely lock off huge pieces of my application that Google could no longer find. There are two problems with coding a style of AJAX that doesn't degrade gracefully.
00:18:43.760
The first one is reachability. If Google can't see it, well, it's like that picture of a tree falling in a forest with nobody hearing it. You could have a beautiful application with nobody to use it. But the other problem is a little bit more subtle. If I use a lot of AJAX, it's not always easy for a user to build a link.
00:19:09.680
That link is how you're going to build your PageRank. So you have to do things explicitly to let users build links to specific content on your site. A couple of things I want to talk about before we move on to faceted navigation.
00:19:34.640
The first thing is a sitemap. With heavy AJAX applications, a sitemap really becomes a must, and really, it's just an XML file called sitemap.xml.
00:19:52.400
There's a very standard template for it, and Google and all the major search engines do accept it. Essentially, all you're doing is building an XML template; you dump in the URLs, priorities for those URLs, and timestamps of when they last changed.
00:20:14.960
As Rails developers, most of us have that information captured in the updated-at attribute anyway. At least Google can see everything, even though this approach won't pass the PageRank, and that's pretty important.
00:20:36.720
If you want people to be able to link to you, look to the Google Maps API. If you look all the way into the upper right, there's this button called link, and if you click it, it provides a non-AJAX way to reach this precise page.
00:21:00.320
That's pretty cool! This allows that feature, which is great for search engine optimization and great for users at the same time. And that's pretty much ideal.
00:21:20.960
So the last thing you can think about is building two versions of the content: one for your users and one for Google. Now, at first blush, this seems like a very radical thing to do, but this is precisely what a lot of us are doing.
00:21:44.080
As we've gone to the jQuery API or the more modern information implementations of the Prototype API, we start with a hard link and then code the regular implementation before layering on the AJAX so that this doesn't degrade gracefully.
00:22:06.400
That's two versions of the same content. Now, what we did for "Dig the Dirt" was that since we didn't really know how Google was going to handle our fast navigation, we wanted to be able to tweak it and see where Google would take it.
00:22:30.160
We had a link at the bottom of the page that said 'JavaScript Free Version,' and really what I was saying was, 'Google, click me, please!' In our faceted navigation, Google could kind of unwind through that, and we could see what Google was doing.
00:22:56.640
So this first section, what you want to remember is when you think 'relevance,' think smart URLs, page titles, and a single H1 tag. Make smart use of your pictures and links, and be thoughtful with your AJAX.
00:23:12.960
Okay, are you guys dying to know what this bullet is going to be about? Racist navigation? Has anybody heard the story, by the way? Can anybody guess what it's going to be about?
00:23:31.440
Okay, that's good! It's kind of an obscure one. But faceted navigation is basically about letting your user query data in multiple dimensions.
00:23:45.600
If you bought a pair of Nike shoes, then you've used faceted navigation. It would ask you, you know, which shoe size do you want? You could click on, you know, what sport you're playing, and you can do this in any order.
00:24:04.320
So I'm going to shift gears a little bit and show you what we're doing on Dig the Dirt. Oh, it's there.
00:24:25.760
I am going to turn off my mirroring, or I'm going to blow my own mind! Okay, so this is my faceted navigation. Here, the idea is that there are a whole lot of plant databases out there, but it's hard for people to find exactly what they're looking for.
00:24:42.640
I might want to say, and I'd appreciate it if you don't go click this because you will melt my server! I have one tiny little slice, right? You can melt my server after this part of the talk.
00:25:05.920
So I'm going to click on the hardiness zone, and we're in 8B here in Austin. Maybe since, you know, the deer are pretty thirsty, somebody's not following this.
00:25:36.240
So, since we're in Austin and the deer are pretty thirsty, maybe I want something that's deer-resistant, and maybe I want to look for things that are blooming red and yellow.
00:25:58.240
But the idea is, you know, as I click through this, these counts are changing, and my search results narrow as I go. We found that our facet menu was complex enough that if too much was open at one time, it confused the user.
00:26:24.720
So we just closed one menu automatically as another opens. You know, maybe I want something full sun, and I can also remove different elements of my search at any time; they all appear up here.
00:26:50.400
So, you know, maybe I'm not so interested in the deer-resistant stuff because, as we know in Austin, deer resistance is really a myth! So anyway, the ideas are that the user has multiple facets.
00:27:11.680
They can drill down to any facet in any order and limit the choices as they go. Normally, what you'd see with a simpler facet search is lists where things start to disappear, but in our particular implementation it doesn't make as much sense.
00:27:33.600
We do have some of the things down here disappear, like if I click this and then go to, you know, plant characteristics, certain things wouldn't show up because there aren't any plants in those categories anymore.
00:27:51.680
Okay, so that's faceted navigation, and now you get to find out why it's racist! Okay, where are you? There's my new book.
00:28:11.920
Okay, so I'm not seeing what you're seeing, so I've got to look over my shoulder, grab this guy, pull them down. And now we can restart. Sometimes my technology gets the better of me.
00:28:36.160
So there's a company called Pumpkin Labs that was asked to build a facet navigation site for great search engine performance for a furniture store, a small local store. They had a good amount of money to dedicate to marketing, so they decided to spend it this way.
00:29:00.240
So they wanted a system that would let you choose different kinds, different dimensions of furniture. For instance, you might be able to select your color, size, or type of furniture.
00:29:20.960
The idea is that as Google crawls this and starts to attach keywords to things, you can update your URL, H1, and your page title. You can come up with some interesting combinations, some of which you might plan in advance, like a leather sofa or a teen dresser that's red.
00:29:45.680
Some combinations you might never conceive, like a black baby bed and a white baby bed. These started to show up on Google, and everyone started calling the CEO of Pumpkin Labs saying, 'You have built me a racist website!' when that was never the intention at all.
00:30:17.360
But the point of this part of the talk is that you can get these interesting results with a simple facet grouping, and it’s incredible for Google because you start to get these combinations that you didn’t think of when you initially developed the site.
00:30:45.360
Now, this is what our implementation looks like. The hardest part about building this system is that some of the implementation is on Active Record, while some is backed by the session.
00:31:12.560
We didn’t want to go all the way out to Active Record every time somebody expanded an individual piece of the facet because that would be too expensive, and we couldn’t afford to put everything in session.
00:31:37.440
So we settled on an Active Record model for the individual plant. One of the common implementations with faceted navigation is to use an index search, like a Solr or a Ferret, or the hot one now is Sphinx.
00:31:55.920
I knew that I wanted my plants to be cross-linked from a variety of different places, so the tagging model was very important to me. I built a model where every attribute on the plant was represented as a tag.
00:32:23.680
So if something has a tag of 'full sun' in the context of sun exposure, when it’s time to write an article about full sun plants, I can grab a chunk of plants— a paginated list for all that rich cross-linking that all the search engines are going to love.
00:32:42.720
That's my model; it's a plant with a lot of tags. The plants are very skinny, and the tags are skinny as well. But when I'm building that facet of navigation and counting things, I can do so quickly.
00:33:06.400
Rather than hardcode the list of facets on the left-hand side, I let my admins build it. This looks exactly like you'd expect; on the left-hand side, there’s the name of this thing—maybe soil—which has soil pH and soil composition underneath it.
00:33:24.640
So it's really a grouping. Then there's soil pH, which is a multi-value field with values of acidic, neutral, and alkaline. Then there’s edible, which is just a boolean with yes or no.
00:33:45.760
So there's an Active Record model with all the different kinds of facets. I do this with single table inheritance, and it seems to work pretty well. Now, the facet implementation is a little trickier.
00:34:11.680
Where does this kind of code live? Well, the logical place for user interface code to live is either in the view or in the helper. But you don’t want to drop this much code in the helper.
00:34:31.380
So what I actually did was build an object model for this, which is located in the helper directory. It’s consumed by a thin wrapper in the helper that basically says 'render the facet free.'
00:34:52.880
So, this implementation is backed by the search, which is an Active Record, and the session, which includes things like which particular node is expanded.
00:35:10.120
There’s really not a lot of behavior; it's mostly just rendering, with things that support the rendering process, such as expanding and contracting, which saves us.
00:35:36.560
So, I wanted Google to capture the state. I let the URL parameters carry the state, which allows Google access to that.
00:36:01.040
Now, the downside of this is no matter what I did to that facet tree, Google was absolutely fascinated with it and would get into that facet tree and never get out.
00:36:31.360
There were too many combinations; it was too rich. You could have one version that’s full sun and then tack on blooms in the early summer. Changing those parameters takes you to the same place, but Google has to try all of them to tell.
00:36:58.240
Sadly, after all that work, we ended up turning it off. We think that the facet navigation is still part of the secret sauce of the business because you have to be able to search this plant database.
00:37:25.960
Right now, we’ve only got about four thousand plants; by the end of the year, we expect to have eight to ten thousand. But even then, that's a lot of different plants and a lot of different circumstances. Our database is adaptive.
00:37:46.560
So, you might have the experience that if African daisies are in your yard, being from Canada, you know they don’t grow well unless they’re in full sun. A person from Mexico might note that African daisies can’t stand full sun and could use partial shade.
00:38:05.760
As our database adapts, our facets will play nice with that kind of model. What we do is we expose those individual searches for things that we think we might need to optimize for search engines.
00:38:29.440
For real estate, you might have a safe search that's, you know, 'Lakefront Austin Homes.' You can grab a faceted search of that and drop the URL into place for hot tags.
00:38:46.960
We find that we’re going to start experimenting with user-saved searches because we believe our users want to be able to save those searches. We think that if they do, we could specifically target those for SEO.
00:39:05.440
That will help our PageRank precisely in the places we want to, and as we grab those, that will also reap benefits for the plants that show up in those searches.
00:39:16.800
So, my mom says disable the fasteners for complex sites because it can lead to random results. The last part of the talk is about smashing mirrors. This is essentially about controlling duplication.
00:39:32.800
Duplication is horrible for websites because it dilutes the juice. If you have four pages that are essentially the same thing, you'd like to have one strong search result rather than four weak ones further down.
00:39:53.680
You want one page with all the juice poured into that single page! The last couple of months, I've been really trying to smash the mirrors.
00:40:13.760
So I'll go into what search engine tools you guys use. Does anybody use analytics? Webmaster tools? Just a couple? Now, this is something you have to go get.
00:40:28.520
This is really the developer end of search engines, focusing on HTML suggestions. I've been watching this first purple link for a long time. In this case, all 479 of those are from an earlier crawl and problems that have already been fixed.
00:40:49.440
Once it comes around again, it'll find other mistakes that I've made, and there will be more, right? But I find that one of the most valuable things I can do is control the duplication. This focuses Google on the pages I want crawled.
00:41:12.080
Second, it should give me better juice in the long term. Duplication can show up in surprising places. One area I’ve been thinking about is a comment or a rating.
00:41:29.480
On this particular page, this description stuff is what I really want to optimize for. I want a rich chunk of text that repeats some important text, in this case, the Sequoia tree is repeated twice in H2s.
00:41:51.480
But what's going to happen if I let Google crawl this, and this list of comments gets longer due to pagination? I'll start duplicating this. Does that make sense? If there's 17 pages of comments, this plant page is replicated 17 times, and that would be fatal for the SEO for this application.
00:42:12.720
So, what I’m watching closely is the ratings. There are a couple of solutions, none of which are perfect, but some of which are promising. The one I like the best is actually rendering all your comments and using JavaScript to show and hide the various chunks.
00:42:34.560
We had a search engine expert tell us this is not going to get us busted for cloaking. Cloaking means that you're showing Google one thing and rendering something else. This is about helping Google with one particular fetch.
00:42:56.320
It's great for SEO because I have one copy of the thing, and I also get the richness of all those comments fed in as well. The next thing that I can do is use AJAX, and if I wanted to, I could create a sitemap for the individual comments, having a separate interface to the comments.
00:43:21.120
But maybe the content of the comments isn't that important to your SEO; it depends on the types of comments you're receiving. If you're getting simple comments, it might not matter as much. But if your AJAX is not accessible and doesn't degrade gracefully, this can be a good technique to keep Google out of where you want it.
00:43:41.840
One thing you can do is say, 'We're not getting much juice down here at this level anyway. I'll bite that bullet but don’t want the duplication.' I want to make sure that Google knows these are different pages.
00:44:05.360
So, with the one without a page number being shorter, the keyword density will be higher, which should help that result show up first. The solution we use is a robots meta tag with 'follow,' which means go ahead and crawl these pages and pass the juice through to the individual pages.
00:44:24.640
Google is still crawling our sidebars and passing the juice to the places that we want them, but we don't want to include these extra pages in the index. This is what that looks like.
00:44:44.720
In my layout, if there’s a page parameter passed in greater than one, I add a robots meta tag with 'no index, follow'.
00:45:02.560
The last couple of things I want to talk about: you know the two-param solution I showed you will lead to duplication if you change the spelling, right? So my example was when I misspelled the name of my dog.
00:45:24.240
Here’s a real-world example: I am a hardcore dyslexic and have written 12 books, but my wife had to edit everything. I actually misspelled Impatiens; I spelled it impatient.
00:45:44.880
I’m also not a hardcore gardener, so that's a bad combination! I found duplication because when I changed the name, Google kept coming back to the first one, even though the definitive version didn't exist anymore.
00:46:09.600
You solve that with a simple redirect, saying okay unless the brand ID matches the plant to the parameter, you're going to redirect back to the plant, and the problem is solved.
00:46:31.120
Normally, I would one-line this, but for presentation purposes, I broke this out like that. I know that we're short on time, and I want to save time for questions.
00:46:50.160
But first, let me leave you with some final thoughts: if you're ever driving in Memphis, watch this lady.
00:47:09.760
Any questions or comments? Yeah, what’s that?
00:47:29.120
I don't know the answer to that. If you're trying to game the search engine, I would be very careful. If you're doing it for usability, I would probably get another opinion on that. I'd definitely ask a search engine expert. Other questions or comments?
00:47:37.280
Okay, thank you!
