Dan Sharp is Co-Founder of Screaming Frog.
Dan Sharp has created one of the most widely used SEO tools with his brother Liam, and his business partners, Graeme Radford and Jason Lock.
The SEO spider tool is synonymous with SEO, and Screaming Frog is also a PPC and SEO agency with a staff of just under 40 people in Henley-on-Thames in the UK.
Outside of SEO, Dan loves sport, in particular football, and is a gooner (Arsenal fan). He is also a fan of music, plays drums and has been learning guitar for a couple of years.
This post is part of a series called ‘Conversations in Search’. I discuss the current state of SEO practice with other SEO experts and discover their views on the future of SEO.
Damien: Tell us a little bit about your background, Dan? How did you get into SEO?
Dan: Well, like everybody else I just felt into it, you know, and that was it.
I went to uni, Bristol UWE. I had a great time in Bristol and did a marketing degree for four years and then didn’t want to get straight into marketing, you know? I didn’t know what I wanted to do at the end of it, and I did the usual thing, I went travelling as everybody did.
I was travelling for a year and came back, was desperate for money, and got hired to work for a company called Guava. Which was an agency that Graeme Radford owned, he is one of the founders of Screaming Frog, so Graeme hired me at Guava to work on the team.
I started initially by doing PPC. I started a blog and I used to write about PPC and do loads of PPC experiments and affiliate work in my own time and got really into it. After a year and a half or something like that, I started understanding and learnt more about SEO, and this was 2004-2005.
I set up my website and started getting it [SEO]. We also started doing more SEO at Guava, and taking on more SEO clients, it became a natural fit, and I became more technical and headed up the SEO, and so forth. SEO became a bit of an obsession for me and I spent most of my time in evenings, nights and weekends learning, experimenting and generally being a massive nerd.
Damien: So you created your agency, Screaming Frog, after leaving Guava?
Dan: Yes, I left Guava, and initially went freelance, under the name Screaming Frog and I had a few clients that kept me super busy. While freelance, my brother Liam and I started developing the SEO Spider tool that everybody calls Screaming Frog. I had this idea for the tool already as I’d used Xenu in-house at Guava and we had an internal toolset as well, at the time.
It did some SEO checks via a web browser rather than an application, but I love desktop applications. I love how flexible, how granular they are and help to dig into the data, and I can do things quickly. That was the approach we wanted to take.
So something that I wanted to do was to build a tool to help me do SEO for clients, automating my processes to be more efficient. It wasn’t for anyone else at the time, and I didn’t have any plans to licence it or anything. It was more ‘Can we build this, and create something better for us?’. For example, Xenu didn’t even pick up canonicals back in its day! It was awesome, but quite limited, and SEO was evolving.
By that point, Graeme had sold Guava. I met up with him and Jason Lock who was the PPC Director at Guava, we quickly got into a conversation of ‘Should we set up a new agency, should we do something?’.
We were working separately, doing our things, I’d already come up with a name Screaming Frog, Graeme and Jason were like ‘That’s a good name’ after everyone else had laughed at me, and that’s how the agency started.
Damien: Dan, we’re talking amid the Coronavirus lockdown in the UK, how are you finding it, how’s business, and how are your team?
Dan: It’s completely changed the way everybody’s working and their lives and everything. Before this, lots of companies were thinking ‘I wonder whether we could work remotely, I wonder whether our company can be completely remote’?
So, in a forced way, we’re finding out how that works. I’ve been pretty impressed. The team is really on it, and they’re outstanding at the moment, I’m genuinely impressed with them.
Our business is strong, and we’re fortunate to be able to work from home, so we’ve got no furloughed staff; thankfully we haven’t got to that level. We have lost a few clients, while some others have had to pause and freeze their budgets. We’re trying to help our clients refocus to get through this period, and on the longer term objective. Obviously, if they’re having cash flow problems, it makes seeing the longer-term more difficult.
There are concerns across our industry. I speak to a few agency owners, and I know a few people in our industry, I do know that some have had layoffs, and some staff furloughed. Agencies are losing some clients and generating new business is more complicated. It’s gonna be a tough time for all agencies, as it is for a lot of companies.
Companies in travel, they’re not making any money, they can’t sell anything. Or in retail, they’ve shut their shops, or they’ve been indirectly affected. If you look at the automotive sector, nobody’s making cars because it relies on people being close together to be able to build them.
For example, we have a client who does event badges, passes and things like that. They also do the key cards that you get for hotels. Given the downturn to hotels and events, it’s no surprise they’re finding it tough. You might think for lots of businesses ‘Oh they should be alright’ and then you think about it, and you go, ‘Oh, no, wait there’, it impacts everyone.
Damien: Given the market unpredictability and upheaval to many businesses, how do you keep a positive outlook?
Dan: It’s a tough one because there’s a lot of negativity at the moment. There’s a lot of small wins to focus the mind at the same time.
For our clients, we’re looking for those things you could do with a campaign to have an impact now. The chances are you would have already done most things for immediate effect if it was that obvious or easy though.
There are some things, perhaps there’s some content campaigns that we were going to create where we are like, ‘Well, you know, we’re not going to be able to outreach this in the next couple of months because of the climate’. So, we put those things on pause to focus on, for example, content that might resonate now with the behaviour changes we are seeing, or a CRO project or whatever that may be. We’re working with clients to move items around and re-prioritise.
For our team, there are just under 40 of us now, we’re doing fun things, like pop quizzes, you know, keeping up sort of positivity. Everyone is on Slack. Every Friday someone is the quizmaster and we make time to have chats on video conferencing and not just meetings. We’re still having a beer or a soft drink at four o’clock on a Friday.
We’re just doing little things to kind of keep us feeling as a team and keeping us positive as well as talking more to clients and trying to keep them positive too and keep things on track.
Damien: Tell us more about the change of prioritisation and focus?
Dan: It’s different for every client. We’ve seen their existing campaigns or other campaigns for our clients not perform as well as we’d like, or we just know that it won’t work because it’s obvious or whatever it might be. We know that we need to review strategy during this somewhat unprecedented time for clients anyway.
Our team has just been reviewing what they’re doing and what we had planned, we generally plan three, six months in advance for all sorts of campaigns.
So we just had a look at what we have planned, and said, ‘Does this make sense? Do we need, we need to change things around?’ Then we’ve had a chat with the client and made adjustments.
So what are we doing? We’re looking at ways where we can make more of an impact for clients. It might make sense to focus on doing keyword research, building out new landing pages or adjusting page alignments.
On the technical stuff, we have seen some progress on things that were on hold. Where the client previously said, ‘Look, we’re not gonna do this realistically for six months’, they’re now saying ‘We’ve got a bit more resource to look at doing some of that’. We see some movement there which is nice.
Damien: What are some of those pre-planned technical things you’re now seeing movement on, Dan?
Dan: It just depends on the client; it could be any technical elements. Some clients do put forward virtually any technical fix if you will that they haven’t previously been able to get done.
Now that people are not buying, they have a little bit more resource free, but actually, they still have developers. They can get stuff done in what would otherwise be a bit of downtime. There’s not necessarily a specific tactical response, like ‘Let’s add structured data for everyone’.
Damien: Do you see a divide in SEO skill sets, technical, content and promotion-focused SEO?
Dan: Indeed, there’s far more division between very technical SEOs, content/PR and SEO project managers. The skill sets in SEO are so vast. I think there’s been way more focus on technical SEO over the past few years. There have been some exciting changes. Then you have people who do link building who are generally creative.
It’s been quite impressive over the last year as you see many SEOs on Twitter talking about moving into almost development, they’re talking about NLP, machine learning and going mad for educating themselves to code using Python.
You see every day SEOs building out their models, doing machine learning, trying to integrate that, and using it for keyword mapping, categorisation of keywords and classification of keyword search intents or contexts.
The effect of this is they’re building efficiency into the work that they do, using more dev skills which are fresh, because if you looked back even five years ago, you really wouldn’t have seen as much of that.
In terms of technical SEO, it focused more over the last year on mobile. Google’s mobile-first indexing has helped to push site owners to build responsive formats. With our clients there’s been a big focus on mobile-first, mobile page speed, making things faster, better security too.
People are also paying far more attention to structured data. Google has been releasing its search features that help you achieve more prominent spaces in the SERP.
SEOs are big into at the moment ‘on-SERP SEO’ and standing out from the crowd as a benefit to structuring your content and site architecture, and even if you choose to apply structured data.
Damien: What are your thoughts on the current state of the SEO practices?
Dan: I think SEOs are doing, on the whole, exciting work and they’re creating great content, and PR, they’re getting it — they’re far more integrated.
I’m seeing more traditional PR agencies or PR teams getting it, and links make a difference. However, I would say since the Coronavirus, unfortunately, I do think that it’s going to take a bit of a toll on the industry as well as most industries!
Damien: Yes, the world over industries are going to, unfortunately, have failures and need to regrow and get back to base again.
Dan: These are challenges everybody’s facing at the moment with Coronavirus. What’s the first thing that people do when there are money problems? It’s to cut back on marketing spend. I know that we’ve had some clients come to us and say, ‘We don’t want to cut the SEO budget, we know how important it is, but we have no choice’.
You know, everybody needs to get back to where they were, this [COVID-19] might go on for six months, there may as economists are forecasting a recessionary period. It’d be optimistic if this went on for three months and then everything came back to normal, but I suspect that won’t be the case.
Pre to this happening I would have said we were in a period of growth. There’s a lot of agencies trying to sell agency offerings as something different in the way they sell and do SEO. It’s fun seeing various SEO agencies growing in the industry, and there’s great camaraderie.
Damien: If you had to choose between developing Screaming Frog the tool, or Screaming Frog the SEO agency, what do you choose?
Dan: Oh, that’s so difficult to split between them! I was doing the tool pre to the agency. But what makes the agency are the people, isn’t it, really? Screaming Frog is ten years old this year. So it’s the many people who have worked at Screaming Frog over the years that make it great. I can’t choose, but it’s all about people.
Damien: Changing gear, what are the common challenges clients have that you see when it comes to implementing SEO recommendations that you make?
Dan: That’s an easy answer, as the most common one is development resources. Then it’d be a mix of budget and internal support to make the changes. Those are, and have been the most significant challenges to getting SEO done. Often the client has scarce developer resources, and they’ve got loads of other things that they need to prioritise.
For clients who understand SEO, where they have an influential in-house person who’s pushing and owning recommendations from the client-side, it gets done.
When we see clients having difficulty to get things done, we ask the client ‘Hey can we talk to the developers? Can we help make a case for this?’. Sometimes that works, but I would say obviously it’s a challenge. The biggest problem is getting technical recommendations done.
Let’s not forget that obviously, there are some genuine reasons why people can’t get stuff done. It might be that they’ve got a legacy website. It’s just not possible. They got a redesign coming up, and they say it’s going to be six months, but turns out to be two years!
The other side is we see a lot of companies do not get content and PR driven link building still. It’s a weird one, and they get that links are essential, but we see some of the biggest companies in the most competitive verticals where they don’t have anybody who understands the value of links.
It’s the basic things, often they don’t have an owner or focus. For example, unlinked brand mentions often fall between the cracks of responsibility.
They don’t tend to carry out content or PR campaigns around their products and services from a linking perspective, and it more rests on hope. They just assume that their PR agency or the marketing team will get those links. The PR and marketing team generally don’t have that focus. Some PR agencies do, some PR agencies are almost like SEO agencies these days without the technical SEO skill sets to do an on-site audit or anything like that.
There’s an overlap between PR and SEO. I think the PR agencies have been in a great place to do link building for brands, but I would just say that I believe SEOs are great at evolving. They’re so flexible moving with the times and probably are doing a much better job of building content-driven links while focusing on the PR value in the content activity anyway.
Damien: SEO is multidisciplinary, bridging gaps between many different teams, are you seeing changes that suggest there’s deeper integration?
Dan: I’m not sure it’s changed loads. If I look at the longer term, over the last five years, I would say there have been some changes. A decade ago SEO was far more siloed. Often SEO agencies were never even asked the questions like:
- Do you have a PR agency?
- What are you doing with content?
- Do you have a content calendar?
- What are you planning?
- Do you have spokespeople?
- What other marketing activity are you doing?
- Do you have products to give away?
- Does anybody do thought leadership?
I think there’s going to be some significant changes, it’s going to be an evolution rather than revolution, not a sudden dramatic change, but you can see changes already.
Sometimes at the moment with PR and content campaigns, there’s too much focus on getting coverage from the kind of ‘general’ mainstream media who will link out to lots of different sites on any subject. Relevancy already does and will be more critical than just mainstream media coverage.
Google has a better understanding of relevancy, experts in topic areas and entities than most SEOs think. Entities that are synonymous with subjects, for example, will help sites to rank well consistently — if they invest in processes to create and surface the coverage of those topics.
I think there are some other changes, we’ve been talking about voice search and visual search for a while, and they have a lot more to come.
Damien: These different modes of search introduce a paradigm shift for search, how do you see that influencing the future of SEO?
Dan: How it changes search behaviour and how it changes audience behaviours and what the opportunities are for that I don’t know.
Things like voice search or video there will be changes to the things you optimise. With the shift to mobile, for example, people have been witnessing for years, you’ve had to rethink things like content layouts radically.
I see growth in ‘visual search’ will be very interesting, and search intent has become an increasing focus. Google has, for a long time, added contexts to the documents they index and want to rank pages with the correct or as best they can assess intent for queries. The level of ambiguity between language and relationships to concepts, topics and entities will continue to improve.
Damien: If you were to advise small business owners about SEO, what should they do now to improve their SEO visibility?
Dan: For a small business, immediately I think they’re not going to have huge budgets, it’s probably the small changes that are going to make the most impact for them.
Small business, I also immediately think ‘local’ and so local SEO strategy is really important, and targeting local based queries, with local landing pages etc. Setting up a Google My Business account, and their Search and Webmaster consoles with Google and Bing will give them insights.
As a local business, let’s be honest it’s probably the fundamentals of keyword research, page alignment, all that type of stuff that will go a long way for them. Small business is rooted in the communities they serve, so they need to deliver the best possible product and service to their customers. They should be actively inviting reviews from their customers.
I guess after that, you could move to link building, but you’re probably looking at solving their problems that their users have by creating helpful content that addresses user needs.
It depends on the local business, but they might be able to create guides that will gain rank for relevant long-tail queries and have them seen as an expert, people will link to that content naturally. Keep an eye out for what experts are doing, and you can pick those topics and become the subject matter expert for that thing.
Damien: What have I not asked you, anything about the future of search you’d like to talk to?
Dan: There’s a general move to ‘web user experience’ with SEO far more connected to other disciplines. Everything impacts SEO, right? It’s beyond just content and links.
What you do as a brand and how good your product is will have a bearing on your SEO performance. If your product is weak, you’re going to struggle to maintain rank. Everything needs to line up today, and if it doesn’t, you won’t rank for very long.
Let’s say even if you had a great brand, a great product, you managed to rank on the first page, but your website sucks, and the site has a crappy user experience, and everybody bounces from your website, you’re not going to rank on the first page for very long.
Google will soon see from the signals that they monitor and drop your ranks. Everything needs to line up, which is pretty cool. So there’s a holistic move toward an enriched experience for users.
We’ve seen it with businesses who’ve got great products, but they got no brand. If you’ve got no brand, then it’s not going to work.
Then you’ve got businesses who have a tremendously fresh brand, they bring out a new product, but their new product is not that great.
They kind of do okay, but you know they struggle, everything needs to line up today because ultimately Google has a much better understanding about what people want to see and your users have never had more control.
Damien: You’re terrific, Dan, thank you so much for your time.
I’m Damien Anderson, the founder of echwa. My goal is to help website owners get the best out of their search optimisation, and help grow their businesses.