If you read my recent case study that looked into how I analyse the content that I write, you will have probably heard of the URL Profiler tool.
To accompany the case study, I’ve recorded a full video tutorial on how to use the tool, including how to configure all the settings for optimal results, how to import URLs and use the tool for content analysis, link prospecting, link auditing and much more.
As well as this, I’ve got an exclusive 10% discount on URL Profiler for all the readers of my blog. Share the post below to unlock your discount coupon:
Video Transcription
Hi. This is Matthew Barby, and I’m going to show you how to use the URL Profiler tool. The URL Profiler tool is a nice bit of software that you may have seen in a recent post I did. This was all focused around analyzing the content on your site, also being to analyze content from other websites you can then use to find trends, whether it’s based on links, based on social shares just looking at the specific factors of the content, your Google Analytics, or a whole host of different things.
What I’m going to do is I’m going to talk you through actually setting up the URL Profiler tool so that you can actually have everything running in the way that you want it to. Then I’ll go on to showing you the kind of set up that I have within mine. Once you’ve installed URL Profiler, the first thing that you need to go into is the settings tab. What you’ll have here is the connections, and really you’re going to have to probably play around with this. It all depends on the processing power of your PC.
If you’ve got a really high-end machine, then you may want to look at the connections of 30 upwards, but I probably wouldn’t go much further than that. If you’re running this on more of a basic system, I’d probably bring that down to ten. Essentially, the more connections that you can run at the same time, the quicker the tool will run. You’ve then got the trade of whether that’s just going to crash your PC. or your Mac.
I’d probably start by testing it out with just, say, something like five or ten connections, and then scale it up if it works okay. Generally, if you’re working on a smaller set of URLs, you should be okay with higher connections. When you’re working on big data sets, that’s when you’re probably going to start seeing issues come in. The connection timer, you can just leave this at ten seconds. I wouldn’t worry too much about that. Link analysis, again, I would leave these at the default settings of zero. This will just speed everything up a lot more. Don’t need to go into the details too much here.
Link metrics, this actually a really nice new feature of the tool of the recent updates that’s being released. Before, what was being gathered is loads and loads of URL data, every single kind of URL data that you can possibly get, from Majestic, from Moz, Ahrefs. What you now have the opportunity to do is bring this down to a basic level. Depending on the type of analysis you want to do, you can still have all the available link metrics. If you’re just looking at this for some back link analysis, the basic settings will be more than enough, likewise, with Moz and also Ahrefs.
Then going into the social, again, you can choose which social media platforms you want to gather stats for. Delicious, I don’t care at all about so I just leave that out. In some circumstances, I won’t care if I stumble upon Reddits, even in some Pinterest for some specific piece of content, I may even only look at the likes of Facebook or Twitter or LinkedIn and Google Plus. What I would say is, tailor this to the specific analysis that you’re doing, and you can just make sure that you’re not gathering stats on things when you don’t need them. Because it really does effect the actually processing time of URL Profiler.
uClassify, this is if you want to gather some extra information around each URL If you want to have a little look at uClassify’s stats around the topics, tonality, language, sentiment, mood, age, this is all the analysis that U Classify offers into content. Personally, for most of the projects that I do within URL Profiler, I don’t use many of these features, but they are really good to have. If you are doing some specific profiling around the tone of content that’s working more specifics into the types of language used, the sentiment involved within each of your articles, this can be a really invaluable piece of information that you can gather.
The final one is proxies. If you do use any private proxies then you can load those in. I’ve only loaded in about ten private proxies here. What I would just do is just select it for when you’re doing a Google index check. You don’t really need to do it on social shares. If some websites you’re scrapping through a load of URLs and a load of content off of them, you may find that they will block off your IP In some circumstances, you can use your proxies on the URL scrapping. You don’t need to have proxies. Ideally, if you were going to use proxies, you probably want a bit more than ten just loading up a few for test processes, really. Another great feature just to stop any problems coming up when you’re scrapping through all the information. I’ll close this down.
The next thing you want to look at is the accounts. One thing to note with URL Profiler is it gathers all of these stats from a whole host of different data sources: Majestic SEO , Mozscape, Ahrefs, uClassify, Share Count API, Google, from analytics and also from scrapping the [inaudible 00:06:03], Copyscape, My-Addr. All of this information… this isn’t bundled into URL Profiler. You have to access to these tools to able to use them. Now, you don’t need to have access to Majestic SEO , Moz and Ahrefs to be able to get linked data. You could have access just to Moz or just Ahrefs, and you’ll still be able to get information.
Now, I just have a basic Majestic SEO , I think it’s the silver account, which is about £35 a month. That’s just off the top of my head. I have a Moz Pro account, but I don’t have a pro account on Ahrefs, so I don’t use that. All you need to do is go through, if you’ve got Majestic SEO , then you just press get access token. It will bring out the pop up window, will take you through to Majestic SEO where you’ll be able to grab your access token, paste it in there, and then you can link up directly to the API It’s really as simple as that. There loads of documentation on the website just in case you are having a bit trouble. Or you can get in touch with me directly, and I can point you in the right direction. It’s the exact same with Moz.
When you go through, you can have a free Moz account and actually have the basic metrics. All you need to do is sign up for a free account on Moz. Or if you’ve got a pro account, you get slightly more metrics. You can go in and the same as Majestic, you can create… well, you’ve got you own access I.D., you just need to create a secret key and then move on from there. Really simple to do, again. Likewise, with Ahrefs, same process, just press get access token. If you go through here, it will open up a Ahrefs and you can go in and do stuff directly there.
The next one is Google. You’ll want to do this. Again, free to do with the Google A.P.I. key. Go through, just click get A.P.I. key, and you’ll come through to the developers console within Google. You can follow through all of the details which are all on the URL Profiler website to be able to set up a free A.P.I. key. It really takes no time at all. I’m not going to run through all of that, because literally you just click here, and you’ve got all the access and all the details step by step on how to create your API keys. I’m not going to waste time going through all of that.
My-Addr, again, this is another things where you can gather information on page rank, Alexa and Whois data. You just sign up. You can add a few credits, that’s what I did. Then you can just add your log in details there. Copyscape, I haven’t used here. If you do have Copyscape information or a Copyscape account, add that in, you can get even more data. The same with uClassify. As I mentioned, I didn’t really use uClassify. If you do, again, I have the information in here and you can select which information you want to gather. Once you’ve got everything set up, that’s all you need to do. You won’t need to do that every time, that’s just a one off thing. You’re already to go.
The next thing you’re going to want to do is sort out adding some URLs to gather information around. Don’t worry about this dashboard area, we’ll go through that in a second. One of the great things about URL Profiler is if you come over into the pane over here where you’ve URL, you can right click. You can either paste a load of URLs from the clipboard. If you’ve had a load of URLs, maybe, in a .txt file or something like that and you just want to paste a few in, or you’ve just grabbed a few and you want to plug straight into the pane here without having to create a file, straight away, you can just paste that in from the clipboard. If I was to just go find my findmyblogway.com, copy that, paste from clipboard. Simple. I’m just going to remove that, delete all URLs.
Alternatively, import from file, so it can go through here. I can import a list of URLs from a.csv or .txt file. One thing I really like is imports from the X.ML site map. So, you notice here, I’ve got the site map open on Find My Blog Way that just has my articles within it. I’m not just plugging in the domain. You’re going to have all of the pagination pages, you have the home page obviously. On top of that, you’ll have random about pages, contact pages, all of the information that, actually, I don’t care about finding the information on. I can just plug that straight in. press import. What URL Profiler will do is pull in all of the URLs from that site map, so really, really handy actually. They’re all plugged in, ready to go.
However every you want to do that. This is just 88 URLs. I’ve actually ran 80,000 URLs through URL Profiler. It’s took a fair amount of time. However, it did complete and gave me a huge insight. I’ll actually be doing a case study on that in the next few weeks. Make sure you check back onto the blog for that. Now you’ve got all your URLs in there. You want to start having a look at what information you’re gathering. If you’re just doing it from one single domain, so you’ve got here just a list, for example, if I’m just looking at Find My Blog Way doing a content analysis of my own, what I would say is don’t worry about gathering all the domain levels stats. What it will do is it will gather domain level statistics for every single one of these URLs., but will only be gathering information on the domain.
You’ll get linked data based on the Find My Blog Way domain 88 times. It’s just a waste of time gathering all that information. What you may as well do is two separate runs and just put the homepage in there and then just do one run on the domain level. It’ll be exactly the same every single time. On the URL data, what you’ll want to do here, and it’s also worth noting actually, if you do have multiple different domains, maybe you’re gathering information around a prospect list that’s loads of different domains, that’s when it’s probably a bit more relevant to to do this domain level data analysis.
On the URL level, this is where you get the real gold in terms of analysis. What I run, because I have Majestic SEO and Moz, I don’t have Ahrefs, I will gather Majestic SEO data and Moz data. What I’ll get there is with the basic data, you’ll get, if we go into the link metrics, you get citation flow, trust flow, external links and referring domains. If you went all available link metrics, which I’ll show you in an example in a moment, you’ll have referring dot gov domains, referring dot edu domains, all the metrics that you’d look at. The same with Moz, domain authority, page authority, Moz rank, Moz trust, external links. You’ll then also have a whole host of different things, linking C blocks, linking IP addresses, all sorts of stuff, similarly with Ahrefs.
You’ll get all of that data for every single one of these URLs. Social shares, what I did within this analysis when I was doing analysis with my content, just want to double check that none of it was blocked in the Robot’s TXT. You don’t need to have that, that was just something I had in this analysis. If you want to check that each of these URLs have been indexed in Google then, like what I mentioned earlier, it’s recommended you use proxies to do this. I’m just going to press no quickly here. Then you just take that and you can check if each URL’s been indexed. You can gather Google Anayltics data if you’ve set up analytics, which I have done.
You’ll bring up this box here. I’ve already set up all of the information but it’s exactly the same and all of the documentation shows you how to set up. Simply press connect, and then I can go through… I can gather all the different information on Find My Blog Way from specific dates for each of these URLs. So, I can get information on the bounce rate, I can get information on the total time on the page, any conversion metrics, all kinds of things. That’s when you can starting doing cross comparisons with social shares, content length, headline length and all kinds of different things that you can compare against these base metrics. It’s up to you if you want to have that information in. Obviously, you could only use that if you own the domain. It’s only going to be relevant at one point or another.
HTTP Status. If you want to check to make sure none of these are going to a 404 or a 500 error, if you’re doing your own domain, I would imagine you might want to check that out. Page feed, another great bit of insight that you can get into each of your pages so you can do some correlation between things like social shares versus page load time to see if more bloated pages are actually costing you on social shares on runs and having multivariate testing against that. Likewise, you can bring in page rank data. For me, I don’t really see a huge amount value in doing that, but depends on the analysis you’re doing. A particularly good feature here is e-mail addresses.
If you’ve got a big prospect list there, what URL Profiler will do is to… if you use things like BuzzStream where they’ll harvest email addresses or Matthew Woodward’s ranking tool, his tool will actually go in… Rank Cracker it’s called, that’s what it is. It will actually find any e-mail addresses that are within the domains that you’re trying to find. You can use it as a… not just a content analysis tool, but also as a full on prospecting tool. Then, when you go down to the content analysis you can have readability scores, flush readability. I can’t remember the names of the other ones but I’ll show you those shortly. Probably one of my favourite settings here, that I haven’t seen anywhere else, is the content CSS selector.
Now, what you can do is if you’re doing readability, you might be thinking, “Well, how is it going to know which area is the bit of content that I’m trying to focus on?” For example, if we load up all the URLs here, if we’re looking at doing some content analysis … If I’m just loading in the URL, it’s going to be looking at all of the HTML. It’ll take in things like all this header area, it’ll take in the sidebar, it’ll take in the comments and all the extra information that actually isn’t a valid representation of the actual content of the page. What you can do is you can go in and just open up the developer tools and find which div is actually the main content area. You can see here the dot entry dash content area, class equals entry dash content.
That is the bit of CSS that’s wrapped around the main content of my URL. What I can go and do is go in and open up URL Profiler again, put in class dot entry dash content. For the content analysis, it will only look in that bit of CSS, making it as accurate as possible, which is great. Finally, this is the back link checker. If I were to put in here findmyblogway.com, what I can do is do a full link analysis of the site so I can find out a breakdown of the different anchor text that’s used for being linked to the domain. What I can also do here is type in some anchor text variations. If you’ve ever used the link detox tool, which allows you to classify loads of your back link data so that you can find out what spam-y links are pointing to you or toxic links or suspicious links.
I can type in here Find My Blog Way, Matthew Barby, Find My Blog Way, and it will classify all of these anchors as being branded [inaudible 00:19:48] commercial. It could be digital marketing blog, blogging tips, things like that. Then you could type in some variations for generic anchor texts. You then press apply, I’m not going to do it in this amount time. If I’ve got a disavow list, I can then paste in all of the disavow domains so that they’re being taken into consideration within the backlink analysis. On top of that, you can also create a white list of domains that you know are good in [inaudible 00:20:23]. If they come up, they’re instantly marked as okay.
On the other side of things, there’s a black list. You may want to go …yYou can actually grab… I think actually the link detox tool, the guys over there, they actually put together a domain black list. Also, the guys at Virante that do Remove’em, they’ve actually made a big black list that you could easily paste into here that will flag will flag up any potential dodgy domains. Then, all that’s left to do it for you to press run profiler. What the tool will start to do is it will start to gather some information all around. What I’ll do here… do not worry about this. All I have to do is save this into a specific area, so I’m just going to go into the desktop for now, test run, save. What it will start to do is it will start to do its thing.
It will start to import all the URLs, check all of the domains, go through and gather different information, and you’ll see it’s really speeding through all these URLs, runs really quickly. I’m just going to cancel this for now. You see it already running onto the URL social metrics. It will go through step by step, going through each of these. I’ll cancel that for now. For my… laptop has a heart attack. What I’ll show you is an output that I did around a few weeks ago now. You’ve probably seen the blog post I did and there was everything that’s terrible about my content that pulled in some findings from this information. Just from this… this is just the information that I got without expanding on it at all. We’ve got things like a breakdown of the URLs, we’ve got HTTP Status. Bear in mind, I’ve disabled loads of the settings here.
We’ve got the page title, the title length, the descriptions, top words used, sentiment, we got the different readability scores. Alongside this, we’ve got the word count, sentences, number of headers, paragraphs. It automatically pulls in offers on the blog and their offer URL We’ve got all of the different statistics from Google Analytics for the past week, which is great. I’ve then got… this is all the extensive Majestic and Mozscape links. Just to show you, this is URL citation flow, trust flow, trust metric, external back links, referring domains, A.C. rankings next to URLs, tons and tons of information. Moving then onto Mozscape, domain authority, page authority, equity links, sub domain external links. It really goes on, and we’re only halfway through this data sheet now.
Then go into page deed score, which you can go down to see if each of these different URLs and the total requests and bits, the HTML. response, it came up with loads and loads of statistics. All the different social stats for each of these URLs across a whole host of different social networks. The [inaudible 00:23:54] headers, if robot spy is allowed or not allowed and then total social stats. You can start to see the kind of value that you can get from just doing a small bit of analysis on your own content. Then, taking this a step further, you can go in actually use the tool for prospect lists, you can do it for detoxing. Doing some kind of domain link analysis to check and update your disavow file, or do some contingency planning.
Work on some specific Google penalty account. You could have a look at content readability, look at just linking metrics and you could do that directly for harvesting e-mail addresses, Whois data, Wayback Machine information, the list really goes on. From what I’ve found in the tools so far, there’s constantly new features being added. Make sure you check out URL Profiler. I know that they offer a free 30-day trial. Then, there’s a full paid account that you can use for 99 pounds at the moment. I think some of the pricing may be changing over the coming few months. Make sure you give that a try. Take a look, also, at my case study that’s focused around the content analysis I did on Find My Blog Way and let me know if you have any issues or you find anything new that you can do on the tool. I’m always interested to find out new ways to use it.