Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
UK Affiliate programmes as an advertiser
Hi, I'm looking into setting up an affiliate programme (as an advertiser). The company offers commercial mortgages for a UK market only. I was wondering if you had any experience with any UK affiliate platforms and had any recommendations? I want to test the water with a platform first as we don't really know whether it will work for us or not. We're looking at it for lead generation. Potentially offering payment on a completed mortgage or a lower payment per lead. I think it may be worthwhile offering both options. I'll have to work out the model, but we can make it work nicely for the publisher, I think. Thanks, Amelia
Affiliate Marketing | | CommT0 -
What impact do affiliate links have on SEO
We commonly reference engineering standards on our site (because half our products are known and searched for by those standard names) and have just signed up as an affiliate to an organisation that sell the standards. Which seems a nice match We're going through and making sure all are no follow now because that seems the general advice - but should we be doing anything else? Yoast says you should mask affiliate links - https://yoast.com/affiliate-links-and-seo/ - but that was from 2010. Matt Cutts is fairly clear on nonfollow - http://www.webpronews.com/heres-what-googles-matt-cutts-says-about-affiliate-links-and-nofollow-2012-06 - but not on whether it affects SERPS Does it? And what is the best way to handle it? This is a typical example of a page where we would have an affiliate link or two. http://www.oakleysteel.co.uk/fake-mill-certificates-en10168
Affiliate Marketing | | Zippy-Bungle0 -
What is the negative effect of getting affiliate links to your domain?
Are the negative effects of affiliate links to your domain bigger than the positive effects? Is it better to use a subdomain for your affiliate campaigns? (Because of linkprofile no follow links or time on page effects?) Thanks for responce!
Affiliate Marketing | | ltom0 -
Ecommerce affiliate marketing comparison engine
Does anyone know if there is such a thing as an ecommerce comparison platform for affiliate programs? Something that would say, take a particular product, and let the user compare between Amazon, eBay, Walmart (all would be through affiliate programs). Does such a thing exist? Or is such a thing even feasible? oops, sorry for multiple posts, please let me know if i can delete somehow
Affiliate Marketing | | Mozzin0 -
Best way to start affiliate marketing a new mens magazine website?
I have just started up a new mens magazine website featuring articles, videos, sports news and photos to help drive traffic to the website. I write all the articles myself and do the social media updates in my spare time. I was wondering if anyone has any tips on the best way to start doing affiliate marketing for a new website? I was inspired to do this by successful internet marketers who make a passive income off their websites. So far I have added Amazon and William Hill as affiliate links as I think my target market will be looking for these types of product links. Also, for anyone who has experience in this, how long it taken for your blog to become successful and make money? Many thanks, Alex
Affiliate Marketing | | AlexBarnes0 -
How do the engines look at pages populated using content from an affiliate site?
We are building out a section of our site using an affiliate partnership in the travel space. How would you suggest organizing the content so we do not get penalized for duplicate content? Do you suggest not suing the content at all?
Affiliate Marketing | | TSDigital0 -
Limit of links in the same page
Hi, I have some pages that have more than 100 links. They are links to some stores (affiliate links). I set all of them as rel="nofollow" but I think Google is penalizing me because the link quantity. What is a good number of URL in the same page? Thanks
Affiliate Marketing | | GDB0 -
Gaining juice from affiliate links
Afternoon Mozzers, We run a basic affiliate program whereby we distribute banners and tools to affiliate sites who then link back to our homepage via these, often through a tool that incorporates an HTML link with our primary keyword term embedded. This banner/tool/link heads to our homepage, and will then incorporate that particular affiliate's unique code in the URL, eg. www.example.com/1A-265-GNH Seems pretty standard practice so far? So, considering this, that we include our primary keyword term in HTML in some banners, some also link to our homepage via an embedded image (and others then just work off cookies as per usual), can anyone hazard a guess, or know for certain, the beneficence of this as a link building means? I ask in terms of trying to determine whether to continue punting our tools and banners as a decent link building opportunity, or concentrate more on other link building means. Hope I've been clear enough, but am happy to elaborate if need be. Cheers
Affiliate Marketing | | Martin_S0