# Solved: Crawling websites ...



## JiminSA (Dec 15, 2011)

Does anyone know about the legality of crawling or scanning through someone else's website looking for information?


----------



## allnodcoms (Jun 30, 2007)

It's not the looking mate, that's fine (Google have actually made a reasonably decent living out of doing exactly that), it's what you intend to do with the data you retrieve. That's where the headaches start.

As long as you follow (or no-follow if you get me!) the server's privacy settings then it's perfectly legal form a personal use perspective. It's all up there on the net (public domain) anyway, and automated searching, scraping and so on is fine. Just respect the robot.txt directives and don't dig too deep, it can get pretty dark down there...

Danny


----------



## JiminSA (Dec 15, 2011)

Yeh, Danny that's what concerns me - legal suits etc, But as you mention if it's displayed it's common data hey? 

I'm not keen but a client has asked me to write something to scan a Classified Ads site for private car sales so that the mobile numbers could be used for an sms (text) group advertising campaign.

I rather think that it's not viable anyway because of the problem of getting that data into the campaign - unless the sellers have software to receive csv's or dat's or whatever (I shall approach them)??


----------



## JiminSA (Dec 15, 2011)

Ok, the seller interface won't be a problem, but the information I need will almost certainly be on a database, to which I do not have legal access, no?

I'll tell my client it's a no-go unless he wants to go to court!


----------



## allnodcoms (Jun 30, 2007)

JiminSA said:


> I'm not keen but a client has asked me to write something to scan a Classified Ads site for private car sales so that the mobile numbers could be used for an sms (text) group advertising campaign.


That is a Judge Judy moment waiting to happen mate. Bad Idea.

Danny


----------



## truebluexxx (Aug 6, 2007)

That is definitely illegal in the UK. You can only contact someone for marketing purposes if they have previously contacted you regarding your business, I think.

Which is why Yell.com give businesses free listings, they can then legally bother you constantly with marketing calls.

But if you only provide the code to get the numbers, and are outside the UK, then I have no idea.

I can't find the UK law covering it, I keep finding similar laws e.g.

http://www.out-law.com/page-11893

This talks about unwanted calls, which isn't the exact offence in the example you give. But I am sure it is in the Data protection act somewhere.

Have a read of this, http://www.tsplegal.com/services_fo...klist_data_protection_and_direct_marketing__1
It is not the law, but a solicitors view of it.


----------



## JiminSA (Dec 15, 2011)

Thanks for your input truebluexxx - interesting point


> You can only contact someone for marketing purposes if they have previously contacted you regarding your business


 consumer protection, I suppose - but if we contact them about their advertisement for their vehicle, having gleaned it from the web, surely that's not illegal? We are simply responding to their Ad.


----------



## truebluexxx (Aug 6, 2007)

Anyone can contact them about the advertisement, but it doesn't sound like the company you are thinking of working with want to buy the cars. It sounds like they want to advertise to these people.

Found the ICO's guidelines (they enforce the law, or don't, as it seems but that doesn't make it legal)
http://www.ico.gov.uk/for_organisations/sector_guides/marketing.aspx

Read the part headed "Electronic mail marketing" read all of it, and pay attention to the all the 'or's and 'and's

Someone I know only read bullet point 2 up to the semi colon and got totally the wrong meaning.

"Electronic mail marketing" confusingly means, electronic messages sent by telephone, fax, email or text.


----------



## JiminSA (Dec 15, 2011)

> It sounds like they want to advertise to these people.


No, I've checked out the bona-fide of my potential clients and they need stock urgently and want to acquire it, legally (in fact the Director is an Advocate).

Getting back to the original question of legality - it pertained to the legality of scanning a website database...


> but the information I need will almost certainly be on a database, to which I do not have legal access, no?


So my need to know concerns the legality of accessing data which the owner has already made public on their site???


----------



## truebluexxx (Aug 6, 2007)

Sorry for focusing on the legality of how the information is going to be used.(One of the few laws I know a bit about) 
I agree with everything Allnodcoms said, "respect the robot.txt".


----------



## JiminSA (Dec 15, 2011)

Yup - I have already approached the site owners asking permission and will continue according to their response a) for common courtesy and b) for legality... But thanks for your participation - most enlightening!


----------

