Screen scraping: how to stop the internet's invisible data leeches

Data is your business's most valuable asset, so it's never a good idea to let it slip into the hands of competitors.

Sometimes, however, that can be difficult to prevent due to an automated technique known as 'screen scraping' that has for years provided a way of extracting data from website pages to be indexed over time.

This poses two main problems: first, that data could be used to gain a business advantage - from undercutting prices (in the case of a price comparison website, for example) to obtaining information on product availability.

Persistent scraping can also grind down a website's performance, which recently happened to LinkedIn when hackers used automated software to register thousands of fake accounts in a bid to extract and copy data from member profile pages.

Ashley Stephenson, CEO of Corero Network Security, explains the origins behind the phenomenon, how it could be affecting your business right now and how to defend from it.

TechRadar Pro: What is screen scraping? Can you talk us through some of the techniques, and why somebody would do it?

Ashley Stephenson: Screen scraping is a concept that was pioneered by early terminal emulation programs decades ago. It is a programmatic method to extract data from screens that are primarily designed to be viewed by humans.

Basically the screen scraping program pretends to be a human and "reads" the screen, collecting the interesting data into lists that can be processed automatically. The most common format is name:value pairs. For example, information extracted from a travel site reservation screen might look like the following -

Origin: Boston, Destination:Atlanta, Date:10/12/13, Flight:DL4431, Price:$650

Screen scraping has evolved significantly over the years. A major historical milestone occurred when the screen scraping concept was applied to the Internet and the web crawler was invented.

Web crawlers originally "read" or screen scraped website pages and indexed the information for future reference (e.g. search). This gave rise to the search engine industry. Today webcrawlers are much more sophisticated and websites include information (tags) dedicated to the crawler and never intended to be read by a human.

Another subsequent milestone in the evolution of screen scraping was the development of e-retail screen scraping, perhaps the most well know example being the introduction of price comparison websites.

These sites employ screen scraping programs to periodically visit a list of known e-retail sites to obtain the latest price and availability information for a specific set of products or services. This information is then stored in a database and used to provide aggregated comparative views of the e-retail landscape to interested customers.

In general the previously described screen scraping techniques have been welcomed by website operators who want their sites to be indexed by the leading search engines such as Google or Bing, similarly e-retailers typically want their products to be displayed on the leading comparison shopping sites.

TRP: Have there been any recent developments in competitive screen scraping?

AS: In contrast over the past few years, recent developments in competitive screen scraping are not necessarily so welcome. For a site to be scraped by a search engine crawler is OK if the crawler visits are infrequent.

For a site to be the target of a price comparison site scraper is OK if the information obtained is used fairly. However as the number of specialized search engines continues to increase and the frequency of price check visits skyrockets these automated page views can rise to levels which impact the intended operation of the target site.

More specifically, if the target site is the victim of competitive scraping the information obtained can be used to undermine the business of the site owner. For example, undercutting prices, beating odds, aggressively acquiring event tickets, reserving inventory, etc.

In general, we believe there is a significant increase in the use of automated bots to gather website content to seed other services, fuel competitive intelligence, and aggregate product details like pricing, features and inventory. Increasingly this information is used to get a leg up over the competition, or to increase website hit rates.

For example, in the travel and tourism industry, price scraping is a real issue as travel sites are constantly looking to beat out the competition by offering the 'best price'. Additionally, the idea of inventory scraping is becoming more common. The concept of bots being used to purchase volumes of a high value item to resell, or to increase online pricing to deter potential buyers.

With the high availability of seemingly legal software bundles and services to facilitate the screen scraping process, and the motives we've just described, it's really a pretty powerful combination.

TRP: How long has screen scraping been going on for and is it becoming more or less of a problem for companies?

AS: Screen scraping has been going on for years but it is only more recently that victims, negatively impacted by this type of behaviour, are beginning to react. Some claim copyright infringement and unfair business practices while in contrast, organizations doing the scraping claim freedom of information.

Many website owners have written usage policies on their sites that prohibit aggressive scraping but have no ability to enforce their policies - the problem doesn't seem to be going away anytime soon.

TRP: How does screen scraping impact negatively on a business's IT systems?

AS: Competitive or abusive Screen scraping is just another example of unwanted traffic. Recent studies show that 61% of Internet Traffic is generated by bots. Bad-bot scrapers consume valuable resources and bandwidth intended to serve genuine web site users, this can result in increased latency for real customers, due to large numbers of non-human visits to the site. The business impact manifests itself as additional IT investment needed to serve the same number of customers.

TRP: Ebay introduced an API years ago to combat screen scraping. Is creating an API to provide access to data a recommended form of defense?

AS: Providing a dedicated API allows "good" scrapers access to your data programmatically and voluntarily observes resource utilization limits however it does not stop malicious information harvesting to be used for competitive advantage.

Real defense can be obtained by taking advantage of technology that can identify and block unwanted non-human visitors to your website. This would allow real or 'good' users to access the site for their intended purposes, while blocking the bad crawlers and bots from causing damage.

TRP: How else can an organisation defend itself from screen scraping?

AS: Using techniques such as IP reputation intelligence, geolocation enforcement, spoofed IP source detection, real time threat-level assessment, request-response behaviour analysis and bi-directional deep packet inspection.

Many organizations today are relying on Corero's First Line of Defense technology block unwanted website traffic including excessive scraping. Corero helps identify human visitors vs. non-human bots (e.g. running scripts) and blocks the unwanted offenders real-time.

TRP: Are there any internet rules governing the use (or misuse) of screen scraping?

AS: Screen scraping has been the topic of some pretty high-profile lawsuits for example Craigslist vs. PadMapper, and in the travel space for example, Ryanair vs. Budget Travel.

However, most court cases to date have not been fully resolved to the satisfaction of the victims. The courts often refuse to grant injunctions for said activity most likely because they have no precedent to work with. This is primarily due to the fact that there few if any internet rules really governing this type of activity.

Source:http://www.techradar.com/news/internet/web/screen-scraping-how-to-stop-the-internet-s-invisible-data-leaches-1214404

Internet Marketing - A Beginners Guide

Every new website owner is faced with the problem of getting visitors to their site, and if the website happens to be a commercial venture it needs to happen fast, even non commercial sites need visitors to survive. A quick look at some of the online tools to find expired domains show just how many websites fail every hour of every day. To help prevent your site joining the daily list of failures effective marketing is essential.

Marketing does not necessarily mean spending lots of money, however if fast results are needed then some money will have to be spent. As this article is aimed at beginners, I'll only briefly look at paid marketing and concentrate mainly on the free or low cost options to promote your site.

Paid Marketing:

In many respects online marketing is not dissimilar to offline marketing and many of the tactics used to promote an offline business will work equally well for websites. For example newspaper or magazine adverts although sometimes expensive can produce excellent results for websites, and occasionally magazines will supply cover cd's with links back to your site, an excellent way to get more visitors.

For larger campaigns television or radio ads can work, and with the expansion of satellite TV stations costs for this type of campaign are coming down all the time. For most new site owners however these options will prove to be too expensive and specialist online advertising will be the preferred choice. In the online paid sector directory listings and Pay Per Click (PPC) are probably the most popular. PPC quite simply work as the name describes and you pay a set amount every time a potential customer clicks your link. This has the advantage you only pay when someone visits your site and the obvious disadvantagethat it's open the click fraud where others click the links to increase your bill.

Most of the providers of PPC offer protection against click fraud and block multiple clicks from the same IP address, although for the site owner this is difficult to monitor, and suspicion is always there that the PPC provider has little incentive to police their policies as it would lead to loss of revenue. From experience PPC advertising can become very expensive and although it's quite easy to set up yourself through companies like google, it can sometimes be worth paying a specialist to run your campaign, as they will know how best to target the clicks to make best use of your money.

Free Marketing:

In business like all walks of life the general rule is that you get nothing for nothing, with Internet marketing this rule somewhat goes out the window as there are many free resources for promoting your website. In this section I'm going to take a look at a few of the most popular, this is of coarse only scraping the surface and as you develop your strategy further you'll find new outlets and tools that will get lots of visitors popping by your website.

Search Engines:

Without question the most effective free source of visitors are the many search engines like Google, Yahoo and MSN... the list really is almost endless. Most new website owners believe to get listed in these popular search engines the best thing to do is submit your site to them, this is not the case. All the main search engines these days use crawlers which automatically browse the web and store the contents of the sites they visit.

The search engines then use this content (along with 100's of other criteria ) to rank sites for specific search terms. Search engine optimization (SEO) is a whole different subject, however in brief what you need is links from other website, and some of the marketing tips below, along with attracting visitors to your site will help with your SEO efforts. The truth is that many people use website marketing solely to increase ranking in search engines, in my experience however I've found that if you promote your site for real visitors the search engines will follow.

There are many SEO companies who specialise in getting your site up the rankings, you do need to be careful as many will make great and exaggerated claims, some of which are simply not possible. The best advice I can give here is to research thoroughly and look at, and if possible contact past customers for a reference. Good SEO can be the most cost effective way to promote your site but you do need to work hard at it or employ someone else to do it for you.

Forums:

Internet forums can be very useful for getting visitors to your site, the biggest advantage with forums is that you can target forums related to your area of interest and most will allow members who contribute useful posts and replies to have a link back to their own website in their signature. You should take care not to just post links as most good forums will consider this as spam, and even if the moderators don't delete the link, visitor looking at it will clearly see what's going on and your sites reputation will suffer.

Blogs:

In my opinion every website owner should have a blog, and for SEO purposes it's probably best to have the blog on a different domain. There are lots of free blog sites like blogger.com which offer you easy to use blogs. The biggest advantage with your own blog is you can write articles and provide links to different areas of your site, this provides different entry points and is also very good SEO practice.

Another use of blogs is comments, a great way of getting visitors back to your site is to search for other blogs relevant to your website and leave comments with a link back to your site. As with forums care should be taken not to spam the comments as it's a bad practice and unlikely to help you long term.

Article Submissions:

If you're up to it, writing an article can be a good way to get links back to your website, most good article distribution sites like EzineArticles.com will allow you to have a short bio at the end of the article, which can direct visitors back to your websites. If possible the article should be about a subject related to your business, but as you can see from my bio below it's not essential :-)

Well that's about it, as I said at the beginning this is only an introduction for beginners and as your site and business grows you'll find many new and exciting ways to market your online business.

Source: http://ezinearticles.com/?Internet-Marketing---A-Beginners-Guide&id=1729653

Essay Assistance - Help With Essay Writing

Help with essay writing? Surely this must be frowned upon by the authorities. Academic writing should be the result of an individual's work, and a student should not ask for another writer to 'fix my essay'. That is certainly true of course, but there is a degree to which essay assistance is allowed, governed by a code of conduct set out by the universities.

There should be no plagiarism, of course, nor any 'ghosting', but online academic writing services exist for the important task of editing for 'clarity, flow and consistency.' The student can submit their essay for assessment in the vital areas of grammar, spelling and punctuation - and turnaround could be within 12 hours if necessary.

It can be very difficult to spot mistakes within one's own writing, academic or otherwise. This is one of the strengths of a professional proof-reading and editing service, which can correct grammar and spelling, sentence structure, and punctuation. This type of online service is always on hand, available 24-hours a day, all year round.

With essay assistance it is possible to submit your work for assessment, yet retain complete control of the finished assignment. The track changes function in Microsoft Word can be used to highlight any changes which have been made. These changes are suggestions only, which can be approved or amended when the document has been returned.

A lengthy essay such as a dissertation can certainly benefit from presentational essay assistance. Maintaining consistency throughout a dissertation is one of the challenges which can be difficult to optimise, and is easy to overlook. Such essay assistance may include the creation of pre-linked contents pages, management of heading and text formatting, inserting page breaks and cover pages, adding headers and footers, and creating dynamic referencing.

Grammatical style is another key element in the clear presentation of your work. Clarity of thought and the coherence of a well-plotted argument can be disguised by extended sub-clauses and the over-use of parentheses. If your reader is distracted from your main purpose then your most powerful points could be lost. This is an important aspect of essay assistance, and having your attention drawn to flaws in your grammatical style are far easier for a third party to identify. After all, you may know what you mean, but your reader must also be able to follow your line of thought.

There is no need to allow this to happen. If marks are lost due to failures in presentation or grammar, then a student will not only have undersold their true worth, but also wasted some of their energies. This is the significance of essay assistance, and with the ease of online access throughout the year it is a potential asset which should not be overlooked.

Source:http://ezinearticles.com/?Essay-Assistance---Help-With-Essay-Writing&id=6349475

How to scrap a car: Top tips

Unsure about what to do when it comes to getting rid of your old car? Read our tips

The process of buying a new car often involves disposing of your old one, and if your car has reached the end of its life then you may be left with no alternative to having it scrapped. This may also be the case if your car fails its MOT and the cost of the repairs is more than the car is worth.

EU End of Life directive

In the past it was common to pay for someone to scrap your car but this meant many cars weren’t disposed of properly. Legislation based on the EU End of Life directive was implemented in the UK in 2002 and addressed the issue by making sure cars could be disposed of for free at licensed scrapyards. The increase in the value of certain metals since then means scrap merchants will usually pay for the metal they’ll recover when they scrap your car.

Use an authorised treatment facility

If your car is to be scrapped, it must be done at an authorised treatment facility (ATF), which is a scrapyard that's registered and monitored by the Environment Agency. There is a database of ATFs on the agency's website, and it's well worth a look. The facility will recycle your car in an environmentally friendly way and issue you with a Certificate of Destruction, which is important you keep, else you could find yourself liable for road tax and a fine, even when your car no longer exists.

Try online scrap merchants

There are now a number of online agents who will collect and scrap your car. You can usually find out how much they’ll pay for your car by entering the registration number and its location on their websites. There are also comparison sites for this so you can see who is offering the highest quote for your car.

Individual parts can make more cash

Depending on the condition of your car, you may make extra money by selling certain parts before it's scrapped. Getting a mechanic to take a look over the car will give you an idea of the value you could expect when negotiating at a scrapyard. The bigger, more fundamental parts like the engine, gearbox and brakes are likely to be worth the most.

Consider using auction sites

Whether you choose to scrap your car as a complete vehicle or to sell some of the parts separately first, auction sites offer an alternative to the scrapyard. Some people list their cars on auction sites at their scrap value in the hope of getting higher bids during the auction period.

Get the correct documentation

Remember that if you sell the car to someone, even if just for scrap, you need to let the DVLA know that you no longer have it by completing section three of the V5C vehicle registration document. It has been known for people to collect cars for scrap and then continue using them without a valid MOT – if you haven’t completed the right paperwork, you will still be responsible for the car. If you sell the parts and then scrap it, you need to get a Certificate of Destruction.

Source: http://www.carbuyer.co.uk/tips-and-advice/138478/how-to-scrap-a-car-top-tips

Data Mining As a Process

The data mining process is also known as knowledge discovery. It can be defined as the process of analyzing data from different perspectives and then summarizing the data into useful information in order to improve the revenue and cut the costs. The process enables categorization of data and the summary of the relationships is identified. When viewed in technical terms, the process can be defined as finding correlations or patterns in large relational databases. In this article, we look at how data mining works its innovations, the needed technological infrastructures and the tools such as phone validation.

Data mining is a relatively new term used in the data collection field. The process is very old but has evolved over the time. Companies have been able to use computers to shift over the large amounts of data for many years. The process has been used widely by the marketing firms in conducting market research. Through analysis, it is possible to define the regularity of customers shopping. How the items are bought. It is also possible to collect information needed for the establishment of revenue increase platform. Nowadays, what aides the process is the affordable and easy disk storage, computer processing power and applications developed.

Data extraction is commonly used by the companies that are after maintaining a stronger customer focus no matter where they are engaged. Most companies are engaged in retail, marketing, finance or communication. Through this process, it is possible to determine the different relationships between the varying factors. The varying factors include staffing, product positioning, pricing, social demographics, and market competition.

A data-mining program can be used. It is important note that the data mining applications vary in types. Some of the types include machine learning, statistical, and neural networks. The program is interested in any of the following four types of relationships: clusters (in this case the data is grouped in relation to the consumer preferences or logical relationships), classes (in this the data is stored and finds its use in the location of data in the per-determined groups), sequential patterns (in this case the data is used to estimate the behavioral patterns and patterns), and associations (data is used to identify associations).

In knowledge discovery, there are different levels of data analysis and they include genetic algorithms, artificial neural networks, nearest neighbor method, data visualization, decision trees, and rule induction. The level of analysis used depends on the data that is visualized and the output needed.

Nowadays, data extraction programs are readily available in different sizes from PC platforms, mainframe, and client/server. In the enterprise-wide uses, size ranges from the 10 GB to more than 11 TB. It is important to note that two crucial technological drivers are needed and are query complexity and, database size. When more data is needed to be processed and maintained, then a more powerful system is needed that can handle complex and greater queries.

With the emergence of professional data mining companies, the costs associated with process such as web data extraction, web scraping, web crawling and web data mining have greatly being made affordable.

Source:http://ezinearticles.com/?Data-Mining-As-a-Process&id=7181033

ScrapeDefender Launches Cloud-Based Anti-Scraping Solution To Protect Web Sites From Content Theft

ScrapeDefender launched today a new cloud-based anti-scraping monitoring solution that identifies and blocks suspicious activity to protect websites against content theft from mass scraping. The product provides triple protection levels against web scraping in the areas of vulnerability scanning, monitoring and security.

ScrapeDefender estimates that losses from web scraping content theft are close to $5 billion annually. According to a recent industry study, malicious non-human-based bot traffic now represents 30% of all website visits. Scrapers routinely target online marketplaces including financial, travel, media, real estate, and consumer-product arenas, stealing valuable information such as pricing and listing data.

ScrapeDefender stops website scraping by identifying and alerting site owners about suspicious activity in near real time. The monitoring system uses intrusion detection-based algorithms and patented technology to analyze network activity for both human and bot-like activity. It was designed from the ground up to work passively with web servers so that the underlying business is not impeded in any way. ScrapeDefender does not require any DNS changes or new hardware.

"Web scraping is growing at an alarming rate and if left unchecked, it is just a matter of time until all sites with useful content will be targeted by competitors harvesting data," said Robert Kane, CEO of ScrapeDefender. "We provide the only solution that scans, monitors and protects websites against suspicious scraping activity, in a way that isn't intrusive."

Irv Chasen, a board member at Bondview, the largest free provider of municipal bond data, said, "Our business is built on providing accurate municipal bond pricing data and related information to professional and retail investors. If competitors are scraping our information and then using it to gain an advantage, it creates a challenging business problem for us. With ScrapeDefender we can easily monitor and stop any suspicious scraping. Their support team made it easy for us to stay proactive and protect our website content."

ScrapeDefender is available as a 24 X 7 managed service or can be customer controlled. Customers are assigned a ScrapeDefender support staff member to help monitor network activity and alerts are automatically sent when suspicious activity is identified. Today's announcement extends ScrapeDefender's scanner, which was introduced in 2011 and remains the only anti-scraping assessment tool on the market that singles out web scraping vulnerabilities.

The ScrapeDefender Suite is available now at www.scrapedefender.com, starting at $79 per month for one domain.

About ScrapeDefender

ScrapeDefender was created by a team of computer security and web content experts with 20 years of experience working at leading organizations such as RSA Security, Goldman Sachs and Getty Images. Our web anti-scraping experts can secure your website to ensure that unauthorized content usage is identified and blocked.

Source: http://www.darkreading.com/vulnerability/scrapedefender-launches-cloud-based-anti/240165737

Data scraping tool for non-coding journalists launches

A tool which helps non-coding journalists scrape data from websites has launched in public beta today.

Import.io lets you extract data from any website into a spreadsheet simply by mousing over a few rows of information.

Until now import.io, which we reported on back in April, has been available in private developer preview and has been Windows only. It is now also available for Mac and is open to all.

Although import.io plans to charge for some services at a later date, there will always be a free option.

The London-based start-up is trying to solve the problem of the fact that there is "lots of data on the web, but it's difficult to get at", Andrew Fogg, founder of import.io, said in a webinar last week.

Those with the know-how can write a scraper or use an API to get at data, Fogg said. "But imagine if you could turn any website into a spreadsheet or API."

Uses for journalists

Journalists can find stories in data. For example, if I wanted to do a story on the type of journalism jobs being advertised and the salaries offered, I could research this by looking at various websites which advertise journalism jobs.

If I were to gather the data from four different jobs boards and enter the information manually into a spreadsheet it would take would take hours if not days; if I were to write a screen scraper for each of the sites it would require knowledge and would probably take a couple of hours. Using import.io I can create a single dataset from multiple sources in a few minutes.

I can then search and sort the dataset and find out different facts, such as how many unpaid internships are advertised, or how many editors are currently being sought.

How it works

When you download the import.io application you see a web browser. This browser allows you to enter a URL for any site you want to scrape data from.

To take the example of the jobs board, this is structured data, with the job role, description and salaries displayed.

The first step is to set up 'connectors' and to do this you need to teach the system where the data is on the page. This is done by hitting a 'record' button on the right of the browser window and mousing over a few examples, in this case advertised jobs. You then click 'train rows'.

It takes between two and five examples to teach import.io where all of the rows are, Fogg explained in the webinar.

The next step is to declare the type of data and add column names. For example there may be columns for 'job title', 'job description' and 'salary'. Data is then extracted into the table below the browser window.

Data from different websites can then be "mixed" into a single searchable database.

In the example used in the webinar, Fogg demonstrated how import.io could take data relating to rucksacks for sale on a shopping website. The tool can learn the "extraction pattern", Fogg explained, and apply that to to another product. So rather than mousing over the different rows of sleeping bags advertised, for example, import.io was automatically able to detect where the price and product details were on the page as it had learnt the structure from how the rucksacks were organised. The really smart bit is that the data from all products can then be automatically scraped and pulled into the spreadsheet. You can then search 'shoes' and find the data has already been pulled into your database.

When a site changes its code a screen scraper would become ineffective. Import.io has a "resilience to change", Fogg said. It runs tests twice a day and users get notified of any changes and can retrain a connector.

It is worth noting that a site that has been scraped will be able to detect that import.io has extracted the data as it will appear in the source site's web logs.

Case studies

A few organisations have already used import.io for data extraction. Fogg outlined three.

    British Red Cross

The British Red Cross wanted to create an iPhone app with data from the NHS Choices website. The NHS wanted the charity to use the data but the health site does not have an API.

By using import.io, data was scraped from the NHS site. The app is now in the iTunes store and users can use it to enter a postcode to find hospital information based on the data from the NHS site.

"It allowed them to build an API for a website where there wasn't one," Fogg said.

    Hewlett Packard

Fogg explained that Hewlett Packard wanted to monitor the prices of its laptops on retailers' websites.

They used import.io to scrape the data from the various sites and were able monitor the prices at which the laptops were being sold in real-time.

    Recruitment site

A US recruitment firm wanted to set up a system so that when any job vacancy appeared on a competitor's website, they could extract the details and push that into their Salesforce software. The initial solution was to write scrapers, Fogg said, but this was costly and in the end they gave up. Instead they used import.io to scrape the sites and collate the data.


Source: http://www.journalism.co.uk/news/data-scraping-tool-for-non-coding-journalists-launches/s2/a554002/