Using External Input Data in Off-the-shelf Web Scrapers

There is a question I’ve wanted to shed some light upon for a long time already: “What if I need to scrape several URL’s based on data in some external database?“.

For example, recently one of our visitors asked a very good question (thanks, Ed):

    “I have a large list of amazon.com asin. I would like to scrape 10 or so fields for each asin. Is there any web scraping software available that can read each asin from a database and form the destination url to be scraped like http://www.amazon.com/gp/product/{asin} and scrape the data?”

This question impelled me to investigate this matter. I contacted several web scraper developers, and they kindly provided me with detailed answers that allowed me to bring the following summary to your attention:
Visual Web Ripper

An input data source can be used to provide a list of input values to a data extraction project. A data extraction project will be run once for each row of input values. You can find the additional information here.
Web Content Extractor

You can use the -at”filename” command line option to add new URLs from TXT or CSV file:

    WCExtractor.exe projectfile -at”filename” -s

projectfile: the file name of the project (*.wcepr) to open.
filename – the file name of the CSV or TXT file that contains URLs separated by newlines.
-s – starts the extraction process

You can find some options and examples here.
Mozenda

Since Mozenda is cloud-based, the external data needs to be loaded up into the user’s Mozenda account. That data can then be easily used as part of the data extracting process. You can construct URLs, search for strings that match your inputs, or carry through several data fields from an input collection and add data to it as part of your output. The easiest way to get input data from an external source is to use the API to populate data into a Mozenda collection (in the user’s account). You can also input data in the Mozenda web console by importing a .csv file or importing one through our agent building tool.

Once the data is loaded into the cloud, you simply initiate building a Mozenda web agent and refer to that Data list. By using the Load page action and the variable from the inputs, you can construct a URL like http://www.amazon.com/gp/product/%asin%.
Helium Scraper

Here is a video showing how to do this with Helium Scraper:


The video shows how to use the input data as URLs and as search terms. There are many other ways you could use this data, way too many to fit in a video. Also, if you know SQL, you could run a query to get the data directly from an external MS Access database like
SELECT * FROM [MyTable] IN "C:\MyDatabase.mdb"

Note that the database needs to be a “.mdb” file.
WebSundew Data Extractor
Basically this allows using input data from external data sources. This may be CSV, Excel file or a Database (MySQL, MSSQL, etc). Here you can see how to do this in the case of an external file, but you can do it with a database in a similar way (you just need to write an SQL script that returns the necessary data).
In addition to passing URLs from the external sources you can pass other input parameters as well (input fields, for example).
Screen Scraper

Screen Scraper is really designed to be interoperable with all sorts of databases. We have composed a separate article where you can find a tutorial and a sample project about scraping Amazon products based on a list of their ASINs.


Source: http://extract-web-data.com/using-external-input-data-in-off-the-shelf-web-scrapers/

Handy Web Extractor

Handy Web Extractor is a simple tool for everyday web content monitoring. It will periodically download the web page, extract the necessary content and display it in the window on your desktop. One may consider it as the data extraction software, taking its own nitch in the scraping software and plugins.

What is it for?

Have you ever needed to track some web site changes without visiting it again and again? If so, you may find this program useful. The idea is simple: it periodically downloads the specified web page, extracts the part you need and displays it for you in a small window. You can easily move, resize, hide or show this window according to your needs.

It’s totally free and available for download.
How does it work?

After installing the program you will see the following window:


At the top of the window (on white background) you may see the extracted web page itself (in this case it’s a header of this article) followed by program settings (on light yellow background). If you don’t see the program settings click on the gear icon (if you want to hide them click it again).

Settings available:

    Web Site URL – type the URL of the target web-page you want to scrape
    Extract using XPath – use this option if you want to specify the portion of the web-page using XPath expression
    Extract using Regex – use this option if you want to specify the portion of the web-page using Regex expression
    Update every N min - to specify how often the program will scrape the target website
    Autostart – check this box if you want the program to start automatically when Windows starts

After you change either the web site URL or XPath/Regex expression click the “Update now” link at the bottom to rescrape the web site.

You may always access this window via the “magnet” icon in the system tray. Click the icon to show/hide the window and right-click it to display an additional menu.

That’s it. The only thing I’d like to mention here is that the program remembers all your settings right away (including window position and size) and you don’t need to “save” them manually.
Usage Examples
Stocks

You can use Handy Web Extractor as a stock tracker:


Hot news

Here is an example of how to monitor hot news using Handy Web Extractor:

Number Tracker

With Handy Web Extractor you can easily extract a single number using Regex expressions. Here is an example of how to track your program downloads:

Here is how it may look on your desktop:

Picture of the day

You may even use Handy Web Extractor to display a picture of the day from any web site :) :



Source: http://extract-web-data.com/handy-web-extractor/

Three Common Methods For Web Data Extraction

Probably the most common technique used traditionally to extract data from web pages this is to cook up some regular expressions that match the pieces you want (e.g., URL's and link titles). Our screen-scraper software actually started out as an application written in Perl for this very reason. In addition to regular expressions, you might also use some code written in something like Java or Active Server Pages to parse out larger chunks of text. Using raw regular expressions to pull out the data can be a little intimidating to the uninitiated, and can get a bit messy when a script contains a lot of them. At the same time, if you're already familiar with regular expressions, and your scraping project is relatively small, they can be a great solution.

Other techniques for getting the data out can get very sophisticated as algorithms that make use of artificial intelligence and such are applied to the page. Some programs will actually analyze the semantic content of an HTML page, then intelligently pull out the pieces that are of interest. Still other approaches deal with developing "ontologies", or hierarchical vocabularies intended to represent the content domain.

There are a number of companies (including our own) that offer commercial applications specifically intended to do screen-scraping. The applications vary quite a bit, but for medium to large-sized projects they're often a good solution. Each one will have its own learning curve, so you should plan on taking time to learn the ins and outs of a new application. Especially if you plan on doing a fair amount of screen-scraping it's probably a good idea to at least shop around for a screen-scraping application, as it will likely save you time and money in the long run.

So what's the best approach to data extraction? It really depends on what your needs are, and what resources you have at your disposal. Here are some of the pros and cons of the various approaches, as well as suggestions on when you might use each one:

Raw regular expressions and code

Advantages:

- If you're already familiar with regular expressions and at least one programming language, this can be a quick solution.

- Regular expressions allow for a fair amount of "fuzziness" in the matching such that minor changes to the content won't break them.

- You likely don't need to learn any new languages or tools (again, assuming you're already familiar with regular expressions and a programming language).

- Regular expressions are supported in almost all modern programming languages. Heck, even VBScript has a regular expression engine. It's also nice because the various regular expression implementations don't vary too significantly in their syntax.

Disadvantages:

- They can be complex for those that don't have a lot of experience with them. Learning regular expressions isn't like going from Perl to Java. It's more like going from Perl to XSLT, where you have to wrap your mind around a completely different way of viewing the problem.

- They're often confusing to analyze. Take a look through some of the regular expressions people have created to match something as simple as an email address and you'll see what I mean.

- If the content you're trying to match changes (e.g., they change the web page by adding a new "font" tag) you'll likely need to update your regular expressions to account for the change.

- The data discovery portion of the process (traversing various web pages to get to the page containing the data you want) will still need to be handled, and can get fairly complex if you need to deal with cookies and such.

When to use this approach: You'll most likely use straight regular expressions in screen-scraping when you have a small job you want to get done quickly. Especially if you already know regular expressions, there's no sense in getting into other tools if all you need to do is pull some news headlines off of a site.

Ontologies and artificial intelligence

Advantages:

- You create it once and it can more or less extract the data from any page within the content domain you're targeting.

- The data model is generally built in. For example, if you're extracting data about cars from web sites the extraction engine already knows what the make, model, and price are, so it can easily map them to existing data structures (e.g., insert the data into the correct locations in your database).

- There is relatively little long-term maintenance required. As web sites change you likely will need to do very little to your extraction engine in order to account for the changes.

Disadvantages:

- It's relatively complex to create and work with such an engine. The level of expertise required to even understand an extraction engine that uses artificial intelligence and ontologies is much higher than what is required to deal with regular expressions.

- These types of engines are expensive to build. There are commercial offerings that will give you the basis for doing this type of data extraction, but you still need to configure them to work with the specific content domain you're targeting.

- You still have to deal with the data discovery portion of the process, which may not fit as well with this approach (meaning you may have to create an entirely separate engine to handle data discovery). Data discovery is the process of crawling web sites such that you arrive at the pages where you want to extract data.

When to use this approach: Typically you'll only get into ontologies and artificial intelligence when you're planning on extracting information from a very large number of sources. It also makes sense to do this when the data you're trying to extract is in a very unstructured format (e.g., newspaper classified ads). In cases where the data is very structured (meaning there are clear labels identifying the various data fields), it may make more sense to go with regular expressions or a screen-scraping application.

Screen-scraping software

Advantages:

- Abstracts most of the complicated stuff away. You can do some pretty sophisticated things in most screen-scraping applications without knowing anything about regular expressions, HTTP, or cookies.

- Dramatically reduces the amount of time required to set up a site to be scraped. Once you learn a particular screen-scraping application the amount of time it requires to scrape sites vs. other methods is significantly lowered.

- Support from a commercial company. If you run into trouble while using a commercial screen-scraping application, chances are there are support forums and help lines where you can get assistance.

Disadvantages:

- The learning curve. Each screen-scraping application has its own way of going about things. This may imply learning a new scripting language in addition to familiarizing yourself with how the core application works.

- A potential cost. Most ready-to-go screen-scraping applications are commercial, so you'll likely be paying in dollars as well as time for this solution.

- A proprietary approach. Any time you use a proprietary application to solve a computing problem (and proprietary is obviously a matter of degree) you're locking yourself into using that approach. This may or may not be a big deal, but you should at least consider how well the application you're using will integrate with other software applications you currently have. For example, once the screen-scraping application has extracted the data how easy is it for you to get to that data from your own code?

When to use this approach: Screen-scraping applications vary widely in their ease-of-use, price, and suitability to tackle a broad range of scenarios. Chances are, though, that if you don't mind paying a bit, you can save yourself a significant amount of time by using one. If you're doing a quick scrape of a single page you can use just about any language with regular expressions. If you want to extract data from hundreds of web sites that are all formatted differently you're probably better off investing in a complex system that uses ontologies and/or artificial intelligence. For just about everything else, though, you may want to consider investing in an application specifically designed for screen-scraping.

As an aside, I thought I should also mention a recent project we've been involved with that has actually required a hybrid approach of two of the aforementioned methods. We're currently working on a project that deals with extracting newspaper classified ads. The data in classifieds is about as unstructured as you can get. For example, in a real estate ad the term "number of bedrooms" can be written about 25 different ways. The data extraction portion of the process is one that lends itself well to an ontologies-based approach, which is what we've done. However, we still had to handle the data discovery portion. We decided to use screen-scraper for that, and it's handling it just great. The basic process is that screen-scraper traverses the various pages of the site, pulling out raw chunks of data that constitute the classified ads. These ads then get passed to code we've written that uses ontologies in order to extract out the individual pieces we're after. Once the data has been extracted we then insert it into a

Source: http://ezinearticles.com/?Three-Common-Methods-For-Web-Data-Extraction&id=165416

Data Management Services

In recent studies it has been revealed that any business activity has astonishing huge volumes of data, hence the ideas has to be organized well and can be easily gotten when need arises. Timely and accurate solutions are important in facilitating efficiency in any business activity. With the emerging professional outsourcing and data organizing companies nowadays many services are offered that matches the various kinds of managing the data collected and various business activities. This article looks at some of the benefits that accrue of offered by the professional data mining companies.

Entering of data

These kinds of services are quite significant since they help in converting the data that is needed in high ideal and format that is digitized. In internet some of this data can found that is original and handwritten. In printed paper documents and or text are not likely to contain electronic or needed formats. The best example in this context is books that need to be converted to e-books. In insurance companies they also depend on this process in processing the claims of insurance and at the same time apply to the law firms that offer support to analyze and process legal documents.

EDC

That is referred to as electronic data. This method is mostly used by clinical researchers and other related organization in medical. The electronic data and capture methods are used in the utilization in managing trials and research. The data mining and data management services are given in upcoming databases for studies. The ideas contained can easily be captured, other services being done and the survey taken.

Data changing

This is the process of converting data found in one format to another. Data extraction process often involves mining data from an existing system, formatting it, cleansing it and can be installed to enhance both availability and retrieving of information easily. Extensive testing and application are the requirements of this process. The service offered by data mining companies includes SGML conversion, XML conversion, CAD conversion, HTML conversion, image conversion.

Managing data service

In this service it involves the conversion of documents. It is where one character of a text may need to be converted to another. If we take an example it is easy to change image, video or audio file formats to other applications of the software that can be played or displayed. In indexing and scanning is where the services are mostly offered.

Data extraction and cleansing

Significant information and sequences from huge databases and websites extraction firms use this kind of service. The data harvested is supposed to be in a productive way and should be cleansed to increase the quality. Both manual and automated data cleansing services are offered by data mining organizations. This helps to ensure that there is accuracy, completeness and integrity of data. Also we keep in mind that data mining is never enough.

Web scraping, data extraction services, web extraction, imaging, catalog conversion, web data mining and others are the other management services offered by data mining organization. If your business organization needs such services here is one that can be of great significance that is web scraping and data mining




Source: http://ezinearticles.com/?Data-Management-Services&id=7131758

Outsourcing Data Entry Services

Data or raw information is the backbone of any industry or business organization. However, raw data is seldom useful in its pure form. For it to be of any use, data has to be recorded properly and organized in a particular manner. Only then can data be processed. That is why it is important to ensure accurate data entry. But because of the unwieldy nature of data, feeding data is a repetitive and cumbersome job and it requires heavy investment, both in terms of time and energy from staff. At the same time, it does not require a high level of technical expertise. Due to these factors, data entry can safely be outsourced, enabling companies to devote their time and energy on tasks that enhance their core competence.

Many companies, big and small, are therefore enhancing their productivity by outsourcing the endless monotonous tasks that tend to cut down the organization's productivity. In times to come, outsourcing these services will become the norm and the volume of work that is outsourced will multiply. The main reason for these kinds of development is the Internet. Web based customer service and instant client support has made it possible for service providers to act as one stop business process outsourcing partners to parent companies that require support.

Data entry services are not all alike. Different clients have different demands. While some clients may require recording information coupled with document management and research, others may require additional services like form processing or litigation support. Data entry itself could be from various sources. For instances, sometimes information may need to be typed out from existing documents while at other times, data needs to be extracted from images or scanned documents. To rise up to these challenges, service providers who offer these services must have the expertise and the software to ensure rapid and accurate data entry. That is why it is important to choose your service provider with a lot of care.

Before hiring your outsourcing partner, you need to ask yourself the following questions.

* What kind of reputation does the company enjoy? Do they have sufficient years of experience? What kind of history and background does the company enjoy?

* Do they have a local management arm that you can liaise with on a regular basis?

* Do the service personnel understand your requirements and can they handle them effectively?

* What are the steps taken by the company to ensure that there is absolutely no compromise in confidentiality and security while dealing with vital confidential data?

* Is there a guarantee in place?

* What about client references?

The answers to these questions will help you identify the right partner for outsourcing your data entry service requirements.



Source: http://ezinearticles.com/?Outsourcing-Data-Entry-Services&id=3568373

Data Mining Social Networks, Smart Phone Data, and Other Data Base, Yet Maintaining Privacy

Is it possible to data mine social networks in such a way to does not hurt the privacy of the individual user, and if so, can we justify doing such? It wasn't too long ago the CEO of Google stated that it was important that they were able to keep data of Google searches so they can find disease, flu, and food born medical clusters. By using this data and studying the regions in the searches to help fight against outbreaks of diseases, or food borne illnesses in the distribution system. This is one good reason to store the data, and collect it for research, as long as it is anonomized, then theoretically no one is hurt.

Unfortunately, this also scares the users, because they know if the searches are indeed stored, this data can be used against them in the future, for instance, higher insurance rates, bombardment of advertising, or get them put onto some sort of future government "thought police" watch-list. Especially considering all the political correctness, and new ways of defining hate speech, bullying, and what is, what isn't, and what might be a domestically home-grown terrorist. The future concept of the thought police is very scary to most folks.

Usually if you want to collect data from a user, you have to give them something back in return, and therefore they are willing to sign away certain privacy rights on that data in trade for the use of such services; such as on their cell phone, perhaps a free iPhone app or a virtual product in an online social network.

Artificially Intelligent Search Features

It is no surprised that AI search features are getting smarter, even able to anticipate your next search question, or what you are really trying to ask, even second guessing your question for instance. Now then, let's discuss this for a moment. Many folks very much enjoy the features of Amazon.com search features, which use artificial intelligence to recommend potential other books, which they might be interested in. And therefore the user probably does not mind giving away information about itself, for this upgraded service or ability, nor would the person mind having cookies put onto their Web browser.

Nevertheless, these types of systems are always exploited for other purposes. For instance consider the Federal Trade Commission's do not call list, and consider how many corporations, political party organizations, and all of their affiliates and partners were able to bypass these rules due to the fact that the consumer or customer had bought something from them in the last six months. This is not what consumers or customers had in mind when they decided they wanted to have this "do not call list" and the resultant and response from the market place, well, it proves we cannot trust the telecommunication companies, their lobbyists, or the insiders within their group (many of which over the years have indeed been somehow connected to the intelligence agencies - AT&T - NSA Echelon for example.)

Now then, this article is in no way to be considered a conspiracy theory, it is just a known fact, yes national security does need access to such information, and often it might be relevant, catching bad guys, terrorists, spies, etc. The NSA is to protect the American People. However, when it comes to the telecommunication companies, their job is to protect shareholder's equity, maximize quarterly profits, expand their business models, and create new profit centers in their corporations.

Thus, such user data will be and has been exploited for future profits against the wishes of the consumer, without the consumer benefiting from free services for lower prices in any way. If there is an explained reason, trade-off, and a monetary consideration, the consumer might feel obliged to have additional calls bothering them while they are at home, additional advertising, and tracking of their preferences for ease of use and suggestions. What types of suggestions?

Well, there is a Starbucks two-blocks from here, turn right, then turn left and it is 200 yards, with parking available; "Sale on Frappachinos for gold-card holders today!" In this case the telecommunication company tracks your location, knows your preferences, and collects a small fee from Starbucks, and you get a free-phone, and 20% off your monthly 4G wireless fee. Is that something a consumer might want; when asked 75% of consumers or smart phone users say; yes. See that point?

In the future smart phones may have data transferred between them, rather than going through a given or closest cell tower. In other words, packets of information may go from your cell phone, to the next nearest cell phone, to another near cell phone, to the person which is intended to receive it. And the data passing through each mobile device, will not be able to read any of the information which was it is not assigned to receive as it wasn't sent to it. By using such a scheme telecommunication companies can expand their services without building more new cell towers, and therefore they can lower the price.

However, it also means that when you lay your cell phone on the table, and it is turned on it would be constantly passing data through it, data which is not yours, and you are not getting paid for that, even though you had to purchase the smart phone. But if the phone was given to you, with a large battery, so it wouldn't go dead during all those transmissions, you probably wouldn't care, as long as your data packets of information were indeed safe and no one else could read them.

This technology exists now, and is being discussed, and consider if you will that the whole strategy of networking smart cell phones or personal tech devices together is nothing new. For instance, the same strategies have been designed for satellites, and to use an analogy, this scheme is very similar to the strategies FedEx uses when it sends packages to the next nearest FedEx office if that is their destination, without sending all of the packages all the way across the country to the central Memphis sort, and then all the way back again. They are saving time, fuel, space, and energy, and if cell phones did this it would save the telecommunication companies mega bucks in the savings of building new cell towers.

As long as you got a free cell phone, which many of us do, unless we have the mega top of the line edition, and if they gave you a long-lasting free battery it is win-win for the user. You probably wouldn't care, and the telecommunication companies could most likely lower the cost of services, and not need to upgrade their system, because they can carry a lot more data, without hundreds of billions of dollars in future investments.

Also a net centric system like this is safer to disruption in the event of an emergency, when emergency communications systems take precedence, putting every cell phone user as secondary traffic at the cell towers, which means their calls may not even get through.

Next, the last thing the telecommunication company would want to do is to data mine that data, or those packets of information from people like a soccer mom calling her son waiting at the bus stop at school. And anyone with a cell phone certainly wouldn't want their packets of information being stolen from them and rerouted because someone near them hacked into the system and had a cell phone that was displaying all of their information.

You can see the problems with all this, but you can also see the incredible economies of scale by making each and every cell phone a transmitter and receiver, which it already is in principle anyway, at least now for all data you send and receive. In the new system, if all the data which is closest by is able to transfer through it, and send that data on its way. The receiving cell phone would wait for all the packets of data were in, and then display the information.

You can see why such a system also might cause people to have a problem with it because of what they call net neutrality. If someone was downloading a movie onto their iPad using a 3G or 4G wireless network, it could tie up all the cell phones nearby that were moving the data through them. In this case, it might upset consumers, but if that traffic could be somewhat delayed by priority based on an AI algorithm decision matrix, something simple, then such a tactic for packet distribution plan might allow for this to occur without disruption from the actual cell tower, meaning everyone would be better off. Therefore we all get information flow faster, more dispersed, and therefore safer from intruders. Please consider all this.





Source: http://ezinearticles.com/?Data-Mining-Social-Networks,-Smart-Phone-Data,-and-Other-Data-Base,-Yet-Maintaining-Privacy&id=4867112

The A B C D of Data Mining Services

If you are very new to the term 'data mining', let the meaning be explained to you. It is form of back office support services that are being offered by many call centers to analyze data from numerous resources and amalgamate them for some useful task. The business establishments in the present generation need to develop a strategy that helps them to cooperate with the market trends and allow them to perform well. The process of data mining is actually the retrieval process of essential and informative data that helps an organization to analyze the business perspectives and can further generate better interests in cutting cost, developing revenue and to acquire valuable data on business services/products.

It is a powerful analytical tool that permits the user to customize a wide range of data in different formats and categories as per their necessity. The data mining process is an integral part of a business plan for companies that need to undertake a diverse research on the customer building process. These analytical skills are generally performed by skilled industrial experts who assist the firms to accelerate their growth through the critical business activities. With a vast applicability in the present time, the back office support services with the data mining process is helping the businesses in understanding and predicting valuable information. Some of them include:

    Profiles of customers
    Customer buying behavior
    Customer buying trends
    Industry analysis

For a layman it is somewhat the process of processing some statistical data or methods. These processes are implemented with some specific tools that preform the following:

    Automated model scoring
    Business templates
    Computing target columns
    Database integration
    Exporting models to other applications
    Incorporating financial information

There are some benefits of Data Mining. Few of them are as follows:

    To understand the requirements of the customers which can help in efficient planning.
    Helps in minimizing risk and improve ROI.
    Generate more business and target the relevant market.
    Risk free outsourcing experience
    Provide data access to business analysts
    A better understanding of the demand supply graph
    Improve profitability by detect unusual pattern in sales, claims, transactions
    To cut down the expenses of Direct Marketing

Data mining is generally a part of the offshore back office services and outsourced to business establishments that require diverse data base on customers and their particular approach towards any service or product. For example banks, telecommunication companies, insurance companies, etc. require huge data base to promote their new policies. If you represent a similar company that needs appropriate data mining process then it is better that you outsource back office support services from a third party and fulfill your business goals with excellent results.

Katie Cardwell works as a senior sales and marketing analyst for a multinational call center company, based in United States of America. She takes care of all the business operations and analysis the back office support services that power an organization. Her extensive knowledge and expertise on Non -voice call center services such as Data Mining Services, Back office support services, etc, have helped many business players to stand with a straight spine and thus making a foothold in the data processing industry.




Source: http://ezinearticles.com/?The-A-B-C-D-of-Data-Mining-Services&id=6503339