Tuesday, 18 November 2008

Neeraj Nathai SmartBridge Picking Winners In Big Data



Picking Winners In Big Data
Big data solutions are picking up speed in the IT industry. There’s a Cambrian explosion of interesting start-ups, and all the database and business intelligence incumbents have moved to create big data offerings, or rebrand into the new universe.
The key to seeing the value of big data is understanding that it’s a business problem, not a matter of picking the right tools and waiting for the magic to happen. That said, technology choices still need to be made in an increasingly crowded and confused market.
The biggest question for anybody wanting to invest or adopt technology in this area is how to pick the winner? Glancing through marketing materials will do little to help you: everybody claims relevance to big data.
As I am often asked my opinion about big data companies, I thought I’d share some of the principles I use to help me think about the industry.
Where’s the value?
We need to understand where the actual value is in the data world. For the most part, this value doesn’t lie purely in the software. Over the past decade we have seen a rising tide of commoditization of the software stack: from operating system, to relational databases, to Hadoop itself. In fact, without this, we wouldn’t have the big data revolution as we know it.
As Hadoop has become a de facto standard, so has the notion of building on top of it with open source. There is some advantage in software innovation, but it is momentary. Once something is known to be possible, there are enough smart programmers out there that reproducing it becomes straightforward. (Because of this, I gloomily predict no shortage of patent battles in the not-too-distant future.)
Despite the flux in the software world, two things about big data remain constant: the need for compute and storage, and data itself. It’s ultimately to ownership of one or both of these factors that IT industry value will gravitate.
Compute and storage
The ever growing need for computing power and storage bodes well for companies providing the basics. These fall into two categories: hardware manufacturers, and cloud infrastructure providers. Not that these two markets are immune to their own fluctuations, thanks to standardization and commoditization, but fundamentally, getting paid for use of metal is the name of the game.
In this respect, it’s not too much of a mystery why storage company EMC has plowed so early and so deep into the big data world. Neither is it hard to see the reasoning behind Intel creating its own optimized Hadoop distribution.
Data
Value resting in data is the more subtle of the two axes of big data success.
Big data is ultimately about the smart use of data to drive a business. There are two kinds of data: data about your business, and data external to your business that you can create value from.
It’s easy to see who might get success from the latter, external data. We can expect that massive data owners such as Google, Facebook, Thomson Reuters, Bloomberg will experience ongoing success for as long as they are able to create product from their data.
Who owns your data, though? The obvious answer, you, isn’t the only answer. In fact, your data is locked up inside the platform choices you make, at both the hardware and software level. If your systems are based on Oracle, Microsoft, you are very unlikely to move in a hurry. Data likes to stay where it is, and tends to attract more data as you build systems around it. Production systems are expensive to replace.

So, the vendors of your software platforms of choice also get long term value from your data. For this reason, it’s hard to bet against existing enterprise application platform incumbents in the big data world. Big data, for most of today’s organizations, is an additive phenomenon, not a challenge to the core of IT.
We’ll see more large platforms ensuring nobody needs to move away. Examples include SAP adding HANA, in order to enable their existing customers for big data, or Amazon Web Services’ addition of their data warehouse Redshift, to ensure that the entire data needs of a company can be met on their platform.
Finally, it would be lunacy to bet against either Oracle, who have been portentously quiet in the big data world over the last year, or Microsoft, who in Excel have the world’s most popular data manipulation environment.
It is all business as usual?
I’m not saying there is no new opportunity in the big data industry. What I am saying is that, as an additive technology, big data is unlikely to enable anybody to challenge Oracle or Microsoft for the throne.
There will be change, though, and it will bring both winners and losers.
The area of value we haven’t yet looked at yet is the point where data interacts with the actual mechanism of a business. That’s where it transfers value to you and enables you to leverage data to get ahead. This breaks down into several areas of opportunity for big data innovators.
  • Domain specific: tools that enable the manipulation and exploitation of data in a way that’s specific to a business segment. We see initial evidence of this market opportunity in the evolution of web and customer analytics products.
  • Machine learning: more data means more to understand, and the only way an organization can realistically do this is with the aid of computers. Machine learning helps automate many parts of the data wrangling process. Furthermore, cross-company data sharing can significantly boost the effectiveness of machine-learning, creating the opportunity for companies gaining early market share. A recent example of this is Sift Science, a fraud detection application.
  • Tools for exploration: human interaction with data is a requirement that’s hard to abstract away. Tableau has a head start in this market for big data, filling the role of “Microsoft Office for Data”, but the field is ripe for new innovation, especially with the increasing power of graphical capabilities and new device formats.
  • Data agility: speed-to-decision is a critical factor in business competitiveness. Any solution that removes laborious steps has an advantage: a particularly problematic area here is data integration, the loading of both internal and external data sources ready for analytics. Most of today’s big data solutions are frankensteined combinations of layers: there’s a great opportunity for vertically integrated solutions that removes needless impedance to data manipulation.
Move up http://i.forbesimg.com t Move down
Areas of risk
What are those riskier big data options? If a solution isn’t scoring high in the categories above, it’s not likely to be around for the long run.
The biggest risk is with solutions that address only a single horizontal part of the data architecture. Time is against companies in this game. By selling just part of a complete solution, they’re working against the rising tide of commoditization. Customers will demand standardization (e.g. Hadoop compatibility) in order to feel safe adopting such solutions, but that prevents the lock-in that will protect that business. It’s not Oracle’s relational database that cements their position: it’s their vertical position up and down the application stack.
Therefore, it’s not surprising to see the pure Hadoop distribution companies making partnerships, and branching out into other vertical layers. In the long term, it’s a tough road they’ve chosen.
One company working actively to solve this problem is DataStax, who have pivoted from being seen as the corporate backer for the Cassandra NoSQL database—a risky horizontal play—to selling an integrated stack of Cassandra, Hadoop and Solr, intended as a complete platform for building enterprise applications. They’re going after some of the platform business: being involved in the actual use of data to drive business value.
For some start-ups, not having long term big data viability might be just fine as a strategy. As the bigger enterprise companies lumber into the arena, they’ll prove handy acquisitions. But given the crowded space and progressive commoditization, this isn’t a certain future. And certainly for their customers, the risks are growing.
More controversially, another area at risk is that of traditional ETL. Or in its broader sense, data integration. This is a hard, hard, problem. It’s not easy to integrate data retrospectively, and it’s not all certain that the integration players from the data warehouse world will be able to translate that success into the big data world. Many early adopters of Hadoop were motivated by the fact that existing ETL solutions couldn’t meet their needs.
In the long term, data integration is likely to be best served by entire new architectures that don’t try to separate and tame the data in the first place. It’s a lot easier to build truly integrated data infrastructure as greenfield.
For those who crack the integration problem, the potential rewards are high. But so is the risk.
Conclusion
In the long term, the additive nature of big data, combined with inertia, makes it a safe bet that current enterprise IT incumbents will continue their reign, as long as they move to embrace big data in their architectures.
There is plenty of opportunity though, especially in greenfield and cloud scenarios. Expect to see increasing returns for those who provide integrated solutions and do a good job of equipping human decision makers.
There’s long term value in metal, and value in data. About everything else, it’s worth thinking carefully.


Source: Wikipedia./Forbes

Posted by at No comments:

Thursday, 14 August 2008

Neeraj Nathai SmartBridge Big data



Big data
Big data[1][2] is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage,[3] search, sharing, transfer, analysis,[4] and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to "spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions."[5][6][7]
As of 2012[update], limits on the size of data sets that are feasible to process in a reasonable amount of time were on the order of exabytes of data.[8][9] Scientists regularly encounter limitations due to large data sets in many areas, including meteorology, genomics,[10] connectomics, complex physics simulations,[11] and biological and environmental research.[12] The limitations also affect Internet search, finance and business informatics. Data sets grow in size in part because they are increasingly being gathered by ubiquitous information-sensing mobile devices, aerial sensory technologies (remote sensing), software logs, cameras, microphones, radio-frequency identification readers, and wireless sensor networks.[13][14] The world's technological per-capita capacity to store information has roughly doubled every 40 months since the 1980s;[15] as of 2012[update], every day 2.5 quintillion (2.5×1018) bytes of data were created.[16] The challenge for large enterprises is determining who should own big data initiatives that straddle the entire organization.[17]
Big data is difficult to work with using most relational database management systems and desktop statistics and visualization packages, requiring instead "massively parallel software running on tens, hundreds, or even thousands of servers".[18] What is considered "big data" varies depending on the capabilities of the organization managing the set, and on the capabilities of the applications that are traditionally used to process and analyze the data set in its domain. "For some organizations, facing hundreds of gigabytes of data for the first time may trigger a need to reconsider data management options. For others, it may take tens or hundreds of terabytes before data size becomes a significant consideration."[19].

Definition

Big data usually includes data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target, as of 2012[update] ranging from a few dozen terabytes to many petabytes of data in a single data set. With this difficulty, new platforms of "big data" tools are being developed to handle various aspects of large quantities of data.
In a 2001 research report[20] and related lectures, META Group (now Gartner) analyst Doug Laney defined data growth challenges and opportunities as being three-dimensional, i.e. increasing volume (amount of data), velocity (speed of data in and out), and variety (range of data types and sources). Gartner, and now much of the industry, continue to use this "3Vs" model for describing big data.[21] In 2012, Gartner updated its definition as follows: "Big data are high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization."[22

Examples

Examples include Big Science, web logs, RFID, sensor networks, social networks, social data (due to the social data revolution), Internet text and documents, Internet search indexing, call detail records, astronomy, atmospheric science, genomics, biogeochemical, biological, and other complex and often interdisciplinary scientific research, military surveillance, forecasting drive times for new home buyers, medical records, photography archives, video archives, and large-scale e-commerce.

Big science

The Large Hadron Collider experiments represent about 150 million sensors delivering data 40 million times per second. There are nearly 600 million collisions per second. After filtering and refraining from recording more than 99.999% of these streams, there are 100 collisions of interest per second.[23][24][25]
  • As a result, only working with less than 0.001% of the sensor stream data, the data flow from all four LHC experiments represents 25 petabytes annual rate before replication (as of 2012). This becomes nearly 200 petabytes after replication.
  • If all sensor data were to be recorded in LHC, the data flow would be extremely hard to work with. The data flow would exceed 150 million petabytes annual rate, or nearly 500 exabytes per day, before replication. To put the number in perspective, this is equivalent to 500 quintillion (5×1020) bytes per day, almost 200 times higher than all the other sources combined in the world.

Science and research

  • When the Sloan Digital Sky Survey (SDSS) began collecting astronomical data in 2000, it amassed more in its first few weeks than all data collected in the history of astronomy. Continuing at a rate of about 200 GB per night, SDSS has amassed more than 140 terabytes of information. When the Large Synoptic Survey Telescope, successor to SDSS, comes online in 2016 it is anticipated to acquire that amount of data every five days.[5]
  • Decoding the human genome originally took 10 years to process; now it can be achieved in one week.[5]
  • Computational social science — Tobias Preis et al. used Google Trends data to demonstrate that Internet users from countries with a higher per capita gross domestic product (GDP) are more likely to search for information about the future than information about the past. The findings suggest there may be a link between online behaviour and real-world economic indicators.[26][27][28] The authors of the study examined Google queries logs made by Internet users in 45 different countries in 2010 and calculated the ratio of the volume of searches for the coming year (‘2011’) to the volume of searches for the previous year (‘2009’), which they call the ‘future orientation index’.[29] They compared the future orientation index to the per capita GDP of each country and found a strong tendency for countries in which Google users enquire more about the future to exhibit a higher GDP. The results hint that there may potentially be a relationship between the economic success of a country and the information-seeking behavior of its citizens captured in big data.

Government

Private sector

  • Amazon.com handles millions of back-end operations every day, as well as queries from more than half a million third-party sellers. The core technology that keeps Amazon running is Linux-based and as of 2005 they had the world’s three largest Linux databases, with capacities of 7.8 TB, 18.5 TB, and 24.7 TB.[37]
  • Walmart handles more than 1 million customer transactions every hour, which is imported into databases estimated to contain more than 2.5 petabytes (2560 terabytes) of data – the equivalent of 167 times the information contained in all the books in the US Library of Congress.[5]
  • Facebook handles 50 billion photos from its user base.
  • FICO Falcon Credit Card Fraud Detection System protects 2.1 billion active accounts world-wide.[38]
  • The volume of business data worldwide, across all companies, doubles every 1.2 years, according to estimates.[39]
  • Infosys has also launched the BigDataEdge to analyse the Big data.[40][41]
  • Windermere Real Estate uses anonymous GPS signals from nearly 100 million drivers to help new home buyers determine their typical drive times to and from work throughout various times of the day [42]

International development

Following decades of work in the area of the effective usage of information and communication technologies for development (or ICT4D), it has been suggested that Big Data can make important contributions to international development.[43][44] On the one hand, the advent of Big Data delivers the cost-effective prospect to improve decision-making in critical development areas such as health care, employment, economic productivity, crime and security, and natural disaster and resource management.[45] On the other hand, all the well-known concerns of the Big Data debate, such as privacy, interoperability challenges, and the almighty power of imperfect algorithms, are aggravated in developing countries by long-standing development challenges like lacking technological infrastructure and economic and human resource scarcity. "This has the potential to result in a new kind of digital divide: a divide in data-based intelligence to inform decision-making."[45]
Technologies
File:DARPA’s Topological Data Analysis program seeks the fundamental structure of massive data sets and is developing the tools to exploit that knowledge.tiff
DARPA’s Topological Data Analysis program seeks the fundamental structure of massive data sets.
Big data requires exceptional technologies to efficiently process large quantities of data within tolerable elapsed times. A 2011 McKinsey report[49] suggests suitable technologies include A/B testing, association rule learning, classification, cluster analysis, crowdsourcing, data fusion and integration, ensemble learning, genetic algorithms, machine learning, natural language processing, neural networks, pattern recognition, anomaly detection, predictive modelling, regression, sentiment analysis, signal processing, supervised and unsupervised learning, simulation, time series analysis and visualisation. Multidimensional big data can also be represented as tensors, which can be more efficiently handled by tensor-based computation,[50] such as multilinear subspace learning.[51] Additional technologies being applied to big data include massively parallel-processing (MPP) databases, search-based applications, data-mining grids, distributed file systems, distributed databases, cloud based infrastructure (applications, storage and computing resources) and the Internet.[citation needed]
Some but not all MPP relational databases have the ability to store and manage petabytes of data. Implicit is the ability to load, monitor, back up, and optimize the use of the large data tables in the RDBMS.[52]
DARPA’s Topological Data Analysis program seeks the fundamental structure of massive data sets and in 2008 the technology went public with the launch of a company called Ayasdi.
The practitioners of big data analytics processes are generally hostile to slower shared storage[citation needed], preferring direct-attached storage (DAS) in its various forms from solid state disk (SSD) to high capacity SATA disk buried inside parallel processing nodes. The perception of shared storage architectures—SAN and NAS—is that they are relatively slow, complex, and expensive. These qualities are not consistent with big data analytics systems that thrive on system performance, commodity infrastructure, and low cost.
Real or near-real time information delivery is one of the defining characteristics of big data analytics. Latency is therefore avoided whenever and wherever possible. Data in memory is good—data on spinning disk at the other end of a FC SAN connection is not. The cost of a SAN at the scale needed for analytics applications is very much higher than other storage techniques.
There are advantages as well as disadvantages to shared storage in big data analytics, but big data analytics practitioners as of 2011[update] did not favour it.[53]


Source: Wikipedia.

Posted by at No comments:

Friday, 11 July 2008

Neeraj Nathai SmartBridge RETAILING



RETAILING

Retail is the sale of goods and services from individuals or businesses to the end-user. Retailers are part of an integrated system called the supply chain. A retailer purchases goods or products in large quantities from manufacturers directly or through a wholesale, and then sells smaller quantities to the consumer for a profit. Retailing can be done in either fixed locations like stores or markets, door-to-door or by delivery. Retailing includes subordinated services, such as delivery. The term "retailer" is also applied where a service provider services the needs of a large number of individuals, such as a public. Shops may be on residential streets, streets with few or no houses or in a shopping mall. Shopping streets may be for pedestrians only. Sometimes a shopping street has a partial or full roof to protect customers from precipitation. Online retailing, a type of electronic commerce used for business-to-consumer (B2C) transactions and mail order, are forms of non-shop retailing.
Shopping generally refers to the act of buying products. Sometimes this is done to obtain necessities such as food and clothing; sometimes it is done as a recreational activity. Recreational shopping often involves window shopping (just looking, not buying) and browsing and does not always result in a purchase
Types of retail outlets
A marketplace is a location where goods and services are exchanged. The traditional market square is a city square where traders set up stalls and buyers browse the merchandise. This kind of market is very old, and countless such markets are still in operation around the whole world.
In some parts of the world, the retail business is still dominated by small family-run stores, but this market is increasingly being taken over by large retail chains.
Retail is usually classified by type of products as follows:
  • Food products
  • Hard goods or durable goods ("hardline retailers") - appliances, electronics, furniture, sporting goods, etc. Goods that do not quickly wear out and provide utility over time.
  • Soft goods or consumables - clothing, apparel, and other fabrics. Goods that are consumed after one use or have a limited period (typically under three years) in which you may use them.
There are the following types of retailers by marketing strategy:
  • Department stores - very large stores offering a huge assortment of "soft" and "hard goods; often bear a resemblance to a collection of specialty stores. A retailer of such store carries variety of categories and has broad assortment at average price. They offer considerable customer service.
  • Discount stores - tend to offer a wide array of products and services, but they compete mainly on price offers extensive assortment of merchandise at affordable and cut-rate prices. Normally retailers sell less fashion-oriented brands.
  • Warehouse stores - warehouses that offer low-cost, often high-quantity goods piled on pallets or steel shelves; warehouse clubs charge a membership fee;
  • Variety stores - these offer extremely low-cost goods, with limited selection;
  • Demographic - retailers that aim at one particular segment (e.g., high-end retailers focusing on wealthy individuals).
  • Mom-And-Pop : is a retail outlet that is owned and operated by individuals. The range of products are very selective and few in numbers. These stores are seen in local community often are family-run businesses. The square feet area of the store depends on the store holder.
  • Specialty stores: A typical speciality store gives attention to a particular category and provides high level of service to the customers. A pet store that specializes in selling dog food would be regarded as a specialty store. However, branded stores also come under this format. For example if a customer visits a Reebok or Gap store then they find just Reebok and Gap products in the respective stores.
  • General store - a rural store that supplies the main needs for the local community;
  • Convenience stores: is essentially found in residential areas. They provide limited amount of merchandise at more than average prices with a speedy checkout. This store is ideal for emergency and immediate purchases as it often works with extended hours, stocking everyday;
  • Hypermarkets: provides variety and huge volumes of exclusive merchandise at low margins. The operating cost is comparatively less than other retail formats.
  • Supermarkets: is a self-service store consisting mainly of grocery and limited products on non food items. They may adopt a Hi-Lo or an EDLP strategy for pricing. The supermarkets can be anywhere between 20,000 and 40,000 square feet (3,700 m2). Example: SPAR supermarket.
  • Malls: has a range of retail shops at a single outlet. They endow with products, food and entertainment under a roof.
  • Category killers or Category Specialist: By supplying wide assortment in a single category for lower prices a retailer can "kill" that category for other retailers. For few categories, such as electronics, the products are displayed at the centre of the store and sales person will be available to address customer queries and give suggestions when required. Other retail format stores are forced to reduce the prices if a category specialist retail store is present in the vicinity.
  • E-tailers: The customer can shop and order through internet and the merchandise are dropped at the customer's doorstep. Here the retailers use drop shipping technique. They accept the payment for the product but the customer receives the product directly from the manufacturer or a wholesaler. This format is ideal for customers who do not want to travel to retail stores and are interested in home shopping. However it is important for the customer to be wary about defective products and non secure credit card transaction. Example: Amazon, Pennyful and eBay.
  • Vending Machines: This is an automated piece of equipment wherein customers can drop the money in the machine and acquire the products.
Some stores take a no frills approach, while others are "mid-range" or "high end", depending on what income level they target.
Other types of retail store include:
  • Automated Retail stores are self-service, robotic kiosks located in airports, malls and grocery stores. The stores accept credit cards and are usually open 24/7. Examples include ZoomShops and Redbox.
  • Big-box stores encompass larger department, discount, general merchandise, and warehouse stores.
Retailers can opt for a format as each provides different retail mix to its customers based on their customer demographics, lifestyle and purchase behaviour. A good format will lend a hand to display products well and entice the target customers to spawn sales.
Global Top Five Retailers
Worldwide Top Five Retailers[2]
Retail Sales Rank
Company
Country of Origin
2010 group revenue (US $mil)
1
Walmart
US
$421,849
2
Carrefour
France
$121,519
3
Tesco
UK
$94,244
4
Metro AG
Germany
$89,311
5
Kroger
US
$82,189

Operations

Retail pricing

The pricing technique used by most retailers is cost-plus pricing. This involves adding a markup amount (or percentage) to the retailer's cost. Another common technique is suggested retail pricing. This simply involves charging the amount suggested by the manufacturer and usually printed on the product by the manufacturer.
In Western countries, retail prices are often called psychological prices or odd prices. Often prices are fixed and displayed on signs or labels. Alternatively, when prices are not clearly displayed, there can be price discrimination, where the sale price is dependent upon who the customer is. For example, a customer may have to pay more if the seller determines that he or she is willing and/or able to. Another example would be the practice of discounting for youths, students, or senior citizens..

Staffing

Because patronage at a retail outlet varies flexibility in scheduling is desirable. Employee scheduling software is sold which, using known patterns of customer patronage, more or less reliably predicts the need for staffing for various functions at times of the year, day of the month or week, and time of day. Usually needs vary widely. Conforming staff utilization to staffing needs requires a flexible workforce which is available when needed but does not have to be paid when they are not, part-time workers; as of 2012 70% of retail workers in the United States were part-time. This may result in financial problems for the workers, who while they are required to be available at all times if their work hours are to be maximized, may not have sufficient income to meet their family and other obligations.[3]

Transfer mechanisms

There are several ways in which consumers can receive goods from a retailer:
  • Counter service, where goods are out of reach of buyers and must be obtained from the seller. This type of retail is common for small expensive items (e.g. jewelry) and controlled items like medicine and liquor. It was common before the 1900s in the United States and is more common in certain countries like India.[which?]
  • Delivery, where goods are shipped directly to consumer's homes or workplaces. Mail orderfrom a printed catalog was invented in 1744 and was common in the late 19th and early 20th centuries. Ordering by telephone is now common, either from a catalog, newspaper, television advertisement or a local restaurant menu, for immediate service (especially for pizza delivery). Direct marketing, including telemarketing and television shopping channels, are also used to generate telephone orders. started gaining significant market share in developed countries in the 2000s.
  • Door-to-door sales, where the salesperson sometimes travels with the goods for sale.
  • Self-service, where goods may be handled and examined prior to purchase

Source: Wikipedia.

Posted by at No comments:
Subscribe to: Posts (Atom)