https://en.wikipedia.org/wiki/Data_mining
	Discover ‘new’ and useful information
https://en.wikipedia.org/wiki/Data_warehouse
	integrate data from a whole business for perspective and decision making

Both involve use of Statistics; 
Both involve massive amounts of data;
Both are helpful with decision making.

Data Mining
	Look for patterns in large scale data
		Statistical analysis
		AI
	Possible users:
		Amazon
			People who bought X often bought Y
		Grocery store
			Have ALL of your purchases
			Target you for future sales
		Politician
		Law enforcement
			Facial recognition
	Knowledge Discovery in Database KDD
		Selection –what is interesting?
		Cleaning—eliminate invalid/incorrect
		Enrichment—add info from other sources
		Transformation/encode—Group, regions, ranges
		Reporting
	What can you get?
		Association
		Sequential patterns
		Classification trees
			Frequency of visit
			Amount spent
			Loyal
			Types of items
			Credit risk
				Amount of purchase
				Financing available
				Income

GOALS
	Predictions
		Discounts to offer
		Volume of sales
		Product lines
	Identification
		Was there a hacker?
		Hunt for submarines
		Gene identification
		Authentication
	Classification
		Discount, rush, loyal, regular, brand fans, healthy, organic, party,
		Convenience, school lunch, snow storm
	Optimization
		Best investment of time
			Space
			Advertising
			Materials
			Sales, profits, exposure…
Knowledge
	Deductive or inductive
	Association rules
		Statistics and other forms of analysis
			Support
			Confidence 
			(probability versus conditional probability)
			Clustering/ natural grouping
Association:
	N items of a certain object
	Look at ALL items in the same order
	Average over many orders
	What % of orders for N also contain M?
	(Problem is 2^N combinations!)
Sampling
	Select ‘a few’ to see if there is a trend
	Statistics allows you to reason from sample data
Frequency pattern tree
	Start most sold
	For each start with the most frequently combined
	Make groups
Partition algorithm
	Non-overlapping subsets
	Organic foods
	Soft drinks
	Dairy products
	Watch for patterns in DIFFERENT areas
		Organic food=> bottled water
		Rich ice cream => wine cooler

Multi-dimensional associations
	Time of day
	Weather forecast
	Day of week
	Holiday coming up
Harder to find negative correlation
	People who buy potato chips are less likely to buy filtered water
	Can merge together to look for pattern
	(all soft drinks; all snacks)
Complications
	Cardinality LARGE
	Transactions LARGE
	Vary by geography
	Vary by season
Classification
	Supervised learning
	(connectionism)
	(any bias in examples, that means there is a bias in the result)
Classification without supervised training
	Similarity
	Randomly pick items
	Add items that are similar
	Naturally develop cluster
Regression/correlation
	Smoking correlates with cancer
	RHO correlation r^2 is supposed to explain 
		How much of the variability is due to that one factore
	Medicines
		Often find side effects
		Viagra was originally a blood pressure medicine
		It’s more famous use was originally a side effect

Data Warehousing
Transactions
Managing the data of the business
	OLTP on line transaction processing
Use data to make decisions
Abstracting, consider different perspectives, statistics
Decision Support Systems DSS
Enterprise Information System EIS
Management Information Systems

DW is
Relatively non-volatile
Trend analysis
Multiple sources
Multi-dimensional
	Product X quarters X regions
Kinds
	Enterprise wide
	Virtual data warehouses
	Data mart 
		Subset of organization; tightly focused
Terms
	ODS Operational Data Store
		Raw info
	ADS analytical data store
		Built to do analysis
Complexity:
	Pivot to reorient X,Y,Z axis 
	Slice and dice—look at subsets of data
	Roll up display—gets to a higher level of organization
	Drill down—gets to a lower level or organization

	Fact table—data
star schema, snowflake, constellation, …
	bitmap indexing
		4 vectors for cars
			Econ, compact, mid, full

BUILD
	Get data; diverse sources
	Format consistency
	Clean to ensure validity
		Errors, incompletes, contradictory. Domain, trace and verify
	Fit into a data model
		Relational
		Object oriented
		Legacy
		Hierarchical
	Load into a warehouse
		Up to date
		Storage
		Loading time
Issues to consider
	Usage projection
	Fit of data model
Available sources
Design for change

See above 

Is DW like Views?
	DW persistent not on demand
	DW is not just relational, multi-dimensional
	DW can be indexed
	DW supports functionality
	DW integrated and temporal
	DW multiple sources, view pre-defined queries

Difficulties
	LOTS of data
	Quality control
	Hard to predict usage
	Software continues to evolve
	Specialist needed

Can data be correctly gathered and still wrong?
Can statistical analysis be done correctly and still be misleading?

Correlation
	Eating of bread with criminal records
		Most criminals eat bread
		Should we ban the sale of bread?
	[Compare non-criminal to criminal consumption of bread]
Game playing
	Are violent people more likely to want to play violent games?
	Are peaceful people able to get aggression out?

Study of the national debt
	Republican: study which party is in control of the House
	Correlate with the national debt
	CONCLUSION: the Democrats are responsible for the national debt
	Democrat: study which party is in control of the White House
	Correlate with the national debt
	CONCLUSION: the Republicans are responsible for the national debt

(IN historical data, the national debt went up the MOST
When there was a republican in the white house and a democratic majority in the House)

POINT:
Make sure the statistics make sense