Nocoug 201311 Mysql Bigdata

8/10/2019 Nocoug 201311 Mysql Bigdata

1/47

Using MySQL for Big Data AdvantageIntegrate for Insight

Sastry [email protected] Copyright 2013, Oracle and/


2/47

Agenda The rise of Big Data & Hadoop

MySQL in the Big Data Lifecy

MySQL Solutions for Big Data

Q&A

Copyright 2013, Oracle and/or


3/47

The following is intended to outline our general product direction. It

intended for information purposes only, and may not be incorporated into any

contract. It is not a commitment to deliver any material, code, or functionality

and should not be relied upon in making purchasing decision. Th

development, release, and timing of any features or functionality described for

Oracle s products remains at the sole discretion of Oracle.

Safe Harbor Statement

Copyright 2013, Oracle and/or its affiliates. All rights reserved.


4/47


McKinsey Glo

MySQL Enterprise Monitor 2.2MySQL Cluster 7.1MySQL Cluster Manager 1.0MySQL Workbench 5.2MySQL Database 5.5MySQL Enterprise Backup 3.5

MySQL Enterprise Monitor 2.3MySQL Cluster Manager 1.1

DRIVING MySQLINNOVATION

All GA!

MySQL Enterprise Backup 3.7

Oracle VM Template for MySQLEnterprise Edition

MySQL Enterprise OracleCertificationsMySQL Windows Installer

MySQL Enterprise Security

MySQL Enterprise Scalability

MySQL Database 5.6 DMR*

MySQL Cluster 7.2 DMR

MySQL Labs!

(

early and often

)

All GA!

MySQL Clus

MySQL Clus

MySQL Utili

MySQL Migr

MySQL EnteMySQL Ente

MySQL Data

MySQL Clus

MySQL Data

A BEM

*Development Milestone Release

2010 2011

All


5/47

Filesystems, Files and LogsRedo, Undo, Data, Index, Binary, Error, Query and Slow

Pluggable Storage Engines Architecture

ConnectorsNative C API, JDBC, ODBC, .Net, PHP, Ruby, Python, VB, Perl

Connection Pool Authentication Thread Reuse Connection Limits Check Memory

Enterprise ManagementServices and Utilities

Backup & RecoverySecurity

ReplicationCluster

PartitioningInstance Manager

Information_SchemaMySQL Workbench

SQL InterfaceDDL, DML, StoredProcedures, Views,

Triggers, Etc..

ParserQuery Translation,Object Privileges

Optimizer Access Paths,

Statistics

Pluggable Storage EnginesMemory, Index and Storage Management

InnoDB MyISAM Cluster Etc Partners Comm

MySQL ServerClients and Apps

2010 Oracle Corporation Proprietary and Confidential


6/47

Cloud

Web & Enterprise OEM & ISV

Industry Leaders Rely on MySQL
http://images.google.com/imgres?imgurl=http://www.infinitytechnology.com/uploadedImages/Home/TechFocus_News/AlcatelLucent_Hor_2col_lrg.gif&imgrefurl=http://www.infinitytechnology.com/phonesystems.aspx&h=251&w=720&sz=11&hl=en&start=4&sig2=urYvX47BOQRqvADY_UMA0w&um=1&tbnid=pA91nzVKrUFImM:&tbnh=49&tbnw=140&ei=94SgSOaRCaqkpATZu-jgBw&prev=/images?q=alcatel+lucent&um=1&hl=en&suggon=0&rls=com.microsoft:en-us&sa=N


7/47


McKinsey G

Scales to 48 CPU Threads Up to 230% performance gain over MySQL 5.5

IMPROVED PERFORMANCE AND SCALABILITY

Better transactional throughput and availabilityIMPROVED INNODB

Better query exec times and diagnostics for query tuning and debuggingIMPROVED OPTIMIZER

Higher performance, availability and data integrity

IMPROVED REPLICATION

Better Instrumentation, User/Application level statistics and monitoring

IMPROVED PERFORMANCE SCHEMA

Fast, Key Value access with full ACID compliance, better developer agility

New! NoSQL ACCESS TO INNODB

MySQL 5.6: In Summary


8/47


McKinsey Glo

Multi-Threaded Slaves Binary Log Group Commit

Optimized Row-Based Replication

PERFORMANCE

Global Transaction Identifiers Replication Failover & Admin Utilities Crash Safe Slaves

FAILOVER & RECOVERY

Replication Event ChecksumsDATA INTEGRITY

Time Delayed Replication Remote Binlog Backup

Informational Log Events

DEV/OPS AGILITY

MySQL 5.6: Best Replication Features Ever


9/47

Leading Use-Case, On-Line Retail

Users

Browsing

RecommendationsProf

PurchHist

Web Logs:Pages Viewed

Comments Posted

Social media updatesPreferences

Brands Liked


Telephony Stream


10/47

BI Solutions

ACQUIRE

ORGANIZANALYZE

DECIDE

MySQL in the Big Data Lifecycle


11/47

*Leading Hadoop Vendor


Download the MySQL Guide to Big Data:http://www.mysql.com/why-mysql/white-papers/mysql-and-hadoop-guide-to-big-data-integra


12/47

NoSQLInterfaceMySQL Da

MySQL Clu


ANALYZE

DECIDE

ORGANIZE

ACQUIRE



13/47

MySQL NoSQL InterfaceDesign Goals: Fast, Flexible and Safe

Fully transactional / ACID

Blazing FastKey / Value

Queries

NoSQLacross

Dat

Combined with Schema Flexibility: Online DDL



14/47

Memcached is in-memory key-value storefor small dataIt is one of the most widely used In-Memory cache implementations for socialnetwork websitesMemcached has a simple and openprotocol as opposed to a rich client bound

to a specific language, andimplementation makes it portable across awide variety of languages andenvironments

How Memcached is used with MySQL separately

MySQL

clientSimple protocol

Set/get/add/delete

SQL query

Save/retrieve copy


15/47

InnoDB as a Key Value store

Combine the best of the NoSQL world and SQL worldMemcached listens on specific ports as the front end,directs requests directly to InnoDBSimple commands, much smaller network transmitpackagesPersistent storage from InnoDBIndex on the key column

Full ACID complianceBypass Optimizer and QP layer of MySQL anddirectly access the storage engineDual access of data (SQL and Memcached)


16/47

Key-value accessBypasses SQL pa

Implemented via:- Memcached plu- Memcached m

InnoDB API- Use existing M- Shared process

latency

MySQL 5.6: NoSQL Interface to InnoDBMemcached API



17/47

Performance

Up to 9x Higher SET / INSERT Throughpu

0

10000

20000

30000

40000

50000

60000

70000

80000

8 32 128 512

T P S

Client Connections

MySQL 5.6: NoSQL Benchmarking

Memcached API

SQL


M SQL Cl M l i l N SQL I f


18/47

MySQL Cluster: Multiple NoSQL Interfaces

Mix&

Match



19/47

NoSQL C++ API, flexaSynch benchmark 30 x Intel E5-2600 Intel Servers, 2 socket, 64GB ACID Transactions, with Synchronous Replication

0

5

10

15

20

25

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

M i l l i o n s o

f U P D A T E s p e r

S e c o n

d

MySQL Cluster Data Nodes

1.2 Billion UPDATEs per Minute



20/47

ImportApache SqoMySQL Ha


ANALYZE

DECIDE ACQUIRE

ORGANIZE



21/47

Apache Sqoop

Apache TLP, part of Hadoop project Developed by Cloudera

Bulk data import and export Between Hadoop (HDFS) and external data stores

JDBC Connector architecture Supports plug-ins for specific functionality

Fast Path Connector developed for MySQL


I ti g D t


22/47

Transactional

Data

HDFSStorage

Sqoop Job

Map

Map

Map

Map

Hadoop Cluster

Sqoop Import

Gather Metadata

Submit Map Only JobImporting Data



23/47

Hadoop Applier: Design


24/47

Hadoop Applier: Design

Uses MySQL replication techniques forreal time integration

Binlog API uses Binary Log to rapidly

fetch new data from a running servervia the replication protocol MySQL Binlog comprised of events,

each event represents a databasechange

Hadoop Applier receives the eventsusing the Binlog API, and writes thechanges into a file in HadoopDistributed File System

Other tools in Hadoop Ecosystem,suchas Apache Hive, can then consumethis data


25/47

Hadoop Applier: Implementation


26/47

Hadoop Applier: Implementation

Replicates rows inserted into a table in MySQL to Hadoop Distributed File Syste

Uses an API provided by libhdfs, a C library to manipulate files in HDFS

The library comes pre-compiled with Hadoop Distributions

Connects to the MySQL master (or reads the binary log generated by MySQL) toFetch the row insert events occurring on the masterDecode these events, extracting data inserted into each field of the rowSeparate the data by the desired field delimiters and row delimitersUse content handlers to get it in the format required

Append it to a text file in HDFS


27/47

Integration with HIVE

Hive runs on top of Hadoop. Install HIVE onthe hadoop master node

Set the default datawarehouse directorysame as the base directory into whichHadoop Applier writesCreate similar schema's on both MySQL &

HiveTimestamps are inserted as first field in

HDFS filesData is stored in 'datafile1.txt' by defaultThe working directory is

base_dir/db_name.db/tb_name

SQL Query

CREATE TABLE t (iINT);

CREtimeINT[RO

DELSTOTEX

d h


28/47

Mapping Between MySQL and HDFS Schema



29/47


AnalyzeExport DDecide

ANALYZE

DECIDE


A l Bi D


30/47

Analyze Big Data


Export Data


31/47

Export Data

HDStor

Sqoop Job

Map

Map

Map

Map

Hadoop Cluster

Sqoop Export

Gather Metadata

Submit Map Only Job

Transactional

& AnalyticsData


M SQL R ti D t b f BI


32/47

MySQL Reporting Database for BI


MySQL Operational Database for Web


33/47

MySQL Operational Database for Web


l d


34/47

Oracle PremierLifetime Support

Oracle ProductCertifications/Integrations

MySQL EnterpriseHigh Availability

MySQL EnterpriseSecurity

MySQL EnterpriseScalability

MySQL EntBacku

MySQL EnterpriseMonitor/Query Analyzer

MySQL

Data Analysis: MySQL Enterprise EditionHighest Levels of Security, Performance and Availability

MySQL EnterpriseAudit


MySQL Enterprise Monitor with Query Analyze


35/47

MySQL Enterprise Monitor with Query Analyze Tune Analytical Queries Enhance DevOps Agility


Scaling Security and Data Protection


36/47

Scaling, Security and Data Protection

MySQL Enterprise Scalability

MySQL Enterprise Security

MySQL Enterprise Audit

MySQL Enterprise Backup


MySQL Enterprise Backup


37/47

MySQL Enterprise Backup Online Backup for InnoDB Full, Incremental, Partial Backups (scriptable interface Compression Point in Time, Full, Partial Recovery options Metadata on status, progress, history

Unlimited Database Size Cross-Platform

Windows, Linux, Unix

Certified with Oracle Secure Backup

MEB Backup

Files

mysqlbackup

Ensures quick , on l ine backup and recovery of your MySQ

M SQL Enterprise Sec rit


38/47

PAM (Pluggable Authentication Modules)

Access external authentication methods Standard interface (Unix, LDAP, others) proxied and non-proxied users

Windows

Access native Windows services Authenticate users already logged intoWindows (Windows Active Directory)

Pluggable Authentication API

MySQL Enterprise SecurityMySQL External Authentication

Integr ates MySQL with existing security infrastructures and SO

5 5 MySQL Enterprise Scalability


39/47

MySQL default thread-handling excellent

performance, can limit scalability as connections grow MySQL Thread Pool improves sustainedperformance/scale as user connections grow

5.5 MySQL Enterprise ScalabilityMySQL Thread Pool

Thread Pool


40/47

Thread Pool

MySQL Enterprise Audit


41/47

Out-of-the-box logging of connections, logins, queryactivity across all or specific MySQL serversUser defined policies, filtering and log rotationDynamically enabled, disabled: no server restartXML-based audit stream per Oracle audit specificatio

Easily implemented via MySQL 5.5 Audit APIMySQL 5.5.28 and higher

Get it here: support.oracle.com and edelivery.oracle.comAdd s regu la tory com pl iance to MySQL appl icat ions

MySQL Enterprise AuditPolicy-based Auditing for MySQL Applications

Oracle Premier Support for MySQL


42/47

Straight from the Source Largest Team of MySQL Experts Backed by MySQL Developers Forward Compatible Hot Fixes MySQL Maintenance Releases MySQL Support in 29 Languages 24/7/365 Unlimited Incidents Knowledge Base MySQL Consultative Support

pp y QRely on The Experts - Get Unique Benefits

"The MySQL support service hessential in helping us with troand providing recommendatioproduction cluster, Thanks."

-- Carlos Morales Playf

O


Summary


43/47

Summary

MySQL + Hadoop: widely deployed solution

Best of both worlds SQL + NoSQL Access

Tools and expertise to support you

End to end Oracle solutions for Big Data

Integrate for Insight


Next Steps


44/47

Next Steps

Download the Guidehttp://www.mysql.com/why-mysql/white-

papers/mysql_wp_hadoop.php

Try Out MySQL 5.6http://www.mysql.com/downloads/mysql/

Engage MySQL Consultinghttp://www.mysql.com/consulting/


Thank you!


45/47

Thank you!



46/47



47/47

Nocoug 201311 Mysql Bigdata

Documents

Transcript of Nocoug 201311 Mysql Bigdata