OpenStack & MySQL

57
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | OpenStack & MySQL Airton Lastori [email protected] Copyright © 2015, Oracle and/or its affiliates. All rights reserved.

Transcript of OpenStack & MySQL

Page 1: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

OpenStack & MySQL

Airton Lastori [email protected]

Copyright © 2015, Oracle and/or its affiliates. All rights reserved.

Page 2: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

3

Page 3: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Program Agenda

An Introduction to OpenStack

An Introduction to Trove

Why MySQL Enterprise Edition?

Integrating MySQL Enterprise with Trove

What’s Coming Next?

1

2

3

4

5

4

Page 4: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

An Introduction to OpenStack

5

Page 5: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

The OpenStack Mission: to produce the ubiquitous Open Source Cloud Computing platform that will meet the needs of public and private clouds regardless of size, by being simple to implement and massively scalable.

6

Page 6: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Contribuidores

• Oracle

• HP

• Microsoft

• IBM

• Redhat

• AT&T

• Muitos outros...

Usuários BR

• Instituto Federal de Santa Catarina

• PUCRS

• UFMG

• Globo.com

• R7

• Talvez mais...

7

Fundação Openstack.org

www.openstack.org/user-stories www.openstack.org/foundation/companies

Page 7: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

OpenStack Momentum • Launched in 2010, OpenStack has grown to become one of the largest open source

project ever created. Some numbers from http://www.stackalytics.com/ as of May 2015:

– Over 400 companies contributed code to OpenStack (Kilo)

– Almost 2700 individuals contributed code to the project (Kilo)

8

0

20

40

60

80

100

120

140

Companies Contributing Up to Juno

0

200

400

600

800

1000

1200

1400

1600

Individuals Contributing Up to Juno

Page 8: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

OpenStack Momentum

Supporting Companies: http://www.openstack.org/foundation/companies/

9

Page 9: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Top 5 Business Drivers for OpenStack

Source: http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014

10

Page 10: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

OpenStack Deployments

Source: http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014

11

Page 11: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Workloads & Databases

Source: http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014

12

Page 12: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

What is OpenStack?

• OpenStack is an open source cloud operating system designed to control large scale cloud deployments

• OpenStack controls compute, storage, network, identity management, orchestration, and much more … pool of resources ready-to-use

13

Page 13: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 14

Page 14: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Cloud Privada

• On Premise

– Distros & Appliances

– Instalação “na unha”

• Hosted Private Clouds

– Rackspace

– Ubuntu

Outros...

Cloud Pública

• Rackspace

• HP

• Vexxhost

• Ultimum

• KIO

• Data Centred

Vários outros...

15

Como usar

www.openstack.org/marketplace

Page 15: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Distros & Appliances

• devstack.org

• Oracle (Linux e Solaris)

• Tesora

• HP

• Redhat

• VMWare

• Muito mais...

Guias de Instalação

• Oracle Linux, Redhat, CentOS 7

• Ubuntu 14.04 (LTS)

• openSUSE 13.2 e Enterprise SUSE

• Debian 8

16

Distros, Appliances ou Instalação “na unha”

docs.openstack.org www.openstack.org/marketplace/distros

Page 16: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

OpenStack Cloud Operating System

17

Page 17: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

OpenStack Services

• Compute Service : Nova

• Object Storage Service : Swift

• Block Storage Service : Cinder

• Image Service : Glance

• Identity Service : Keystone

• Network Service : Neutron

• Dashboard : Horizon

• Telemetry Service : Ceilometer

• Orchestration Service : Heat

• Database Service : Trove

• Data Processing : Sahara

• Bare-Metal Provisioning : Ironic

And many more …

18

core

Page 18: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Most OpenStack Services Use MySQL Availability and Scalability of MySQL Is Crucial for Operation and Performance

• Very large transactions from services going into MySQL • To avoid bottleneck

• Automatically add nodes for scalability and availability • Instead of downtime to restructure

• Important to have back ups (MySQL Enterprise Backup) • If MySQL goes down your service and cloud is down

http://docs.openstack.org/openstack-ops/content/architecture.html

Page 19: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

OpenStack is About Abstraction and Integration

20

Block Storage service

Cinder

Network service

Neutron

Compute service

Nova Application Programming Interfaces

Plugins then Implement the API (partial list)

Common Interface – OpenStack API

Page 20: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Amazon OpenStack Trove

21

How Does OpenStack Trove Compare to Amazon RDS …and DynamoDB …and RedShift

Page 21: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Goals: OpenStack With Oracle Linux and Oracle VM

Interoperability

• Help make OpenStack a first class solution for enterprise customers

• Work with the OpenStack community to support enterprise applications in an OpenStack cloud

• Provide choice for managing hypervisors with Oracle Linux and Oracle VM

• Ensure Oracle Linux and Oracle VM can work with any OpenStack solution customers choose

• Work with third party vendors to offer joint solutions

22

Page 22: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

OpenStack Across Oracle’s Portfolio

Horizon

Cloud Management

Oracle Linux , Oracle Solaris, Oracle VM,

Nova Compute Virtualization

Oracle Linux , Oracle Solaris, Oracle Virtual

Networking

Neutron Cloud Networking

Oracle Linux, Oracle Solaris, Oracle ZS3,

Oracle Axiom, Oracle StorageTek Tape,

Oracle FS1

Cinder/Swift Cloud Storage

Oracle Linux, Oracle Solaris, Oracle VM

Templates

Glance Image Deployment

23

Page 23: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

End to End Full Stack Support

24

Control node Compute node n Compute node 1

VM VM

VM

VM VM

VM

Clusterware

Page 24: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Oracle OpenStack High Availability MySQL Cluster Active/Active

MySQL Cluster RabbitMQ Keepalived Nova Neutron Memcached

Cinder Swift Keystone Glance Heat Horizon

Docker Containers

Controller Node(s)

MySQL Cluster RabbitMQ Keepalived Nova Neutron Memcached

Cinder Swift Keystone Glance Heat Horizon

Docker Containers

Container life cycle management (Ansible)

Management Controller Nodes

API

MySQL Cluster Data Nodes

Management

Data Layer

HA

Pro

xy

Current A/A: Galera

MySQL Cluster

Scaling Limited write Linear write

Performance Standard Real-time

Online DDL No Yes

Auto Sharding No Yes

NoSQL APIs No Yes

Load Balancing No Yes

Page 25: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

An Introduction to Trove

26

Page 26: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Current State of Database Deployments

• Ever Growing Database Population

• Multiple versions and patch levels

• Compliance challenges

• Poor resource utilization

• High cost of deployment and operation

Siloed, Dispersed, Varied and Complex

27

Page 27: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

DBaaS Key Capabilities

Self Service Metering

Elasticity

Service Catalog High Availability

Security

End-users can provision, monitor & manage for greater agility

Allows chargeback for cost allocation

Support dynamic workloads with minimal excess capacity

Standardized database services for rapid, error-free deployment

Avoiding costly unplanned downtime

Protecting against external attacks, misuse of information & ensuring regulatory compliance

28

Page 28: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Highlights:

• Scalable and reliable cloud DBaaS provisioning

• Built for needs of Cloud users and DBAs

• Provision/manage multiple MySQL EE instances

• Automates administration across lifecycle:

– Deployment

– Configuration

– Patching

– Backups / restores

– Monitoring

Benefits:

• Reduces database sprawl

• Supports rapid provisioning

• Enhances security

• Improves performance and availability

• Automates management

Improves agility, security, and reduces TCO

29

DBaaS with OpenStack Trove and MySQL Enterprise

* MarketsandMarkets forecasts the Cloud Database and DBaaS Market to grow from $1.07 billion in 2014 to $14.05 billion by 2019, at a Compound Annual Growth Rate (CAGR) of 67.30% during the forecast period of 2014-2019.

Page 29: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Tuning

• Automatically tune my.cnf

• InnoDB Buffer Pool size

• Replication options

• Max Connections

• …

• Sane defaults

• InnoDB only

• Disable LOAD DATA INFILE

• Disable SELET INTO OUTFILE

• API to programmatically set configuration groups

Management

• Create database / schema

• Create users

• Grant permissions

• Enable root user

• Resize flavor

• Resize volume

• Full and incremental backups

• Create replicas

• Create clusters

30

Security

• Security groups

• Turn off SSH

• Remove anonymous user(s)

• Remove non-localhost user(s)

• Remove local file access

• Mangle root user password

• Apply security patches automatically

Trove is more than just provisioning…

Page 30: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Trove Architecture

31

• Built on OpenStack – Backups to Swift

– Images in Glance

– Nova management

– Data in Cinder

• Major components – API

– Taskmanager

– Conductor

– Guest Agent

Page 31: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Provisioning Models

32

PDB

Pre-provisioned

hardware

Oracle 11g

Pre-provisioned

hardware

Mongo DB

Pre-provisioned

hardware

MySQL

OpenStack Cloud

Oracle 12c Container Database

PDB PDB

Trove DBaaS

Trove Instance

Trove Instance

Oracle Trove

Instance

MySQL

Cluster

Nova Instance

Oracle

Nova Instance

Oracle

Trove Instance

MySQL

Cluster

Single Instance

Provisioning

Cluster

Provisioning Multi-tenant

DBMS Provisioning

Provision from

Inventory

PDB

PDB

* Oracle Database 11g and 12c are currently only supported in Tesora DBaaS Enterprise

*

*

Page 32: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Enterprise Hardened Trove: Tesora DBaaS Platform

OpenStack Trove Community Edition

Enterprise DBaaS Features

Early Access to Upstream

24/7 Support

Guest Images Proprietary DBs

Simplified Install/Config

Community Code

Enterprise Edition

Simplified Install/Config

Extensive Testing/Bug Fixes

Guest Images O/S DBs

Community Code Community Code

Extensive Testing/Bug Fixes

Guest Images O/S DBs

Enhanced GUI Management

33

Page 33: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

• As a Trove user

– OpenStack distribution that includes Trove (e.g. Oracle OpenStack) • http://www.oracle.com/us/technologies/linux/op

enstack/overview/index.html

– Tesora DBaaS platform, a standalone Trove-based DBaaS product • http://tesora.com/download-tesora-dbaas-

platform-community-edition

• As a Trove developer $ git clone http://github.com/openstack/trove-integration

$ cd trove-integration/scripts

$ ./redstack install

$ ./redstack kick-start mysql

• On top of DevStack

– Add to localrc: • ENABLED_SERVICES+=,trove,tr-api,tr-tmgr,tr-cond

– Swift should also be enabled for Backup and Restore. • ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-account

• SWIFT_HASH=<swift-hash-here>

34

Getting Started with OpenStack Trove

Page 34: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Why MySQL Enterprise?

35

Page 35: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Why MySQL Enterprise Edition? In Addition to all the MySQL Features you Love

Insure Your Deployments

Get the Best Results

Delight Customers

Improve Performance & Scalability

Enhance Agility & Productivity

Reduce TCO

Mitigate Risks

Get Immediate

Help if/when Needed

Increase Customer

Satisfaction

36

Page 36: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Management Tools Advanced Features Support

• Scalability • High Availability • Security • Audit • Encryption

•Monitoring • Backup • Development • Administration •Migration

• Technical Support • Consultative Support •Oracle Certifications

MySQL Enterprise Edition

37

Page 37: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

MySQL HA in OpenStack

• High Availability

– Server monitoring with auto-promotion and transparent application failover

• Optionally scale-out through sharding

– Application provides shard key

– Tools for resharding

– Global updates & tables

• Connection options

– Fabric-aware connectors

– MySQL Router

• Server provisioning using OpenStack

– Supports the Nova and Neutron APIs

MySQL Enterprise High Availability + Sharding-Based Scale-out

Router

Application

Read-slaves

SQL

HA group

Connector

Application

38

Group Replication cluster

Fabric node cluster

Page 38: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

• Active/Active Update Anywhere

– Conflict detection and resolution (transaction rollback)

– Optimistic State Machine Replication

• Automatic group membership management and failure detection

– No need for server fail-over

– Elastic scale out/in

– No single point of failure

– Automatic reconfiguration

• Well integrated

– InnoDB

– GTID-based replication

– PERFORMANCE_SCHEMA

– Row based replication

– Multi-threaded slaves

MySQL Group Replication

Application

MySQL Nodes Replication

Plugin

API

MySQL Server

Group Comms

labs.mysql.com

39

For Active/Active OpenStack Setups

Page 39: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

• Connection and Transaction routing

• Transparently improve your MySQL apps

– Transparent MySQL Fabric support • Transparent HA

• Transparent Sharding

– Transparent support for MySQL Group Replication clusters

– Transparent support for custom clusters and HA setups

• Easily extendable using plugin APIs

• Many new plugins to come

– Aggregation, Binary Log, Intelligent Load Balancing, Read/Write splitting, …

40

MySQL Router Transparent Routing for Complex MySQL Installations

Page 40: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Scaling, Security, and Data Protection

MySQL Enterprise Scalability

MySQL Enterprise Monitor

MySQL Enterprise Backup

MySQL Enterprise Security

MySQL Enterprise Encryption

MySQL Enterprise Audit

MySQL Enterprise Authentication

MySQL Enterprise High Availability

Oracle Enterprise Manager for MySQL

41

Page 41: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Integrating MySQL Enterprise & Trove

42

Page 42: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

MySQL Enterprise & Trove

43

• Create a MySQL Enterprise machine image for use with Trove

– Pull a standard image from Glance and add/install: • MySQL Enterprise Server

• Default MySQL config (/etc/mysql/my.cnf) that enables all Enterprise Plugins

– Can later use a Trove configuration group to sync and manage options across all instances

• MySQL Enterprise Backup

• MySQL Enterprise Monitor Agent

– Configured to know how to reach out to the MySQL Enterprise Monitor Server

• Trove Guest Agent

– Configured to know how to reach the Trove Server (API, Taskmanager, Conductor)

– Customized backup strategy definitions for MySQL Enterprise Backup

• trove/guestagent/strategies/backup/mysql_impl.py

• See: http://dev.mysql.com/doc/mysql-enterprise-backup/3.12/en/mysqlbackup.usage.html

Page 43: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

• Store customized MySQL Enterprise Trove image in Glance

– Or skip all of the image related steps • Official certified Oracle Linux + MySQL Enterprise Trove images

• Tesora DBaaS Enterprise Certified MySQL Enterprise Trove images

• Create necessary entries in Keystone • User, role, tenant, service, endpoint

• Create MySQL datastore within Trove

– Use trove-manage client • Sync Trove information to OpenStack’s MySQL data store (trove-manage

db_sync)

• Create MySQL datastore using new Trove machine image in Glance

MySQL Enterprise & Trove

44

Page 44: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

• Configure MySQL Enterprise Backups

– Specify MySQL Enterprise Backup as tool for backups (/etc/trove/trove.conf):

[mysql]

backup_strategy=mysqlbackup

backup_incremental_strategy={'mysqlbackup': 'incremental'}

• Instances can then be created with trove client or via Horizon web UI

MySQL Enterprise & Trove

45

Page 45: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

MySQL Enterprise & Trove

46

• Launch MySQL EE instance

– Flavor from Nova

– Cinder volume for the datadir

• Initialize the instance – Specify default accounts

– Specify default schemas

• Optionally restore from a backup

Page 46: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 47

Page 47: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

What’s Next?

48

Page 48: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Plans for MySQL Enterprise

• MySQL 5.7 (GA)

– Improved scalability, performance, security, and ease-of-use

– Server Team Blog : http://mysqlserverteam.com

• MySQL Group Replication (Labs) – Easy HA with Active/Active clusters

– Replication & HA Team Blog : http://mysqlhighavailability.com

• MySQL Router (GA)

– Load balancing, transparent failover, aggregation, transparent sharding, … stay tuned!

• MySQL Fabric (GA) – Fabric 1.6 supports multi-node clusters

49

Making MySQL “Cloud Native”

Page 49: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

The Future of MySQL Scaling (HA + Sharding)

Global Data Shard 1 Shard 2

MySQL Fabric controller cluster

SQL Queries

Server/Shard State & Mapping

Global Group HA Group

Coordination and Control

HA Group

Group Replication cluster

50

Group Replication cluster

Group Replication cluster

MySQL Router

Page 50: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Plans for OpenStack

• Liberty (October 2015) and beyond …

– Queue Service (Zaqar)

– Shared File System (Manila)

– DNS Service (Designate)

– Key Management (Barbican)

– Container Service (Docker and Kubernetes: Magnum)

– Docker deployment and management tools (Kolla) • Used heavily in Oracle OpenStack for Oracle Linux R2

– Additional details: http://status.openstack.org/release/

• Next release is Mitaka (April 2016)

51

Page 51: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Plans for Trove

• High Availability

– Auto-scaling up and down based on demand

– Failure detection and auto-failover

• Disaster Recovery – Cross region replication and failover

• Support for additional datastores

• Better integration with other OpenStack modules (e.g. Ceilometer)

• Improved security, monitoring, metering

• API for accessing datastore log files

– For example: Enterprise Audit log, Slow Query log, Error log, etc.

52

Page 52: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Appendix

53

Page 53: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

OpenStack Resources

• Docs

– http://docs.openstack.org/

• Wiki

– https://wiki.openstack.org/wiki/Main_Page

• Users Stories and Case Studies – http://www.openstack.org/user-stories/

• Community help

– https://wiki.openstack.org/wiki/IRC

– https://wiki.openstack.org/wiki/Mailing_Lists

54

Page 54: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Trove Resources

• Docs

– http://docs.openstack.org/developer/trove/

• Wiki

– https://wiki.openstack.org/wiki/Trove

• Launchpad – https://launchpad.net/trove

• Community help

– #openstack-trove on Freenode

• Events

– http://www.tesora.com/troveday/

55

Page 55: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Oracle OpenStack Resources

• Product web page (data sheet, white paper, videos, FAQ)

– http://www.oracle.com/us/technologies/linux/openstack/overview/index.html

• Documentation: Installation and User Guide, HOL

– http://www.oracle.com/technetwork/server-storage/openstack/linux/documentation/index.html

• Downloads – http://www.oracle.com/technetwork/server-

storage/openstack/linux/downloads/index.html

56

Page 56: OpenStack & MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

MySQL Enterprise Resources

• Manual

– https://dev.mysql.com/doc/refman/5.7/en/

• Additional info

– https://www.mysql.com/products/enterprise/

• Community forums – http://forums.mysql.com/

• Server Team Blog

– http://mysqlserverteam.com

• Replication and HA Team Blog

– http://mysqlhighavailability.com

57

Page 57: OpenStack & MySQL