Manage MySQL like a devops sysadmin

Percona Live London 2011

Frédéric Descamps

www.percona.com

Who am I ?

Frédéric Descamps

@lefred

http://about.be/lefred

Managing MySQL since 3.23 (as far as I remember)

devops believer

www.percona.com

Thank You to Our Sponsors

Platinum Sponsor

Gold Sponsor

Silver Sponsors

www.percona.com

Percona Live London Sponsors

Exhibitor Sponsors

Friends of Percona Sponsors

Media Sponsors

www.percona.com

To start with...

Let's use devops like

slides...

6

Let's use devops like slides...

7

from

8

9

to

10

11

devopsis a movement willing to help

aligning IT with company needs

(starting by aligning devs and ops)

@endemics - Gildas Le Nadan - “The Devops movement”, Mixit Lyon 2011

12

devops is about communication !

13

devops is about communication !

betweendev and ops

14

devs ops

Small and frequent successful iterations

This is one of the most fundamental ideas in devops: the feedback loop !

15

We can say that devops is bringing

together people and practices from

development and operations

16

We can say that devops is bringing

together people and practices from

development and operations

I am a happy dev !I am a happy op !

17

devops now

=● being professional● improving the collaboration by increasing trust and respect

18

/me likes !

HOW can I reach that goal ?

19

devops requires

visibility !

20

Everybody should have a status overview

Is there any problem ? How are the test going ? How is performance ?

21

devops is about CAMS

22

CULTUREAUTOMATION

MEASUREMENTSHARING

@botchagalupe @damonedwards http://www.opscode.com/blog/2010/07/16/what-devops-means-to-me/

23

What techniques can I use ?

24

MonitoringInfrastructure as

CodeContinuous

Improvement

25

What about DBAs ?

26

Join the team, follow the

movement !

27

You are already between devs

and ops

28

You are already between devs

and ops

I am a happy dba !

29

But we are still closer to the

dark side of the force: sysadmins

30

But we are still closer to the

dark side of the force: sysadmins

31

How can we apply devops

ideas to our dba job ?

32

1. communication2. configuration

management3. testing4. visibility

33

1. Communication

Usually DEVS communicatewith managers, quality managers, products managers, ...

34

OPS “communicate” with support, ISP's and... their servers

35

DBAs don't communicate

36

The interaction between developers, sysadmins, dbas and all the actors

should begin from the start of any projects

37

38

39

Discuss everything database related with developers and

sysadmins:

● disks and raid design● schema design & partitions● additional columns & indexes● tables repair, ...

40

2. configuration management

Can we automate DBAs job ? Should we ? Even if we have only one database ?

41

42

43

A lot of things to configure● binaries● databases ● configs● cron jobs● monitoring● grants● replication

44

Documentation is needed and must be up to date

45

Needs to be reproducible

quickly without missing anything !

46

Keep track of any changes

Mind the gap

47

3. testing

Test every changes

every new package... EVERYTHING

48

4. visibility

Expose monitoring, trending, test results and project status

to EVERYONE in your company!!

Everybody should know why the database is slooooow !

49

Use all these metrics to plan ahead all the changes !

NOT ALONE BUT WITH EVERYBODY

50

notions practices

ideas methodologies

51

We are DBAs we work with data,

we need something more

concrete !

52

We are DBA's we work with data, we need

something more concrete !

53

TRUE !

54

communication

55

Use an agile method to

handle your projects

56

For operations the best method

is kanban

57

58

59

Use dashboards and make them

available to EVERYONE

60

share all your dashboards with

everybody in your company

61

configuration management

62

consider your infrastructure as

code

63

Use puppet (or any other

configuration management tool) to deploy

everything related to your database

64

Save your puppet recipes into a version control system

65

but also save your database

schema changes in that system

66

test

67

68

What should you test as a devops

dba ?

69

Test if you can deploy

everything and if you didn't forget

anything

70

Test also the performance !Is it faster with this change ?

71

use Percona Toolkit

72

automate your tests

73

visibility

74

monitoring

75

Monitor your database with an alerting system (nagios).

Disk/IO, CPU load, Network traffic, disk usage

76

But do you want to be woken up at night

because replication of the machine taking backups is 60sec

behind ?

77

NO!

78

Then do you want to be woken up in the middle

of the night because they are 1000 more selects than usual ?

79

NO!

80

Your alerting should answer those questions:

Is the database server running ?Does it allow connections ?

Is replication lagging on production slaves ?

81

trending

82

The most interesting facts for a DBA are the trending

results !

Use cacti ... with mysql templates !

They provide very good metrics

83

Correlate changes in the trending results

with your configuration changes...

everything should be tracked

84

others ?

85

You can also use other “devops”

oriented tools like :

86

puppet-dashboard

with kanban plugin

87

If there is only one thing you

remember from this presentation

88

Yes, even a schema change

MUST be considered like a

migration !

89

review ittest itplan itdo it

90

and now that we have a

devdbaops team

91

we can start working on

really interesting tasks for

everyone

92

the smooth

migration

93

When this goal is achieved

DEVS have less pressureOPS have less pressureDBAs have less pressure

94

collaboration =

less pressure

95

96

v 1.9application v 2.0

db schema

v 1.9 v 2.0

present future

97

v 1.9application v 2.0

db schema

v 1.9 v 2.0

present future

98

v 1.9application v 2.0

db schema

v 1.9 v 2.0

present

99

v 1.9application v 2.0

db schema

v 1.9 v 2.0

present Issue !!

100

v 1.9application v 2.0

db schema

v 1.9 v 2.0

101

v 1.9application v 2.0

db schema

v 1.9 v 2.0

Issue !!

102

v 1.9application v 2.0

db schema

v 1.9 v 2.0

103

This can only be the result of

104

collaboration between DEVS,

DBAs & OPS

105

demo

106

gitpuppet

Percona serverJenkins

VirtualBoxvagrant

mcollective

107

Change the schema (add an index for example)

Commit !

108

Launches a new virtual machine from scratch using vagrant

Install everything and populate a test db

109

Start some tests with pt-query-digest and compares to old values

Faster is a success

Slower is a fail

110

Photo Credit: http://www.flickr.com/photos/kuma-kigurumi/http://www.flickr.com/photos/airforceone/2125791018/http://www.flickr.com/photos/akosvaradi/2617404213/http://www.flickr.com/photos/doberdad/2264130166http://www.flickr.com/photos/tomsaint/2714401733http://www.flickr.com/photos/10335017@N07/4570943373/http://www.bxel.net/metro-de-bruxelles/metro_plan_bruxelles.jpghttp://www.flickr.com/photos/youngkimstudio/6216328782http://www.flickr.com/photos/26676383@N00/3690701110/http://www.flickr.com/photos/blubbla/209864856http://www.flickr.com/photos/chrishuffman/2336990347http://geekandpoke.typepad.com/.a/6a00d8341d3df553ef0153922fa2ee970b-pi

Projects links: http://puppetlabs.com/resources/overview-2/http://puppetlabs.com/puppet/related-projects/dashboard/http://theforeman.org/http://git-scm.com/http://www.redmine.org/http://www.redmine.org/projects/redmine/wiki/PluginKanbanhttp://vagrantup.comhttps://www.virtualbox.org/http://jenkins-ci.org/http://www.percona.com/software/percona-toolkit/http://code.google.com/p/mysql-cacti-templates/

www.percona.com

You can find all these recipes here !

lefred@percona.com@lefred

We're Hiring! www.percona.com/about-us/careers/

www.percona.com

Annual MySQL Users Conference Presented by Percona Live

The Hyatt Regency Hotel, Santa Clara, CA

April 10th-12th, 2012

Featured SpeakersMark Callaghan, Facebook

Jeremy Zawodny, Craigslist

Marten Mickos, Eucalyptus Systems

Sarah Novotny, Blue Gecko

Peter Zaitsev, Percona

Baron Schwartz, Percona

The Call for Papers is Now Open!

Visit www.percona.com/live/mysql-conference-2012/

114

Questions ?

114

Thank you !

Manage MySQL like a devops sysadmin Percona Live London 2011Frédéric Descamps

www.percona.comWho am I ?Frédéric Descamps@lefredhttp://about.be/lefredManaging MySQL since 3.23 (as far as I remember)devops believer

www.percona.comThank You to Our SponsorsPlatinum SponsorGold SponsorSilver Sponsors

www.percona.comPercona Live London SponsorsExhibitor SponsorsFriends of Percona SponsorsMedia Sponsors

www.percona.comTo start with...Let's use devops like slides...

6Let's use devops like slides...

7from

8

9to

10

11devopsis a movement willing to help aligning IT with company needs(starting by aligning devs and ops) @endemics - Gildas Le Nadan - “The Devops movement”, Mixit Lyon 2011

12devops is about communication !

13devops is about communication !betweendev and ops

14devs opsSmall and frequent successful iterationsThis is one of the most fundamental ideas in devops: the feedback loop !

15We can say that devops is bringing together people and practices from development and operations

16We can say that devops is bringing together people and practices from development and operationsI am a happy dev !I am a happy op !

17devops now =● being professional● improving the collaboration by increasing trust and respect

18/me likes ! HOW can I reach that goal ?

19devops requiresvisibility !

20Everybody should have a status overviewIs there any problem ? How are the test going ? How is performance ?

21devops is about CAMS

22CULTUREAUTOMATIONMEASUREMENTSHARING@botchagalupe @damonedwards http://www.opscode.com/blog/2010/07/16/what-devops-means-to-me/

23What techniques can I use ?

24MonitoringInfrastructure as CodeContinuous Improvement

25What about DBAs ?

26Join the team, follow the movement !

27You are already between devs and ops

28You are already between devs and opsI am a happy dba !

29But we are still closer to the dark side of the force: sysadmins

30But we are still closer to the dark side of the force: sysadmins

31How can we apply devops ideas to our dba job ?

321. communication2. configuration management3. testing4. visibility

331. CommunicationUsually DEVS communicatewith managers, quality managers, products managers, ...

34OPS “communicate” with support, ISP's and... their servers

35DBAs don't communicate

36The interaction between developers, sysadmins, dbas and all the actors should begin from the start of any projects

37

38

39Discuss everything database related with developers and sysadmins:● disks and raid design● schema design & partitions● additional columns & indexes● tables repair, ...

402. configuration managementCan we automate DBAs job ? Should we ? Even if we have only one database ?

41

42

43A lot of things to configure● binaries● databases ● configs● cron jobs● monitoring● grants● replication

44Documentation is needed and must be up to date

45Needs to be reproduciblequickly without missing anything !

46Keep track of any changesMind the gap

473. testingTest every changes every new package... EVERYTHING

484. visibilityExpose monitoring, trending, test results and project status to EVERYONE in your company!!Everybody should know why the database is slooooow !

49Use all these metrics to plan ahead all the changes !NOT ALONE BUT WITH EVERYBODY

50notions practicesideas methodologies

51We are DBAs we work with data, we need something more concrete !

52We are DBA's we work with data, we need something more concrete !

53TRUE !

54communication

55Use an agile method to handle your projects

56For operations the best method is kanban

57

58

59Use dashboards and make them available to EVERYONE

60share all your dashboards with everybody in your company

61configuration management

62consider your infrastructure as code

63Use puppet (or any other configuration management tool) to deploy everything related to your database

64Save your puppet recipes into a version control system

65but also save your database schema changes in that system

66test

67

68What should you test as a devops dba ?

69Test if you can deploy everything and if you didn't forget anything

70Test also the performance !Is it faster with this change ?

71use Percona Toolkit

72automate your tests

73visibility

74monitoring

75Monitor your database with an alerting system (nagios). Disk/IO, CPU load, Network traffic, disk usage

76But do you want to be woken up at night because replication of the machine taking backups is 60sec behind ?

77NO!

78Then do you want to be woken up in the middle of the night because they are 1000 more selects than usual ?

79NO!

80Your alerting should answer those questions:Is the database server running ?Does it allow connections ?Is replication lagging on production slaves ?

81trending

82The most interesting facts for a DBA are the trending results !Use cacti ... with mysql templates !They provide very good metrics

83Correlate changes in the trending results with your configuration changes... everything should be tracked

84others ?

85You can also use other “devops” oriented tools like :

86puppet-dashboardwith kanban plugin

87If there is only one thing you remember from this presentation

88Yes, even a schema change MUST be considered like a migration !

89review ittest itplan itdo it

90and now that we have a devdbaops team

91we can start working on really interesting tasks for everyone

92the smooth migration

93When this goal is achievedDEVS have less pressureOPS have less pressureDBAs have less pressure

94collaboration =less pressure

95

96v 1.9applicationv 2.0db schemav 1.9v 2.0presentfuture

97v 1.9applicationv 2.0db schemav 1.9v 2.0presentfuture

98v 1.9applicationv 2.0db schemav 1.9v 2.0present

99v 1.9applicationv 2.0db schemav 1.9v 2.0presentIssue !!

100v 1.9applicationv 2.0db schemav 1.9v 2.0

101v 1.9applicationv 2.0db schemav 1.9v 2.0Issue !!

102v 1.9applicationv 2.0db schemav 1.9v 2.0

103This can only be the result of

104 collaboration between DEVS, DBAs & OPS

105demo

106gitpuppetPercona serverJenkinsVirtualBoxvagrantmcollective

107Change the schema (add an index for example)Commit !

108Launches a new virtual machine from scratch using vagrantInstall everything and populate a test db

109Start some tests with pt-query-digest and compares to old valuesFaster is a successSlower is a fail

110Photo Credit: http://www.flickr.com/photos/kuma-kigurumi/http://www.flickr.com/photos/airforceone/2125791018/http://www.flickr.com/photos/akosvaradi/2617404213/http://www.flickr.com/photos/doberdad/2264130166http://www.flickr.com/photos/tomsaint/2714401733http://www.flickr.com/photos/10335017@N07/4570943373/http://www.bxel.net/metro-de-bruxelles/metro_plan_bruxelles.jpghttp://www.flickr.com/photos/youngkimstudio/6216328782http://www.flickr.com/photos/26676383@N00/3690701110/http://www.flickr.com/photos/blubbla/209864856http://www.flickr.com/photos/chrishuffman/2336990347http://geekandpoke.typepad.com/.a/6a00d8341d3df553ef0153922fa2ee970b-piProjects links: http://puppetlabs.com/resources/overview-2/http://puppetlabs.com/puppet/related-projects/dashboard/http://theforeman.org/http://git-scm.com/http://www.redmine.org/http://www.redmine.org/projects/redmine/wiki/PluginKanbanhttp://vagrantup.comhttps://www.virtualbox.org/http://jenkins-ci.org/http://www.percona.com/software/percona-toolkit/http://code.google.com/p/mysql-cacti-templates/

www.percona.comYou can find all these recipes here !

lefred@percona.com@lefred

We're Hiring! www.percona.com/about-us/careers/

www.percona.comAnnual MySQL Users Conference Presented by Percona LiveThe Hyatt Regency Hotel, Santa Clara, CAApril 10th-12th, 2012Featured SpeakersMark Callaghan, FacebookJeremy Zawodny, CraigslistMarten Mickos, Eucalyptus SystemsSarah Novotny, Blue GeckoPeter Zaitsev, PerconaBaron Schwartz, PerconaThe Call for Papers is Now Open!Visit www.percona.com/live/mysql-conference-2012/

114Questions ?114Thank you !