Pluk2011 deploy-mysql-like-a-devops-sysadmin
228
Manage MySQL like a devops sysadmin Percona Live London 2011 Frédéric Descamps
-
Upload
frederic-descamps -
Category
Technology
-
view
101 -
download
0
Transcript of Pluk2011 deploy-mysql-like-a-devops-sysadmin
Manage MySQL like a devops sysadmin
Percona Live London 2011
Frédéric Descamps
www.percona.com
Who am I ?
Frédéric Descamps
@lefred
http://about.be/lefred
Managing MySQL since 3.23 (as far as I remember)
devops believer
www.percona.com
Thank You to Our Sponsors
Platinum Sponsor
Gold Sponsor
Silver Sponsors
www.percona.com
Percona Live London Sponsors
Exhibitor Sponsors
Friends of Percona Sponsors
Media Sponsors
www.percona.com
To start with...
Let's use devops like
slides...
6
Let's use devops like slides...
7
from
8
9
to
10
11
devopsis a movement willing to help
aligning IT with company needs
(starting by aligning devs and ops)
@endemics - Gildas Le Nadan - “The Devops movement”, Mixit Lyon 2011
12
devops is about communication !
13
devops is about communication !
betweendev and ops
14
devs ops
Small and frequent successful iterations
This is one of the most fundamental ideas in devops: the feedback loop !
15
We can say that devops is bringing
together people and practices from
development and operations
16
We can say that devops is bringing
together people and practices from
development and operations
I am a happy dev !I am a happy op !
17
devops now
=● being professional● improving the collaboration by increasing trust and respect
18
/me likes !
HOW can I reach that goal ?
19
devops requires
visibility !
20
Everybody should have a status overview
Is there any problem ? How are the test going ? How is performance ?
21
devops is about CAMS
22
CULTUREAUTOMATION
MEASUREMENTSHARING
@botchagalupe @damonedwards http://www.opscode.com/blog/2010/07/16/what-devops-means-to-me/
23
What techniques can I use ?
24
MonitoringInfrastructure as
CodeContinuous
Improvement
25
What about DBAs ?
26
Join the team, follow the
movement !
27
You are already between devs
and ops
28
You are already between devs
and ops
I am a happy dba !
29
But we are still closer to the
dark side of the force: sysadmins
30
But we are still closer to the
dark side of the force: sysadmins
31
How can we apply devops
ideas to our dba job ?
32
1. communication2. configuration
management3. testing4. visibility
33
1. Communication
Usually DEVS communicatewith managers, quality managers, products managers, ...
34
OPS “communicate” with support, ISP's and... their servers
35
DBAs don't communicate
36
The interaction between developers, sysadmins, dbas and all the actors
should begin from the start of any projects
37
38
39
Discuss everything database related with developers and
sysadmins:
● disks and raid design● schema design & partitions● additional columns & indexes● tables repair, ...
40
2. configuration management
Can we automate DBAs job ? Should we ? Even if we have only one database ?
41
42
43
A lot of things to configure● binaries● databases ● configs● cron jobs● monitoring● grants● replication
44
Documentation is needed and must be up to date
45
Needs to be reproducible
quickly without missing anything !
46
Keep track of any changes
Mind the gap
47
3. testing
Test every changes
every new package... EVERYTHING
48
4. visibility
Expose monitoring, trending, test results and project status
to EVERYONE in your company!!
Everybody should know why the database is slooooow !
49
Use all these metrics to plan ahead all the changes !
NOT ALONE BUT WITH EVERYBODY
50
notions practices
ideas methodologies
51
We are DBAs we work with data,
we need something more
concrete !
52
We are DBA's we work with data, we need
something more concrete !
53
TRUE !
54
communication
55
Use an agile method to
handle your projects
56
For operations the best method
is kanban
57
58
59
Use dashboards and make them
available to EVERYONE
60
share all your dashboards with
everybody in your company
61
configuration management
62
consider your infrastructure as
code
63
Use puppet (or any other
configuration management tool) to deploy
everything related to your database
64
Save your puppet recipes into a version control system
65
but also save your database
schema changes in that system
66
test
67
68
What should you test as a devops
dba ?
69
Test if you can deploy
everything and if you didn't forget
anything
70
Test also the performance !Is it faster with this change ?
71
use Percona Toolkit
72
automate your tests
73
visibility
74
monitoring
75
Monitor your database with an alerting system (nagios).
Disk/IO, CPU load, Network traffic, disk usage
76
But do you want to be woken up at night
because replication of the machine taking backups is 60sec
behind ?
77
NO!
78
Then do you want to be woken up in the middle
of the night because they are 1000 more selects than usual ?
79
NO!
80
Your alerting should answer those questions:
Is the database server running ?Does it allow connections ?
Is replication lagging on production slaves ?
81
trending
82
The most interesting facts for a DBA are the trending
results !
Use cacti ... with mysql templates !
They provide very good metrics
83
Correlate changes in the trending results
with your configuration changes...
everything should be tracked
84
others ?
85
You can also use other “devops”
oriented tools like :
86
puppet-dashboard
with kanban plugin
87
If there is only one thing you
remember from this presentation
88
Yes, even a schema change
MUST be considered like a
migration !
89
review ittest itplan itdo it
90
and now that we have a
devdbaops team
91
we can start working on
really interesting tasks for
everyone
92
the smooth
migration
93
When this goal is achieved
DEVS have less pressureOPS have less pressureDBAs have less pressure
94
collaboration =
less pressure
95
96
v 1.9application v 2.0
db schema
v 1.9 v 2.0
present future
97
v 1.9application v 2.0
db schema
v 1.9 v 2.0
present future
98
v 1.9application v 2.0
db schema
v 1.9 v 2.0
present
99
v 1.9application v 2.0
db schema
v 1.9 v 2.0
present Issue !!
100
v 1.9application v 2.0
db schema
v 1.9 v 2.0
101
v 1.9application v 2.0
db schema
v 1.9 v 2.0
Issue !!
102
v 1.9application v 2.0
db schema
v 1.9 v 2.0
103
This can only be the result of
104
collaboration between DEVS,
DBAs & OPS
105
demo
106
gitpuppet
Percona serverJenkins
VirtualBoxvagrant
mcollective
107
Change the schema (add an index for example)
Commit !
108
Launches a new virtual machine from scratch using vagrant
Install everything and populate a test db
109
Start some tests with pt-query-digest and compares to old values
Faster is a success
Slower is a fail
110
Photo Credit: http://www.flickr.com/photos/kuma-kigurumi/http://www.flickr.com/photos/airforceone/2125791018/http://www.flickr.com/photos/akosvaradi/2617404213/http://www.flickr.com/photos/doberdad/2264130166http://www.flickr.com/photos/tomsaint/2714401733http://www.flickr.com/photos/10335017@N07/4570943373/http://www.bxel.net/metro-de-bruxelles/metro_plan_bruxelles.jpghttp://www.flickr.com/photos/youngkimstudio/6216328782http://www.flickr.com/photos/26676383@N00/3690701110/http://www.flickr.com/photos/blubbla/209864856http://www.flickr.com/photos/chrishuffman/2336990347http://geekandpoke.typepad.com/.a/6a00d8341d3df553ef0153922fa2ee970b-pi
Projects links: http://puppetlabs.com/resources/overview-2/http://puppetlabs.com/puppet/related-projects/dashboard/http://theforeman.org/http://git-scm.com/http://www.redmine.org/http://www.redmine.org/projects/redmine/wiki/PluginKanbanhttp://vagrantup.comhttps://www.virtualbox.org/http://jenkins-ci.org/http://www.percona.com/software/percona-toolkit/http://code.google.com/p/mysql-cacti-templates/
www.percona.com
You can find all these recipes here !
www.percona.com
Annual MySQL Users Conference Presented by Percona Live
The Hyatt Regency Hotel, Santa Clara, CA
April 10th-12th, 2012
Featured SpeakersMark Callaghan, Facebook
Jeremy Zawodny, Craigslist
Marten Mickos, Eucalyptus Systems
Sarah Novotny, Blue Gecko
Peter Zaitsev, Percona
Baron Schwartz, Percona
The Call for Papers is Now Open!
Visit www.percona.com/live/mysql-conference-2012/
114
Questions ?
114
Thank you !
Manage MySQL like a devops sysadmin Percona Live London 2011Frédéric Descamps
www.percona.comWho am I ?Frédéric Descamps@lefredhttp://about.be/lefredManaging MySQL since 3.23 (as far as I remember)devops believer
www.percona.comThank You to Our SponsorsPlatinum SponsorGold SponsorSilver Sponsors
www.percona.comPercona Live London SponsorsExhibitor SponsorsFriends of Percona SponsorsMedia Sponsors
www.percona.comTo start with...Let's use devops like slides...
6Let's use devops like slides...
7from
8
9to
10
11devopsis a movement willing to help aligning IT with company needs(starting by aligning devs and ops) @endemics - Gildas Le Nadan - “The Devops movement”, Mixit Lyon 2011
12devops is about communication !
13devops is about communication !betweendev and ops
14devs opsSmall and frequent successful iterationsThis is one of the most fundamental ideas in devops: the feedback loop !
15We can say that devops is bringing together people and practices from development and operations
16We can say that devops is bringing together people and practices from development and operationsI am a happy dev !I am a happy op !
17devops now =● being professional● improving the collaboration by increasing trust and respect
18/me likes ! HOW can I reach that goal ?
19devops requiresvisibility !
20Everybody should have a status overviewIs there any problem ? How are the test going ? How is performance ?
21devops is about CAMS
22CULTUREAUTOMATIONMEASUREMENTSHARING@botchagalupe @damonedwards http://www.opscode.com/blog/2010/07/16/what-devops-means-to-me/
23What techniques can I use ?
24MonitoringInfrastructure as CodeContinuous Improvement
25What about DBAs ?
26Join the team, follow the movement !
27You are already between devs and ops
28You are already between devs and opsI am a happy dba !
29But we are still closer to the dark side of the force: sysadmins
30But we are still closer to the dark side of the force: sysadmins
31How can we apply devops ideas to our dba job ?
321. communication2. configuration management3. testing4. visibility
331. CommunicationUsually DEVS communicatewith managers, quality managers, products managers, ...
34OPS “communicate” with support, ISP's and... their servers
35DBAs don't communicate
36The interaction between developers, sysadmins, dbas and all the actors should begin from the start of any projects
37
38
39Discuss everything database related with developers and sysadmins:● disks and raid design● schema design & partitions● additional columns & indexes● tables repair, ...
402. configuration managementCan we automate DBAs job ? Should we ? Even if we have only one database ?
41
42
43A lot of things to configure● binaries● databases ● configs● cron jobs● monitoring● grants● replication
44Documentation is needed and must be up to date
45Needs to be reproduciblequickly without missing anything !
46Keep track of any changesMind the gap
473. testingTest every changes every new package... EVERYTHING
484. visibilityExpose monitoring, trending, test results and project status to EVERYONE in your company!!Everybody should know why the database is slooooow !
49Use all these metrics to plan ahead all the changes !NOT ALONE BUT WITH EVERYBODY
50notions practicesideas methodologies
51We are DBAs we work with data, we need something more concrete !
52We are DBA's we work with data, we need something more concrete !
53TRUE !
54communication
55Use an agile method to handle your projects
56For operations the best method is kanban
57
58
59Use dashboards and make them available to EVERYONE
60share all your dashboards with everybody in your company
61configuration management
62consider your infrastructure as code
63Use puppet (or any other configuration management tool) to deploy everything related to your database
64Save your puppet recipes into a version control system
65but also save your database schema changes in that system
66test
67
68What should you test as a devops dba ?
69Test if you can deploy everything and if you didn't forget anything
70Test also the performance !Is it faster with this change ?
71use Percona Toolkit
72automate your tests
73visibility
74monitoring
75Monitor your database with an alerting system (nagios). Disk/IO, CPU load, Network traffic, disk usage
76But do you want to be woken up at night because replication of the machine taking backups is 60sec behind ?
77NO!
78Then do you want to be woken up in the middle of the night because they are 1000 more selects than usual ?
79NO!
80Your alerting should answer those questions:Is the database server running ?Does it allow connections ?Is replication lagging on production slaves ?
81trending
82The most interesting facts for a DBA are the trending results !Use cacti ... with mysql templates !They provide very good metrics
83Correlate changes in the trending results with your configuration changes... everything should be tracked
84others ?
85You can also use other “devops” oriented tools like :
86puppet-dashboardwith kanban plugin
87If there is only one thing you remember from this presentation
88Yes, even a schema change MUST be considered like a migration !
89review ittest itplan itdo it
90and now that we have a devdbaops team
91we can start working on really interesting tasks for everyone
92the smooth migration
93When this goal is achievedDEVS have less pressureOPS have less pressureDBAs have less pressure
94collaboration =less pressure
95
96v 1.9applicationv 2.0db schemav 1.9v 2.0presentfuture
97v 1.9applicationv 2.0db schemav 1.9v 2.0presentfuture
98v 1.9applicationv 2.0db schemav 1.9v 2.0present
99v 1.9applicationv 2.0db schemav 1.9v 2.0presentIssue !!
100v 1.9applicationv 2.0db schemav 1.9v 2.0
101v 1.9applicationv 2.0db schemav 1.9v 2.0Issue !!
102v 1.9applicationv 2.0db schemav 1.9v 2.0
103This can only be the result of
104 collaboration between DEVS, DBAs & OPS
105demo
106gitpuppetPercona serverJenkinsVirtualBoxvagrantmcollective
107Change the schema (add an index for example)Commit !
108Launches a new virtual machine from scratch using vagrantInstall everything and populate a test db
109Start some tests with pt-query-digest and compares to old valuesFaster is a successSlower is a fail
110Photo Credit: http://www.flickr.com/photos/kuma-kigurumi/http://www.flickr.com/photos/airforceone/2125791018/http://www.flickr.com/photos/akosvaradi/2617404213/http://www.flickr.com/photos/doberdad/2264130166http://www.flickr.com/photos/tomsaint/2714401733http://www.flickr.com/photos/10335017@N07/4570943373/http://www.bxel.net/metro-de-bruxelles/metro_plan_bruxelles.jpghttp://www.flickr.com/photos/youngkimstudio/6216328782http://www.flickr.com/photos/26676383@N00/3690701110/http://www.flickr.com/photos/blubbla/209864856http://www.flickr.com/photos/chrishuffman/2336990347http://geekandpoke.typepad.com/.a/6a00d8341d3df553ef0153922fa2ee970b-piProjects links: http://puppetlabs.com/resources/overview-2/http://puppetlabs.com/puppet/related-projects/dashboard/http://theforeman.org/http://git-scm.com/http://www.redmine.org/http://www.redmine.org/projects/redmine/wiki/PluginKanbanhttp://vagrantup.comhttps://www.virtualbox.org/http://jenkins-ci.org/http://www.percona.com/software/percona-toolkit/http://code.google.com/p/mysql-cacti-templates/
www.percona.comYou can find all these recipes here !
www.percona.comAnnual MySQL Users Conference Presented by Percona LiveThe Hyatt Regency Hotel, Santa Clara, CAApril 10th-12th, 2012Featured SpeakersMark Callaghan, FacebookJeremy Zawodny, CraigslistMarten Mickos, Eucalyptus SystemsSarah Novotny, Blue GeckoPeter Zaitsev, PerconaBaron Schwartz, PerconaThe Call for Papers is Now Open!Visit www.percona.com/live/mysql-conference-2012/
114Questions ?114Thank you !
![_Akurate_Akudorik_Review_Apr15.pdf · close mic'd, upfront music like 'Royals' by Lorde [Pure Heroine, Universal], and is less successful with big soundstage audiophile works like](https://static.fdocuments.nl/doc/165x107/5f87cd27dabf2a251e2e30e6/akurateakudorikreviewapr15pdf-close-micd-upfront-music-like-royals-by.jpg)
