AdrienneTannenbaum2 - Congres BI

8/3/2019 AdrienneTannenbaum2 - Congres BI

1/119

2002Database Design Solutions, Inc.40 Morristown Road, Suite 2B

Bernardsville, NJ 07924www.dbdsolutions.com

Metadata Solutions areHere!

And we are finallyready.


2/119

22002Database Design Solutions, Inc.

40 Morristown Road, Suite 2BBernardsville, NJ 07924www.dbdsolutions.com

Remember

o Data affects thequality andextensiveness of

informationo We need information

to widen ourknowledge

o Knowledge is whatmakes us intelligent


3/119



Whats In Store

for this Afternoon?

o The evolution ofintelligence

o Metadata a full

discussion

o The Metadata Solution

o Metadata SolutionArchitectures

o Bringing the MetadataSolution to YourOrganization


4/119



Case Study 1

Data and Metadata

o Pharmaceutical Company

o Quite distributedo Functionally

o Financiallyo Information Creation


5/119



FDA Requirement

o How much does it cost to release a drug?

o How do you justify your prices

o Costs involved:o Researcho Manufacturing

o Clinical Trials

o Marketingo Sales

o Employee Costs


6/119



How Do We Do That?

o Each division keeps their own data

o Each division stores data in a different way

o Each database is designed differentlyo Not all data exists everywhere

o Some data by researcher

o Some data by employee

o Some data by drugo Some data by manufacturing site

o Etc.


7/119



Putting the

Data Together

o What data do wehave?

o What does it mean?

o Where is it?

o How did it get there?

o How do I get it?


8/119



Their Solution

o A data warehouse

o Major extract andtranslation efforts

o Solid metadata supporto Reporting Tools did nothave appropriatemetadata coverage

o Separate customizedmetadata store

o Metadata access beyondthe data warehouse

Ive got it!


9/119



Case Study 2 IntegratingData Without Metadata

o Internet ServiceProvider

o Another month,

another acquisitiono Another acquisition,

another billing system

o Another billing system,more customer data

o More customer data,more integrationproblems


10/119



Their System

Architecture

Billing System 2

Billing System 1

Billing System n

Std.Translation

Std.Extract

Customer DW


11/119



Their

Environmental Snapshot

o Big Web Believers

o No Formal SystemsDevelopment

Methodology

o Anti-Big Picture

o DevelopmentSupport Focused onTechnical

Infrastructure


12/119



Their Information Picture

o Which numbersshould we use?

o Why are they alldifferent?


13/119



Their

Information Dilemmas

o Definition of Customer

o Definition of Service

o Product Variationso Revenue Calculations


14/119



Their Metadata Solution

Data Warehouse

Development

Tool

Development Library

DW Reporting Tool

Metadata Store

Standard Reports


15/119



Lessons Learned

o Metadata can be moreimportant than data

o Metadata does not have

to all be stored in oneplace

o Metadata can bestandardized and shared

o Information Managementis a practice not amethodology


16/119



Case Study 3Beginning a Painful Process

o A Major Health CareProvider

o HIPAA reporting

requirements

o Data everywheremetadata in someplaces


17/119



Getting Started

o HIPAA ComplianceTaskforce

o Begin with a data

search


18/119



How Painful Is It?

o Still assessing theamount of timeneeded to LOCATE

the datao Cannot assess yet the

amount of timerequired to

INTERPRET andCONNECT the data


19/119



Learning Lessons?

o Maybe.

o Determine whatmetadata you need

o Create the metadataBEFORE you need it

o What comes first, thepain or the prevention?


20/119



Beyond Data

More 5 Questions

1. What metadata do I have?

2. What does it mean?

3. Where is it?4. How did it get there?

5. How do I get it? (Go get it for me)


21/119



Information

TheUltimate

Objective


22/119



Wheres the info?

o Locating information

o Interpreting

informationo Validating

interpretations

o Finalizing information


23/119



A Formal Definition?

o The set of available data,whether internal orexternal, numeric,

textual, graphic, visual,audio, and so on, thatwhen combined with anyset of manual ormechanized processesand integrated with pre-existing informationassists the decisionmaker.

Information


24/119



The 7 Rules

wrt Information

1. Information can takeon virtually anyformat (as long as it

is identifiable,storable, andretrievable)

2. Information shouldnot be restricted to

fulfill the roleidentified by itscreating process

3. Not all data isintended to becomeinformation

4. Information shouldnot be derived fromother informationall information

should be derivedfrom data


25/119



The 7 Rules wrt

Information - contd

5. Informationsusability is directlyrelated to its

relationship to basicdata

6. The averageinformationconsumer will go to

extreme lengths tofind it, get it, andapply it

7. Unless theinformationconsumer is aware of

the informationsexistence, it is highlyunlikely that even thebest information will

be used optimally


26/119



Typical

Information Requests?

o Who are our threelargest customers?

o How long did it take

to roll-out our latestrelease of ProductXYZ?

o How many of our

executives were hiredvia employmentagencies?


27/119



Information vs. Knowledge

o Today, nothing becomes information unlesssome processing or context, usually of aparticular specialty, is applied to it.

o Information evolves.

Art

AttendanceList

ReportCard

Student Name

Course Name


28/119



Information EvolutionJ 5 0 4 3

Part Inventory Guide

Characters without Context

Context

InformationJ 5 0 4 3

Part Number for Windshield Wiper


29/119



Corporate Information

o Information Technology (IT) adds some newtypes of context

o Applications/Systems

o Tools

o Packages

o Databases


30/119



Informations Role Today

o Providing the Answer

o Supplying the Trend

o Pointing Us to a Solutiono Getting Us There

o Assisting the Completion of a Task

o Solving a Problem


31/119



The Role of Information

o Helps us do our jobs!

(or at least it should)

P tti


32/119



Putting

Information Together

o First we locate ito What types of

information are welooking for?

o Then we identifywhich pieces we wanto There are often

supplemental piecesof information

involved

o We connect andrelate as we interpret


33/119



Types of Information

o Structured

o Database contents

o Documents

o Models

o

o Unstructured**

o Raw data, raw text

o Program Code

o Graphicso Sound

o Web sites**

o .

**What makesthese thingsunstructured?

C ti Diff t


34/119



Connecting Different

Types of Information

o Traditionally it was amatter of matchingcommon values

o Gets complicatedwhen the data doesnot equate or when itresides in physically

different types ofstructures

I f ti


35/119



Information

Integration Styles

o DatabaseConsolidation

o Data Management

o Data Modeling

o Data Warehousing

o Corporate Intranets


36/119



Database Consolidation

CorporateAccountingMaster

FieldSales DB Wholesale

Sales DBCustomerHistory DB

WholesaleSales DB

Corporate Sales


37/119



CRMA Special Type

FieldSales DB Customer

History DB

WholesaleSales DB

CorporateAccounting

CRM Package

Local Sales Data Mart

Document ManagementSystem


38/119



Data Management

o Data Modeling

o Enterprise Data Modelso Standards


39/119



Data Warehousing

Data Warehouse Result

Application DB2

Application DB 3

CleanMergeTransitionAggregate

Application DB 1

Th T i l


40/119



The Typical

Data Warehouse -contd

Data Warehouse Result

End-User Query ToolsMulti-DimensionalAnalysis

ToolsDirectoryData

WarehouseMiddleware


41/119



Data Warehousing

o What happened?

o Data Accuracy?

o Data Credibility?

o Integrating Unplanned Data?


42/119



Corporate Intranets

o Best Way to answer Question Number 1:o What data do I have?

o Many also answered Question Number 2:o What does it mean?

o From whose perspective?

o Fewer answered Question Number 3:o Where is it?

o Even fewer answered Question Number 5:o How do I get it (Go get it for me!)?


43/119



Metadata

The reason weare here


44/119



What is Metadata?

o Its one of those

things we need andwant but just cant

get our fingers on..

R b


45/119



Remember

Metadatas Role

J 5 0 4 3

Part Inventory Guide

Characters withoutContext

Context

InformationJ 5 0 4 3Part Number forWindshield Wiper

Instance Data

Metadata

Information


46/119

472002Database Design Solutions, Inc. 40 Morristown Road, Suite 2BBernardsville, NJ 07924www.dbdsolutions.com

This Example of Instance Data

is What Most People Relate to:

Employee SS#: 387-33-0595

Employee Name: Jacqueline ReedEmployee Start Date: 03-09-1994Department Code: 476Payroll Code: 256


47/119


Employee SS#: 387-33-0595

Employee Name: Jacqueline Reed

Employee Start Date: 03-09-94

Department Code: 476

Payroll Code: 256

Table Name: EMPLOYEE

EMPLOYEE_SSNO, X(9) Primary Key

EMPLOYEE_LAST_NAME, X(20) Index

EMPLOYEE_FIRST_NAME, X(15)

EMPLOYEE_DEPT_CODE, X(3) Foreign Key

EMPLOYEE_PAY_CODE, X(3) Foreign Key

Table Name: DEPARTMENT

DEPT_CODE, X(3) Primary Key

DEPT_NAME, X(15)PAYROLL_CODE, X(3) Foreign Key

InstanceData example

Is this Metadata?

Or this?


48/119


Intermediate Processes

CustomerMaster

Customer Number: 8938

Customer Name:Smith, RussellTable Name:CustomerColumn Name:Customer_No

Column Name:Customer_Name

Edit

SalesTransactionsETL

ETL

Sales

DataWarehouse


49/119


Generic Metadata Role

o Making sure that the covered metadata is:

o Identifiable (What metadata do we have?)

o Understandable (What does it mean?)

o Locatable (Where is it?)

o Traceable (How did it get there?)

o Accessible (How do I get it?)

R l ti


50/119


Relating

Metadata to Data

o Gives us Information

o Provides answers tothe 5 Questions

from eachperspective

o Data will be:o Identified

o Defined

o Locatedo Sourced

o Accessed

Th Lib A l


51/119


o Instance Data: TheBook

o Metadata: The CardCatalog Entry

The Library Analogy

Comparing Data to Metadata


52/119


The Information/MetadataConnection

Why this allhas a

purpose


53/119


Data and Metadata

Data + Metadata =Information


54/119


Data/Metadata/Information

o What data do I have?


o Where is it?o How did it get there?

o How do I get it?

o Product_price

o The gross charge for

the firms offerings,before any discounts

o SalesDatabase

Product Table

o Update_ProductProgram

o Product_price Query

Data:$46.30


55/119


Data/Metadata/

Informationcontd

o What data do I have?


o Where is it?

o How did it get there?

o How do I get it?

o Entity Name

o Any person or organizationwith whom we do business

o Marketing Data Model

mktg.erx File

ERWin Modeling Tool

o David Myers ModelerdmyersUserid

o ERWin Executable

Data:Customer


56/119


The Metadata Solution

What We StriveFor

What is a


57/119


What is a

Metadata Solution?

o An organized andintegrated set ofrelated metadata

o Logically connected,often physicallyseparate

o Common access

points and methods

The Metadata


58/119


The Metadata

Solution Role

o Providing a unifiedviewof the metadata

that we need toidentify, access, andinterpret theinformation that weneed to do our jobs

Metadata


59/119


Metadata

Solution Role - contd

o Puts some purposeand order arounddisparate metadata!


60/119


Job/Role Variability

o Depending on who I

am, I can use thesame metadatasolution for adifferent purpose

Metadata


61/119


Metadata

Solution Components

o Metadata

o Metamodel(s)

o Meta-metamodelo Metadata stores

o Software


62/119


Component Roleso Metadata You should know that by now!

o Metamodel The organized collection of metadata,representing a particular perspective (source, function,or other metadata grouping)

o Meta-Metamodel The organized collection ofmetamodels, used as gateway info in a MetadataSolution

o Metadata Store where the metadata is physically kept

o Software Everything required to create, access,update, and display metadata, populate itsoccurrences, and share/exchange them with othermetadata solution architectural components


63/119


Component Variability

o Metamodel Standard, Customized, or Built-from-scratch?

o Metadata Store none, one, or many?

Standard or custom built?o Software Herein lies the rub!

o Metadata Display (with or without the web)

o Metadata Access and Processing

o Metadata Exchange and Update

o Metadata Security

Types of


64/119


Types of

Metadata Solutions

o Metadata Repository(ies)

o Enterprise Portals

o Standalone or Distributed Metadata Storeso XML-Based Metadata Exchange

o Information Directories

o Others

o Integrated Metadata Solution Architectures

o a combination of the above


65/119


The MetadataRepository

Option 1

The Metadata


66/119



The Metadata

Repository

o The vendor-neutral metadata store

o Can be virtual

o Can store metadata gateways to external

metadata

o Integrated

o Standard Architecture

The Generic


67/119



The Generic

Metadata Repository

UnderlyingDatabase

Repository Metamodel

Repository Software

The Metadata


68/119



The Metadata

Repository Database

o Stores metadata contents as well asaddressing and execution data necessary toget to external metadata

o Open (commercial) or closed (proprietary)


69/119



Repository Metamodel

o The Meta-metamodel

o Correlates with theunderlying database

o Tracks relationshipso Handles security

o Tool focused

o RepositoryBlueprint


70/119



Repository Software

o Go Get It Routines

- APIs, RPCs

o Repository Policieso Repository

Templates

o Repository Utilities


71/119



Distributed Repositories

o Not that simple with todays tools

o Special distribution considerations dependon tool architecture

o Application Logic

o Presentation Logic

o Database Logic

Client/Server


72/119



Client/Server

Processing Layers

Server

DatabaseServer

Execute database

Database extract access program

Application Logic

Display results

in GUI format

Point and click

request for

information

Presentation Logic

Database Processing

Database

Calls

DBMS Access Language


73/119



Enterprise Portals

Option 2


74/119



Enterprise Portals

o InformationDistribution Hubs

o Web Browser or

Customized URLfront-end

o Customized Searchand Retrieve

o Tag or URL based


75/119



Enterprise Portals - contd

o Fancy name for acompanys intranet?

o Accurate name for an

organizedmetamodel-basedintranet

o Able to get to the

same stuff with thesame request everytime


76/119



Portal Componentso Meta-metamodel (of

sorts)

o Taken care of whenyou buy portalsoftware

o Metadata/datatransfer mechanism

o Portal display/front-end

Metadata/Data


77/119



XML Based Portals

o Most Popular

o XML is used toexchange queries

and query results toand from the portalsoftware

o Metadata wrappers

(tags) are generatedthroughout the portalarchitecture


78/119



An XML Based Portal

DesktopWord

ProcessingTool

DocumentManagement

System

CorporateLibrarySystem

XML-BasedPortal

Access

PUBLICATIONS

AUTHOR = Adrienne

Tannenbaum

Standard

Publication

DTD

Another XML


79/119



Based Portal

Delimited

File

Common

Metadata

Store

DBMS

Catalog

Program

Library

Shared

Metadata

Store

Extract/Translate

Process

Extract/

Translate

Process

XML

Parser

Extract

Process

Extract/

Translate

Process

Application

Development

Application

Design

Spreadsheets

Word Processing

Simple databases

Graphics

Desktop Office

Productivity Tools

Intranet Communication

(HTML)


80/119



Metadata Stores

Option 3


81/119

82



Metadata Stores

o Standalone or Distributed

o Most Popular

o Easiest to implement (depending on design)o Home-grown databases which follow a

home-grown metamodel

Distributed


82/119

83


40 Morristown Road, Suite 2B


Distributed

Metadata Stores

o Plan first, Design next, Implement last

o Each metadata store must be relatedto theothers

o Common metamodel

o Different ways of populating each metadatastore

Distributing


83/119

84




Distributing

Metadata Stores - contd

o Common Metamodel Variations

o Same constructs, different metadatainstances

o Different constructs, different metadatainstances

Common


84/119

85




Metamodel Population

SubjectArea

DataElement

DataStore

Application

ApplicationComponents

Categorizes

Resides In

Accessed By

Consists of

Common

Metamodel

Relational

MetadataStore 1

SubjectArea

DataElement

Data

Store

Application


Categorizes

Resides In

Accessed By

Consists of

CommonMetamodel

Relational

MetadataStore 2

SubjectArea

DataElement

Data

Store

Application


Categorizes

Resides In

Accessed By

Consists of

CommonMetamodel

Relational

MetadataStore 3

Common


85/119

86




Metamodel Population

Object

Object

Type

Metamodel

Data

Store

Physical

Location

Integrating Meta-Metamodel Meta-metaData Store


86/119




XML Based Exchange

Option 4

XML Based


87/119

88




Metadata Exchange

o Has been a silverbullet for quite some

time

o Leave everything asis, just pass it backand forth?

o Standard XML

documents totranslate from andto?

eXtensible


88/119

89




Markup Language

o The W3C calls XML a common syntax forexpressing structurein data

o XML is a method of putting structure in a text

file

o XML is a markup specification language andXML files contain both data and metadata


89/119

90




More important.

o XML is ametalanguage whichdescribes a markup

language

o XML is anarchitecture not anapplication


90/119

91




Why The Popularity

o It is a standardized way to exchange data

o It is Flexible You can do a lot with it

o It is Highly adaptable

o Custom usage is not dependent on acommittee

o It permits the automated exchange of databetween two organizations via the web


91/119

92




How XML works

o XML is composed of markup and content

o Markup is used to describe content at thedesired level of detail

o XML has nameable tags that are used to definewhat the content is

o An XML document must be well-formed


92/119

93




How XML works - contd

o If a Document Type Declaration (DTD) exists andthe XML document conforms to the DTD the XMLdocument is said to be valid.

o Very Simplified DTD exampleo

o


93/119

94




Simplified XML Sample

File

File

type

has

File has

File Type

XML Based


94/119

95




Information Exchange

o Choices in scopeaffect the overallmetadata solution

architecture


95/119




Information Directories

Option 5


96/119

Information


97/119

98




Directories - contd

o Directories are a webbrowser without the

access

o A portal that does notfetch

o Optional use of theweb

o Retail repositoriesusually offer thiscapability

Metadata Categorization


98/119

99




Within a Directory

o Categorization of a set of scoped information(could be metadata or data)

o Association of the categories with the

information

o Identification of the categorized information

o Optional access of the metadata-identified

information


99/119




Other Metadata

Solutions

Just as an aside..

Other


100/119

101




Metadata Solutions

o File ManagementSystems

o Database

ManagementSystems

o Object OrientedComponent Libraries

o Etc.

File


101/119

102




Management System

o A file typeis afiletypeis a file type

o Why are file types soimportant to FileManagement

Systems?

File typesguarantee

reusability and

standardization

The File


102/119

103




System Metamodel

File

File Type

AccessProcedure

Has Associated

Has Associated

Database


103/119

104




Management Systems

o Cant forget that catalog!

o Physical definitions of all defined databases

o Database is related to all of its components

Object Oriented


104/119

105




Component Libraries

o Potential standardization of the way data isdefined and accessed in an organization

o Standard software components

o Standard object classes

o Who has access to the library?


105/119




Integrated Metadata

Solution Architectures

Tying them all

together

Integrated Metadata


106/119

107




Solution Architecture

o A Unified Set of Software

o Shared Metadata Store

o Reflective of Common Metamodel

o Active Component Connections


107/119

108




What Goes on Inside?

DBMS Metamodel

Metadata Store

DBMS Catalog

ETLApplication

Metadata Store

Planning Analysis

Construction Application

Development

MetamodelIntegrated Development

Environment

Decision Support

Metamodel

Data Warehouse

Reporting Tool Shared Metadata

Repository

DataWarehouse

What Goes


108/119

109




on Inside? - contd

Metamodel

MetamodelConstruct

MetadataStore

RepositoryProcedure

Contains

Resides In

Implemented As

Accessed by

Illustrated Metadata

S


109/119

110




Solution

Microsoft SQL ServerCustom Schema

ModelMart

System CatalogsDB2OracleMSSQLTandem

Windows NTUser client

iexplore.exe

Batchscheduler

Cool:GenPublic Interface

Tables

Windows NTAdministrative client

Maintenance

0

ModelMart API

Load program

MTS

COMcomponents

MS InternetInformation

Server 4

PresentationActive Server

Pages

Metadata Solution

A E l


110/119

111




Access Example

Get library collectionSet libraryCollection = ISession.Objects.Collect(,

LibraryTypeCode)\

Loop through libraries

For Each libraryReference in libraryCollectionDebug.Print libraryReference.Name

Get diagram collection for librarySet diagramCollection = Isession.Objects.Collect(

libraryReference.Handle, DiagramTypeCode)

Loop through diagramsFor Each diagramReference in diagramCollection

Debug.Print vbTab & diagramReference.Name

Introducing Metadata


111/119




Introducing Metadata

Solutions to Your

Organization


112/119

113




Most Common Mistake

o Making theinvestment

o Purchasing/developin

g the solutiono Just letting it loose!

o Without thinking abouthow it will changethings

Thi T C id


113/119

114




Things To Consider

o Changes in daily activitieso How will the metadata be actively maintained?

o Will application or database development

processes need to change?o Changes in organization structure

o Who will be using the metadata solution?

o Who will administer the metadata solution?

o Who will enhance or maintain the metadatasolution?

o Etc.

A i R di


114/119

115




Assessing Readiness

o They wanted metadata

o You gave them metadata

o What will they do with it?

o When?

o How?

o How will you ensure their objectives?

B i R l


115/119

116




Basic Rules

o Make standardized metadata part of allinformation creation

o Information access should be the result of

metadata-based searches

o Industry standards should have a positiveimpact

o Needing common metadata stores shouldbecome of less and less importance

Finally


116/119

117




Finally

o Be sure the originalobjectives are met

o Now

o As new architecturalcomponents are added

o Always rememberwhy the metadatasolution was created

to begin with

Q ti ?


117/119

118




Questions?

Wh t W C d


118/119

119




What Weve Covered

o The evolution ofintelligence

o Metadata a full

discussiono The Metadata Solution

o Metadata SolutionArchitectures

o Bringing the MetadataSolution to YourOrganization

F M I f


119/119

For More Info..

o Metadata Solutions (2001, Addison Wesley)

o www.dbdsolutions.com

o Publications

o Discussion Group

o Upcoming Conferences (USA)

o Metadata Conference, San Antonio, Texas

o Email:o [email protected]

AdrienneTannenbaum2 - Congres BI

Documents

Transcript of AdrienneTannenbaum2 - Congres BI