MSc in RSIP Jie Zhang s0567061 10/02/06

Advanced Image ProcessingStudent Seminar:

Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Morito

shi AndoFujitsu Laboratories Ltd., Atsugi, 243-0197 Japan )

MSc in RSIP

Jie Zhang

s0567061

10/02/06

Background Why Lip reading?

Human computer interaction Automatically recognize speech contents by

processing lip movement images Potential application for disable facilities

What is the principle? Dealing with frame images sampled from video Lip Feature extraction Lip Feature recognition Various methodology

System overview

Lip colour extraction algorithm Colour characteristics of lip fe

ature Roughly mouth detection with

colour algorithm Precise mouth position detecti

on Eigen image Eigen template Matching algorithm

Eigenwaveform recognition Mouth status Dictionary data Thresholding

Training Lip images

Test Lip images

Colour extraction

Eigenvector template build

Colour extraction

Eigentemplate Detection

Eigenwaveform

recognition

Colour extraction algorithm

Colour components of face featureStrongly brightness dependence !

Colour system HSV or RGB Evenly luminance Brightness normalization

Lip region colour characteristics Three classes: lip, skin, tee

th Looking for the separation

of spectrum componentsFigure 1


Figure 2: Brightness dependent face feature colour distribution of normalised RGB image.

Well distinguished in R &G!


Lip Extraction with colour distribution Threshold function of R an

d G to separate skin and mouth

Label and Extract largest bright area to appropriate size as it is deem to be mouth

Remove teeth and lips with second R and G threshold function

Position determination with four key points of the lips

Figure 3

Lip edge is the combination of teeth and mouth cavity.

Eigentemplate method Aim

----- precisely detect the location of lip! Creation of eigen image

Use appropriate colour extracted image (trimmed image)

PCA + one dimension vector for a single trimmed image

Form image series matrix with x vectors

Eigentemplate method

Calculate eigenvector

Convert eigenvectors to eigenimage


Figure 4


Eigentemplate Trimmed test image

Recover eigenimage to template image

Similarity calculationpkp yb

ky

Searching and trimming the image, compare trimmed image will template. Largest similarity gives the exact location.


Figure 5

Figure 6

Eigenwaveform recognition

Brief introduction to preprocessing A threshold method to detect mouth states and henc

e recognize particular mouth shape Define mouth state: i.e. open, wide open, close, tight

closed… Aspect ratio: distance b between upper and lower ed

ge points over distance a between left and right edge points

Projection vector components: project the image of mouth and lip into eigen space through time scale.

Thresholding

Eigenwaveform recognition

Dictionary matching

Figure 7

Small difference sw recognize the test utterance as the same as the template utterance

Summary & Analysis

Lip reading system Colour extraction method roughly classify lip and other face feat

ure Eigentemplate method precisely detect the location Eigenwaveform algorithm recognize the utterance

Analysis Widely used image processing technique Hard to get high precision Difficult for language different, culture different, appearance diff

erence user Potential problems of algorithm: brightness dependence; overla

p between lip & skin; similar words; uncommon mouth shape; various speak speed and so on.

Need more improvement!

That’s all!Thank you very much!

Question?

MSc in RSIP Jie Zhang s0567061 10/02/06

Documents

Transcript of MSc in RSIP Jie Zhang s0567061 10/02/06