Download - LINUX & Parallel Processing

LINUX & Parallel LINUX & Parallel ProcessingProcessing

Wouter BrissinckWouter Brissinck

Vrije Universiteit BrusselVrije Universiteit Brussel

Dienst INFODienst [email protected]@info.vub.ac.be

+32 2 629 2965+32 2 629 2965

OverviewOverview

The DreamThe Dream The FactsThe Facts The ToolsThe Tools The Real ThingThe Real Thing The ConclusionThe Conclusion

History History

‘‘6060 1 computer N users 1 computer N users ‘‘80 80 N computers N computers N users N users ‘‘90 90 N computers N computers 1 user 1 user

past past problem = f(computer) problem = f(computer) future future computer = f(problem)computer = f(problem)

The Sequential FrustrationThe Sequential Frustration

Faster CPU : High development Faster CPU : High development cost + timecost + time

Speed of light : faster Speed of light : faster smaller smaller Size of Si atom is the ultimate limitSize of Si atom is the ultimate limit

ParallelismParallelism

Offers scalable performanceOffers scalable performance Solves as yet unsolvable problemsSolves as yet unsolvable problems

• problem sizeproblem size• computation timecomputation time

Hardware cost Hardware cost Software developmentSoftware development

ScalabilityScalability

PROBLEMPROBLEM * p

CPUCPUCPU

CPUCPUCPU

CPU1

p

ScalabilityScalability

problem on 1 CPU takes T timeproblem on 1 CPU takes T time p times larger problem on p CPUs p times larger problem on p CPUs

takes T timetakes T time

Same technology ( only more of it )Same technology ( only more of it ) Same code ( if ... )Same code ( if ... ) Economy of scale Economy of scale

Hardware cost ?Hardware cost ?

Parallel machines :Parallel machines :• expensive to buyexpensive to buy• expensive to maintainexpensive to maintain• expensive to upgradeexpensive to upgrade• unreliableunreliable• lousy operating lousy operating

systemssystems

N.O.W :N.O.W :• you own one !you own one !• Its maintained for youIts maintained for you• You just did, didn’t You just did, didn’t

you ?you ?• You rely on it every day You rely on it every day

!!• How did you call How did you call

LINUX ?LINUX ?A network-of-workstations is a virtually free A network-of-workstations is a virtually free parallel machine !parallel machine !

OverviewOverview


100 painters 100 painters 1 day ? 1 day ?

SynchronizationSynchronization• shared resourcesshared resources• dependenciesdependencies

CommunicationCommunication

ProblemsProblems

1 painter 1 painter 100 days 100 days

Amdahl’s LawAmdahl’s Law

pp

T

TSpeedup

p

111

= sequential fraction

pTp

TTTp

)1(

TTT )1(1

= communication cost

Amdahl’s LawAmdahl’s Law

1

#processors=p

Speedup=S

pp

SSpeedup

1

1 = sequential fraction = communication cost

GranularityGranularity

Granularity =Granularity = ionCommunicat

nComputatio

N.O.W Dedicated MIMDPipelining in CPU

coarse fine

Network latencyNetwork latency

L

T = L + len / B

B

len

L=Latency(s) len=length(bits) B=Bandwidth(bits/s)

proc1

proc2

The N of N.O.WThe N of N.O.W

?

Software DesignSoftware Design

Sequential specification Sequential specification Parallel Parallel implementationimplementation

Parallel specification Parallel specification (less) (less) parallel implementationparallel implementation• Preferably : 1 process/processorPreferably : 1 process/processor

Software Design : Software Design : classificationclassification

Master - SlaveMaster - Slave PipeliningPipelining Systolic ArraySystolic Array

Functional parallelismFunctional parallelism Data parallelismData parallelism

OverviewOverview


P.V.MP.V.M

PVM : overviewPVM : overview PVM PVM

installationinstallation Using PVMUsing PVM

• consoleconsole• compilingcompiling• writing codewriting code

Message Message passingpassing• in Cin C• in C++in C++

gdb and pvmgdb and pvm

Parallel Virtual MachineParallel Virtual Machine

Most popular Message Passing Most popular Message Passing EngineEngine

Works on MANY platforms ( LINUX, Works on MANY platforms ( LINUX, NT, SUN, CRAY, ...)NT, SUN, CRAY, ...)

Supports C, C++, fortranSupports C, C++, fortran Allows heterogenious virtual Allows heterogenious virtual

machinesmachines FREE software ( ftp.netlib.org )FREE software ( ftp.netlib.org )

Heterogenious NOWsHeterogenious NOWs

PVM provides data format PVM provides data format translation of messagestranslation of messages

Must compile application for each Must compile application for each kind of machinekind of machine

Can take advantage of specific Can take advantage of specific resources in the LANresources in the LAN

PVM featuresPVM features

User-configured host poolUser-configured host pool Application = set of (unix) processesApplication = set of (unix) processes

• automatically mappedautomatically mapped• forced by programmerforced by programmer

Message Passing modelMessage Passing model• strongly typed messagesstrongly typed messages• asynchronous (blocking/non-blocking asynchronous (blocking/non-blocking

receive)receive) Multiprocessor supportMultiprocessor support

How PVM worksHow PVM works

Console for resource managementConsole for resource management Deamon for each host in the virtual Deamon for each host in the virtual

machinemachine• message passingmessage passing• fault tolerancefault tolerance

PVM library provides primitives forPVM library provides primitives for• task spawningtask spawning• send/receivesend/receive• VM info, ...VM info, ...

How PVM worksHow PVM works

Tasks are identified by TIDs (pvmd Tasks are identified by TIDs (pvmd supplied)supplied)

Group server provides for groups Group server provides for groups of tasksof tasks• Tasks can join and leave group at willTasks can join and leave group at will• Groups can overlapGroups can overlap• Useful for multicastsUseful for multicasts

N.O.W SetupN.O.W Setup

PC PCPCPC

LINUX LINUXLINUXLINUX

PVM

Master

Slave Slave Slave Slave

Example : masterExample : master

#include "pvm3.h”main(){

int cc, tid;char buf[100];cc = pvm_spawn("hello_other",

(char**)0, 0, "", 1, &tid);if (cc == 1) {

cc = pvm_recv(-1, -1);pvm_bufinfo(cc, (int*)0, (int*)0, &tid);pvm_upkstr(buf);printf("from t%x: %s\n", tid, buf);

} pvm_exit();

}

Example : SlaveExample : Slave

#include "pvm3.h"main(){

int ptid;char buf[100];ptid = pvm_parent();strcpy(buf, "hello, world from ");gethostname(buf + strlen(buf), 64);

pvm_initsend(PvmDataDefault);pvm_pkstr(buf);pvm_send(ptid, 1);

pvm_exit();}

PVM : installingPVM : installing

Set environment variables (before Set environment variables (before build)build)• PVM_ROOTPVM_ROOT• PVM_ARCHPVM_ARCH

Make defaultMake default Make sure you can rsh without Make sure you can rsh without

authenticationauthentication

PVM and rshPVM and rsh

PVM starts tasks using rshPVM starts tasks using rsh Fails if a password is requiredFails if a password is required .rhosts on remote account must .rhosts on remote account must

mention the machine where the mention the machine where the console is runningconsole is running

Typical error : Can’t start pvmdTypical error : Can’t start pvmd

Typical situationTypical situation

All machines share the same user All machines share the same user accountsaccounts

Account has these directories/files :Account has these directories/files :/shared/pvm3/lib/LINUX/pvmd3/shared/pvm3/lib/LINUX/pvmd3

/shared/pvm3/lib/SUN4/pvmd3/shared/pvm3/lib/SUN4/pvmd3

~/pvm3/bin/LINUX/Eg.App~/pvm3/bin/LINUX/Eg.App

~/pvm3/bin/SUN4/Eg.App~/pvm3/bin/SUN4/Eg.App

.rhosts file looks like this : ( mymachine .rhosts file looks like this : ( mymachine is the machine which runs the console )is the machine which runs the console )

mymachinemymachine

mymachine.my.domainmymachine.my.domain

PVM ConsolePVM Console

add hostname : add a host to VMadd hostname : add a host to VM conf : show list of hostsconf : show list of hosts ps -a : show all pvm processes runningps -a : show all pvm processes running

• Manually started (i.e. not spawned) tasks are Manually started (i.e. not spawned) tasks are shown by a ‘-’shown by a ‘-’

reset : kill all pvm tasksreset : kill all pvm tasks quit : leave console, but keep deamon quit : leave console, but keep deamon

alivealive halt : kill deamonhalt : kill deamon

PVM Console : hostfilePVM Console : hostfile

hostfile contains hosts to be hostfile contains hosts to be added, with optionsadded, with options

mymachine ep=$(SIM_ROOT)/bin/$(PVM_ARCH)faraway lo=woutie pw=whoknows* lo=wouter

On machine without network (just On machine without network (just loopback):loopback):

* ip=127.0.0.1

Compiling a programCompiling a program

Application must be recompiled for Application must be recompiled for each architectureeach architecture• binary goes in binary goes in ~/pvm3/bin/$(PVM_ARCH)~/pvm3/bin/$(PVM_ARCH)• aimk determines architecture, then calls aimk determines architecture, then calls

‘make’‘make’ Must be linked with -lpvm3 and -Must be linked with -lpvm3 and -

lgpvm3lgpvm3 Libraries in Libraries in $(PVM_ROOT)/lib/$$(PVM_ROOT)/lib/$(PVM_ARCH)(PVM_ARCH)

The PVM libraryThe PVM library

Enrolling and TIDsEnrolling and TIDs

int tid=pvm_mytid()

int tid=pvm_parent()

int info=pvm_exit()


Spawning a taskSpawning a task

flag = flag = PvmTaskDefaultPvmTaskDefault

PvmTaskHost (‘where’ is host)PvmTaskHost (‘where’ is host)

PvmTaskDebug (use gdb to spawn)PvmTaskDebug (use gdb to spawn)

typical error : tids[i] = -7 : executable not foundtypical error : tids[i] = -7 : executable not found

int numt=pvm_spawn(char *task, char **argv, int flag, char* where,int ntask, int *tids)


Message sending :Message sending :• initialize a buffer (initsend)initialize a buffer (initsend)• pack the datastructurepack the datastructure• send (send or mcast)send (send or mcast)

Message receiving :Message receiving :• blocking/non-blocking receiveblocking/non-blocking receive• unpack (the way you packed)unpack (the way you packed)


• Use pvm_initsend() to initiate sendUse pvm_initsend() to initiate sendint bufid = pvm_initsend(int encoding)int bufid = pvm_initsend(int encoding)

• Use one of the several pack functions Use one of the several pack functions to pack to pack

int info=pvm_pkint(int *np,int nitem, int int info=pvm_pkint(int *np,int nitem, int stride)stride)

• To send To send int info=pvm_send( int tid, int msgtag)int info=pvm_send( int tid, int msgtag)

int info=pvm_mcast(int *tids, int ntask, int int info=pvm_mcast(int *tids, int ntask, int msgtag)msgtag)


pvm_recv waits till a message with a pvm_recv waits till a message with a given given source-tidsource-tid and and tagtag arrives (-1 is arrives (-1 is wildcard) :wildcard) :int bufid=pvm_recv (int tid, int msgtag)int bufid=pvm_recv (int tid, int msgtag)

pvm_nrecv returns if no appropriate pvm_nrecv returns if no appropriate message has arrivedmessage has arrivedint bufid=pvm_nrecv (int tid, int msgtag)int bufid=pvm_nrecv (int tid, int msgtag)

Use one of the several unpack functions Use one of the several unpack functions to unpack (exactly the way you packed) to unpack (exactly the way you packed) int info=pvm_upkint(int *np,int nitem, int stride)int info=pvm_upkint(int *np,int nitem, int stride)

Message passing : an Message passing : an exampleexample

struct picture{

int sizex,sizey;char* name;colourMap* cols;

};


void pkPicture(picture* p){

pvm_pkint(&p->sizex,1,1);pvm_pkint(&p->sizey,1,1);pvm_pkstr(p->name);pkColourMap(p->cols);

};


picture* upkPicture(){

picture* p=new picture;pvm_upkint(&p->sizex,1,1);pvm_upkint(&p->sizey,1,1);p->name=new char[20];pvm_upkstr(p->name);p->cols=upkColourMap();return p;

};


//I’m Apicture myPic;...pvm_initsend(PvmDataDefault);pkPicture(&myPic);pvm_send(B,PICTAG);


//I’m Bpicture* myPic;

pvm_recv(A,PICTAG);myPic=upkPicture();

In C++ ?In C++ ?

Messages contain data, not codeMessages contain data, not code

No cute way to send and receive No cute way to send and receive

objectsobjects Provide constructors that build from Provide constructors that build from

receive bufferreceive buffer Receiving end must have a list of Receiving end must have a list of

possible objectspossible objects

PVM and GDBPVM and GDB

Spawn tasks with PvmTaskDebugSpawn tasks with PvmTaskDebug Pvmd will call Pvmd will call $PVM_ROOT/lib/debugger taskname$PVM_ROOT/lib/debugger taskname

Make sure this scripts starts an Make sure this scripts starts an xterm with a debugger running xterm with a debugger running ‘taskname’‘taskname’

GREAT debugging tool !!GREAT debugging tool !!

OverviewOverview


Ray-TracingRay-Tracing

Ray-TracingRay-Tracing• master slavemaster slave• Data ParallelData Parallel

MS

SSS

SS

Simulation : the ProblemSimulation : the Problem

“Unknown” system

par1 par3

par2

input1

input2

input3

Behaviour ?

Modeling : exampleModeling : example

EntranceShopping

Area Queues andCash desks

Exit

How many queues to achieve average queue length < 4 ?

Modeling : viewsModeling : views

Modeling viewpoints :Modeling viewpoints :• event viewevent view• process viewprocess view

Model classificationModel classification• Logic gatesLogic gates• Queueing netsQueueing nets• Petri netsPetri nets• ……..

Simulation : Design CycleSimulation : Design Cycle

“Real” System

Model

Results

Modeling

Simulation Validation

Simulation : Time Simulation : Time InvestmentInvestment

Modeling & ValidationModeling & Validation==

Human ThinkingHuman Thinking

SimulationSimulation==

Machine ThinkingMachine Thinking

Simulation : Time Simulation : Time Investment (2)Investment (2)

Simple Model :Simple Model :• Short Simulation TimeShort Simulation Time• Large Modeling TimeLarge Modeling Time• Large Validation TimeLarge Validation Time

Complex Model :Complex Model :• Large Simulation TimeLarge Simulation Time• Short Modeling TimeShort Modeling Time• Short Validation TimeShort Validation Time

Solution ???

Solution !!

Parallel SimulatorsParallel Simulators

Two Methods :Two Methods :• Multiple-Run :Multiple-Run :

– same simulation, different same simulation, different parameters/input stimuliparameters/input stimuli

– easy (stupid but effective)easy (stupid but effective)

• ““event-parallelism” :event-parallelism” :– distribute model over processorsdistribute model over processors– hard : fine granularhard : fine granular

Modeling : causalityModeling : causality

Process(state)

e1

e

e

e2

21 ee tt

Global Event Ordering

Parallel Simulators : event Parallel Simulators : event parallelismparallelism

Idea : simulate different events in Idea : simulate different events in parallelparallel

Condition : causal independence !Condition : causal independence ! Local Time Local Time Global Time Global Time

lvt1

lvt2

e1

e2

e1’

e2’

Parallel Simulators : event Parallel Simulators : event parallelismparallelism

Very fine grainVery fine grain Strong synchronization (=limited Strong synchronization (=limited

parallelism)parallelism) Strongly model dependent Strongly model dependent

performanceperformance Hard to implementHard to implement

Software TOOL

Parallel Simulators : Event Parallel Simulators : Event ParallelismParallelism

Fine grain specification :


Coarse grain implementation : (top view)


Coarse grain implementation : (side view)

Parallel Simulator

Sequential Sequential Sequential

Parallel Simulators : some Parallel Simulators : some resultsresults

0

50000

100000

150000

200000

250000

300000

350000

400000

2 4 6 8

processors

sp

ee

d (

ev/s

)

seq

da=10

da=5

da=0

OverviewOverview


ConclusionConclusion

NOW + LINUX + PVM = Scalability for NOW + LINUX + PVM = Scalability for free !free !

Drawbacks :Drawbacks :• Software is hard to implementSoftware is hard to implement• Performance is sensitive toPerformance is sensitive to

– problemproblem– machinemachine– implementationimplementation

Solution : software toolsSolution : software tools

ReferencesReferences

See our web-page atSee our web-page at

http://infoweb.vub.ac.be/~parallelhttp://infoweb.vub.ac.be/~parallel