LINUX & Parallel Processing
description
Transcript of LINUX & Parallel Processing
LINUX & Parallel LINUX & Parallel ProcessingProcessing
Wouter BrissinckWouter Brissinck
Vrije Universiteit BrusselVrije Universiteit Brussel
Dienst INFODienst [email protected]@info.vub.ac.be
+32 2 629 2965+32 2 629 2965
OverviewOverview
The DreamThe Dream The FactsThe Facts The ToolsThe Tools The Real ThingThe Real Thing The ConclusionThe Conclusion
History History
‘‘6060 1 computer N users 1 computer N users ‘‘80 80 N computers N computers N users N users ‘‘90 90 N computers N computers 1 user 1 user
past past problem = f(computer) problem = f(computer) future future computer = f(problem)computer = f(problem)
The Sequential FrustrationThe Sequential Frustration
Faster CPU : High development Faster CPU : High development cost + timecost + time
Speed of light : faster Speed of light : faster smaller smaller Size of Si atom is the ultimate limitSize of Si atom is the ultimate limit
ParallelismParallelism
Offers scalable performanceOffers scalable performance Solves as yet unsolvable problemsSolves as yet unsolvable problems
• problem sizeproblem size• computation timecomputation time
Hardware cost Hardware cost Software developmentSoftware development
ScalabilityScalability
PROBLEMPROBLEM * p
CPUCPUCPU
CPUCPUCPU
CPU1
p
ScalabilityScalability
problem on 1 CPU takes T timeproblem on 1 CPU takes T time p times larger problem on p CPUs p times larger problem on p CPUs
takes T timetakes T time
Same technology ( only more of it )Same technology ( only more of it ) Same code ( if ... )Same code ( if ... ) Economy of scale Economy of scale
Hardware cost ?Hardware cost ?
Parallel machines :Parallel machines :• expensive to buyexpensive to buy• expensive to maintainexpensive to maintain• expensive to upgradeexpensive to upgrade• unreliableunreliable• lousy operating lousy operating
systemssystems
N.O.W :N.O.W :• you own one !you own one !• Its maintained for youIts maintained for you• You just did, didn’t You just did, didn’t
you ?you ?• You rely on it every day You rely on it every day
!!• How did you call How did you call
LINUX ?LINUX ?A network-of-workstations is a virtually free A network-of-workstations is a virtually free parallel machine !parallel machine !
OverviewOverview
The DreamThe Dream The FactsThe Facts The ToolsThe Tools The Real ThingThe Real Thing The ConclusionThe Conclusion
100 painters 100 painters 1 day ? 1 day ?
SynchronizationSynchronization• shared resourcesshared resources• dependenciesdependencies
CommunicationCommunication
ProblemsProblems
1 painter 1 painter 100 days 100 days
Amdahl’s LawAmdahl’s Law
pp
T
TSpeedup
p
111
= sequential fraction
pTp
TTTp
)1(
TTT )1(1
= communication cost
Amdahl’s LawAmdahl’s Law
1
#processors=p
Speedup=S
pp
SSpeedup
1
1 = sequential fraction = communication cost
GranularityGranularity
Granularity =Granularity = ionCommunicat
nComputatio
N.O.W Dedicated MIMDPipelining in CPU
coarse fine
Network latencyNetwork latency
L
T = L + len / B
B
len
L=Latency(s) len=length(bits) B=Bandwidth(bits/s)
proc1
proc2
The N of N.O.WThe N of N.O.W
?
Software DesignSoftware Design
Sequential specification Sequential specification Parallel Parallel implementationimplementation
Parallel specification Parallel specification (less) (less) parallel implementationparallel implementation• Preferably : 1 process/processorPreferably : 1 process/processor
Software Design : Software Design : classificationclassification
Master - SlaveMaster - Slave PipeliningPipelining Systolic ArraySystolic Array
Functional parallelismFunctional parallelism Data parallelismData parallelism
OverviewOverview
The DreamThe Dream The FactsThe Facts The ToolsThe Tools The Real ThingThe Real Thing The ConclusionThe Conclusion
P.V.MP.V.M
PVM : overviewPVM : overview PVM PVM
installationinstallation Using PVMUsing PVM
• consoleconsole• compilingcompiling• writing codewriting code
Message Message passingpassing• in Cin C• in C++in C++
gdb and pvmgdb and pvm
Parallel Virtual MachineParallel Virtual Machine
Most popular Message Passing Most popular Message Passing EngineEngine
Works on MANY platforms ( LINUX, Works on MANY platforms ( LINUX, NT, SUN, CRAY, ...)NT, SUN, CRAY, ...)
Supports C, C++, fortranSupports C, C++, fortran Allows heterogenious virtual Allows heterogenious virtual
machinesmachines FREE software ( ftp.netlib.org )FREE software ( ftp.netlib.org )
Heterogenious NOWsHeterogenious NOWs
PVM provides data format PVM provides data format translation of messagestranslation of messages
Must compile application for each Must compile application for each kind of machinekind of machine
Can take advantage of specific Can take advantage of specific resources in the LANresources in the LAN
PVM featuresPVM features
User-configured host poolUser-configured host pool Application = set of (unix) processesApplication = set of (unix) processes
• automatically mappedautomatically mapped• forced by programmerforced by programmer
Message Passing modelMessage Passing model• strongly typed messagesstrongly typed messages• asynchronous (blocking/non-blocking asynchronous (blocking/non-blocking
receive)receive) Multiprocessor supportMultiprocessor support
How PVM worksHow PVM works
Console for resource managementConsole for resource management Deamon for each host in the virtual Deamon for each host in the virtual
machinemachine• message passingmessage passing• fault tolerancefault tolerance
PVM library provides primitives forPVM library provides primitives for• task spawningtask spawning• send/receivesend/receive• VM info, ...VM info, ...
How PVM worksHow PVM works
Tasks are identified by TIDs (pvmd Tasks are identified by TIDs (pvmd supplied)supplied)
Group server provides for groups Group server provides for groups of tasksof tasks• Tasks can join and leave group at willTasks can join and leave group at will• Groups can overlapGroups can overlap• Useful for multicastsUseful for multicasts
N.O.W SetupN.O.W Setup
PC PCPCPC
LINUX LINUXLINUXLINUX
PVM
Master
Slave Slave Slave Slave
Example : masterExample : master
#include "pvm3.h”main(){
int cc, tid;char buf[100];cc = pvm_spawn("hello_other",
(char**)0, 0, "", 1, &tid);if (cc == 1) {
cc = pvm_recv(-1, -1);pvm_bufinfo(cc, (int*)0, (int*)0, &tid);pvm_upkstr(buf);printf("from t%x: %s\n", tid, buf);
} pvm_exit();
}
Example : SlaveExample : Slave
#include "pvm3.h"main(){
int ptid;char buf[100];ptid = pvm_parent();strcpy(buf, "hello, world from ");gethostname(buf + strlen(buf), 64);
pvm_initsend(PvmDataDefault);pvm_pkstr(buf);pvm_send(ptid, 1);
pvm_exit();}
PVM : installingPVM : installing
Set environment variables (before Set environment variables (before build)build)• PVM_ROOTPVM_ROOT• PVM_ARCHPVM_ARCH
Make defaultMake default Make sure you can rsh without Make sure you can rsh without
authenticationauthentication
PVM and rshPVM and rsh
PVM starts tasks using rshPVM starts tasks using rsh Fails if a password is requiredFails if a password is required .rhosts on remote account must .rhosts on remote account must
mention the machine where the mention the machine where the console is runningconsole is running
Typical error : Can’t start pvmdTypical error : Can’t start pvmd
Typical situationTypical situation
All machines share the same user All machines share the same user accountsaccounts
Account has these directories/files :Account has these directories/files :/shared/pvm3/lib/LINUX/pvmd3/shared/pvm3/lib/LINUX/pvmd3
/shared/pvm3/lib/SUN4/pvmd3/shared/pvm3/lib/SUN4/pvmd3
~/pvm3/bin/LINUX/Eg.App~/pvm3/bin/LINUX/Eg.App
~/pvm3/bin/SUN4/Eg.App~/pvm3/bin/SUN4/Eg.App
.rhosts file looks like this : ( mymachine .rhosts file looks like this : ( mymachine is the machine which runs the console )is the machine which runs the console )
mymachinemymachine
mymachine.my.domainmymachine.my.domain
PVM ConsolePVM Console
add hostname : add a host to VMadd hostname : add a host to VM conf : show list of hostsconf : show list of hosts ps -a : show all pvm processes runningps -a : show all pvm processes running
• Manually started (i.e. not spawned) tasks are Manually started (i.e. not spawned) tasks are shown by a ‘-’shown by a ‘-’
reset : kill all pvm tasksreset : kill all pvm tasks quit : leave console, but keep deamon quit : leave console, but keep deamon
alivealive halt : kill deamonhalt : kill deamon
PVM Console : hostfilePVM Console : hostfile
hostfile contains hosts to be hostfile contains hosts to be added, with optionsadded, with options
mymachine ep=$(SIM_ROOT)/bin/$(PVM_ARCH)faraway lo=woutie pw=whoknows* lo=wouter
On machine without network (just On machine without network (just loopback):loopback):
* ip=127.0.0.1
Compiling a programCompiling a program
Application must be recompiled for Application must be recompiled for each architectureeach architecture• binary goes in binary goes in ~/pvm3/bin/$(PVM_ARCH)~/pvm3/bin/$(PVM_ARCH)• aimk determines architecture, then calls aimk determines architecture, then calls
‘make’‘make’ Must be linked with -lpvm3 and -Must be linked with -lpvm3 and -
lgpvm3lgpvm3 Libraries in Libraries in $(PVM_ROOT)/lib/$$(PVM_ROOT)/lib/$(PVM_ARCH)(PVM_ARCH)
The PVM libraryThe PVM library
Enrolling and TIDsEnrolling and TIDs
int tid=pvm_mytid()
int tid=pvm_parent()
int info=pvm_exit()
The PVM libraryThe PVM library
Spawning a taskSpawning a task
flag = flag = PvmTaskDefaultPvmTaskDefault
PvmTaskHost (‘where’ is host)PvmTaskHost (‘where’ is host)
PvmTaskDebug (use gdb to spawn)PvmTaskDebug (use gdb to spawn)
typical error : tids[i] = -7 : executable not foundtypical error : tids[i] = -7 : executable not found
int numt=pvm_spawn(char *task, char **argv, int flag, char* where,int ntask, int *tids)
The PVM libraryThe PVM library
Message sending :Message sending :• initialize a buffer (initsend)initialize a buffer (initsend)• pack the datastructurepack the datastructure• send (send or mcast)send (send or mcast)
Message receiving :Message receiving :• blocking/non-blocking receiveblocking/non-blocking receive• unpack (the way you packed)unpack (the way you packed)
The PVM libraryThe PVM library
• Use pvm_initsend() to initiate sendUse pvm_initsend() to initiate sendint bufid = pvm_initsend(int encoding)int bufid = pvm_initsend(int encoding)
• Use one of the several pack functions Use one of the several pack functions to pack to pack
int info=pvm_pkint(int *np,int nitem, int int info=pvm_pkint(int *np,int nitem, int stride)stride)
• To send To send int info=pvm_send( int tid, int msgtag)int info=pvm_send( int tid, int msgtag)
int info=pvm_mcast(int *tids, int ntask, int int info=pvm_mcast(int *tids, int ntask, int msgtag)msgtag)
The PVM libraryThe PVM library
pvm_recv waits till a message with a pvm_recv waits till a message with a given given source-tidsource-tid and and tagtag arrives (-1 is arrives (-1 is wildcard) :wildcard) :int bufid=pvm_recv (int tid, int msgtag)int bufid=pvm_recv (int tid, int msgtag)
pvm_nrecv returns if no appropriate pvm_nrecv returns if no appropriate message has arrivedmessage has arrivedint bufid=pvm_nrecv (int tid, int msgtag)int bufid=pvm_nrecv (int tid, int msgtag)
Use one of the several unpack functions Use one of the several unpack functions to unpack (exactly the way you packed) to unpack (exactly the way you packed) int info=pvm_upkint(int *np,int nitem, int stride)int info=pvm_upkint(int *np,int nitem, int stride)
Message passing : an Message passing : an exampleexample
struct picture{
int sizex,sizey;char* name;colourMap* cols;
};
Message passing : an Message passing : an exampleexample
void pkPicture(picture* p){
pvm_pkint(&p->sizex,1,1);pvm_pkint(&p->sizey,1,1);pvm_pkstr(p->name);pkColourMap(p->cols);
};
Message passing : an Message passing : an exampleexample
picture* upkPicture(){
picture* p=new picture;pvm_upkint(&p->sizex,1,1);pvm_upkint(&p->sizey,1,1);p->name=new char[20];pvm_upkstr(p->name);p->cols=upkColourMap();return p;
};
Message passing : an Message passing : an exampleexample
//I’m Apicture myPic;...pvm_initsend(PvmDataDefault);pkPicture(&myPic);pvm_send(B,PICTAG);
Message passing : an Message passing : an exampleexample
//I’m Bpicture* myPic;
pvm_recv(A,PICTAG);myPic=upkPicture();
In C++ ?In C++ ?
Messages contain data, not codeMessages contain data, not code
No cute way to send and receive No cute way to send and receive
objectsobjects Provide constructors that build from Provide constructors that build from
receive bufferreceive buffer Receiving end must have a list of Receiving end must have a list of
possible objectspossible objects
PVM and GDBPVM and GDB
Spawn tasks with PvmTaskDebugSpawn tasks with PvmTaskDebug Pvmd will call Pvmd will call $PVM_ROOT/lib/debugger taskname$PVM_ROOT/lib/debugger taskname
Make sure this scripts starts an Make sure this scripts starts an xterm with a debugger running xterm with a debugger running ‘taskname’‘taskname’
GREAT debugging tool !!GREAT debugging tool !!
OverviewOverview
The DreamThe Dream The FactsThe Facts The ToolsThe Tools The Real ThingThe Real Thing The ConclusionThe Conclusion
Ray-TracingRay-Tracing
Ray-TracingRay-Tracing• master slavemaster slave• Data ParallelData Parallel
MS
SSS
SS
Simulation : the ProblemSimulation : the Problem
“Unknown” system
par1 par3
par2
input1
input2
input3
Behaviour ?
Modeling : exampleModeling : example
EntranceShopping
Area Queues andCash desks
Exit
How many queues to achieve average queue length < 4 ?
Modeling : viewsModeling : views
Modeling viewpoints :Modeling viewpoints :• event viewevent view• process viewprocess view
Model classificationModel classification• Logic gatesLogic gates• Queueing netsQueueing nets• Petri netsPetri nets• ……..
Simulation : Design CycleSimulation : Design Cycle
“Real” System
Model
Results
Modeling
Simulation Validation
Simulation : Time Simulation : Time InvestmentInvestment
Modeling & ValidationModeling & Validation==
Human ThinkingHuman Thinking
SimulationSimulation==
Machine ThinkingMachine Thinking
Simulation : Time Simulation : Time Investment (2)Investment (2)
Simple Model :Simple Model :• Short Simulation TimeShort Simulation Time• Large Modeling TimeLarge Modeling Time• Large Validation TimeLarge Validation Time
Complex Model :Complex Model :• Large Simulation TimeLarge Simulation Time• Short Modeling TimeShort Modeling Time• Short Validation TimeShort Validation Time
Solution ???
Solution !!
Parallel SimulatorsParallel Simulators
Two Methods :Two Methods :• Multiple-Run :Multiple-Run :
– same simulation, different same simulation, different parameters/input stimuliparameters/input stimuli
– easy (stupid but effective)easy (stupid but effective)
• ““event-parallelism” :event-parallelism” :– distribute model over processorsdistribute model over processors– hard : fine granularhard : fine granular
Modeling : causalityModeling : causality
Process(state)
e1
e
e
e2
21 ee tt
Global Event Ordering
Parallel Simulators : event Parallel Simulators : event parallelismparallelism
Idea : simulate different events in Idea : simulate different events in parallelparallel
Condition : causal independence !Condition : causal independence ! Local Time Local Time Global Time Global Time
lvt1
lvt2
e1
e2
e1’
e2’
Parallel Simulators : event Parallel Simulators : event parallelismparallelism
Very fine grainVery fine grain Strong synchronization (=limited Strong synchronization (=limited
parallelism)parallelism) Strongly model dependent Strongly model dependent
performanceperformance Hard to implementHard to implement
Software TOOL
Parallel Simulators : Event Parallel Simulators : Event ParallelismParallelism
Fine grain specification :
Parallel Simulators : Event Parallel Simulators : Event ParallelismParallelism
Coarse grain implementation : (top view)
Parallel Simulators : Event Parallel Simulators : Event ParallelismParallelism
Coarse grain implementation : (side view)
Parallel Simulator
Sequential Sequential Sequential
Parallel Simulators : some Parallel Simulators : some resultsresults
0
50000
100000
150000
200000
250000
300000
350000
400000
2 4 6 8
processors
sp
ee
d (
ev/s
)
seq
da=10
da=5
da=0
OverviewOverview
The DreamThe Dream The FactsThe Facts The ToolsThe Tools The Real ThingThe Real Thing The ConclusionThe Conclusion
ConclusionConclusion
NOW + LINUX + PVM = Scalability for NOW + LINUX + PVM = Scalability for free !free !
Drawbacks :Drawbacks :• Software is hard to implementSoftware is hard to implement• Performance is sensitive toPerformance is sensitive to
– problemproblem– machinemachine– implementationimplementation
Solution : software toolsSolution : software tools
ReferencesReferences
See our web-page atSee our web-page at
http://infoweb.vub.ac.be/~parallelhttp://infoweb.vub.ac.be/~parallel