FFakultät für Mathematik und Informatikakultät für ...

Fakultät für Mathematik und InformatikFakultät für Mathematik und Informatik

Preprint 2015-15

Alexander Heinlein, Axel Klawonn, and Oliver Rheinbach

ISSN 1433-9307

Parallel Two-Level Overlapping Schwarz Methods in Fluid-Structure Interaction

Alexander Heinlein, Axel Klawonn,

and Oliver Rheinbach

Parallel Two-Level Overlapping Schwarz Methods in Fluid-Structure Interaction

TU Bergakademie Freiberg

Fakultät für Mathematik und Informatik

Prüferstraße 9

09596 FREIBERG

http://www.mathe.tu-freiberg.de

ISSN 1433 – 9307

Herausgeber: Dekan der Fakultät für Mathematik und Informatik

Herstellung: Medienzentrum der TU Bergakademie Freiberg

Parallel Two-Level Overlapping Schwarz

Methods in Fluid-Structure Interaction

Alexander Heinlein1, Axel Klawonn1, and Oliver Rheinbach2

1 Mathematisches Institut, Universitat zu Koln, Weyertal 86-90, 50931 Koln,Germany. E-mail: alexander.heinlein,[email protected]

2 Institut fur Numerische Mathematik und Optimierung, Fakultat furMathematik und Informatik, Technische Universitat Bergakademie Freiberg,Akademiestr. 6, 09596 Freiberg, Germany.E-mail: [email protected]

December 2, 2015

Abstract. Parallel overlapping Schwarz preconditioners are considered and ap-plied to the structural block in monolithic fluid-structure interaction (FSI). Thetwo-level overlapping Schwarz method uses a coarse level based on energy minimiz-ing functions. Linear elastic as well as nonlinear, anisotropic hyperelastic structuralmodels are considered in an FSI problem of a pressure wave in a tube. Using ourrecent parallel implementation of a two-level overlapping Schwarz preconditionerbased on the Trilinos library, the total computation time of our FSI benchmarkproblem was reduced by more than a factor of two compared to the algebraic one-level overlapping Schwarz method used previously. Finally, also strong scalabilityfor our FSI problem is shown for up to 512 processor cores.

1 The Two-Level Overlapping Schwarz Preconditioner

The GDSW preconditioner [4] is a two-level additive Schwarz preconditioner

M−1GDSW = Φ

(

ΦTAΦ)

−1ΦT +

N∑

i=1

RTi A

−1i Ri, (1)

with a special choice of energy minimizing coarse space functions Φ. Thecoarse space functions are discrete harmonic extensions of the restrictions ofthe null space of A to connected components (vertices, edges, and faces) ofthe interface Γ of a nonoverlapping domain decomposition. For the elasticityproblems considered here, the null space is spanned by the three translationsr1, r2, r3 and three (linearized) rotations r4, r5, r6.

Let I = Ω \ Γ be the set of degrees of freedom (d.o.f.) in the interior of asubdomain. Then the basis functions of the GDSW coarse space are definedby

Φ =

[

−A−1II A

TΓIΦΓ

ΦΓ

]

=

[

ΦI

ΦΓ

]

, (2)

2 A. Heinlein, A. Klawonn, and O. Rheinbach

where ΦΓ =[

RTΓ1GΓ1 ... RT

ΓMGΓN

]

, and RΓj is the restriction from Γ ontoΓj , the j-th interface component. The matrices GΓj are chosen such thattheir columns form a basis of the restriction of the matrix G to the indicescorresponding to Γj . For GDSW the condition number bound

κ(

M−1GDSWK

)

≤ C

(

1 +H

δ

)(

1 + log

(

H

h

))

,

was shown in [4]. Here, H and h are the typical subdomain and finite elementdiameters and δ denotes the overlap.

We have implemented a parallel GDSW preconditioner based on Trili-nos [7] and report on parallel scalability for model problems in [6]. Cur-rently, we use UMFPACK 5.3.0 to solve the problems on the subdomainsand MUMPS 4.10.0 in MPI parallel mode for the coarse problem. We use aone-to-one correspondance of subdomains to cores.

2 Monolithic Fluid-Structure Interaction

We use the software environment from [1], i.e., we use the LifeV softwarelibrary 3.6.2 coupled to FEAP 8.2. As opposed to [1], where a convectiveexplicit (CE) approach was used for the fluid, we now use a fully implicitscheme, and the linearized systems are now preconditioned using a FaCSIpreconditioner applying a SIMPLE preconditioner for the fluid; see [3].

2.1 Model Description

The fluid-structure interaction (FSI) problem consists of the fluid problem

ρf

(

∂u

∂t

∣

∣

∣

∣

X

+ ((u−w) · ∇)u

)

−∇ · σf (u, p) = 0 in Ωft × (0, T ],

∇ · u = 0 in Ωft × (0, T ],

(3)

which corresponds to the incompressible Navier-Stokes equations in ArbitraryLagrangian Eulerian (ALE) formulation, cf. [5], and the structural problem

ρs∂2ds

∂t2−∇ · (FS) = 0 in Ωs × (0, T ]. (4)

Here, Ωft and Ωf are the fluid domain in actual and reference configuration,

respectively, Ωs is the structural reference configuration, and Γ = ∂Ωf ∩∂Ωs

is the FSI interface. In (3), ρf denotes the density of the fluid, u and p are

the velocity and pressure, respectively, w =∂df

∂t|X is the velocity and df the

displacement of the fluid mesh movement, and σf (u, p) is the Cauchy stresstensor. In (4), ds is the displacement of the structure, ρs is the density of thestructure, and FS are the first Piola-Kirchhoff stresses.

Two-Level Overlapping Schwarz Methods in Fluid Structure Interaction 3

The ALE mapping At = id + df is obtained by solving an additionalgeometry problem

−∆df = 0 in Ωf ,df = ds on Γ,df · nf = 0 on ∂Ωf\Γ,

(5)

i.e., by means of discrete harmonic extensions. The fluid, structural, andgeometry problems are coupled by the geometric adherence (6), the continuityof the velocities (7), and the continuity of the stresses (8) on Γ ,

df = ds, (6)

∂ds

∂t= u At, (7)

(det[F])−1F−Tσf nf At + (FS)ns = 0. (8)

2.2 Monolithic Coupling in FSI

We use finite differences, in a fully implicit scheme, for the approximationof the time derivatives of both the fluid and the structure equations. Weuse piecewise quadratic (P2) finite elements for the structure and geometryproblems and P2–P1 mixed finite elements for the fluid, using conformingmeshes at the FSI interface.

The monolithic approach leads to a single nonlinear system containingthe fluid (F ), the structure (S), the geometry (G) problem, and the couplingconditions,

F (un+1f , pn+1,dn+1

f ) + 0 + CT1 λ

n+1 + 0

0 + S (dn+1s ) + CT

3 λn+1 + 0

C1 un+1f + C2 d

n+1s + 0 + 0

0 + C4dn+1s + 0 + H dn+1

f

=

bf

bs

C2 dns

0

.

(9)Here, λ is the vector of Lagrange multipliers. For conforming meshes, we haveC1|Γ = I|Γ , C3|Γ = −I|Γ , C2|Γ = 1/∆tC3, C4|Γ = I|Γ , where I|Γ is theidentity matrix defined on the interface Γ .

2.3 Linearization and Parallel Monolithic Preconditioner

As in [1], we solve the nonlinear monolithic FSI problem (9) using an inexactNewton method. The corresponding tangent JM, associated with (9), reads

JM =

DdfF 0 CT

1 DdfF

0 DdsS CT

3 0C1 C2 0 00 C4 0 H

≈

DdfF 0 CT

1 DdfF

0 DdsS 0 0

C1 C2 0 00 C4 0 H

=: PDN .

(10)


Inflow

Outflow

Mesh #1: Interior radius of the structure 0.15cmOuter radius of the structure 0.21cmLength 2.5cm

Mesh #2: Interior radius of the structure 0.08cmOuter radius of the structure 0.1cmLength 5cm

Mesh #3: Interior radius of the structure 0.08cmOuter radius of the structure 0.11cmLength 10cm

Fig. 1. Geometry of the FSI problem. The number of d.o.f. is almost identical forall geometries and well-balanced between fluid (F) and structure (S), cf. Table 1.

Mesh Velocity (F) Pressure (F) Displacement (S) Displacement (G)

#1 393 903 17 261 379 080 393 903#2 401 763 17 775 373 032 401 763#3 376 623 17 352 346 320 376 623Table 1. Number of degrees of freedom of the different meshes.

Here, DdfF denotes the linearization of the fluid operator, Ddf

F the shapederivatives, and Dds

S the linearization of the structural operator.We solve the linearized system using a GMRES iteration with the FaCSI

preconditioner [3], which is based on a factorization of the matrix PDN . Thefluid block is treated further by static condensation of the interface degrees offreedom and the use of a SIMPLE preconditioner for the fluid block; see [3].The inverses appearing in the application of the FaCSI preconditioner arereplaced by algebraic (for fluid or structure, separately) and geometric one-level overlapping Schwarz preconditioner (for the structure) or the GDSWpreconditioner (for the structure).

3 Numerical Results for Fluid-Structure Interaction

We consider our FSI problem while applying different preconditioners to thestructural block – but without changing the preconditioners for the fluid andgeometry blocks. We then report on the resulting performance of the fullmonolithic FSI simulation.

The default preconditioner for the structural block is IFPACK, a parallelalgebraic overlapping Schwarz preconditioner from Trilinos [7]. Our parallelpreconditioner has two potential advantages over IFPACK: it uses a geometricoverlap, and it can use a coarse space, for better robustness and improvednumerical scalability. We consider three different meshes; cf. Fig. 1. We applyzero-displacement Dirichlet boundary conditions to the structure at the inletand the outlet. A pressure wave in a tube (see Fig. 5) is created by applyinga constant normal stress σ · n = 0.133kPa at the inflow for t ≤ 0.003s. Weconsider three different material models for the wall, i.e., a linear elastic(LE), a Neo-Hookean (NH), and a realistic anisotropic nonlinear material


model (ΨA), cf. [8], which we have already considered in FSI for arterial wallsin [1]. For linear elasticity, we use E = 40kPa and ν = 0.3, for Neo-Hooke,µ = 7.72kPa and κ = 383.3kPa, and for the ΨA model, we use the parametersfrom [2, (ΨA Set 2)].

Time to Solution using Different Preconditioners for the Struc-

ture Block We perform simulations of the pressure wave in a tube for atotal simulation time T = 0.01s. We use a time step ∆t = 0.0001, 0.0002,0.0004, or 0.0005, i.e., we solve 100, 50, 25 or 20 monolithic nonlinear sys-tems. The nonlinear problems are solved using, on average, 5.1 (LE 0.0001s),5.6 (NH 0.0001s), 6.6 (ΨA 0.0001s), 6.1 (LE 0.0002s), 6.3 (NH 0.0002s), 7.9(ΨA 0.0002s), 7.4 (LE 0.0004s), 7.9 (NH 0.0004s), 11.9 (ΨA 0.0004s), 8.4(LE 0.0005s), or 9.5 (NH 0.0005s) Newton iterations, and a preconditionedGMRES iteration is used to solve the linearized monolithic systems in eachNewton step. Our stopping criterion for Newton is a mixed criterion with arelative and absolute tolerance of 10−8 and for GMRES a relative tolerance of10−6. We compare the number of iterations and the computing times in Ta-ble 2; see also Fig. 2. When using our preconditioner, we consider three cases:only using the first level (OS1), using the first and coarse level but neglectingthe rotations (r4, r5, r6) when constructing the coarse level (GDSW-nr), andusing the full GDSW preconditioner (GDSW), i.e., with first and coarse level.In the case where rotations are neglected (GDSW-nr) no geometric informa-tion is needed for the construction of the coarse problem. For all methods,we specify an overlap of δ = 2h. We perform the comparison using Mesh #1and 128 cores.

In Table 2, for a small time step, all preconditioners show a very similarperformance with respect to the number of GMRES iteration as well as thetimings. However, for a larger time step, where the weight in front of themass matrix is small, the number of iterations and the timings for IFPACKquickly deteriorate. The other methods, which use a geometric overlap, showa better performance. The use of a coarse space gives further improvements:for the largest time steps the GDSW preconditioner is the fastest method.Neglecting the rotations in the GDSW preconditioner (GDSW-nr), whichmakes the preconditioner more algebraic, yields a number of iterations whichfalls between the one-level preconditioner and the GDSW with the full coarsespace. For our experiments, it thus seems that for GDSW-nr it is not easy toamortize the cost for the coarse level compared to OS1.

The use of the GDSW preconditioner with the full coarse space, however,can be recommended as a new default. For smaller time steps the performanceof all preconditioners is similar and for larger time steps it is clearly the fastestoption: For the most challenging structural model (ΨA) combined with thelargest time step the monolithic FSI simulation is more than 2.5 times fasterwhen using the GDSW preconditioner instead of IFPACK; see Table 2. Thisis especially remarkable since, in our monolithic preconditioner, we have onlyexchanged the preconditioner for the structural block whereas the timingsare for the complete FSI simulation.


Fig. 2. Total number of GMRES iterations (top) and total runtime (bottom) forthe pressure wave in a tube FSI problem using Mesh #1 and 128 cores; see alsoTable 2. We use different preconditioners for the structure block. “OS1” is the one-level Schwarz preconditioner, “GDSW-nr” is the GDSW preconditioner withoutrotations, and “OS2” is the GDSW preconditioner with full coarse space.

Strong Scaling for the Fluid-Structure Interaction problem InFig. 3 and Fig. 4, we present strong parallel scaling results for the pressure

wave in a tube using ∆t = 0.0001s and ∆t = 0.0002s, respectively, for a linearelastic tube. For the structure, we have used our new default preconditioner,i.e., the GDSW preconditioner including rotations, with overlaps of δ = 1hand δ = 2h. We present the GMRES iterations per Newton step and the totalruntime for a time step. The timings are for the fully coupled FSI simulation.

For all cases, we observe good scalability results, whereas the scaling isslightly worse for a time step of 0.0002s. This is partially a result of thenumber of Newton iterations, which varies from 3 to 5. We also observe a sig-nificant influence of the shape of the geometry on the performance. For Mesh#3, we observe the lowest number of iterations, the best numerical scalability,the lowest computing computing times and the best parallel scalability.Acknowledgements The authors gratefully acknowledge the Gauss Centre for Su-

percomputing (GCS) for providing computing time through the John von Neumann

Institute for Computing (NIC) on the GCS share of the supercomputer JUQUEEN

at Julich Supercomputing Centre (JSC). The authors also gratefully acknowledge

the Cray XT6m at Universitat Duisburg-Essen and the financial support by the

German Science Foundation (DFG), project no.KL2094/3-1 and RH122/4-1.


# Cores101 102

GM

RE

S It

erat

ions

20

30

40

50

60

70

80

90

100Mesh 1, GDSW, overlap 1hMesh 1, GDSW, overlap 2hMesh 2, GDSW, overlap 1hMesh 2, GDSW, overlap 2hMesh 3, GDSW, overlap 1hMesh 3, GDSW, overlap 2h

# Cores101 102

Tim

e in

s

102

103

Optimal ScalingMesh 1, GDSW, overlap 1hMesh 1, GDSW, overlap 2hMesh 2, GDSW, overlap 1hMesh 2, GDSW, overlap 2hMesh 3, GDSW, overlap 1hMesh 3, GDSW, overlap 2h

Fig. 3. Strong scaling (16 to 512 cores) for FSI using linear elasticity and ∆t =0.0001s The computing time for one time step is shown. Always 3 Newton steps.

# Cores101 102

GM

RE

S It

erat

ions

20

30

40

50

60

70

80

90

100Mesh 1, GDSW, overlap 1hMesh 1, GDSW, overlap 2hMesh 2, GDSW, overlap 1hMesh 2, GDSW, overlap 2hMesh 3, GDSW, overlap 1hMesh 3, GDSW, overlap 2h

# Cores101 102

Tim

e in

s

102

103

Optimal ScalingMesh 1, GDSW, overlap 1hMesh 1, GDSW, overlap 2hMesh 2, GDSW, overlap 1hMesh 2, GDSW, overlap 2hMesh 3, GDSW, overlap 1hMesh 3, GDSW, overlap 2h

Fig. 4. Same as Fig. 3 but using ∆t = 0.0002s. Newton steps vary from 3 to 5.

Fig. 5. Fluid pressure (top) and structural deformation (bottom) for the linearelastic (left), the Neo-Hookean (middle), and the ΨA (right) material model att = 0.003s. The structural displacement is magnified by a factor of 10. The figurealso illustrates the significantly different behavior for the material models.


∆t Struct. IFPACK One-level Schwarz GDSW w/o rot. GDSW(OS1) (GDSW-nr)

Time GMRES Time GMRES Time GMRES Time GMRESiter iter iter iter

0.0001 LE 5.0m 53.4 5.1m 50.8 5.4m 51.8 5.3m 50.8

NH 8.6m 89.8 6.8m 59.3 7.1m 55.m 7.0m 52.7

ΨA 19.7m 214.7 9.9m 82.0 10.5m 81.0 10.6m 79.1

0.0002 LE 8.9m 95.8 7.8m 74.5 7.0m 60.7 6.8m 58.0

NH 14.2m 152.4 9.8m 87.5 9.6m 77.2 9.0m 66.0

ΨA 33.3m 316.7 13.2m 96.9 13.8m 94.1 13.9m 90.7

0.0004 LE 15.3m 147.2 14.1m 124.5 10.9m 84.4 9.6m 71.9

NH 24.7m 226.5 17.8m 145.7 16.2m 117.9 13.6m 88.4

ΨA 63.0m 399.9 27.0m 145.4 27.1m 135.5 23.5m 108.5

0.0005 LE 19.4m 169.0 17.7m 142.0 13.0m 93.7 11.3m 76.3

NH 33.5m 261.5 24.2m 171.0 20.9m 133.2 17.1m 96.1

Table 2. Average computing time per time step (in minutes) and average numberof GMRES iterations per Newton step for the pressure wave in a tube problem;see Fig. 2 for the total runtimes. Linear elasticity (LE), Neo-Hooke (NH), and anonlinear, anisotropic hyperelastic material law to model an arterial wall (ΨA);also see Fig. 5. The time step is ∆t and the final simulation time is T = 0.01s.We compare IFPACK with the one-level overlapping Schwarz preconditioner (OS1)and the GDSW preconditioner with and without rotations (OS2/OS2-nr) on 128cores of a Cray XT6m. No convergence for ΨA and ∆t = 0.0005. Best numbers inbold face.


References

1. D. Balzani, S. Deparis, S. Fausten, D. Forti, A. Heinlein, A. Klawonn,

A. Quarteroni, O. Rheinbach, and J. Schroder, Numerical modeling offluid-structure interaction in arteries with anisotropic polyconvex hyperelasticand anisotropic viscoelastic material models at finite strains, Internat. J. Nu-mer. Meth. Biomed. Engrg. (2016), Accepted 10/2015.

2. D. Brands, A. Klawonn, O. Rheinbach, and J. Schroder, Modelling andconvergence in arterial wall simulations using a parallel FETI solution strategy,Comput. Methods Biomech. Biomed. Engin. 11 (2008), 569–583.

3. S. Deparis, D. Forti, G. Grandperrin, and A. Quarteroni,FaCSI : A block parallel preconditioner for fluid-structure interaction in hemodynamics,Tech. Report 13, 2015.

4. C. R. Dohrmann, A. Klawonn, and O. B. Widlund, Domain decompositionfor less regular subdomains: overlapping Schwarz in two dimensions, SIAM J.Numer. Anal. 46:4 (2008), 2153–2168.

5. L. Formaggia, A. Quarteroni, and A. Veneziani,Cardiovascular Mathematics, vol. 3, 2009.

6. A. Heinlein, A. Klawonn, and O. Rheinbach, Parallel overlapping Schwarzwith an energy-minimizing coarse space, 2015, To be submitted to the Proceed-ings of the 23rd International Conference on Domain Decomposition Methods,Springer Lect. Notes Comput. Sci. Eng.

7. M. A. Heroux, R. A. Bartlett, V. E. Howle, R. J. Hoekstra, J. J. Hu,

T. G. Kolda, R. B. Lehoucq, K. R. Long, R. P. Pawlowski, E. T. Phipps,

A. G. Salinger, H. K. Thornquist, R. S. Tuminaro, J. M. Willenbring,

A. Williams, and K. S. Stanley, An overview of the Trilinos project, ACMTrans. Math. Softw. 31:3 (2005), 397–423.

8. G. Holzapfel, T. Gasser, and R. Ogden, A new constitutive framework forarterial wall mechanics and a comparative study of material models, Journalof Elasticity 61 (2000), 1–48.

FFakultät für Mathematik und Informatikakultät für ...

Documents

Transcript of FFakultät für Mathematik und Informatikakultät für ...