A large database of graphs and its use for benchmarkinggraph isomorphism algorithms

M. De Santo a, P. Foggia b, C. Sansone b, M. Vento a,*

a Dipartimento di Ing. dell’Informazione e di Ing. Elettrica, Universit!aa di Salerno, Via Ponte Don Melillo, 1 I-84084 Salerno, Italyb Dipartimento di Informatica e Sistemistica, Universit!aa di Napoli ‘‘Federico II’’ Via Claudio, 21 I-80125 Napoli, Italy

Abstract

Despite of the fact that graph-based methods are gaining more and more popularity in different scientific areas, it hasto be considered that the choice of an appropriate algorithm for a given application is still the most crucial task.

The lack of a large database of graphs makes the task of comparing the performance of different graph matchingalgorithms difficult, and often the selection of an algorithm is made on the basis of a few experimental results available.

In this paper we present an experimental comparative evaluation of the performance of four graph matching al-gorithms. In order to perform this comparison, we have built and made available a large database of graphs, which isalso described in detail in this article.! 2002 Elsevier Science B.V. All rights reserved.

Keywords: Graph database; Benchmarking; Exact graph matching; Graph isomorphism

1. Introduction

Graphs are widely used in several areas forrepresenting many different kinds of information:syntactic structures of a language, chemical com-pounds, geographic maps, computer networks,software systems architectures, database struc-tures, complex objects or scenes found in static ormotion pictures are just a few examples of whathas been described using graphs (Rouvray and

Balaban, 1979; Wong, 1992; Shearer et al., 1998;Abdulrahim and Misra, 1998).

Many of these applications require some kindof comparison between graphs, which can be for-mulated as a graph matching problem. In parti-cular, in several important cases, the applicationentails the search for graph isomorphism or forgraph–sub-graph isomorphism. In the last threedecades, plenty of algorithms for this problem,using either optimal or approximate methods,have been proposed with the specific aim of re-ducing computational complexity of the matchingprocess. Far from giving an exhaustive list, somewidely known algorithms are described in (Hop-croft and Wong, 1974; Luks, 1982; Ullmann, 1976;Bunke and Messmer, 1995; Cordella et al., 1999).

Pattern Recognition Letters 24 (2003) 1067–1079

www.elsevier.com/locate/patrec

*Corresponding author. Tel.: +39-08996-4254; fax: +39-08996-4218.

E-mail addresses: desanto@unisa.it (M. De Santo), foggi-apa@unina.it (P. Foggia), carlosan@unina.it (C. Sansone),mvento@unisa.it (M. Vento).

0167-8655/03/$ - see front matter ! 2002 Elsevier Science B.V. All rights reserved.PII: S0167-8655 (02 )00253-2

Although the authors of each novel algorithmusually provide experimental results supportingthe claim that their method can outperform theprevious ones at least under some circumstances, itis almost always very difficult for a user to choosethe algorithm which is best suited for dealing withthe problem at hand. In fact, a report of the al-gorithm performance on a specific test case canoften provide no useful clues about what willhappen in a different domain.

Unfortunately, only a few papers face theproblem of making an extensive comparison ofgraph matching algorithms in terms of key per-formance indices (memory and time requirements,maximum graph size, etc.) (Bunke and Vento,1999). So, it seems that the habit of proposingmore and more new algorithms is prevailingagainst the need of assessing the performance ofthe existing ones in an objective way. As a con-sequence, the users of graph-based approaches canonly use qualitative criteria to select the algorithmthat seems to better fit with the application con-straints.

As the popularity of graphs as a representationtool grows also for complex problems requiringhundreds or even thousands of nodes, it becomesmore and more important for building robust andeffective systems that we are able to choose themost appropriate algorithms to be used as buildingblocks. Hence, some effort must be put in the taskof comparing graph matching algorithms.

If we try to understand the reasons why up tonow no serious attempt has been made for com-paring graph matching algorithms, we can easilyrecognize that one of the main difficulties is thelack of standard databases of graphs specificallydesigned for this purpose. In other research fields(for example, OCR), the availability of large de-facto standard databases improves the verifiabilityand the comparability of the experimental resultsof each method; why this cannot be done also forgraph matching?

A possible answer is that the creation of a graphdatabase is definitely not a simple task, since sev-eral issues have to be faced. The first question is:should the graphs be collected from real-worldapplications or should they be generated accordingto some probabilistic model? The latter choice,

besides being simpler to implement, permits a finercontrol over the features of the graphs; but thenwhich model(s) will faithfully reflect the perfor-mance obtainable in a given real application?Further questions: how large should the graphsbe? How dense? How many graphs should becomprised in the database to make it statisticallyrepresentative? Should semantic attributes be at-tached to nodes and edges, and if so, how shouldthose attributes be obtained?

Since these problems have no simple answer onwhich the majority of the research community canagree, it is unreasonable that a single comprehen-sive database can be built which will satisfy all theneeds of everyone. Nevertheless, it is our opinionthat it is important to start building and sharinggraph data sets, to provide a common groundfor comparing the algorithms’ performance. Adatabase which lacks some property desirable forsome class of users is better than no database atall.

Even after settling the question of the database,some other issues remain open. In order to providea fair comparison, it should be taken into accountthat algorithms often are optimized for differentusage patterns. Inexact graph matching algorithmsare more robust, but also considerably slower thanexact matching algorithms. Suboptimal algorithmscan be quite faster, but may fail in finding a solu-tion even if it exists. Some kinds of algorithms canbe quite slow when matching two graphs, but showa considerable speed-up when matching one graphagainst a large set of prototypes. Other algorithmscan be impressive on small graphs, but, due to asignificant memory usage, can result definitelyinapplicable to larger ones. As a consequence, acomparison is meaningful only if the algorithmsbeing compared have similar characteristics; oth-erwise little or no useful information can begained.

In this paper we present an experimental com-parison of four exact graph matching algorithms.In order to perform this benchmarking activity, wehave built and made available to the public do-main a large database of graphs.

In the rest of the paper, we will describe in moredetail the composition of the database. Then wewill provide an overview of graph matching algo-

1068 M. De Santo et al. / Pattern Recognition Letters 24 (2003) 1067–1079

rithms, followed by some more information aboutthe algorithms used for this comparison and thereasons behind their choice. Finally, the obtainedperformance of the algorithms on the database willbe presented and discussed.

2. The database

In general, two approaches can be followed forgenerating a database; a first way is to start fromgraphs obtained from real data, otherwise thedatabase can be obtained synthetically. Althoughthe first approach allows us to obtain rather real-istic graphs, it is generally more expensive as itrequires the collection of real data and the selec-tion of the set of algorithms to be used for ob-taining graphs from data. In this case the graphsare dependent on both the domain under consider-ation and the pre-processing algorithm used, re-ducing significantly the generality of the databaseand its reusability in other contexts. On the con-trary, the artificial generation of graphs is not onlysimpler and faster than collecting graphs from realapplications, but also allows us to control thevariation of several critical parameters of theunderlying graph population, such as the averagenumber of nodes, average number of edges pernode, number of different labels, and so on.Starting from these considerations, we decided togenerate the database synthetically, so obtaining aquite large database of graphs, that can be alsoeasily expandable, in a relatively short time.

The choice of the kind of graphs to be in-cluded in the database started with an analysisof the graphs mainly used by members of theIAPR-TC15 community (see http://www.iapr-tc15.unisa.it); a classification of various kinds of graphsused within the pattern recognition field has beenrecently proposed in (Bunke and Vento, 1999).The database is structured in pairs of graphs. Twocategories of pairs of graphs have been introduced.Pairs made of isomorphic graphs and pairs madeof graphs in which the second graph is a sub-graphof the first one. A total of 72,800 pairs of graphshave been generated: 18,200 pairs of isomorphicgraphs and 54,600 pairs for which a sub-graphisomorphism exists. Each category of pairs is made

up of graphs that are different for structure andsize. In particular, the following kinds of graphshave been considered:

• Randomly connected graphs;• Regular meshes, with different dimensionality:2D, 3D and 4D;

• Irregular meshes;• Bounded valence graphs;• Irregular bounded valence graphs.

Each kind of graphs has pairs of different size,ranging from few dozens to about 1000 nodes (i.e.small and medium size graphs according to theclassification presented in (Bunke and Vento,1999)). For each size and kind of graphs 100 dif-ferent pairs have been generated.

A brief description of the properties of eachkind of graph and of the motivation inspiring thechoice of including them in the database is given inSection 2.1, together with the number of generatedisomorphic pairs per kind.

In case of graph sub-graph isomorphism wehave generated pairs in which the two graphs(satisfying the graph–sub-graph condition) havethree different size ratios, namely 0.2, 0.4 and 0.6.So, each kind of graphs contains a number of pairsthree times higher than that obtained in case of theisomorphism.

The graphs composing the whole database havebeen distributed on a CD during the 3rd IAPR-TC15 Workshop on Graph-based Representa-tions in Pattern Recognition (GbR2001) and arealso publicly available on the web at the http://amalfi.dis.unina.it/graph.

Graphs are stored in a binary format. Eachgraph is contained in a file, and the files aregrouped into directories according to the kind ofgraphs and to the value of the parameters used inthe generation.

Each graph file is composed by a sequence of16-bit words; the words are encoded in little-en-dian format (e.g. LSB first). The first word repre-sents the number of nodes in the graph. Then, foreach node, there is a word encoding the number ofedges coming out of that node, followed by a se-quence of words encoding the endpoints of thoseedges.

M. De Santo et al. / Pattern Recognition Letters 24 (2003) 1067–1079 1069

In addition to the graphs, we have also madeavailable the sources of the programs used fortheir generation, which are coded in C++. For thegeneration of the whole database, a Unix shellscript has been written. The database has beengenerated on a computer with a 350 MHz PentiumIII processor and 128 MB of RAM. The under-lying operating system was Linux 2.2.14, and theC++ compiler was egcs release 1.1.2. The totaltime required by the generation has been of about7 h, and the overall size of the compressed archiveis about 420 MB. The database is completed with asimplified ground truth; for each pair of graphs(both in the case of isomorphism and/or sub-graphisomorphism) the number of existing solutions isreported. This information has been obtained byusing the graph matching algorithms described inSection 3.

2.1. Kind of graphs

Randomly connected graphs are graphs in whichthe edges connect nodes without any struc-tural regularity. They have been introduced forsimulating applications in which the entities (rep-resented by nodes) can establish relations (repre-sented by edges) with any other entity (not onlythe surrounding ones) independently of the rela-tive positions. This hypothesis typically occurs inthe middle and high processing levels of a com-puter vision task.

It is assumed that the probability of an edgeconnecting 2 nodes of the graph is independent onthe nodes themselves. The same model proposed in(Ullmann, 1976) has been adopted for generatingthese graphs: it fixes the value g of the probabilitythat an edge is present between two distinct nodesn and n0. The probability distribution is assumed tobe uniform.

According to the meaning of g, if N is the totalnumber of nodes of the graph, the number of itsedges will be equal to gNðN " 1Þ. However, if thisnumber is not sufficient to obtain a connectedgraph, further edges are suitably added until thegraph being generated becomes connected.

Three different values of the edge density g hasbeen considered (0.01, 0.05 and 0.1) and 1000 pairsof isomorphic graphs of different size, ranging

from 20 to 1000 have been generated for eachvalue of g.

Regular meshes are introduced for simulatingapplications dealing with regular structures asthose operating at the lower levels of a vision task.Furthermore, it is generally agreed that graphshaving a regular structure represent a worst casefor general graph matching algorithms (i.e. algo-rithms working on any kind of graphs) (Ullmann,1976). This problem determined the born of spe-cialized graph matching methods, with kind ofalgorithms able to efficiently perform the matchingfor given graph structures. So the database in-cludes, as regular graphs, the mesh connectedgraphs (2D, 3D and 4D).

The considered 2D mesh are graphs in whicheach node (except those belonging to the border ofthe mesh) is connected with its 4 neighborhoodnodes. Similarly, each node of a 3D and 4D graphhas respectively connections with its 6 and 8neighborhood nodes. In all cases the meshes areopen, i.e. the border nodes are connected only withinternal nodes.

In particular, 1000 pairs of isomorphic 2Dmeshes with size ranging from 16 to 1024 nodeshave been generated, as well as 800 pairs of 3Dmeshes (with size ranging from 27 to 1000 nodes)and 500 pairs of 4D meshes (with size rangingfrom 16 to 1296 nodes).

Irregular mesh-connected graphs have been in-troduced for simulating the behavior of the algo-rithms in presence of slightly distorted meshes.

They have been obtained from regular 2D me-shes by the addition of a certain number of edges.Each added edge connects nodes that have beenrandomly determined according to a uniform dis-tribution. The number of added branches is qN ,where q is a constant greater than 0. Note that, thecloser q to 0 is, the more symmetric the graphs are.

Three values of q has been considered (0.2, 0.4and 0.6) and 1000 pairs of 2D meshes of differentsize have been generated for each value of q, aswell as 800 pairs of 3D meshes and 500 pairs of 4Dmeshes, giving rise to a total of 6900 pairs of iso-morphic graphs.

Bounded valence graphs model those applica-tions in which each object (i.e. a node) establisha fixed number of relations (edges) with other

1070 M. De Santo et al. / Pattern Recognition Letters 24 (2003) 1067–1079

objects, not necessarily with those belonging to itsneighborhood. They are graphs in which everynode has a number of edges (among ongoing andoutgoing) lower than a given threshold, calledvalence. A particular case occurs when the numberof edges is equal for all the nodes; in this case thegraph is commonly called fixed valence graph.

The database includes graphs with a fixed va-lence, that have been generated by inserting ran-dom edges (using an uniform distribution) with theconstraint that the valence of a node cannot exceeda selected value; edge insertion continues until allthe nodes reach the desired valence. It is worthnoting that it is impossible to have fixed valencegraphs with an odd number of nodes and an oddvalence, but in our database we have only con-sidered graphs with an even number of nodes.

Three different values of the valence v ð3;6 and 9Þ have been considered and 1000 pairs ofgraphs of different size have been generated foreach value of v. Again, sizes ranging from 20 to1000 nodes have been considered.

In order to introduce some irregularities in thebounded valence graphs, we have considered alsothe Irregular bounded valence graphs. For suchgraphs, the average valence of the nodes (that is,the ratio between the number of edges and thenumber of nodes) is still bounded, but the singlenodes may have a valence which is quite differentfrom the average, and which is not bounded bya constant value. To this aim, first a fixed valencegraph is generated. Then, a certain number ofedges are moved from the nodes they are attachedto, to other nodes. The number of movements isequal to M ¼ 0:1NV , where V is the valence. Thisis equivalent to say that 10% of all the edges aremoved.

The edges to be moved are chosen according toa random distribution with uniform probability.However, the new endpoints to which these edgesare connected are not chosen uniformly, since thischoice would affect only very slightly the overallvariance of the valence of the nodes. Instead, aftera random permutation of the nodes, the movededges are distributed among the nodes using aprobability distribution in which the node whoseindex is i has a probability of receiving an edgeevaluated as a expð"biÞ where a and b depend on

the number N of nodes, and satisfy the follow-ing constraints: (i) the sum of the probabilities ofthe nodes of the graph is equal to 1 and (ii) theprobability of the node i multiplied by the numberof edges to be moved is equal to 0:5

pN . Using this

distribution the maximum valence of the resultinggraph will not be independent of N, and so special-purpose algorithms for bounded valence graphscannot be employed, even though the graph isisomorphic for 90% of its edges to a fixed valencegraph.

As in case of bounded valence graphs, threevalues of v ð3; 6 and 9Þ have been considered and1000 pairs of graphs of different size (from 20 to1000 nodes) have been generated for each valueof v.

3. A brief review of algorithms for graph isomor-phism

During the last decades significant researchefforts have been devoted to improve performanceof the graph matching algorithms, in terms of bothcomputational time and memory requirements.

Some algorithms reduce the computationalcomplexity of the matching process by imposingtopological restrictions on the graphs (Hopcroftand Wong, 1974; Luks, 1982; Jiang and Bunke,1996). An alternative approach is that of using anadequate representation of the searching processand pruning unprofitable paths in the searchspace. In this way, no restrictions must be imposedon the structure of the input graphs and the ob-tained algorithms can be applied in more generalcases.

A procedure that significantly reduces the sizeof the search space is the backtracking algorithmproposed by Ullmann (1976). This algorithmis devised for both graph isomorphism and sub-graph isomorphism and is still today one of themost commonly used for exact graph matching.This is confirmed by the fact that in (Messmer,1996) it is compared with other algorithms and itresults the more convenient in terms of matchingtime in case of one-to-one matching.

Another backtracking algorithm is the onepresented by Schmidt and Druffel (1976). It uses

M. De Santo et al. / Pattern Recognition Letters 24 (2003) 1067–1079 1071

the information contained in the distance matrixrepresentation of a graph to reduce the search treeof possible mappings.

A more recent algorithm, known as VF, isbased on a depth-first search strategy, with a set ofrules to efficiently prune the search tree. Such rulesin case of isomorphism are shown in (Cordellaet al., 1998). An improved version, referred to asVF2, is based on the same rationale, but stores theinformation of the state space search in more ef-fective data structures, so as to significantly reducethe matching time and the memory requirements.Details on these aspects are given in (Cordellaet al., 2001).

As regards the graph isomorphism problem, itis also necessary to mention the McKay’s Nautyalgorithm (McKay, 1981). It is based on a set oftransformations that reduce the graphs to bematched to a canonical form on which the test-ing of the isomorphism is significantly faster. Evenif Nauty is considered one of the fastest graphisomorphism algorithms today available, it hasbeen shown that there are categories of graphs forwhich it requires exponential time (Miyazaki,1997).

Another possible approach to the isomorphismproblem is the one presented in (Bunke andMessmer, 1995); instead of reducing the com-plexity of matching two graphs, the authors at-tempt to reduce the overall computational costwhen matching a sample graph against a large setof prototypes. The method performs the matchingin quadratic time with the size of the input graphand independently on the number of prototypes. Itis obviously convenient in applications requiringthe matching of a graph against a database, butthe memory required to store the pre-processeddatabase grows exponentially with the size of thegraphs, making the method suitable only for smallgraphs. So one of the authors concludes in (Mess-mer, 1996) that in case of one-to-one match-ing other algorithms (in particular, in (Messmer,1996) the Ullmann’s one is cited) are more suit-able.

All the above-cited algorithms are exact ones;i.e. they find the exact solution to the matchingproblem, if any. Besides them, other techniques(as those based on non-deterministic paradigms:

Christmas et al., 1995; De Jong and Spears, 1989),able to find approximate solutions to the matchingproblem have been proposed, especially in the re-cent past. We do not explicitly consider them inthis paper, since they are really so powerful toreduce the complexity (in most cases) from expo-nential to polynomial, but, as said, they do notguarantee finding an exact and optimal solution.

4. Experimental results

The algorithms we have chosen for our experi-mentation are the Ullmann’s algorithm, the algo-rithm by Schmidt and Druffel (in the followingreferred to as SD), the VF2 algorithm and Nauty.

These algorithms possess several commoncharacteristics. They are exact isomorphism algo-rithms (although two of them can also be used,with small modifications, for graph–sub-graphisomorphism). They are always able to find theoptimal solution, if it exists. They are general, inthe sense that they do not rely on any specialproperty of the graphs. They are optimized forone-to-one matching, as opposed to one-to-manymatching (as is the case for the algorithms byMessmer and Bunke: Messmer, 1996; Messmerand Bunke, 1999). Three of them also show somesimilarity at the implementation level, being basedon a depth-first search. While the fourth algorithm(Nauty) has a quite different implementationstructure, it has been included in the experimen-tation because it is widely considered one of thefastest isomorphism algorithms available today.

We used, when available, the original imple-mentations of the algorithms and run them onan Intel Celeron 500 MHz PC, equipped with128 MB of RAM. So, as regards the Nauty algo-rithm, we used the version 2.0b9 made available byB.D. McKay at the http://cs.anu.edu.au/%bdm/nauty. A publicly available implementation of theUllman’s algorithm (Ullmann, 1976) can be foundat the URL http://ftp.iam.unibe.ch/pub/Tools/GUB_toolkit.tar.Z; anyway, we have rewritten itin C++ with the aim of improving its efficiency.Thus, in this paper we refer to our implementationof the Ullman’s algorithm, available at site http://amalfi.dis.unina.it/graph together with the VF2

1072 M. De Santo et al. / Pattern Recognition Letters 24 (2003) 1067–1079

algorithm and an implementation of the SD al-gorithm (Schmidt and Druffel, 1976).

Since not all the selected algorithms can solvethe graph sub-graph isomorphism problem, inall tests only isomorphic pairs have been used.Moreover, two different evaluation criteria havebeen chosen. The first one measures the timeneeded by each algorithm for finding a solution tothe isomorphism problem, while the second oneconsiders pairs of non-isomorphic graphs andtakes into account how long each algorithm needsfor finding out that the two input graphs do notmatch.

In the following of the section plots giving thetimes as a function of the input graphs size areshown. Results on all the kinds of graphs presentin the database are reported, but the ones on the3D and 4D meshes, since in these cases the be-havior of the algorithms is very similar to thatexhibited on the 2D meshes. Note that for eachkind of graphs both matching times and the timesneeded for finding out that the input graphs do notmatch are shown.

Times are always reported in seconds in a log-arithmic scale. It is worth noting that some curvesdo not report the times obtained in correspon-dence with a given size. It happens when the al-gorithm is not able to give us a result (i.e. to find asolution to the isomorphism problem or to rec-ognize that the input graphs are not isomorphic) inless than half an hour.

4.1. Randomly connected graphs

Fig. 1 shows the behavior of the four selectedalgorithms with reference to the randomly con-nected graphs. In particular, Fig. 1a–c respectivelyrefer to values of g equal to 0.01, 0.5 and 0.1 forpairs of isomorphic graphs, while Fig. 1d–f refer tothe same values of g for pairs of non-isomorphicgraphs. As regards the matching times in case ofisomorphic pairs, VF2 and Nauty perform alwaysbetter with respect to SD and Ullmann. Moreover,Ullmann is faster than SD if the size of the graphsis smaller than 700. After this size, in fact, theUllmann’s algorithm is not able to find any iso-morphism. As regards the comparison between

VF2 and Nauty, it can be noted that VF2 obtainsbetter performance for graphs of small size andfor quite sparse graphs, while for dense graphsthe Nauty algorithm exhibits better results. Onthe contrary, in case of non-isomorphic graphs theUllmann’s algorithm is always the best one.Moreover, VF2 always performs better than Na-uty. Finally, it is worth noting that the perfor-mance of the SD algorithm only depends on thesize of the input graphs.

4.2. 2D meshes

In Fig. 2 the performance of the algorithms on2D regular meshes are shown. The behavior of allalgorithms is practically the same either for iso-morphic or for non-isomorphic pairs. In bothcases, in fact, as the size of the graphs grows up to100 nodes, i.e. for graphs of medium size, neitherNauty nor Ullmann’s are able to find solutions.For any input graph size, the VF2 algorithm is thebest one and its performance are significantlybetter in case of non-isomorphic pairs.

Fig. 3 reports the performance on irregular 2Dmeshes. In particular, in Fig. 3a–c pairs of iso-morphic 2D meshes with values of q equal re-spectively to 0.2, 0.4 and 0.6 are considered, whileFig. 3d–f refer to pairs of non-isomorphic 2Dmeshes with the same values of q.

For pairs of isomorphic graphs, the main dif-ference with respect to the case of regular meshesis that the Nauty algorithm is now always able tofind a solution, but it still performs worse thanVF2. However its behavior is better than those ofSD and Ullmann are. Moreover, the maximumgraph size for which the Ullmann’s algorithm isstill able to find a solution grows proportionallyto the irregularity of the graphs. For a value ofq equal to 0.6, Ullmann can solve the isomor-phism problem for graphs whose size is up to500 nodes. If it finds a solution the matchingtime is always better than the one obtained bySD.

Even for non-isomorphic graphs the Nauty al-gorithm is always able to give a result within thehalf an hour timeout, but in this case the differencewith respect to the time needed by VF2 is higher.Also the Ullmann’s algorithm always finds out the

M. De Santo et al. / Pattern Recognition Letters 24 (2003) 1067–1079 1073

correct result within the timeout, but it is charac-terized by an oscillatory behavior. This is due tothe fact that there are very few pairs of graphs forwhich the time needed to give a result is someorders of magnitude higher than the time neededby all the others pairs. So, it is very difficultto predict the behavior of such an algorithm asa function of the number of nodes and/or thevalues of q. This is particularly true for smallvalues of q.

4.3. Bounded valence graphs

In Fig. 4 the performance of the algorithms onbounded valence graphs are shown. In this case theconsidered values of the valence v are 3, 6 and 9, asrespectively reported in Fig. 4a–c for pairs of iso-morphic graphs and Fig. 4d–f for pairs of non-isomorphic graphs.

In case of isomorphic pairs, the Ullmann’s al-gorithm is not always able to find a solution; if it

Fig. 1. The performance of the algorithms on randomly connected graphs, as a function of the graph size and for different values of g:(a,d) g ¼ 0:01, (b,e) g ¼ 0:05, (c,f) g ¼ 0:1. Left column plots refer to pairs of isomorphic graphs, while right column plots to pairs ofnon-isomorphic graphs.

1074 M. De Santo et al. / Pattern Recognition Letters 24 (2003) 1067–1079

Fig. 2. The performance of the algorithms on regular 2D meshes as a function of the graph size in case of pairs of isomorphic graphs(a) and of non-isomorphic graphs (b).

Fig. 3. The performance of the algorithms irregular 2D meshes as a function of the graph size and for different values of q: (a,d)q ¼ 0:2, (b,e) q ¼ 0:4, (c,f) q ¼ 0:6. Left column plots refer to pairs of isomorphic graphs, while right column plots to pairs of non-isomorphic graphs.

M. De Santo et al. / Pattern Recognition Letters 24 (2003) 1067–1079 1075

happens, however, its matching time is typicallysmaller than the one of SD and always higherthan those obtained by the other two algorithms.Moreover, SD performs always worse than VF2and Nauty.

As regards the comparison between VF2 andNauty, the first algorithm performs always betterfor a value of v equal to 3, while for higher valuesof v the matching times obtained by Nauty forquite small graph are smaller than those obtainedby VF2. Anyway, independently of the consideredvalue of v, as the size of the input graphs is biggerthan 600, VF2 is the best algorithm.

On the contrary, in case of non-isomorphicgraphs VF2 always performs better than Nauty.For high values of v Ullmann again exhibits anoscillatory behavior, but its performance is prac-tically always worse than that of VF2. Even in thiscase the performance of the SD algorithm onlydepends on the size of the input graphs.

4.4. Irregular bounded valence graphs

Finally, in Fig. 5 the performance of the algo-rithms on irregular bounded valence graphs isshown. Also in this case the considered values of

Fig. 4. The performance of the algorithms on bounded valence graphs as a function of the graph size and for different values of thevalence m: (a,d) v ¼ 3, (b,e) v ¼ 6, (c,f) v ¼ 9. Left column plots refer to pairs of isomorphic graphs, while right column plots to pairs ofnon-isomorphic graphs.

1076 M. De Santo et al. / Pattern Recognition Letters 24 (2003) 1067–1079

the valence v are 3, 6 and 9, as respectivelyreported in Fig. 5a–c for pairs of isomorphicgraphs and Fig. 5d–f for pairs of non-isomorphicgraphs.

In case of isomorphic graphs, VF2 is the bestalgorithm for low values of v, while its perfor-mance is practically the same of that exhibited byNauty for higher values of v. However, indepen-dently on the values of v, VF2 has the best per-formance as the size of the input graphs grows. Onthe contrary, the Ullmann’s algorithm is never able

to find a solution if the size of the input graphsbecomes higher than 700.

As regards non-isomorphic graphs, both SDand Nauty exhibit the same behavior of the iso-morphic graph case, while VF2 gives rise to bet-ter results, performing always significantly betterthan Nauty. For all values of v, however, thebest algorithm is Ullmann, which is able to findout the correct result in a time that is about twoorders of magnitude lower than the one neededby VF2.

Fig. 5. The performance of the algorithms on irregular bounded valence graphs as a function of the graph size and for different values ofthe valence v: (a,d) v ¼ 3, (b,e) v ¼ 6, (c,f) v ¼ 9. Left column plots refer to pairs of isomorphic graphs, while right column plots to pairsof non-isomorphic graphs.

M. De Santo et al. / Pattern Recognition Letters 24 (2003) 1067–1079 1077

5. Discussion and conclusions

In this paper we have presented a benchmark-ing activity for assessing the performance of somewidely used exact graph matching algorithms. Thecomparison has been carried out on a large dat-abase of artificially generated graphs, which hasbeen built and made publicly available to providea common reference data set for further bench-marking activities.

As it could be expected, it does not exist analgorithm that is definitively better than all theothers. In particular, for randomly connectedgraphs, the Nauty algorithm is the best one if thegraphs are quite dense and/or of quite large size.For smaller and quite sparse graphs, on the con-trary, VF2 performs better.

On more regular graphs, i.e. on 2D meshes,VF2 is definitely the best algorithm: in this casethe Nauty algorithm is often not even able to finda solution for graphs bigger than few dozens ofnodes. In case of bounded valence graph, if thevalence is small, VF2 is always the best algorithm,while for bigger values of the valence the Nautyalgorithm is more convenient if the size of thegraphs is small.

If the two graphs being compared are not iso-morphic, Ullmann’s algorithm shows a quite sin-gular behavior: for most of the cases, it is able tostop the search very early, requiring a time that isconsiderably lower than that of the other algo-rithms. On the other hand, in the remaining fewcases, the time needed to stop the search is ex-tremely longer (often by several orders of magni-tude), making the algorithm unpractical. We havenot yet been able to characterize the graphs whichtrigger such a behavior, nor to provide a reason-able estimate of their occurrence probability. Itcan be noted that for these cases, the algorithmshowing the best performance is definitely VF2.Finally, it is also worth noting that SD and VF2are the only algorithms that have always been ableto find a solution to the isomorphism problem inour tests, independently of the size and the kind ofthe graphs to be matched.

Future steps of this benchmarking activity willinvolve the comparison of other algorithms, andthe extension to other problems. We are also

planning to extend the database with other graphcategories and to add an indexing facility (basedon several graph parameters), for making itsuse more easy and convenient to other research-ers.

References

Abdulrahim, M., Misra, M., 1998. A graph isomorphismalgorithm for object recognition. Pattern Anal. Appl. 1 (3),189–201.

Bunke, H., Messmer, B.T., 1995. Efficient attributed graphmatching and its application to image analysis. In: Braccini,C., De Floriani, L., Vernazza, G. (Eds.), Image Analysisand Processing. Springer, Berlin, pp. 45–55.

Bunke, H., Vento, M. 1999. Benchmarking of graph matchingalgorithms. In: Proceedings of the 2nd Workshop onGraph-based Representations. Haindorf, pp. 109–114.

Christmas, W.J., Kittler, J., Petrou, M., 1995. Structuralmatching in computer vision using probabilistic relaxation.IEEE Trans. Pattern Anal. Mach. Intell. 17 (8), 749–764.

Cordella, L.P., Foggia, P., Sansone, C., Vento, M., 1998.Subgraph Transformations for the inexact matching ofattributed relational graphs. Comput. Suppl. 12, 43–52.

Cordella, L.P., Foggia, P., Sansone, C., Vento, M., 1999.Performance evaluation of the VF graph matching algo-rithm. In: Proceedings of the 10th ICIAP, IEEE ComputerSociety Press, pp. 1172–1177.

Cordella, L.P., Foggia, P., Sansone, C., Vento, M., 2001. Animproved algorithm for matching large graphs. In: Pro-ceedings of the 3rd IAPR-TC-15 International Workshopon Graph-based Representations. Italy, pp. 149–159.

De Jong, K.A., Spears, W.M., 1989. Using genetic algorithmsto solve NP-complete problems. In: Schaffer, J.D. (Ed.),Genetic Algorithms. Morgan Kaufmann, Los Altos, CA,pp. 124–132.

Hopcroft, J., Wong, J., 1974. Linear time algorithm forisomorphism of planar graphs. In: Proceedings of the 6thAnnual ACM Symposium on Theory of Computing, pp.172–184.

Jiang, X., Bunke, H., 1996. Including geometry in graphrepresentations: a quadratic time isomorphism algorithmand its applications. In: Perner, P., Wang, P., Rosenfeld, A.(Eds.), Lecture Notes in Computer Science, 1121, pp. 110–119.

Luks, E.M., 1982. Isomorphism of graphs of bounded valencecan be tested in polynomial time. J. Comput. Syst. Sci., 42–65.

McKay, B.D., 1981. Practical graph isomorphism. CongressusNumerantium 30, 45–87.

Messmer, B.T., 1996. Efficient Graph Matching Algorithms forPreprocessed Model Graphs, Ph.D. Thesis, Institute ofComputer Science and Applied Mathematics, University ofBern.

1078 M. De Santo et al. / Pattern Recognition Letters 24 (2003) 1067–1079

Messmer, B.T., Bunke, H., 1999. A decision tree approach tograph and subgraph isomorphism detection. Pattern Re-cogn. 32, 1979–1998.

Miyazaki, T., 1997. The complexity of McKay’s canonicallabeling algorithm. In: Finkelstein, L., Kantor, W.M.(Eds.), Groups and Computation, II. American Mathemat-ical Society, Providence, RI, pp. 239–256.

Rouvray, D.H., Balaban, A.T., 1979. Chemical applications ofgraph theory. In: Applications of Graph Theory. AcademicPress, New York, pp. 177–221.

Schmidt, D.C., Druffel, L.E., 1976. A fast backtrackingalgorithm to test directed graphs for isomorphism usingdistance matrices. J. ACM 23, 433–445.

Shearer, K., Bunke, H., Venkatesh, S., Kieronska, D., 1998.Graph matching for video indexing. Comput. Suppl. 12, 53–62.

Ullmann, J.R., 1976. An algorithm for subgraph isomorphism.J. Assoc. Comput. Mach. 23, 31–42.

Wong, E.K., 1992. Model matching in robot vision by sub-graph isomorphism. Pattern Recogn. 25 (3), 287–304.

M. De Santo et al. / Pattern Recognition Letters 24 (2003) 1067–1079 1079