Spaces:

Bani57
/

website

Sleeping

App Files Files Community

Andrej Janchevski commited on Feb 23

Commit

5e0b3dd

0 Parent(s):

Initial commit - add source files from research repos

Browse files

This view is limited to 50 files because it contains too many changes. See raw diff

Files changed (50) hide show

.gitattributes +13 -0
.gitignore +3 -0
src/research/COINs-KGGeneration/README.md +60 -0
src/research/COINs-KGGeneration/data/FB15k-237/MSR-LA_Data_Full Rights_FB15K-237 Knowledge Base Completion Dataset (2650).docx +0 -0
src/research/COINs-KGGeneration/data/FB15k-237/README.txt +41 -0
src/research/COINs-KGGeneration/data/FB15k-237/test.txt +0 -0
src/research/COINs-KGGeneration/data/FB15k-237/train.txt +3 -0
src/research/COINs-KGGeneration/data/FB15k-237/valid.txt +0 -0
src/research/COINs-KGGeneration/data/NELL-995/README.md +15 -0
src/research/COINs-KGGeneration/data/NELL-995/entity2id.txt +0 -0
src/research/COINs-KGGeneration/data/NELL-995/entity2vec.bern +3 -0
src/research/COINs-KGGeneration/data/NELL-995/kb_env_rl.txt +3 -0
src/research/COINs-KGGeneration/data/NELL-995/ranked_relations.csv +426 -0
src/research/COINs-KGGeneration/data/NELL-995/raw.kb +3 -0
src/research/COINs-KGGeneration/data/NELL-995/relation2id.txt +400 -0
src/research/COINs-KGGeneration/data/NELL-995/relation2vec.bern +0 -0
src/research/COINs-KGGeneration/data/NELL-995/test.txt +0 -0
src/research/COINs-KGGeneration/data/NELL-995/train.txt +3 -0
src/research/COINs-KGGeneration/data/NELL-995/valid.txt +543 -0
src/research/COINs-KGGeneration/data/WN18RR/test.txt +0 -0
src/research/COINs-KGGeneration/data/WN18RR/train.txt +0 -0
src/research/COINs-KGGeneration/data/WN18RR/valid.txt +0 -0
src/research/COINs-KGGeneration/graph_analysis/KGist/__init__.py +0 -0
src/research/COINs-KGGeneration/graph_analysis/KGist/anomaly_detector.py +83 -0
src/research/COINs-KGGeneration/graph_analysis/KGist/correct_assertion.py +72 -0
src/research/COINs-KGGeneration/graph_analysis/KGist/evaluator.py +271 -0
src/research/COINs-KGGeneration/graph_analysis/KGist/graph.py +188 -0
src/research/COINs-KGGeneration/graph_analysis/KGist/main.py +86 -0
src/research/COINs-KGGeneration/graph_analysis/KGist/model.py +388 -0
src/research/COINs-KGGeneration/graph_analysis/KGist/rule.py +263 -0
src/research/COINs-KGGeneration/graph_analysis/KGist/searcher.py +257 -0
src/research/COINs-KGGeneration/graph_analysis/METIS/libmetis.so +3 -0
src/research/COINs-KGGeneration/graph_analysis/METIS/metis.dll +3 -0
src/research/COINs-KGGeneration/graph_analysis/__init__.py +0 -0
src/research/COINs-KGGeneration/graph_analysis/main.py +566 -0
src/research/COINs-KGGeneration/graph_analysis/metrics.py +950 -0
src/research/COINs-KGGeneration/graph_analysis/stats.py +158 -0
src/research/COINs-KGGeneration/graph_completion/README.md +89 -0
src/research/COINs-KGGeneration/graph_completion/__init__.py +0 -0
src/research/COINs-KGGeneration/graph_completion/configs/freebase.yml +29 -0
src/research/COINs-KGGeneration/graph_completion/configs/freebase_betae.yml +44 -0
src/research/COINs-KGGeneration/graph_completion/configs/freebase_complex.yml +31 -0
src/research/COINs-KGGeneration/graph_completion/configs/freebase_distmult.yml +31 -0
src/research/COINs-KGGeneration/graph_completion/configs/freebase_gnn.yml +34 -0
src/research/COINs-KGGeneration/graph_completion/configs/freebase_gqe.yml +39 -0
src/research/COINs-KGGeneration/graph_completion/configs/freebase_q2b.yml +40 -0
src/research/COINs-KGGeneration/graph_completion/configs/freebase_rotate.yml +29 -0
src/research/COINs-KGGeneration/graph_completion/configs/hep.yml +28 -0
src/research/COINs-KGGeneration/graph_completion/configs/nell.yml +30 -0
src/research/COINs-KGGeneration/graph_completion/configs/nell_betae.yml +44 -0

.gitattributes ADDED Viewed

	@@ -0,0 +1,13 @@

+/src/research/COINs-KGGeneration/data/FB15k-237/text_emnlp.txt filter=lfs diff=lfs merge=lfs -text
+/src/research/COINs-KGGeneration/data/FB15k-237/train.txt filter=lfs diff=lfs merge=lfs -text
+/src/research/COINs-KGGeneration/data/NELL-995/entity2vec.bern filter=lfs diff=lfs merge=lfs -text
+/src/research/COINs-KGGeneration/data/NELL-995/kb_env_rl.txt filter=lfs diff=lfs merge=lfs -text
+/src/research/COINs-KGGeneration/data/NELL-995/raw.kb filter=lfs diff=lfs merge=lfs -text
+/src/research/COINs-KGGeneration/data/NELL-995/train.txt filter=lfs diff=lfs merge=lfs -text
+/src/research/MultiProxAn/checkpoints/qm9.ckpt filter=lfs diff=lfs merge=lfs -text
+/src/research/MultiProxAn/checkpoints/qm9_c.ckpt filter=lfs diff=lfs merge=lfs -text
+*.pdf filter=lfs diff=lfs merge=lfs -text
+*.jpg filter=lfs diff=lfs merge=lfs -text
+*.so filter=lfs diff=lfs merge=lfs -text
+*.dll filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text

.gitignore ADDED Viewed

	@@ -0,0 +1,3 @@

+.idea/
+__pycache__
+.ipynb_checkpoints

src/research/COINs-KGGeneration/README.md ADDED Viewed

	@@ -0,0 +1,60 @@

+# COINs: Knowledge Graph Inference with Model-based Acceleration
+LoG 2024 conference submission
+---------
+## Abstract
+We present a new method, called COINs for **CO**mmunity **IN**formed graph embedding**s**, to enhance the efficiency of
+knowledge graph models for link prediction and conjunctive query answering. COINs uses a community-detection-based graph
+data augmentation and a two-step prediction pipeline: we first achieve node localization through community prediction,
+and subsequently, we further localize within the predicted community. We establish theoretical criteria to evaluate our
+method in our specific context and establish a direct expression of the reduction in time complexity. We empirically
+demonstrate an important scalability-performance trade-off where for the median evaluation sample we preserve 97.18% of
+the baseline accuracy in single-hop query answering, for only 7.52% of the original computational cost on a
+single-CPU-GPU machine.
+## Instructions
+### Obtaining the code
+Clone this repository by running:
+`git clone https://github.com/ResearchWeasel/coins-log-2024.git`
+### Dependencies
+The implementation requires version `3.6.13` of the Python programming language.
+To install it and the dependent Python packages, we recommend having [Anaconda](https://www.anaconda.com/download), then
+running the following commands from the main directory of the repository files:
+1. `conda create --name coins python=3.6.13`
+2. `conda activate coins`
+3. `pip install -r requirements.txt`
+4. `export PYTHONPATH='.'`
+### Reproducing results
+To regenerate the tables and figures provided in the paper, run the Jupyter notebook `graph_completion/Plotting.ipynb`.
+Figure PDFs will be saved to the `graph_completion/results` directory.
+To run end-to-end a COINs training and evaluation experiment from the paper, run the following command from the main
+directory of the repository files:
+- GPU:
+  `CUDA_VISIBLE_DEVICES=<GPU ID> python graph_completion/main.py -cf='graph_completion/configs/<CONFIG FILENAME>.yml'`
+- CPU (after setting the `device` config parameter to `cpu` in the YAML file):
+  `python graph_completion/main.py -cf='graph_completion/configs/<CONFIG FILENAME>.yml'`
+The experiment results will be saved to a directory in `graph_completion/results/<DATASET>/runs`.
+----------
+## Authors
+- Anonymous

src/research/COINs-KGGeneration/data/FB15k-237/MSR-LA_Data_Full Rights_FB15K-237 Knowledge Base Completion Dataset (2650).docx ADDED Viewed

Binary file (38.5 kB). View file

src/research/COINs-KGGeneration/data/FB15k-237/README.txt ADDED Viewed

	@@ -0,0 +1,41 @@

+FB15K-237 Knowledge Base Completion Dataset
+This dataset contains knowledge base relation triples and textual mentions of Freebase entity pairs, as used in the work published in [1] and [2].
+The knowledge base triples are a subset of the FB15K set [3], originally derived from Freebase. The textual mentions are derived from 200 million sentences from the ClueWeb12 [5] corpus coupled with Freebase entity mention annotations [4].
+FILE FORMAT DETAILS
+The files train.txt, valid.txt, and test.text contain the training, development, and test set knowledge base triples used in both [1] and [2].
+The file text_cvsc.txt contains the textual triples used in [2] and the file text_emnlp.txt contains the textual triples used in [1].
+The knowledge base triples contain lines like this:
+/m/0grwj	/people/person/profession	/m/05sxg2
+The format is:
+mid1	relation	mid2
+The separator is a tab character; the mids are Freebase ids of entities, and the relation is a single or a two-hop relation from Freebase, where an intermediate complex value type entity has been collapsed out.
+The textual mentions files have lines like this:
+/m/02qkt        [XXX]:<-nn>:fact:<-pobj>:in:<-prep>:game:<-nsubj>:'s:<ccomp>:pivot:<nsubj>:[YYY]    /m/05sb1    3
+This indicates the mids of two Freebase entities, together with a fully lexicalized dependency path between the entities. The last element in the tuple is the number of occurrences of the specified entity pair with the given dependency path in sentences from ClueWeb12.
+The dependency paths are specified as sequences of words (like the word "fact" above) and labeled dependency links (like <nsubj> above). The direction of traversal of a dependency arc is indicated by whether there is a - sign in front of the arc label "e.g." <-nsubj> vs <nsubj>.
+REFERENCES
+[1] Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. Representing text for joint embedding of text and knowledge bases.  In Proceedings of EMNLP 2015.
+[2] Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and Their Compositionality 2015.
+[3] Antoine Bordes, Nicolas Usunier, Alberto Garcia Duran, Jason Weston, and Oksana Yakhnenko.  Translating embeddings for modeling multirelational data. In Advances in Neural Information Processing Systems (NIPS) 2013.
+[4] Evgeniy Gabrilovich, Michael Ringgaard, and Amarnag Subramanya. FACC1: Freebase annotation of ClueWeb corpora, Version 1 (release date 2013-06-26, format version 1, correction level 0). http://lemurproject.org/clueweb12/FACC1/
+[5] http://lemurproject.org/clueweb12/
+CONTACT
+Please contact Kristina Toutanova kristout@microsoft.com if you have questions about the dataset.

src/research/COINs-KGGeneration/data/FB15k-237/test.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

src/research/COINs-KGGeneration/data/FB15k-237/train.txt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:61099230e4439f90885ca9767739e31e8e32f54736fa1c35952b27997bc7c08a
+size 21005177

src/research/COINs-KGGeneration/data/FB15k-237/valid.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

src/research/COINs-KGGeneration/data/NELL-995/README.md ADDED Viewed

	@@ -0,0 +1,15 @@

+# NELL-995 Dataset for Knowledge Graph Reasoning
+We build this dataset from the 995th iteration of CMU's NELL System
+## Explanation about some files in the directory
+* kb_env_rl.txt: knowledge graph used to build the RL environment
+* raw.kb: the raw KB data
+* entity2vec.bern/relation2vec.bern: embedding used to represent RL states
+* tasks/: this directory contains data for each reasoning task
+    * tasks/${relation}/*.vec: trained TransD(H) Embeddings
+    * tasks/${relation}/*.bern: trained TransR Embedding trained
+    * tasks/${relation}/*.unif: trained TransE Embeddings
+    * tasks/${relation}/transX: triples used to train the KB embeddings
+    * tasks/${relation}/train.pairs: train triples in the PRA format
+    * tasks/${relation}/test.pairs: test triples in the PRA format
+    * tasks/${relation}/path_to_use.txt: reasoning paths found the RL agent

src/research/COINs-KGGeneration/data/NELL-995/entity2id.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

src/research/COINs-KGGeneration/data/NELL-995/entity2vec.bern ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e157b44e75d04d339d4a7a5b4732e85096be090b225f84ca3297374cbfe49617
+size 71753696

src/research/COINs-KGGeneration/data/NELL-995/kb_env_rl.txt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:48b080737899828cc98f269feb4d63418f73cfe2dac3b60fe0e5015b34bd7ecf
+size 26540694

src/research/COINs-KGGeneration/data/NELL-995/ranked_relations.csv ADDED Viewed

	@@ -0,0 +1,426 @@

+generalizations	2202099
+concept:haswikipediaurl	303608
+concept:atdate	15175
+concept:latitudelongitude	6332
+concept:proxyfor	5362
+concept:agentcreated	4668
+concept:atlocation	4272
+concept:subpartof	4073
+concept:teamplaysagainstteam	3411
+concept:mutualproxyfor	3320
+concept:itemfoundinroom	2472
+concept:professionistypeofprofession	2361
+concept:locationlocatedwithinlocation	2273
+concept:clothingtogowithclothing	2242
+concept:subpartoforganization	2087
+concept:bookwriter	2083
+concept:furniturefoundinroom	2037
+concept:athleteplayssport	2008
+concept:agentcollaborateswithagent	1928
+concept:chemicalistypeofchemical	1812
+concept:agentcontrols	1745
+concept:worksfor	1707
+concept:organizationhiredperson	1617
+concept:animalistypeofanimal	1582
+concept:agentactsinlocation	1493
+concept:hasofficeincity	1486
+concept:animalpreyson	1429
+concept:bodypartcontainsbodypart	1393
+concept:teamplaysinleague	1386
+concept:cityliesonriver	1369
+concept:agentparticipatedinevent	1356
+concept:agriculturalproductcookedwithagriculturalproduct	1302
+concept:academicprogramatuniversity	1294
+concept:athleteplaysforteam	1288
+concept:athleteplaysinleague	1269
+concept:companyeconomicsector	1226
+concept:agentinvolvedwithitem	1197
+concept:competeswith	1194
+concept:journalistwritesforpublication	1185
+concept:agentcompeteswithagent	1148
+concept:agentcontributedtocreativework	1030
+concept:televisionstationaffiliatedwith	1001
+concept:weaponssuchasweapons	993
+concept:personbelongstoorganization	969
+concept:plantincludeplant	941
+concept:politicianusendorsedbypoliticianus	913
+concept:persondiedincountry	907
+concept:drughassideeffect	886
+concept:organizationheadquarteredincity	829
+concept:touristattractionsuchastouristattraction	826
+concept:athleteflyouttosportsteamposition	802
+concept:countrylocatedingeopoliticallocation	794
+concept:musicartistgenre	774
+concept:personleadsorganization	771
+concept:sportsgameteam	730
+concept:agentbelongstoorganization	727
+concept:musicianplaysinstrument	703
+concept:organizationterminatedperson	694
+concept:citylocatedinstate	690
+concept:animaldevelopdisease	669
+concept:athletehomestadium	667
+concept:synonymfor	655
+concept:animaleatfood	650
+concept:personborninlocation	642
+concept:hasspouse	638
+concept:persongraduatedfromuniversity	621
+concept:beveragemadefrombeverage	614
+concept:statelocatedingeopoliticallocation	612
+concept:bakedgoodservedwithbeverage	601
+concept:personhasjobposition	595
+concept:sportusesequipment	564
+concept:visualartistartform	562
+concept:drugworkedonbyagent	560
+concept:personbornincity	554
+concept:buildinglocatedincity	550
+concept:citylocatedincountry	531
+concept:politicianholdsoffice	529
+concept:hotelincity	528
+concept:musicianinmusicartist	527
+concept:countryalsoknownas	501
+concept:academicfieldsuchasacademicfield	497
+concept:politicianrepresentslocation	482
+concept:actorstarredinmovie	470
+concept:persongraduatedschool	442
+concept:animalsuchasinsect	433
+concept:agriculturalproductincludingagriculturalproduct	429
+concept:bankbankincountry	428
+concept:bacteriaisthecausativeagentofphysiologicalcondition	427
+concept:persondiedatage	416
+concept:coachwontrophy	414
+concept:conceptprerequisiteof	404
+concept:organizationheadquarteredincountry	402
+concept:producesproduct	400
+concept:athleteledsportsteam	400
+concept:topmemberoforganization	389
+concept:stadiumlocatedincity	384
+concept:attractionofcity	384
+concept:headquarteredin	383
+concept:animalthatfeedoninsect	383
+concept:arthropodcalledarthropod	378
+concept:hasofficeincountry	378
+concept:teamplayssport	371
+concept:languageofuniversity	369
+concept:languageofcountry	367
+concept:teamplaysincity	363
+concept:airportincity	362
+concept:weaponmadeincountry	361
+concept:automakerproducesmodel	358
+concept:personhascitizenship	358
+concept:emotionassociatedwithdisease	352
+concept:teamwontrophy	352
+concept:fooddecreasestheriskofdisease	351
+concept:acquired	348
+concept:parentofperson	347
+concept:arthropodandotherarthropod	343
+concept:drugpossiblytreatsphysiologicalcondition	341
+concept:organizationacronymhasname	341
+concept:animalsuchasfish	339
+concept:politicianusholdsoffice	330
+concept:politicianusmemberofpoliticalgroup	321
+concept:mammalsuchasmammal	320
+concept:agriculturalproductcomingfromvertebrate	315
+concept:riveremptiesintoriver	312
+concept:leaguestadiums	311
+concept:specializationof	301
+concept:fishservedwithfood	289
+concept:personhasresidenceingeopoliticallocation	288
+concept:animalsuchasinvertebrate	287
+concept:televisionstationincity	276
+concept:directordirectedmovie	272
+concept:ceoof	271
+concept:istallerthan	260
+concept:parkincity	259
+concept:personchargedwithcrime	255
+concept:sportfansincountry	250
+concept:personalsoknownas	245
+concept:arterycalledartery	242
+concept:athleteinjuredhisbodypart	242
+concept:sportschoolincountry	239
+concept:agriculturalproductcutintogeometricshape	234
+concept:animaleatvegetable	230
+concept:personmovedtostateorprovince	230
+concept:thinghascolor	230
+concept:citylocatedingeopoliticallocation	229
+concept:automobilemakerdealersincity	225
+concept:radiostationincity	214
+concept:sportsgamewinner	213
+concept:invertebratefeedonfood	213
+concept:thinghasshape	209
+concept:professionusestool	208
+concept:agriculturalproductcontainchemical	206
+concept:organizationdissolvedatdate	205
+concept:haswife	203
+concept:hobbiessuchashobbies	202
+concept:buildingfeaturemadefrombuildingmaterial	200
+concept:athleteplayssportsteamposition	194
+concept:foodcancausedisease	192
+concept:organizationalsoknownas	184
+concept:sportusesstadium	183
+concept:agriculturalproducttoattractinsect	171
+concept:citycapitalofcountry	170
+concept:sporthassportsteamposition	167
+concept:teamalsoknownas	166
+concept:statelocatedincountry	160
+concept:stateorprovinceisborderedbystateorprovince	159
+concept:musicgenressuchasmusicgenres	158
+concept:objectpartofobject	155
+concept:organizationheadquarteredinstateorprovince	148
+concept:fatherofperson	147
+concept:agriculturalproductcamefromcountry	147
+concept:newspaperincity	144
+concept:lakeinstate	144
+concept:objectfoundinscene	143
+concept:teamhomestadium	143
+concept:agriculturalproductgrowninlandscapefeatures	140
+concept:companyalsoknownas	137
+concept:countrycurrency	134
+concept:coachesteam	133
+concept:academicfieldconcernssubject	132
+concept:cityalsoknownas	129
+concept:ismultipleof	128
+concept:athletebeatathlete	127
+concept:athletewinsawardtrophytournament	125
+concept:hashusband	124
+concept:personleadsgeopoliticalorganization	123
+concept:statehascapital	118
+concept:mountaininstate	116
+concept:bankboughtbank	113
+concept:automobilemakerdealersincountry	111
+concept:plantrepresentemotion	111
+concept:transportationincity	109
+concept:awardtrophytournamentisthechampionshipgameofthenationalsport	105
+concept:coachesinleague	103
+concept:sportsgamesport	94
+concept:hassibling	92
+concept:motherofperson	92
+concept:clothingmadefromplant	92
+concept:museumincity	91
+concept:plantgrowinginplant	90
+concept:beveragecontainsprotein	86
+concept:automobilemakercardealersinstateorprovince	85
+concept:agriculturalproductgrowinginstateorprovince	84
+concept:jobpositionusesacademicfield	83
+concept:hasfamilymember	82
+concept:languageschoolincity	81
+concept:animalssuchasmammals	81
+concept:wineryproduceswine	81
+concept:hospitalincity	78
+concept:personhasresidenceincity	72
+concept:personattendsschool	70
+concept:collidedwith	66
+concept:vegetableproductioninstateorprovince	65
+concept:hasbrother	63
+concept:musclelocatedinbodypart	63
+concept:aquariumincity	62
+concept:teammate	59
+concept:objectlookslikeobject	57
+concept:colorassociatedwithvisualizableattribute	53
+concept:writerwasbornincity	52
+concept:itemexistsatlocation	52
+concept:locationofbombing	50
+concept:productinstanceof	49
+concept:countriessuchascountries	49
+concept:personhasage	49
+concept:diseasecausesphysiologicalcondition	47
+concept:athletessuchasathletes	46
+concept:streetincity	45
+concept:attractionmadeofbuildingmaterial	45
+concept:personleadscity	41
+concept:victiminbombing	41
+concept:personhasresidenceinstateorprovince	40
+concept:restaurantincity	39
+concept:commentatorofbombing	39
+concept:locationofprotest	39
+concept:vegetablecanbeservedwithgrain	39
+concept:productbeinglauched	37
+concept:personwrittenaboutinpublication	36
+concept:attractionbefallincity	35
+concept:arthropodthatfeedoninsect	35
+concept:invertebratehascolor	32
+concept:professionssuchasprofessions	32
+concept:companyproducesvideogamesystem	31
+concept:agentcreatedorganization	31
+concept:musicfestivalssuchasmusicfestivals	30
+concept:organizationhasofficialwebsite	30
+concept:attackerinbombing	30
+concept:televisionshowssuchastelevisionshows	30
+concept:mountainatgeopoliticallocation	30
+concept:personhasresidenceincountry	30
+concept:animalsuchasmollusk	29
+concept:conferencessuchasconferences	29
+concept:toolssuchastools	29
+concept:locationofearthquake	29
+concept:locationofroadaccident	28
+concept:malemovedtostateorprovince	28
+concept:organizationcreatedatdate	28
+concept:personhasethnicity	27
+concept:coachworksincountry	27
+concept:amphibiansuchasamphibian	27
+concept:productproducedincountry	27
+concept:criminalssuchascriminals	27
+concept:personbirthdate	26
+concept:bakegoodistypicalincountry	26
+concept:personhasreligion	26
+concept:arteryarisesfromartery	26
+concept:athletecoach	26
+concept:persondiedinstateorprovince	25
+concept:diseasemaybecausedbydrug	25
+concept:actorofprotest	25
+concept:marriedinyear	25
+concept:comediancanbethesameasfemale	24
+concept:arthropodlookslikeinsect	24
+concept:personinacademicfield	24
+concept:invertebrateturnintoinsect	24
+concept:mammalinducesemotion	23
+concept:launchingproductcompany	23
+concept:victiminroadaccident	23
+concept:epicenterofearthquake	22
+concept:personborninstateorprovince	22
+concept:personbornincountry	21
+concept:arthropodbesometimesconfusewithinsect	21
+concept:countryisthehomeofsportsteam	21
+concept:meatcomesfromanimal	21
+concept:musicsongisatrackofmusicalbum	20
+concept:stadiumoreventvenuedisclosescompany	20
+concept:bankchiefexecutiveceo	20
+concept:zooincity	20
+concept:vegetableisproducedatcountry	19
+concept:launchingproducttype	19
+concept:visualartistartmovement	19
+concept:damagedinbombing	18
+concept:professionactsatthesameareaofpoliticissue	18
+concept:vehicleistypeofvehicle	18
+concept:athletealsoknownas	18
+concept:languageofcity	18
+concept:agentstudiesphysiologicalcondition	18
+concept:actorworkedwithdirector	17
+concept:eventdate	17
+concept:arachnidiseatenbybird	17
+concept:shoppingmallincity	17
+concept:clothmadeofplant	17
+concept:agentholdssharesincompany	17
+concept:persondiedincity	17
+concept:magnitudeofearthquaque	17
+concept:celebritysuchascelebrity	16
+concept:injuredinbombing	16
+concept:actorsuchasactor	16
+concept:coachcanspeaklanguage	16
+concept:mammaleatsinsect	16
+concept:numberofinjuredinbombing	16
+concept:mlareaexpert	15
+concept:stateorprovinceattainstrainstation	15
+concept:automobilemakerchiefexecutiveceo	15
+concept:wineryatfarm	15
+concept:hassister	15
+concept:architectssuchasarchitects	15
+concept:buildingispartofuniversity	15
+concept:musicalartisthadapetanimal	15
+concept:astronautwasonamissionwithastronaut	15
+concept:arthropodsuchascrustacean	15
+concept:athletepublishesinblog	15
+concept:numberofkilledinbombing	14
+concept:languagespokenatisland	14
+concept:coachstudiedatuniversity	14
+concept:officebuildingroominbuilding	14
+concept:buildingfeaturebuiltabovelake	14
+concept:productlauchpredecessor	14
+concept:productlauchvenue	14
+concept:politicianuswenttoschool	14
+concept:beveragecansoftendisease	14
+concept:arteriessuchasarteries	14
+concept:politicsbillconcernsissue	14
+concept:agenthaswebsite	13
+concept:sourceofearthquake	13
+concept:newpaperhaswebsite	13
+concept:grainusedtomakecandy	13
+concept:weaponusedinprotest	13
+concept:placeofworshippracticesreligion	13
+concept:productlauchcompetingcompany	13
+concept:magazineisleadbyceo	13
+concept:meetingeventtitlehasspeakingperson	12
+concept:politicsbillsponsoredbypoliticianus	12
+concept:targetinbombing	12
+concept:meetingeventtitleatdate	12
+concept:meetingeventtitlehasmeetingeventtype	12
+concept:prodctlaunchingproductcompetitor	12
+concept:farmlocatedinstate	12
+concept:meetingeventtitleinbuilding	12
+concept:guiltyofroadaccident	12
+concept:meetingeventtitleinofficebuildingroom	12
+concept:mlauthorofsoftware	12
+concept:farmproducesagriculturalproduct	12
+concept:academicfieldhassubfield	12
+concept:zipcodeinlocation	11
+concept:academicfieldusedbyeconomicsector	11
+concept:politicsgroupconcernsissue	11
+concept:awardtrophytournamentwasawardedonstadium	11
+concept:softwareisprogrammedinprogramminglanguage	11
+concept:astronautssuchasaustronauts	11
+concept:sportsgamescore	11
+concept:drugexpectedreleasedate	11
+concept:airportsuchasairport	11
+concept:arthropodcanbeveryirritatingtomammal	11
+concept:commentatorofearthquake	11
+concept:hotelwasbuiltatfarm	11
+concept:bombinghaswitness	11
+concept:hotelbuiltatpark	11
+concept:explosivesinbombing	10
+concept:aquariumssuchasaquariums	10
+concept:sportsgamedate	10
+concept:videogamessuchasvideogames	10
+concept:restaurantlocatedinskyscraper	10
+concept:persondeathdate	10
+concept:buildingbelongstoorganization	10
+concept:moviessuchasmovies	10
+concept:booksuchasbook	10
+concept:religionssuchasreligions	10
+concept:victiminearthquake	10
+concept:archaeasuchasarchaea	10
+concept:reasonofcollision	10
+concept:timeofearthquake	9
+concept:arachnidssuchasarachnids	9
+concept:commentatorofprotest	9
+concept:birdmayconsumeprotein	9
+concept:victiminprotest	9
+concept:athleteplayedforschool	9
+concept:bacteriacancauseillnessfromfood	9
+concept:athletefracturedbone	9
+concept:numberofinjuredinroadaccident	9
+concept:personineconomicsector	9
+concept:timeofbombing	9
+concept:damagedinearthquake	8
+concept:insecteatsgrain	8
+concept:athletecanbethesameasactor	8
+concept:injuredinprotest	8
+concept:televisionshowisbasedonmovie	8
+concept:criminalarrestedincountry	8
+concept:numberofcasualtiesinroadaccident	7
+concept:timeofroadaccident	7
+concept:sportsgameloserscore	7
+concept:sportsgamewinnerscore	7
+concept:injuredinearthquake	7
+concept:automobileenginesuchasautomobileengine	7
+concept:injuredinroadaccident	7
+concept:sportsgameloser	6
+concept:islandhasskiarea	6
+concept:universityledmusicfestival	6
+concept:commentatorofroadaccident	6
+concept:agentowns	5
+concept:numberofinjuredinprotest	5
+concept:sportnamehasoriginlanguage	5
+concept:publicationpublishedinmagazine	5
+concept:coachalsoknownas	5
+concept:securityinbombing	5
+concept:numberofcasualtiesinearthquake	5
+concept:numberofinjuredinearthquake	5
+concept:stadiumalsoknownas	5
+concept:numberofarrestedinbombing	5
+concept:prodctlaunchingproductprice	3
+concept:damagedinprotest	3
+concept:celebritycanbethesameasceo	3
+concept:timeofprotest	2
+concept:persondiedinlocation	2
+concept:scientistcanbethesameaswriter	1
+concept:farmlocatedincountry	1
+concept:eventatlocation	1

src/research/COINs-KGGeneration/data/NELL-995/raw.kb ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6ca558a996843f8fff7185e56e32fe6ede5c266244c8ff6e77070064e300272b
+size 12961921

src/research/COINs-KGGeneration/data/NELL-995/relation2id.txt ADDED Viewed

	@@ -0,0 +1,400 @@

+concept:agriculturalproductcookedwithagriculturalproduct	0
+concept:agriculturalproductgrowninlandscapefeatures	1
+concept:sportfansincountry	2
+concept:beveragemadefrombeverage_inv	3
+concept:emotionassociatedwithdisease_inv	4
+concept:personbelongstoorganization_inv	5
+concept:agentcontrols_inv	6
+concept:clothingtogowithclothing_inv	7
+concept:cityliesonriver	8
+concept:agriculturalproductincludingagriculturalproduct	9
+concept:musicianinmusicartist	10
+concept:radiostationincity	11
+concept:specializationof	12
+concept:specializationof_inv	13
+concept:hasspouse	14
+concept:agentcollaborateswithagent_inv	15
+concept:bankboughtbank_inv	16
+concept:conceptprerequisiteof	17
+concept:automobilemakerdealersincountry	18
+concept:actorstarredinmovie_inv	19
+concept:stateorprovinceisborderedbystateorprovince	20
+concept:televisionstationincity	21
+concept:personleadsgeopoliticalorganization	22
+concept:languageofuniversity	23
+concept:sportschoolincountry_inv	24
+concept:cityliesonriver_inv	25
+concept:headquarteredin_inv	26
+concept:hasspouse_inv	27
+concept:subpartof_inv	28
+concept:objectpartofobject	29
+concept:musicgenressuchasmusicgenres_inv	30
+concept:atdate_inv	31
+concept:atlocation	32
+concept:weaponssuchasweapons_inv	33
+concept:organizationheadquarteredincity	34
+concept:organizationterminatedperson_inv	35
+concept:stadiumlocatedincity_inv	36
+concept:personborninlocation	37
+concept:sportsgameteam_inv	38
+concept:agriculturalproductcontainchemical_inv	39
+concept:musicartistgenre	40
+concept:stadiumlocatedincity	41
+concept:parentofperson_inv	42
+concept:emotionassociatedwithdisease	43
+concept:academicfieldconcernssubject_inv	44
+concept:languageofcountry_inv	45
+concept:haswife	46
+concept:transportationincity	47
+concept:hobbiessuchashobbies	48
+concept:awardtrophytournamentisthechampionshipgameofthenationalsport_inv	49
+concept:parkincity_inv	50
+concept:attractionofcity	51
+concept:countrylocatedingeopoliticallocation_inv	52
+concept:museumincity	53
+concept:companyalsoknownas	54
+concept:athletehomestadium	55
+concept:hashusband	56
+concept:organizationdissolvedatdate	57
+concept:animalthatfeedoninsect_inv	58
+concept:conceptprerequisiteof_inv	59
+concept:personleadsgeopoliticalorganization_inv	60
+concept:professionusestool	61
+concept:professionusestool_inv	62
+concept:teamplaysincity	63
+concept:personmovedtostateorprovince_inv	64
+concept:politicianusholdsoffice_inv	65
+concept:fatherofperson	66
+concept:motherofperson_inv	67
+concept:automobilemakerdealersincity	68
+concept:plantgrowinginplant_inv	69
+concept:hotelincity_inv	70
+concept:objectfoundinscene_inv	71
+concept:citylocatedinstate	72
+concept:hassibling	73
+concept:arthropodandotherarthropod	74
+concept:atlocation_inv	75
+concept:furniturefoundinroom	76
+concept:hobbiessuchashobbies_inv	77
+concept:athleteledsportsteam_inv	78
+concept:plantincludeplant_inv	79
+concept:personbornincity	80
+concept:fishservedwithfood	81
+concept:teamplayssport	82
+concept:persondiedincountry	83
+concept:attractionofcity_inv	84
+concept:persongraduatedfromuniversity	85
+concept:musicianinmusicartist_inv	86
+concept:radiostationincity_inv	87
+concept:personchargedwithcrime_inv	88
+concept:weaponssuchasweapons	89
+concept:countrycurrency	90
+concept:personborninlocation_inv	91
+concept:automakerproducesmodel	92
+concept:countrylocatedingeopoliticallocation	93
+concept:buildinglocatedincity	94
+concept:bankbankincountry_inv	95
+concept:teamplaysagainstteam_inv	96
+concept:ceoof_inv	97
+concept:animalsuchasfish	98
+concept:persondiedatage_inv	99
+concept:hasofficeincity	100
+concept:organizationalsoknownas	101
+concept:agentcreated	102
+concept:organizationacronymhasname	103
+concept:politicianrepresentslocation	104
+concept:agentcompeteswithagent	105
+concept:sportusesstadium	106
+concept:citylocatedingeopoliticallocation_inv	107
+concept:itemfoundinroom_inv	108
+concept:animalistypeofanimal_inv	109
+concept:sportsgameteam	110
+concept:teamalsoknownas_inv	111
+concept:persongraduatedfromuniversity_inv	112
+concept:agentinvolvedwithitem_inv	113
+concept:agentactsinlocation_inv	114
+concept:countryalsoknownas	115
+concept:languageofuniversity_inv	116
+concept:personhasjobposition	117
+concept:latitudelongitude	118
+concept:teamplaysinleague	119
+concept:athletebeatathlete	120
+concept:companyeconomicsector_inv	121
+concept:teamplayssport_inv	122
+concept:personhascitizenship_inv	123
+concept:persondiedincountry_inv	124
+concept:televisionstationincity_inv	125
+concept:organizationheadquarteredincity_inv	126
+concept:motherofperson	127
+concept:clothingmadefromplant	128
+concept:televisionstationaffiliatedwith	129
+concept:personleadsorganization	130
+concept:leaguestadiums_inv	131
+concept:sportusesequipment_inv	132
+concept:museumincity_inv	133
+concept:foodcancausedisease	134
+concept:plantincludeplant	135
+concept:beveragecontainsprotein_inv	136
+concept:directordirectedmovie_inv	137
+concept:animalsuchasinvertebrate	138
+concept:riveremptiesintoriver_inv	139
+concept:bookwriter_inv	140
+concept:worksfor	141
+concept:politicianrepresentslocation_inv	142
+concept:agentinvolvedwithitem	143
+concept:academicfieldsuchasacademicfield_inv	144
+concept:clothingtogowithclothing	145
+concept:airportincity_inv	146
+concept:agentbelongstoorganization	147
+concept:visualartistartform	148
+concept:professionistypeofprofession_inv	149
+concept:athleteinjuredhisbodypart_inv	150
+concept:mutualproxyfor	151
+concept:fooddecreasestheriskofdisease	152
+concept:agriculturalproductgrowinginstateorprovince_inv	153
+concept:musicgenressuchasmusicgenres	154
+concept:agentactsinlocation	155
+concept:mountaininstate	156
+concept:countrycurrency_inv	157
+concept:academicprogramatuniversity	158
+concept:automobilemakercardealersinstateorprovince	159
+concept:drugworkedonbyagent	160
+concept:stateorprovinceisborderedbystateorprovince_inv	161
+concept:bookwriter	162
+concept:teamalsoknownas	163
+concept:agentbelongstoorganization_inv	164
+concept:coachesteam	165
+concept:weaponmadeincountry	166
+concept:sportusesstadium_inv	167
+concept:mammalsuchasmammal	168
+concept:actorstarredinmovie	169
+concept:buildingfeaturemadefrombuildingmaterial_inv	170
+concept:bodypartcontainsbodypart_inv	171
+concept:agentcompeteswithagent_inv	172
+concept:foodcancausedisease_inv	173
+concept:arthropodandotherarthropod_inv	174
+concept:drugworkedonbyagent_inv	175
+concept:producesproduct_inv	176
+concept:animalsuchasinvertebrate_inv	177
+concept:mountaininstate_inv	178
+concept:politicianusmemberofpoliticalgroup	179
+concept:objectfoundinscene	180
+concept:fishservedwithfood_inv	181
+concept:athleteflyouttosportsteamposition_inv	182
+concept:invertebratefeedonfood	183
+concept:animaleatvegetable	184
+concept:plantrepresentemotion_inv	185
+concept:statehascapital	186
+concept:animaleatfood_inv	187
+concept:newspaperincity_inv	188
+concept:animaleatvegetable_inv	189
+concept:teamwontrophy_inv	190
+concept:persongraduatedschool_inv	191
+concept:televisionstationaffiliatedwith_inv	192
+concept:journalistwritesforpublication	193
+concept:agentcontributedtocreativework	194
+concept:agriculturalproductincludingagriculturalproduct_inv	195
+concept:istallerthan_inv	196
+concept:bacteriaisthecausativeagentofphysiologicalcondition	197
+concept:athleteplaysinleague	198
+concept:agriculturalproductcutintogeometricshape	199
+concept:drughassideeffect	200
+concept:sporthassportsteamposition_inv	201
+concept:personhasresidenceingeopoliticallocation	202
+concept:topmemberoforganization	203
+concept:animalsuchasfish_inv	204
+concept:organizationdissolvedatdate_inv	205
+concept:coachesinleague	206
+concept:athleteplayssport	207
+concept:transportationincity_inv	208
+concept:fatherofperson_inv	209
+concept:academicfieldsuchasacademicfield	210
+concept:agentparticipatedinevent_inv	211
+concept:lakeinstate_inv	212
+concept:companyeconomicsector	213
+concept:academicprogramatuniversity_inv	214
+concept:chemicalistypeofchemical_inv	215
+concept:animalsuchasinsect_inv	216
+concept:personbelongstoorganization	217
+concept:athleteplayssportsteamposition_inv	218
+concept:itemfoundinroom	219
+concept:athleteplayssportsteamposition	220
+concept:plantrepresentemotion	221
+concept:personhasresidenceingeopoliticallocation_inv	222
+concept:personalsoknownas_inv	223
+concept:teamwontrophy	224
+concept:thinghascolor_inv	225
+concept:athleteplaysforteam	226
+concept:headquarteredin	227
+concept:arterycalledartery	228
+concept:agriculturalproductcamefromcountry_inv	229
+concept:journalistwritesforpublication_inv	230
+concept:statelocatedingeopoliticallocation	231
+concept:bakedgoodservedwithbeverage	232
+concept:athleteledsportsteam	233
+concept:sportschoolincountry	234
+concept:furniturefoundinroom_inv	235
+concept:statelocatedincountry	236
+concept:countryalsoknownas_inv	237
+concept:touristattractionsuchastouristattraction_inv	238
+concept:agentcontributedtocreativework_inv	239
+concept:citylocatedincountry_inv	240
+concept:animalpreyson_inv	241
+concept:automobilemakercardealersinstateorprovince_inv	242
+concept:personalsoknownas	243
+concept:cityalsoknownas_inv	244
+concept:agriculturalproductgrowinginstateorprovince	245
+concept:parkincity	246
+concept:musicartistgenre_inv	247
+concept:riveremptiesintoriver	248
+concept:organizationalsoknownas_inv	249
+concept:drughassideeffect_inv	250
+concept:coachwontrophy	251
+concept:mutualproxyfor_inv	252
+concept:agriculturalproductcutintogeometricshape_inv	253
+concept:organizationhiredperson_inv	254
+concept:competeswith	255
+concept:organizationhiredperson	256
+concept:politicianusendorsedbypoliticianus	257
+concept:ceoof	258
+concept:automobilemakerdealersincity_inv	259
+concept:chemicalistypeofchemical	260
+concept:bankbankincountry	261
+concept:ismultipleof_inv	262
+concept:teamhomestadium_inv	263
+concept:athletebeatathlete_inv	264
+concept:personmovedtostateorprovince	265
+concept:producesproduct	266
+concept:coachesteam_inv	267
+concept:organizationacronymhasname_inv	268
+concept:musicianplaysinstrument	269
+concept:politicianusendorsedbypoliticianus_inv	270
+concept:personleadsorganization_inv	271
+concept:citylocatedinstate_inv	272
+concept:agriculturalproducttoattractinsect	273
+concept:drugpossiblytreatsphysiologicalcondition	274
+concept:teamplaysagainstteam	275
+concept:arthropodcalledarthropod	276
+concept:proxyfor_inv	277
+concept:teamplaysincity_inv	278
+concept:thinghasshape	279
+concept:clothingmadefromplant_inv	280
+concept:haswife_inv	281
+concept:hasofficeincountry_inv	282
+concept:buildingfeaturemadefrombuildingmaterial	283
+concept:thinghasshape_inv	284
+concept:sportfansincountry_inv	285
+concept:bacteriaisthecausativeagentofphysiologicalcondition_inv	286
+concept:athleteplayssport_inv	287
+concept:weaponmadeincountry_inv	288
+concept:visualartistartform_inv	289
+concept:hasofficeincity_inv	290
+concept:cityalsoknownas	291
+concept:politicianholdsoffice	292
+concept:drugpossiblytreatsphysiologicalcondition_inv	293
+concept:locationlocatedwithinlocation	294
+concept:teamplaysinleague_inv	295
+concept:persondiedatage	296
+concept:statelocatedingeopoliticallocation_inv	297
+concept:buildinglocatedincity_inv	298
+concept:hotelincity	299
+concept:agentparticipatedinevent	300
+concept:airportincity	301
+concept:bakedgoodservedwithbeverage_inv	302
+concept:synonymfor	303
+concept:statehascapital_inv	304
+concept:newspaperincity	305
+concept:objectpartofobject_inv	306
+concept:bankboughtbank	307
+concept:agriculturalproductcookedwithagriculturalproduct_inv	308
+concept:athleteflyouttosportsteamposition	309
+concept:citylocatedingeopoliticallocation	310
+concept:agriculturalproductcomingfromvertebrate_inv	311
+concept:agriculturalproducttoattractinsect_inv	312
+concept:animaldevelopdisease	313
+concept:beveragemadefrombeverage	314
+concept:personbornincity_inv	315
+concept:athletehomestadium_inv	316
+concept:languageofcountry	317
+concept:plantgrowinginplant	318
+concept:beveragecontainsprotein	319
+concept:mammalsuchasmammal_inv	320
+concept:istallerthan	321
+concept:automobilemakerdealersincountry_inv	322
+concept:animaldevelopdisease_inv	323
+concept:directordirectedmovie	324
+concept:coachesinleague_inv	325
+concept:fooddecreasestheriskofdisease_inv	326
+concept:animalthatfeedoninsect	327
+concept:animaleatfood	328
+concept:latitudelongitude_inv	329
+concept:parentofperson	330
+concept:automakerproducesmodel_inv	331
+concept:atdate	332
+concept:athleteplaysinleague_inv	333
+concept:personhasjobposition_inv	334
+concept:awardtrophytournamentisthechampionshipgameofthenationalsport	335
+concept:competeswith_inv	336
+concept:companyalsoknownas_inv	337
+concept:politicianholdsoffice_inv	338
+concept:statelocatedincountry_inv	339
+concept:sportsgamewinner_inv	340
+concept:animalpreyson	341
+concept:synonymfor_inv	342
+concept:agriculturalproductgrowninlandscapefeatures_inv	343
+concept:thinghascolor	344
+concept:athleteinjuredhisbodypart	345
+concept:sportusesequipment	346
+concept:invertebratefeedonfood_inv	347
+concept:organizationheadquarteredincountry	348
+concept:athletewinsawardtrophytournament	349
+concept:animalistypeofanimal	350
+concept:sporthassportsteamposition	351
+concept:politicianusholdsoffice	352
+concept:animalsuchasinsect	353
+concept:citylocatedincountry	354
+concept:academicfieldconcernssubject	355
+concept:subpartoforganization	356
+concept:acquired	357
+concept:subpartof	358
+concept:agentcreated_inv	359
+concept:sportsgamewinner	360
+concept:politicianusmemberofpoliticalgroup_inv	361
+concept:ismultipleof	362
+concept:teamhomestadium	363
+concept:organizationheadquarteredincountry_inv	364
+concept:citycapitalofcountry_inv	365
+concept:professionistypeofprofession	366
+concept:acquired_inv	367
+concept:sportsgamesport_inv	368
+concept:proxyfor	369
+concept:agriculturalproductcamefromcountry	370
+concept:arthropodcalledarthropod_inv	371
+concept:topmemberoforganization_inv	372
+concept:subpartoforganization_inv	373
+concept:persongraduatedschool	374
+concept:coachwontrophy_inv	375
+concept:athletewinsawardtrophytournament_inv	376
+concept:touristattractionsuchastouristattraction	377
+concept:agriculturalproductcontainchemical	378
+concept:personchargedwithcrime	379
+concept:agriculturalproductcomingfromvertebrate	380
+concept:athleteplaysforteam_inv	381
+concept:agentcontrols	382
+concept:locationlocatedwithinlocation_inv	383
+concept:personhascitizenship	384
+concept:worksfor_inv	385
+concept:hassibling_inv	386
+concept:leaguestadiums	387
+concept:bodypartcontainsbodypart	388
+concept:arterycalledartery_inv	389
+concept:hashusband_inv	390
+concept:citycapitalofcountry	391
+concept:organizationheadquarteredinstateorprovince_inv	392
+concept:organizationheadquarteredinstateorprovince	393
+concept:lakeinstate	394
+concept:sportsgamesport	395
+concept:organizationterminatedperson	396
+concept:hasofficeincountry	397
+concept:agentcollaborateswithagent	398
+concept:musicianplaysinstrument_inv	399

src/research/COINs-KGGeneration/data/NELL-995/relation2vec.bern ADDED Viewed

The diff for this file is too large to render. See raw diff

src/research/COINs-KGGeneration/data/NELL-995/test.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

src/research/COINs-KGGeneration/data/NELL-995/train.txt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:826606112803d19cef310b45a908e6793d89c0ce99fed60d71a9c2095b71d8f6
+size 12571114

src/research/COINs-KGGeneration/data/NELL-995/valid.txt ADDED Viewed

	@@ -0,0 +1,543 @@

+concept_athlete_roberto_alomar	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_sportsteam_eastern_michigan_eagles	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_kris_draper	concept:athleteplaysforteam	concept_sportsteam_red_wings
+concept_athlete_gabe_kapler	concept:athleteplaysforteam	concept_sportsteam_tampa_bay_devil_rays
+concept_athlete_zach_miner	concept:athleteplayssport	concept_sport_baseball
+concept_actor_howard_witt	concept:worksfor	concept_website_chicago_tribune
+concept_city_london_city	concept:organizationhiredperson	concept_politicianus_ken_livingstone
+concept_athlete_derrick_harvey	concept:athleteplayssport	concept_sport_football
+concept_sportsteam_dowling_college_golden_lions	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_coach_bill_stewart	concept:worksfor	concept_city_abc
+concept_athlete_jerry_rice	concept:athleteplaysinleague	concept_sportsleague_nfl
+concept_athlete_johnny_estrada	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_coach_mike_martz	concept:worksfor	concept_sportsteam_rams
+concept_company_cnn	concept:organizationhiredperson	concept_personmexico_ryan_whitney
+concept_coach_brady_quinn	concept:athleteplaysforteam	concept_sportsteam_cleveland_browns
+concept_athlete_lebron	concept:personborninlocation	concept_city_york
+concept_coach_tom_renney	concept:worksfor	concept_sportsteam_rangers
+concept_athlete_print	concept:agentbelongstoorganization	concept_company_adobe
+concept_athlete_dustin_pedroia	concept:athleteplayssport	concept_sport_baseball
+concept_sportsteam_utah_valley_wolverines	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_cody_scarpetta	concept:athleteplayssport	concept_sport_baseball
+concept_coach_a_j__burnett	concept:athleteplayssport	concept_sport_baseball
+concept_sportsteam_ivy_league	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_lorenzen_wright	concept:athleteplaysforteam	concept_sportsteam_reds
+concept_person_abbas	concept:personleadsorganization	concept_musicartist_faction
+concept_university_cu	concept:organizationhiredperson	concept_coach_gary_barnett
+concept_sportsteam_mac_gear	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_sportsteam_md_baltimore_county_retrievers	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_person_christopher002	concept:personborninlocation	concept_city_york
+concept_politicianus_william_powers	concept:worksfor	concept_magazine_national_journal
+concept_sportsteam_southeastern_louisiana_lions	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_sportsteam_columbia_university_lions	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_university_msu	concept:organizationhiredperson	concept_coach_dantonio
+concept_athlete_phil_coke	concept:athletehomestadium	concept_stadiumoreventvenue_pete_times_forum
+concept_person_arnold001	concept:personborninlocation	concept_city_york
+concept_athlete_jeff_garcia	concept:athleteplaysforteam	concept_sportsteam_bucs
+concept_blog_audubon	concept:organizationheadquarteredincity	concept_city_new_orleans
+concept_personus_scott_pelley	concept:personleadsorganization	concept_televisionnetwork_cbs
+concept_coach_bobby_gonzalez	concept:worksfor	concept_university_seton_hall
+concept_person_rob_parker	concept:worksfor	concept_televisionstation_detroit_news
+concept_personus_george_lucas	concept:worksfor	concept_magazine_lucasarts
+concept_company_west_virginia	concept:organizationhiredperson	concept_coach_beilein
+concept_musician_agencies	concept:agentbelongstoorganization	concept_landscapefeatures_states
+concept_city_pittsburgh	concept:organizationhiredperson	concept_male_dave_wannstedt
+concept_personmexico_brendan_harris	concept:athletehomestadium	concept_stadiumoreventvenue_yankee_stadium
+concept_coach_boeheim	concept:worksfor	concept_sportsleague_syracuse
+concept_personmexico_craig_counsell	concept:athleteplaysforteam	concept_sportsteam_yankees
+concept_journalist_kathleen_parker	concept:worksfor	concept_newspaper_washington_post_writers_group
+concept_company_west_virginia	concept:organizationhiredperson	concept_athlete_rich_rodriguez
+concept_person_arthur	concept:personborninlocation	concept_county_york_city
+concept_company_clinton	concept:organizationheadquarteredincity	concept_city_brooklyn
+concept_sportsteam_jacksonville_jaguars	concept:teamplaysinleague	concept_sportsleague_nfl
+concept_sportsteam_man_united	concept:teamplaysinleague	concept_sportsleague_fa
+concept_radiostation_kogo	concept:organizationheadquarteredincity	concept_city_san_diego
+concept_sportsteam_shenandoah_hornets	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_sportsteam_edmonton_oilers	concept:teamplayssport	concept_sport_hockey
+concept_athlete_hideo_nomo	concept:athleteplaysforteam	concept_sportsteam_dodgers
+concept_sportsteam_tufts_jumbos	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_sportsteam_cavs	concept:organizationhiredperson	concept_writer_brown
+concept_radiostation_wfp	concept:organizationheadquarteredincity	concept_city_rome
+concept_athlete_graham_cooke	concept:athleteplayssport	concept_sport_golf
+concept_coach_torry_holt	concept:athleteplayssport	concept_sport_football
+concept_sportsteam_crew	concept:teamplayssport	concept_sport_basketball
+concept_sportsteam_miami_marlins	concept:teamplaysinleague	concept_sportsleague_mlb
+concept_coach_gene_chizik	concept:worksfor	concept_university_auburn
+concept_athlete_kevin_hart	concept:athleteplayssport	concept_sport_baseball
+concept_athlete_ethier	concept:athleteplaysforteam	concept_sportsteam_dodgers
+concept_sportsteam_unlv_runnin_rebels	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_red_grange	concept:athleteplaysinleague	concept_sportsleague_nfl
+concept_ceo_sheldon_adelson	concept:personleadsorganization	concept_biotechcompany_las_vegas_sands_corp_
+concept_athlete_dimitar_berbatov	concept:athleteplaysforteam	concept_sportsteam_man_utd
+concept_athlete_ronaldo	concept:athleteplaysforteam	concept_sportsteam_man_utd
+concept_geopoliticallocation_manchester_city	concept:teamplaysinleague	concept_sportsleague_fa
+concept_journalist_jacques_steinberg	concept:worksfor	concept_website_new_york_times
+concept_coach_lenny_moore	concept:athleteplaysforteam	concept_sportsteam_steelers
+concept_university_cu	concept:organizationhiredperson	concept_coach_dan_hawkins
+concept_athlete_marwin_vega	concept:athleteplayssport	concept_sport_baseball
+concept_athlete_johan_petro	concept:athleteplaysinleague	concept_sportsleague_nba
+concept_person_peter_munk	concept:worksfor	concept_biotechcompany_barrick_gold_corporation
+concept_radiostation_kxcv	concept:organizationheadquarteredincity	concept_city_maryville
+concept_athlete_ken_venturi	concept:athleteplayssport	concept_sport_golf
+concept_stateorprovince_illinois	concept:organizationhiredperson	concept_coach_bruce_weber
+concept_journalist_george_will	concept:worksfor	concept_city_abc
+concept_ceo_tod_nielsen	concept:personleadsorganization	concept_company_borland
+concept_personus_john_browne	concept:worksfor	concept_company_bp001
+concept_athlete_tony_esposito	concept:athletehomestadium	concept_stadiumoreventvenue_wrigley_field
+concept_athlete_tony_pena	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_person_carly_fiorina001	concept:personleadsorganization	concept_university_hewlett_packard
+concept_sportsteam_rhode_island_rams	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_edgar_renteria	concept:athleteplayssport	concept_sport_baseball
+concept_person_bill	concept:personborninlocation	concept_city_york
+concept_athlete_gary_sheffield	concept:athleteplaysforteam	concept_sportsteam_yankees
+concept_company_stax	concept:organizationhiredperson	concept_ceo_jim_stewart
+concept_person_edward001	concept:personborninlocation	concept_county_york_city
+concept_athlete_miroslav_klose	concept:athleteplaysforteam	concept_sportsteam_germany
+concept_company_murray	concept:organizationheadquarteredincity	concept_island_manhattan
+concept_personmexico_ryan_whitney	concept:worksfor	concept_governmentorganization_capitol
+concept_athlete_jordan_schafer	concept:athleteplayssport	concept_sport_baseball
+concept_sportsteam_new_paltz_hawks	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_renaldo_balkman	concept:athleteplaysforteam	concept_sportsteam_nuggets
+concept_coach_tyler_kennedy	concept:athleteplaysforteam	concept_sportsteam_pittsburgh_penguins
+concept_stateorprovince_colorado	concept:organizationhiredperson	concept_coach_gary_barnett
+concept_sportsteam_northeast_wisconsin_technical_college	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_jimmy_gobble	concept:athleteplayssport	concept_sport_baseball
+concept_sportsteam_se_louisiana_lions	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_logan_kensing	concept:athleteplayssport	concept_sport_baseball
+concept_terroristorganization_state	concept:agentbelongstoorganization	concept_politicalparty_house
+concept_athlete_guillermo_mota	concept:athleteplayssport	concept_sport_baseball
+concept_sportsteam_new_york_bobcats	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_coach_dave_winfield	concept:athleteplayssport	concept_sport_baseball
+concept_personnorthamerica_kenneth_d__lewis	concept:personleadsorganization	concept_country___america
+concept_personeurope_william_paley	concept:personleadsorganization	concept_company_cnn__pbs
+concept_coach_j_j__hardy	concept:athleteplayssport	concept_sport_baseball
+concept_athlete_rich_gannon	concept:athleteplaysforteam	concept_sportsteam_oakland_raiders
+concept_website_experience	concept:organizationhiredperson	concept_personmexico_ryan_whitney
+concept_televisionstation_wgby_tv	concept:agentbelongstoorganization	concept_company_pbs
+concept_person_andrea	concept:personborninlocation	concept_county_york_city
+concept_athlete_ramon_ramirez	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_televisionstation_wxia	concept:organizationheadquarteredincity	concept_city_atlanta
+concept_politicsblog_rolling_stone	concept:organizationhiredperson	concept_personus_jann_wenner
+concept_athlete_jermaine_clark	concept:athleteplayssport	concept_sport_baseball
+concept_athlete_kevin_cash	concept:athleteplayssport	concept_sport_baseball
+concept_company_borders	concept:organizationhiredperson	concept_comedian_george_jones
+concept_writer_byron_york	concept:personleadsorganization	concept_newspaper_national_review
+concept_city_dallas	concept:organizationhiredperson	concept_personaustralia_avery_johnson
+concept_nongovorganization_u_s_	concept:organizationhiredperson	concept_coach_bob_bradley
+concept_person_cole	concept:personborninlocation	concept_county_york_city
+concept_writer_anne_applebaum	concept:worksfor	concept_website_washington_post
+concept_politicianus_bill_kristol	concept:worksfor	concept_website_new_york_times
+concept_athlete_eduardo_perez	concept:athleteplayssport	concept_sport_baseball
+concept_building_united_states_capitol	concept:organizationheadquarteredincity	concept_city_washington_d_c
+concept_athlete_rick_vandenhurk	concept:athleteplayssport	concept_sport_baseball
+concept_athlete_tyler_flowers	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_athlete_brad_james	concept:athleteplayssport	concept_sport_baseball
+concept_radiostation_king	concept:organizationheadquarteredincity	concept_city_seattle
+concept_athlete_luis_valbuena	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_athlete_jesse_litsch	concept:athleteplayssport	concept_sport_baseball
+concept_athlete_bernard_berrian	concept:athletehomestadium	concept_stadiumoreventvenue_metrodome
+concept_sportsteam_northern_illinois_huskies	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_city_discussion	concept:organizationhiredperson	concept_personmexico_ryan_whitney
+concept_biotechcompany_tesco	concept:organizationhiredperson	concept_ceo_sir_terry_leahy
+concept_personcanada_ian_davis	concept:worksfor	concept_company_mckinsey
+concept_person_sabrina	concept:personborninlocation	concept_city_york
+concept_actor_megyn_kelly	concept:worksfor	concept_governmentorganization_fox_news
+concept_athlete_matt_murton	concept:athletehomestadium	concept_stadiumoreventvenue_wrigley_field
+concept_athlete_rafael_soriano	concept:athleteplayssport	concept_sport_baseball
+concept_person_ahmadinejad	concept:personborninlocation	concept_city_york
+concept_athlete_deron_williams	concept:athleteplaysforteam	concept_sportsteam_utah_jazz
+concept_athlete_jaime_garcia	concept:athleteplayssport	concept_sport_baseball
+concept_athlete_jay_morrish	concept:athleteplayssport	concept_sport_golf
+concept_sportsteam_texas_arlington_mavericks	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_person_jessica001	concept:personborninlocation	concept_city_york
+concept_athlete_henrik_lundqvist	concept:athleteplaysforteam	concept_sportsteam_rangers
+concept_athlete_edwin_encarnacion	concept:athleteplayssport	concept_sport_baseball
+concept_ceo_larry_probst	concept:personleadsorganization	concept_company_electronic_arts
+concept_athlete_felix_pie	concept:athleteplayssport	concept_sport_baseball
+concept_sportsteam_minnesota_vikings	concept:organizationhiredperson	concept_coach_dennis_green
+concept_coach_richard_hamilton	concept:athleteplaysforteam	concept_sportsteam_pistons
+concept_person_turner	concept:personborninlocation	concept_city_york
+concept_personmexico_ryan_church	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_athlete_paul_hornung	concept:athleteplaysinleague	concept_sportsleague_nfl
+concept_athlete_bob_wickman	concept:athleteplayssport	concept_sport_baseball
+concept_personcanada_peter_mansbridge	concept:worksfor	concept_company_cbc
+concept_city_washington_d_c	concept:organizationhiredperson	concept_celebrity_don_james
+concept_sportsteam_rochester_yellow_jackets	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_person_courtney	concept:personborninlocation	concept_city_york
+concept_athlete_magic_johnson	concept:athleteplaysforteam	concept_sportsteam_los_angeles_lakers
+concept_scientist_ingvar_kamprad	concept:personleadsorganization	concept_company_ikea
+concept_athlete_brad_voyles	concept:athleteplayssport	concept_sport_baseball
+concept_person_douglas	concept:personborninlocation	concept_county_york_city
+concept_person_matthews	concept:personborninlocation	concept_city_york
+concept_athlete_jose_garcia	concept:athleteplayssport	concept_sport_baseball
+concept_televisionstation_ktar	concept:organizationheadquarteredincity	concept_city_phoenix
+concept_radiostation_kvoa_tv	concept:organizationheadquarteredincity	concept_city_tucson
+concept_sportsteam_milwaukee_braves	concept:teamplayssport	concept_sport_baseball
+concept_athlete_jerome_bettis	concept:athleteplaysforteam	concept_sportsteam_steelers
+concept_sportsteam_detroit_lions	concept:teamplaysinleague	concept_sportsleague_nfl
+concept_sportsteam_red_sox	concept:teamplayssport	concept_sport_baseball
+concept_athlete_charles_barkley	concept:athleteplaysinleague	concept_sportsleague_nba
+concept_radiostation_kunm	concept:organizationheadquarteredincity	concept_city_albuquerque
+concept_athlete_charles_barkley	concept:athleteplaysforteam	concept_sportsteam_rockets
+concept_sportsteam_north_park_vikings	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_craig_breslow	concept:athleteplayssport	concept_sport_baseball
+concept_coach_matthew_lombardi	concept:athleteplaysforteam	concept_sportsteam_lee_flames
+concept_athlete_philippoussis	concept:athleteplayssport	concept_sport_tennis
+concept_coach_guus_hiddink	concept:worksfor	concept_country_russia
+concept_athlete_travis_blackley	concept:athleteplayssport	concept_sport_baseball
+concept_personaustralia_smith	concept:worksfor	concept_sportsteam_falcons
+concept_athlete_kei_igawa	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_radiostation_kisw_fm	concept:organizationheadquarteredincity	concept_city_seattle
+concept_sportsteam_santa_clara_university	concept:teamplaysinleague	concept_sportsleague_international
+concept_coach_lendale_white	concept:athletehomestadium	concept_stadiumoreventvenue_lp_field
+concept_personmexico_carlos_guevara	concept:athleteplayssport	concept_sport_baseball
+concept_journalist_richard_anderson	concept:personleadsorganization	concept_biotechcompany_delta_air_lines_inc_
+concept_athlete_brian_gionta	concept:athleteplaysforteam	concept_sportsteam_devils
+concept_athlete_dmitri_young	concept:athleteplaysforteam	concept_sportsteam_detroit_tigers
+concept_coach_van_note	concept:athleteplaysforteam	concept_sportsteam_pirates
+concept_scientist_mitch_kapor	concept:worksfor	concept_city_lotus
+concept_blog_today	concept:organizationhiredperson	concept_personmexico_ryan_whitney
+concept_coach_mike_d_antoni	concept:worksfor	concept_sportsteam_knicks
+concept_sportsleague_new	concept:organizationhiredperson	concept_coach_spagnuolo
+concept_televisionstation_kutv	concept:organizationheadquarteredincity	concept_county_salt_lake
+concept_coach_stoops	concept:worksfor	concept_company_arizona
+concept_athlete_jermaine_o_neal	concept:athleteplaysforteam	concept_sportsteam_pacers
+concept_male_kirsten_powers	concept:worksfor	concept_mountain_fox
+concept_athlete_mark_prior	concept:athleteplayssport	concept_sport_baseball
+concept_sportsteam_oakland_athletics	concept:teamplaysinleague	concept_sportsleague_mlb
+concept_country_party	concept:organizationhiredperson	concept_personus_party
+concept_coach_nate_mclouth	concept:athleteplayssport	concept_sport_baseball
+concept_athlete_warren_sapp	concept:athleteplaysinleague	concept_sportsleague_nfl
+concept_person_lee001	concept:personborninlocation	concept_city_york
+concept_personaustralia_david_nagel	concept:personleadsorganization	concept_company_palmsource
+concept_sportsteam_buffalo_bills	concept:organizationhiredperson	concept_coach_jauron
+concept_athlete_sidney_crosby	concept:athleteplaysforteam	concept_sportsteam_pittsburgh_penguins
+concept_personmexico_shaun_marcum	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_sportsteam_oregon_state_beavers	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_lidstrom	concept:athleteplaysforteam	concept_sportsteam_red_wings
+concept_sportsteam_australian_national_university	concept:teamplaysinleague	concept_sportsleague_international
+concept_athlete_farmar	concept:athletehomestadium	concept_stadiumoreventvenue_staples_center
+concept_athlete_victor_diaz	concept:athleteplayssport	concept_sport_baseball
+concept_personmexico_ryan_whitney	concept:worksfor	concept_city_washington_d_c
+concept_athlete_jeff_lacy	concept:athleteplayssport	concept_sport_boxing
+concept_coach_kip_wells	concept:athleteplaysforteam	concept_sportsteam_bruins
+concept_ceo_david_gergen	concept:worksfor	concept_retailstore_united_technologies
+concept_country___america	concept:organizationhiredperson	concept_personmexico_ryan_whitney
+concept_athlete_dwight_howard	concept:athleteplaysinleague	concept_sportsleague_nba
+concept_sportsteam_brewers	concept:teamplaysinleague	concept_sportsleague_mlb
+concept_athlete_bryzgalov	concept:athletehomestadium	concept_stadiumoreventvenue_honda_center
+concept_stateorprovince_minnesota	concept:organizationhiredperson	concept_coach_tim_brewster
+concept_sportsteam_cavaliers	concept:teamplaysinleague	concept_sportsleague_nba
+concept_coach_kevin_youkilis	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_athlete_aaron_fultz	concept:athleteplayssport	concept_sport_baseball
+concept_coach_ed_orgeron	concept:worksfor	concept_stateorprovince_ole_miss
+concept_sportsteam_va_commonwealth_rams	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_website_fortune	concept:organizationheadquarteredincity	concept_city_new_york
+concept_athlete_randall_simon	concept:athleteplaysforteam	concept_sportsteam_pirates
+concept_female_sarah_caldwell	concept:personleadsorganization	concept_organization_opera_company_of_boston
+concept_radiostation_kola	concept:organizationheadquarteredincity	concept_city_redlands
+concept_person_daniel	concept:personborninlocation	concept_airport_jersey
+concept_person_angela	concept:personborninlocation	concept_city_york
+concept_university_novartis	concept:organizationhiredperson	concept_ceo_daniel_vasella
+concept_athlete_brian_moehler	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_personaustralia_smith	concept:worksfor	concept_university_oklahoma_state
+concept_radiostation_kprl	concept:organizationheadquarteredincity	concept_city_paso_robles
+concept_athlete_mike_lowell	concept:athleteplaysforteam	concept_sportsteam_red_sox
+concept_televisionstation_kfmb	concept:organizationheadquarteredincity	concept_city_san_diego
+concept_university_raiders	concept:organizationhiredperson	concept_actor_bill_callahan
+concept_athlete_joe_sakic	concept:athleteplayssport	concept_sport_hockey
+concept_journalist_joe_henderson	concept:worksfor	concept_televisionstation_tampa_tribune
+concept_actor_scott_mcnealy	concept:worksfor	concept_company_sun
+concept_sportsteam_sam_houston_state_bearkats	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_steve_trachsel	concept:athleteplayssport	concept_sport_baseball
+concept_personmexico_ryan_whitney	concept:worksfor	concept_nongovorganization_council
+concept_sportsteam_marshall_university	concept:teamplayssport	concept_sport_basketball
+concept_city_georgetown	concept:organizationhiredperson	concept_person_thompson
+concept_person_mitchell	concept:personborninlocation	concept_county_york_city
+concept_athlete_juan_acevedo	concept:athleteplayssport	concept_sport_baseball
+concept_sportsteam_atlantic_coast_conference	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_sportsteam_rockies	concept:teamplaysinleague	concept_sportsleague_mlb
+concept_university_kansas_city	concept:organizationhiredperson	concept_athlete_paul_brown
+concept_sportsteam_maryland_baltimore_county_retrievers	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_evan_longoria	concept:athleteplayssport	concept_sport_baseball
+concept_person_pat_riley	concept:worksfor	concept_sportsteam_heat
+concept_person_ron_perelman	concept:worksfor	concept_company_revlon
+concept_athlete_chris_davis	concept:athleteplayssport	concept_sport_hockey
+concept_university_kansas_state	concept:organizationhiredperson	concept_actor_ron_prince
+concept_athlete_cedric_simmons	concept:athleteplaysinleague	concept_sportsleague_nba
+concept_university_byu	concept:organizationhiredperson	concept_coach_gary_crowton
+concept_sportsteam_emory_eagles	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_person_otis	concept:personborninlocation	concept_county_york_city
+concept_sportsteam_morehead_state_eagles	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_sportsteam_montana_state	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_chris_davis	concept:athleteplaysinleague	concept_sportsleague_nhl
+concept_city_toronto	concept:organizationhiredperson	concept_coach_mitch_williams
+concept_university_unc	concept:organizationhiredperson	concept_coach_dean_smith
+concept_athlete_carlos_beltran	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_coach_steve_spurrier	concept:worksfor	concept_sportsteam_redskins
+concept_athlete_mewelde_moore	concept:athletehomestadium	concept_stadiumoreventvenue_heinz_field
+concept_sportsteam_hampton_university	concept:teamplayssport	concept_sport_basketball
+concept_company_missouri	concept:organizationhiredperson	concept_person_anderson
+concept_company_time_warner	concept:organizationhiredperson	concept_person_gerald_levin
+concept_athlete_grant_hill	concept:athleteplaysinleague	concept_sportsleague_nba
+concept_coach_marion_barber_iii	concept:athleteplaysforteam	concept_sportsteam_dallas_cowboys
+concept_athlete_james_farrior	concept:athletehomestadium	concept_stadiumoreventvenue_heinz_field
+concept_person_caroline002	concept:personborninlocation	concept_city_york
+concept_athlete_rip_hamilton	concept:athleteplaysinleague	concept_sportsleague_nba
+concept_athlete_stephen_drew	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_sportsteam_drake_bulldogs	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_coach_kevin_youkilis	concept:athleteplaysforteam	concept_sportsteam_red_sox
+concept_sportsteam_seahawks	concept:organizationhiredperson	concept_coach_jim_zorn
+concept_athlete_scott_cassidy	concept:athleteplayssport	concept_sport_baseball
+concept_stateorprovince_indiana	concept:organizationhiredperson	concept_coach_bobby_knight
+concept_athlete_chris_smith	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_televisionstation_kpho	concept:organizationheadquarteredincity	concept_city_phoenix
+concept_sportsteam_gonzaga_bulldogs	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_zach_thomas	concept:athleteplayssport	concept_sport_football
+concept_company_flyglobespan	concept:organizationheadquarteredincity	concept_city_edinburgh
+concept_person_pearson	concept:personborninlocation	concept_city_york
+concept_athlete_rajai_davis	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_nongovorganization_opposition	concept:organizationhiredperson	concept_personmexico_ryan_whitney
+concept_sportsleague_new	concept:organizationhiredperson	concept_coach_sparano
+concept_televisionstation_kera_tv	concept:agentbelongstoorganization	concept_company_pbs
+concept_athlete_david_murphy	concept:athleteplaysinleague	concept_sportsleague_nhl
+concept_athlete_otto_graham	concept:athleteplayssport	concept_sport_football
+concept_athlete_alex_herrera	concept:athleteplayssport	concept_sport_baseball
+concept_sportsteam_mid_eastern_athletic_conference	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_darrick_martin	concept:athleteplaysinleague	concept_sportsleague_nba
+concept_terroristorganization_state	concept:organizationhiredperson	concept_coach_kay_yow
+concept_person_max001	concept:personborninlocation	concept_city_york
+concept_politicianus_newt_gingrich	concept:personleadsorganization	concept_politicalparty_house
+concept_sportsteam_la_clippers	concept:teamplaysinleague	concept_sportsleague_nba
+concept_sportsteam_minnesota_wilds	concept:teamplayssport	concept_sport_hockey
+concept_sportsteam_buccaneers	concept:organizationhiredperson	concept_coach_john_gruden
+concept_country_syria	concept:organizationhiredperson	concept_personmexico_ryan_whitney
+concept_company_focus	concept:organizationhiredperson	concept_personmexico_ryan_whitney
+concept_televisionstation_kshb	concept:organizationheadquarteredincity	concept_county_kansas_city
+concept_person_ter_petrosyan	concept:personleadsorganization	concept_country_republic_of_armenia
+concept_athlete_nomar_garciaparra	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_athlete_roddy_white	concept:athleteplaysforteam	concept_sportsteam_falcons
+concept_personeurope_azim_premji	concept:personleadsorganization	concept_sportsteam_wipro_technologies
+concept_sportsteam_vanguard_lions	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_coach_vincent_lecavalier	concept:athleteplaysforteam	concept_sportsteam_tampa_bay_lightning
+concept_athlete_jonathan_stewart	concept:athleteplayssport	concept_sport_football
+concept_athlete_jo_jo_reyes	concept:athleteplayssport	concept_sport_baseball
+concept_stateorprovince_oregon	concept:organizationhiredperson	concept_coach_ernie_kent
+concept_coach_brian_kelly	concept:worksfor	concept_geopoliticallocation_central_michigan
+concept_athlete_robert_trent_jones_ii	concept:athleteplayssport	concept_sport_golf
+concept_politician_jobs	concept:worksfor	concept_county_god
+concept_person_jeff_zucker	concept:worksfor	concept_company_nbc_universal001
+concept_televisionstation_kfts	concept:agentbelongstoorganization	concept_company_pbs
+concept_athlete_brandon_marshall	concept:athleteplayssport	concept_sport_golf
+concept_politician_john_peyton	concept:worksfor	concept_city_jacksonville
+concept_person_tony_snow	concept:worksfor	concept_musicartist_news
+concept_creditunion_kansas	concept:organizationhiredperson	concept_athlete_roy_williams
+concept_person_albert002	concept:personborninlocation	concept_city_york
+concept_athlete_john_bale	concept:athleteplayssport	concept_sport_baseball
+concept_company_new_york	concept:organizationhiredperson	concept_politician_michael_r__bloomberg
+concept_televisionstation_kxne_tv	concept:agentbelongstoorganization	concept_company_pbs
+concept_personnorthamerica_bret_baier	concept:worksfor	concept_governmentorganization_fox_news
+concept_radiostation_kcsm	concept:organizationheadquarteredincity	concept_city_san_mateo
+concept_athlete_gaby_hernandez	concept:athleteplayssport	concept_sport_baseball
+concept_athlete_dustin_moseley	concept:athleteplayssport	concept_sport_baseball
+concept_radiostation_kska	concept:organizationheadquarteredincity	concept_city_anchorage
+concept_coach_vincent_lecavalier	concept:athleteplayssport	concept_sport_baseball
+concept_personeurope_allan_moss	concept:worksfor	concept_bank_macquarie_bank
+concept_athlete_jerry_rice	concept:athleteplayssport	concept_sport_football
+concept_person_john003	concept:worksfor	concept_politicsblog_ny_times
+concept_person_abbas	concept:personleadsorganization	concept_nongovorganization_fatah_leadership
+concept_person_mitchell001	concept:personborninlocation	concept_city_york
+concept_writer_david_broder	concept:worksfor	concept_city_washington_d_c
+concept_blog_komu_tv	concept:agentbelongstoorganization	concept_company_nbc
+concept_athlete_jason_bere	concept:athleteplayssport	concept_sport_baseball
+concept_creditunion_kansas	concept:organizationhiredperson	concept_personcanada_self
+concept_company_boehringer_ingelheim	concept:organizationheadquarteredincity	concept_city_ridgefield
+concept_sportsteam_texas_el_paso_miners	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_stateorprovince_tennessee	concept:organizationhiredperson	concept_personmexico_bruce_pearl
+concept_journalist_natalie_angier	concept:worksfor	concept_musicartist_times
+concept_sportsteam_temple_university	concept:teamplaysinleague	concept_sportsleague_international
+concept_athlete_kei_igawa	concept:athletehomestadium	concept_stadiumoreventvenue_yankee_stadium
+concept_sportsteam_knicks	concept:teamplaysinleague	concept_sportsleague_nba
+concept_athlete_felix_hernandez	concept:athleteplaysforteam	concept_sportsteam_mariners
+concept_person_ty_willingham	concept:worksfor	concept_city_washington_d_c
+concept_televisionstation_wmeb_tv	concept:agentbelongstoorganization	concept_company_pbs
+concept_person_perenchio	concept:personleadsorganization	concept_city_univision
+concept_journalist_jane_mayer	concept:worksfor	concept_website_new_york_american
+concept_athlete_yoslan_herrera	concept:athleteplayssport	concept_sport_baseball
+concept_athlete_casey_kotchman	concept:athleteplayssport	concept_sport_baseball
+concept_journalist_rocky_anderson	concept:worksfor	concept_biotechcompany_salt_lake_city
+concept_athlete_bill_mazeroski	concept:athleteplayssport	concept_sport_baseball
+concept_person_gundy	concept:worksfor	concept_university_osu
+concept_athlete_ryan_braun	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_athlete_rafer_alston	concept:athleteplaysinleague	concept_sportsleague_nba
+concept_website_new_yorker_magazine	concept:organizationheadquarteredincity	concept_city_new_york
+concept_athlete_john_axford	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_professor_david_levy	concept:worksfor	concept_politicsblog_turner_sports
+concept_sportsteam_miami_dolphins	concept:teamplaysinleague	concept_sportsleague_nfl
+concept_athlete_braden_looper	concept:athleteplaysinleague	concept_sportsleague_nfl
+concept_ceo_sergey_brin	concept:personleadsorganization	concept_university_google
+concept_company_pakistan	concept:organizationhiredperson	concept_personmexico_ryan_whitney
+concept_athlete_pedro_feliz	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_sportsteam_washington_state	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_company_chevron001	concept:organizationheadquarteredincity	concept_city_san_ramon
+concept_personafrica_mike_davis	concept:worksfor	concept_university_uab
+concept_geopoliticalorganization_address	concept:organizationhiredperson	concept_personmexico_ryan_whitney
+concept_professor_greg_palast	concept:worksfor	concept_governmentorganization_bbc
+concept_sportsteam_texas_a_m_international_university	concept:teamplaysinleague	concept_sportsleague_international
+concept_sportsteam_twins	concept:teamplaysinleague	concept_sportsleague_mlb
+concept_sportsteam_quebec_nordiques	concept:teamplaysinleague	concept_sportsleague_nhl
+concept_sportsteam_n2009_all_star	concept:teamplaysinleague	concept_sportsleague_mlb
+concept_company_amazon_com_inc	concept:organizationhiredperson	concept_personus_jeff_bezos
+concept_athlete_sidney_crosby	concept:athletehomestadium	concept_stadiumoreventvenue_mellon_arena
+concept_politicalparty_conservatives	concept:organizationhiredperson	concept_personmexico_ryan_whitney
+concept_sportsteam_cal_state_san_bernardino_coyotes	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_placido_polanco	concept:athleteplayssport	concept_sport_baseball
+concept_athlete_brad_ausmus_gg	concept:athleteplaysforteam	concept_sportsteam_astros
+concept_coach_reggie_herring	concept:worksfor	concept_stateorprovince_arkansas
+concept_sportsteam_curtin_university	concept:teamplaysinleague	concept_sportsleague_international
+concept_personmexico_jack_morris	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_sportsteam_knicks	concept:organizationhiredperson	concept_personmexico_isaiah_thomas
+concept_person_norman001	concept:personborninlocation	concept_city_york
+concept_televisionstation_wha__tv	concept:agentbelongstoorganization	concept_company_pbs
+concept_coach_shanahan	concept:worksfor	concept_city_denver
+concept_athlete_garret_anderson	concept:athletehomestadium	concept_stadiumoreventvenue_old_trafford
+concept_personnorthamerica_maria_shriver	concept:worksfor	concept_company_nbc
+concept_athlete_vicente_padilla	concept:athletehomestadium	concept_stadiumoreventvenue_giants_stadium
+concept_athlete_isaiah_thomas	concept:athleteplaysinleague	concept_sportsleague_nba
+concept_televisionstation_kosu	concept:organizationheadquarteredincity	concept_city_stillwater
+concept_sportsteam_lewis_flyers	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_kyle_farnsworth	concept:athleteplayssport	concept_sport_baseball
+concept_athlete_ben_eager	concept:athletehomestadium	concept_stadiumoreventvenue_wrigley_field
+concept_personeurope_james_wright	concept:worksfor	concept_university_villanova
+concept_athlete_yuniesky_betancourt	concept:athleteplayssport	concept_sport_baseball
+concept_sportsteam_harvard_divinity_school	concept:teamplaysinleague	concept_sportsleague_international
+concept_athlete_darren_mcfadden	concept:athletehomestadium	concept_stadiumoreventvenue_mcafee_coliseum
+concept_athlete_carlos_boozer	concept:athleteplaysforteam	concept_sportsteam_utah_jazz
+concept_athlete_alexei_ramirez	concept:athleteplaysforteam	concept_sportsteam_white_sox
+concept_athlete_mark_mcgwire_and_sammy_sosa	concept:athleteplaysinleague	concept_sportsleague_nfl
+concept_athlete_esteban_yan	concept:athleteplayssport	concept_sport_baseball
+concept_coach_mike_holmgren	concept:worksfor	concept_sportsteam_packers
+concept_university_oklahoma_state	concept:organizationhiredperson	concept_coach_travis_ford
+concept_athlete_tom_glavine	concept:athleteplayssport	concept_sport_baseball
+concept_athlete_pau_gasol	concept:athleteplayssport	concept_sport_basketball
+concept_athlete_mike_cameron	concept:athleteplayssport	concept_sport_baseball
+concept_coach_mike_stoops	concept:worksfor	concept_company_arizona
+concept_athlete_jeremy_bonderman	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_personus_steve_ballmer	concept:worksfor	concept_university_microsoft
+concept_journalist_timothy_egan	concept:worksfor	concept_musicartist_times
+concept_athlete_scott_linebrink	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_coach_steve_spagnuolo	concept:worksfor	concept_city_st_louis
+concept_personnorthamerica_neil_bogart	concept:personleadsorganization	concept_city_casablanca
+concept_professor_david_yu	concept:personleadsorganization	concept_company_betfair
+concept_athlete_nick_young	concept:athleteplaysforteam	concept_sportsteam_washington_wizards
+concept_televisionstation_wfxu	concept:agentbelongstoorganization	concept_televisionnetwork_pbs
+concept_person_oscar_wyatt	concept:worksfor	concept_newspaper_texas
+concept_televisionstation_wmfe_tv	concept:agentbelongstoorganization	concept_company_pbs
+concept_radiostation_kut	concept:organizationheadquarteredincity	concept_city_austin
+concept_writer_joshua_nkomo	concept:worksfor	concept_politicalparty_zimbabwe_african_peoples_union
+concept_sportsteam_grace_lancers	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_politicianus_john_lafferty	concept:worksfor	concept_sportsteam_carnegie_mellon_university
+concept_athlete_corey_maggette	concept:athleteplayssport	concept_sport_basketball
+concept_website_wsb_tv	concept:organizationheadquarteredincity	concept_city_atlanta
+concept_building_houses	concept:organizationhiredperson	concept_personus_democratic
+concept_sportsteam_florida_st__seminoles	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_tommy_maddox	concept:athletehomestadium	concept_stadiumoreventvenue_heinz_field
+concept_personmexico_jose_bautista	concept:athleteplayssport	concept_sport_baseball
+concept_sportsteam_jaguars	concept:teamplaysinleague	concept_sportsleague_nfl
+concept_politicalparty_senate	concept:organizationhiredperson	concept_person_lower
+concept_coach_alexander_frolov	concept:athleteplaysforteam	concept_sportsteam_kings_college
+concept_sportsleague_notice	concept:organizationhiredperson	concept_personmexico_ryan_whitney
+concept_professionalorganization_cal	concept:organizationhiredperson	concept_person_jeff_tedford
+concept_sportsteam_san_jose_state_spartans	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_company_martha_stewart	concept:organizationheadquarteredincity	concept_city_new_york
+concept_person_britney	concept:personborninlocation	concept_city_york
+concept_personmexico_ryan_whitney	concept:personleadsorganization	concept_governmentorganization_agency
+concept_athlete_ryan_rowland_smith	concept:athletehomestadium	concept_stadiumoreventvenue_safeco_field
+concept_person_wendy001	concept:personborninlocation	concept_city_york
+concept_athlete_ed_belfour	concept:athleteplaysforteam	concept_sportsteam_blackhawks
+concept_personmexico_ryan_whitney	concept:personleadsorganization	concept_website_change
+concept_athlete_eric_byrnes	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_televisionstation_wcfe_tv	concept:agentbelongstoorganization	concept_company_pbs
+concept_biotechcompany_section	concept:agentbelongstoorganization	concept_stateorprovince_contact
+concept_male_gerald_green	concept:athleteplaysinleague	concept_sportsleague_nba
+concept_televisionstation_kyve_tv	concept:agentbelongstoorganization	concept_company_pbs
+concept_athlete_udonis_haslem	concept:athleteplaysforteam	concept_sportsteam_heat
+concept_personaustralia_jobs	concept:worksfor	concept_publication_macworld
+concept_person_abbas	concept:personleadsorganization	concept_city_name
+concept_athlete_j_d__martin	concept:athleteplayssport	concept_sport_baseball
+concept_ceo_thomas_m__ryan	concept:worksfor	concept_biotechcompany_cvs_caremark
+concept_athlete_rashard_lewis	concept:athleteplayssport	concept_sport_basketball
+concept_politicianus_jon_scott	concept:worksfor	concept_mountain_fox
+concept_politician_jobs	concept:worksfor	concept_blog_today
+concept_athlete_jorge_julio	concept:athleteplayssport	concept_sport_baseball
+concept_sportsteam_njit_highlanders	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_sportsteam_hershey_bears	concept:teamplayssport	concept_sport_hockey
+concept_sportsteam_northwestern	concept:organizationhiredperson	concept_coach_pat_fitzgerald
+concept_coach_carroll	concept:worksfor	concept_university_southern_cal
+concept_personmexico_jason_taylor	concept:athleteplayssport	concept_sport_football
+concept_coach_andrew_ladd	concept:athletehomestadium	concept_stadiumoreventvenue_wrigley_field
+concept_athlete_bobby_orr	concept:athleteplaysforteam	concept_sportsteam_blackhawks
+concept_writer_luca	concept:personleadsorganization	concept_winery_ferrari
+concept_person_levine	concept:personborninlocation	concept_city_york
+concept_sportsteam_purdue_university	concept:teamplayssport	concept_sport_basketball
+concept_politicianus_mikhail_gorbachev	concept:personleadsorganization	concept_organization_soviet
+concept_coach_brett_lee	concept:worksfor	concept_university_uw
+concept_televisionstation_kwhy	concept:organizationheadquarteredincity	concept_county_los_angeles_county
+concept_televisionstation_kcpq_tv	concept:organizationheadquarteredincity	concept_city_tacoma
+concept_sportsteam_capitals	concept:organizationhiredperson	concept_athlete_ovechkin
+concept_radiostation_word_fm	concept:organizationheadquarteredincity	concept_city_pittsburgh
+concept_sportsteam_dalhousie_university	concept:teamplaysinleague	concept_sportsleague_international
+concept_sportsleague_new	concept:organizationhiredperson	concept_coach_sean_payton
+concept_televisionstation_kmne_tv	concept:agentbelongstoorganization	concept_company_pbs
+concept_governmentorganization_federal	concept:organizationheadquarteredincity	concept_city_boston
+concept_sportsteam_st__louis_cardinals	concept:teamplayssport	concept_sport_baseball
+concept_sportsteam_minnesota_timberwolves	concept:teamplayssport	concept_sport_basketball
+concept_athlete_eric_milton	concept:athleteplayssport	concept_sport_baseball
+concept_athlete_jayson_werth	concept:athleteplayssport	concept_sport_baseball
+concept_sportsteam_dayton_flyers	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_laurence_maroney	concept:athletehomestadium	concept_stadiumoreventvenue_gillette_stadium
+concept_coach_richard_zednik	concept:athleteplaysforteam	concept_sportsteam_florida_international
+concept_athlete_derrick_harvey	concept:athleteplaysinleague	concept_sportsleague_nfl
+concept_county_duke_university	concept:organizationhiredperson	concept_coach_mike_krzyzewski
+concept_person_katsuaki_watanabe	concept:worksfor	concept_automobilemaker_toyota_motor
+concept_athlete_rafael_perez	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_sportsteam_old_dominion_university	concept:teamplaysinleague	concept_sportsleague_international
+concept_personmexico_melky_cabrera	concept:athleteplaysforteam	concept_sportsteam_yankees
+concept_sportsteam_west_virginia_state_yellow_jackets	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_country_russia	concept:organizationhiredperson	concept_coach_hiddink
+concept_county_chicago	concept:organizationhiredperson	concept_politicianus_richard_m__daley
+concept_company_remark	concept:organizationhiredperson	concept_personmexico_ryan_whitney
+concept_person_richard002	concept:worksfor	concept_city_uk
+concept_person_richard002	concept:personborninlocation	concept_city_york
+concept_coach_aaron_harang	concept:athleteplaysforteam	concept_sportsteam_blackhawks
+concept_televisionstation_knxv	concept:organizationheadquarteredincity	concept_city_phoenix
+concept_athlete_david_carr	concept:athleteplaysforteam	concept_sportsteam_texans
+concept_sportsteam_florida_international	concept:organizationhiredperson	concept_coach_john_fox
+concept_athlete_rod_smith	concept:athletehomestadium	concept_stadiumoreventvenue_invesco_field
+concept_athlete_jaromir_jagr	concept:athletehomestadium	concept_stadiumoreventvenue_giants_stadium
+concept_recordlabel_death_row	concept:organizationhiredperson	concept_person_suge_knight
+concept_sportsteam_bond_university	concept:teamplaysinleague	concept_sportsleague_international
+concept_athlete_ivan_rodriguez	concept:athletehomestadium	concept_stadiumoreventvenue_yankee_stadium
+concept_athlete_denny_bautista	concept:athleteplaysinleague	concept_sportsleague_mlb
+concept_athlete_phil_hughes_and_ian_kennedy	concept:athletehomestadium	concept_stadiumoreventvenue_yankee_stadium
+concept_sportsteam_new_york_liberty	concept:teamplaysinleague	concept_sportsleague_wnba
+concept_personmexico_shaun_marcum	concept:athleteplaysforteam	concept_sportsteam_blue_jays
+concept_sportsteam_steelers	concept:teamplaysinleague	concept_sportsleague_nfl
+concept_sportsteam_lsu	concept:organizationhiredperson	concept_person_skip_bertman
+concept_person_lynch	concept:personborninlocation	concept_city_york
+concept_athlete_mike_myers	concept:athleteplayssport	concept_sport_baseball
+concept_athlete_joe_bradley	concept:athleteplaysforteam	concept_sportsteam_pats
+concept_athlete_sean_taylor	concept:athleteplaysforteam	concept_sportsteam_saints
+concept_radiostation_ketr	concept:organizationheadquarteredincity	concept_city_commerce
+concept_coach_lubomir_visnovsky	concept:athleteplaysforteam	concept_sportsteam_oilers
+concept_sportsteam_york_university	concept:teamplaysinleague	concept_sportsleague_international
+concept_personus_rex_tillerson	concept:agentbelongstoorganization	concept_petroleumrefiningcompany_exxonmobil
+concept_sportsteam_cal_poly_slo_mustangs	concept:teamplaysinleague	concept_sportsleague_ncaa
+concept_athlete_jordan_staal	concept:athleteplaysforteam	concept_sportsteam_pittsburgh_penguins
+concept_athlete_mike_wallace	concept:worksfor	concept_company_n60_minutes
+concept_politician_kaine	concept:personleadsorganization	concept_publication_richmond
+concept_athlete_darrent_williams	concept:athletehomestadium	concept_stadiumoreventvenue_invesco_field
+concept_person_brent_staples	concept:worksfor	concept_website_new_york_times
+concept_personasia_rick_ankiel	concept:athleteplayssport	concept_sport_football
+concept_person_levon_ter__petrosyan	concept:worksfor	concept_country_armenia
+concept_person_charles_prince	concept:personleadsorganization	concept_company_citigroup

src/research/COINs-KGGeneration/data/WN18RR/test.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

src/research/COINs-KGGeneration/data/WN18RR/train.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

src/research/COINs-KGGeneration/data/WN18RR/valid.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

src/research/COINs-KGGeneration/graph_analysis/KGist/__init__.py ADDED Viewed

File without changes

src/research/COINs-KGGeneration/graph_analysis/KGist/anomaly_detector.py ADDED Viewed

	@@ -0,0 +1,83 @@

+from math import log2 as log
+from scipy.special import gammaln
+from collections import defaultdict
+import numpy as np
+from itertools import chain, combinations
+from graph_analysis.KGist.evaluator import Evaluator
+class AnomalyDetector:
+    def __init__(self, model):
+        self.idify = model.graph.idify
+        self.model = model
+        self.evaluator = Evaluator(self.model.graph)
+        self.subject_to_rules = dict()
+        for rule in self.model.rules:
+            sub = rule[0] if type(rule) is tuple else rule.root
+            if sub not in self.subject_to_rules:
+                self.subject_to_rules[sub] = set()
+            self.subject_to_rules[sub].add(rule)
+        self.binomial_map = dict()
+        self.edge_to_id = dict()
+        for m, edge in model.graph.id_to_edge.items():
+            sub, pred, obj = edge
+            self.edge_to_id[(sub, pred, obj)] = m
+    def length_binomial(self, n, k):
+        '''
+        :n: n in (n choose k)
+        :k: k in (n choose k)
+        :return: log_2(n choose k)
+        '''
+        if (n, k) in self.binomial_map:
+            return self.binomial_map[(n, k)]
+        length = (gammaln(n + 1) - gammaln(k + 1) - gammaln((n + 1) - k)) / np.log(2)
+        self.binomial_map[(n, k)] = length
+        return length
+    def score_blame_edge(self, node, pred):
+        '''
+        The number of bits describing the node as an exception to rules.
+        '''
+        def powerset(iterable):
+            s = list(iterable)
+            return chain.from_iterable(combinations(s, r) for r in range(1, len(s)+1))
+        score = 0
+        rules = set()
+        for labels in powerset(self.model.graph.labels(node)):
+            if labels in self.subject_to_rules:
+                rules.update(self.subject_to_rules[labels])
+        def has_pred(rule, pred):
+            if type(rule) is tuple:
+                return rule[1][0][0] == pred
+            else:
+                return pred in rule.get_preds()
+        rules = set(filter(lambda rule: has_pred(rule, pred), rules))
+        for rule in rules:
+            cas = set(self.model.graph.candidates[rule]['ca_to_size'].keys()) if type(rule) is tuple else set(real.root for real in rule.correct_assertions)
+            if node not in cas:
+                num_assertions = self.model.graph.nodes_with_type(rule[0] if type(rule) is tuple else rule.root)
+                num_correct = len(self.model.graph.candidates[rule]['ca_to_size'] if type(rule) is tuple else rule.correct_assertions)
+                num_exceptions = num_assertions - num_correct
+                assert(num_assertions > 0)
+                assert(num_exceptions > 0)
+                score += (1 / num_exceptions) * self.length_binomial(num_assertions, num_exceptions)
+        return score
+    def score_edge(self, edge, blame_edge=True):
+        if self.idify:
+            edge = (self.model.graph.node_to_id[edge[0]], self.model.graph.pred_to_id[edge[1]], self.model.graph.node_to_id[edge[2]])
+        eid = self.edge_to_id[edge] if type(edge) is tuple else edge
+        sub, pred, obj = edge if type(edge) is tuple else self.model.graph.id_to_edge[eid]
+        score = 0
+        if eid not in self.model.tensor: # edges not in the model are not explained
+            # number of bits describing the unexplained edge
+            score = (1 / (self.model.graph.m - len(self.model.tensor))) * self.evaluator.length_negative_edge_error(self.model)
+        score += self.score_blame_edge(sub, pred) + self.score_blame_edge(obj, pred)
+        return score

src/research/COINs-KGGeneration/graph_analysis/KGist/correct_assertion.py ADDED Viewed

	@@ -0,0 +1,72 @@

+from collections import defaultdict
+class Node:
+    def __init__(self, name, typ):
+        self.name = name
+        self.type = typ
+        # maps (pred, dir, node_type) to node names
+        self.neighbors_of_type = dict()
+        # node names
+        self.neighbors = set()
+    def add_neighbor(self, pred, dir, node_type, node):
+        if (pred, dir, node_type) not in self.neighbors_of_type:
+            self.neighbors_of_type[(pred, dir, node_type)] = set()
+        self.neighbors_of_type[(pred, dir, node_type)].add(node)
+        self.neighbors.add(node)
+class CorrectAssertion:
+    '''
+    A class which represents realized rules as rooted subgraphs.
+    '''
+    def __init__(self, root, label):
+        self.root = root
+        # maps node names to node objects
+        self.nodes = dict()
+        self.edges = set()
+        self.edge_ids = set()
+        self.labels = set()
+    def add_edge(self, edge, eid=None, labels=False):
+        '''
+        :edge: (u, u_typ, pred, dir, v, v_typ)
+        :eid: the id of the edge in the full graph
+        :labels: boolean specifying whether or not to add node labels
+        '''
+        u, u_typ, pred, dir, v, v_typ = edge
+        sub = u if dir == 'out' else v
+        sub_typ = u_typ if dir == 'out' else v_typ
+        obj = u if dir == 'in' else v
+        obj_typ = u_typ if dir == 'in' else v_typ
+        if u not in self.nodes:
+            self.nodes[u] = Node(u, u_typ)
+        if v not in self.nodes:
+            self.nodes[v] = Node(v, v_typ)
+        self.nodes[sub].add_neighbor(pred, 'out', obj_typ, obj)
+        self.nodes[obj].add_neighbor(pred, 'in', sub_typ, sub)
+        self.edges.add(edge)
+        # edge coverage
+        if eid != None:
+            self.edge_ids.add(eid)
+        # label coverage
+        if labels:
+            if u != self.root:
+                for label in u_typ:
+                    self.labels.add((label, u))
+            if v != self.root:
+                for label in v_typ:
+                    self.labels.add((label, v))
+    def merge(self, correct_assertion):
+        assert(self.root == correct_assertion.root)
+        for edge in correct_assertion.edges:
+            self.add_edge(edge)
+        self.edge_ids.update(correct_assertion.edge_ids)
+        self.labels.update(correct_assertion.labels)
+    def compose(self, correct_assertion):
+        for edge in correct_assertion.edges:
+            self.add_edge(edge)
+        self.edge_ids.update(correct_assertion.edge_ids)
+        self.labels.update(correct_assertion.labels)

src/research/COINs-KGGeneration/graph_analysis/KGist/evaluator.py ADDED Viewed

	@@ -0,0 +1,271 @@

+from math import log2 as log
+from scipy.special import comb
+from scipy.sparse import csr_matrix
+import numpy as np
+from collections import defaultdict
+from scipy.special import gammaln
+from graph_analysis.KGist.rule import Rule
+class Evaluator:
+    '''
+    Evaluates a model or a rule.
+    '''
+    def __init__(self, graph):
+        self.graph = graph
+        self.log_V = log(self.graph.n)
+        self.log_E = log(self.graph.m)
+        self.log_E_plus_1 = log(self.graph.m + 1)
+        self.log_labels_plus_1 = log(self.graph.total_num_labels + 1)
+        self.log_LV = log(self.graph.num_node_labels)
+        # a cache for L_N since this can be slow to do many times
+        self.length_natural_number_map = dict()
+        # a cache for log of binomials
+        self.binomial_map = dict()
+        self.rule_to_length = dict()
+    def evaluate(self, model, with_lengths=False):
+        '''
+        L(M) + L(G|M) - (Section 3.2)
+        '''
+        length_model = self.length_model(model)
+        length_error, neg_edge, neg_node = self.length_graph_with_model(model)
+        val = length_model + length_error
+        if with_lengths:
+            return val, length_model, neg_edge, neg_node
+        return val
+    def evaluate_change(self, model, rule, prev_model_len):
+        '''
+        L(M union {r}) + L(G|M union {r})
+        :return: the MDL objective after inserting a new rule to M
+        '''
+        # update score
+        neg_edge = self.length_negative_edge_error(model)
+        neg_node = self.length_negative_label_error(model)
+        length_error_with_rule = neg_edge + neg_node
+        length_model_with_rule = self.length_model_new_rule(model, rule, prev_model_len)
+        val = length_model_with_rule + length_error_with_rule
+        return val, length_model_with_rule, neg_edge, neg_node
+    def length_model_new_rule(self, model, rule, length_model):
+        '''
+        L(M union {r})
+        :return: the length of the model with a new rule added
+        '''
+        # old length
+        length = length_model
+        length += self.length_rule(rule) + self.length_rule_assertions(rule, model)
+        return length
+    def length_natural_number(self, n):
+        '''
+        :n: a number
+        :return: the number of bits required to transmit the number
+        '''
+        if n <= 0:
+            return 0
+        if n in self.length_natural_number_map:
+            return self.length_natural_number_map[n]
+        c = log(2.865064)
+        i = log(n)
+        while i > 0:
+            c = c + i
+            i = log(i)
+        self.length_natural_number_map[n] = c # cache the value
+        return c
+    def length_binomial(self, n, k):
+        '''
+        :n: n in (n choose k)
+        :k: k in (n choose k)
+        :return: log_2(n choose k)
+        '''
+        # we cache computations that we've already made for speedups
+        if (n, k) in self.binomial_map:
+            return self.binomial_map[(n, k)]
+        # gammln computation of log_e(n choose k) with change of base to log_2
+        length = (gammaln(n + 1) - gammaln(k + 1) - gammaln((n + 1) - k)) / np.log(2)
+        # cache the result for future use
+        self.binomial_map[(n, k)] = length
+        return length
+    def length_model(self, model):
+        '''
+        L(M) - (Section 3.2.1)
+        :model: a list of rules and their (partial-)matches
+        :return: the number of bits required to transmit the model
+        '''
+        rules = model.rules.keys()
+        # num rules
+        length = log(2 * self.graph.num_node_labels * self.graph.num_edge_labels * self.graph.num_node_labels + 1)
+        # rules
+        length += sum(self.length_rule(rule) + self.length_rule_assertions(rule, model) for rule in rules)
+        return length
+    def length_rule(self, rule):
+        '''
+        L(g) - (Section 3.2.1)
+        :rule: (parent, children)
+            - recusrive case
+                - parent: a set of node labels
+                - children: a set of elements like (edge_type, dir, rule)
+            - base case (parent is a leaf)
+                - parent: a set of node labels
+                - children: an empty set
+        :return: the number of bits required to transmit a rule
+        '''
+        if type(rule) is not tuple:
+            '''
+            Level 1+ rules.
+            '''
+            # node labels
+            length = self.log_LV
+            length += sum(-log(self.graph.node_label_counts[label] / self.graph.n) for label in rule.root)
+            # num branches
+            length += self.length_natural_number(len(rule.children) + 1)
+            for child in rule.children:
+                edge_type, _, child = child
+                # edge dir
+                length += 1
+                # edge label
+                length += -log(self.graph.edge_label_counts[edge_type] / self.graph.m)
+                # children
+                length += self.length_rule(child)
+            return length
+        parent, children = rule
+        # parent labels
+        length = self.log_LV
+        length += sum(-log(self.graph.node_label_counts[label] / self.graph.n) for label in parent)
+        # num branches
+        length += self.length_natural_number(len(children) + 1)
+        for child in children:
+            edge_type, _, rule_prime = child
+            # edge type
+            length += -log(self.graph.edge_label_counts[edge_type] / self.graph.m)
+            # edge dir
+            length += 1
+            # visit child
+            length += self.length_rule(rule_prime)
+        return length
+    def length_rule_assertions(self, rule, model, correct_assertions=None, info=False):
+        '''
+        L(alpha(g)) - (Section 3.2.1)
+        '''
+        if type(rule) is not tuple:
+            '''
+            Level 1+ rules.
+            '''
+            if rule.tuplify() in self.rule_to_length:
+                return self.rule_to_length[rule.tuplify()]
+            # num assertions
+            num_assertions = self.graph.nodes_with_type(rule.root)
+            # num exceptions
+            num_correct = len(rule.correct_assertions)
+            num_exceptions = num_assertions - num_correct
+            length = log(num_assertions)
+            # exception ids
+            length += self.length_binomial(num_assertions, num_exceptions)
+            cost_of_exceptions = length
+            # correct assertions
+            def length_correct_assertion(real, real_node, _rule, depth=1):
+                '''
+                :real: a CorrectAssertion, which we descend recusrively to encode
+                :_rule: a Rule, of which real is a relization
+                '''
+                _length = 0
+                # for r' in chi(r)
+                for pred, dir, child in _rule.children:
+                    # num spokes
+                    _length += self.log_V
+                    # spoke ids
+                    num_assertions = len(real.nodes[real_node].neighbors_of_type[(pred, dir, child.root)])
+                    _length += self.length_binomial(self.graph.n - 1, num_assertions)
+                    # visit children
+                    for real_child in real.nodes[real_node].neighbors_of_type[(pred, dir, child.root)]:
+                        _length += length_correct_assertion(real, real_child, child, depth + 1)
+                return _length
+            for real in rule.correct_assertions: # each correct_assertion is a tree
+                length += length_correct_assertion(real, real.root, rule)
+            self.rule_to_length[rule.tuplify()] = length
+            if info:
+                # length, length of excpetions, length of cas
+                return length, cost_of_exceptions, length - cost_of_exceptions
+            return length
+        if rule in self.rule_to_length:
+            return self.rule_to_length[rule]
+        # correct assertions
+        correct_assertions = model.rules[rule] if model else correct_assertions
+        # num assertions
+        num_assertions = self.graph.nodes_with_type(rule[0])
+        # num exceptions
+        num_correct = len(correct_assertions)
+        num_exceptions = num_assertions - num_correct
+        length = log(num_assertions)
+        # exception ids
+        length += self.length_binomial(num_assertions, num_exceptions)
+        cost_of_exceptions = length
+        def length_rule_tree(rule, rho):
+            # num_to_choose_from = self.graph.nodes_with_type(rule[1][0][2][0])
+            # assert(num_to_choose_from >= rho)
+            return self.log_V + self.length_binomial(self.graph.n - 1, rho)
+        # correct assertions
+        for ca in correct_assertions:
+            # size correct assertion
+            length += length_rule_tree(rule, ca)
+        self.rule_to_length[rule] = length
+        if info:
+            return length, cost_of_exceptions, length - cost_of_exceptions
+        return length
+    def length_graph_with_model(self, model):
+        '''
+        L(G|M) - (Section 3.2.2)
+        '''
+        negative_edge_error = self.length_negative_edge_error(model)
+        negative_node_error = self.length_negative_label_error(model)
+        length = negative_edge_error + negative_node_error
+        return length, negative_edge_error, negative_node_error
+    def length_negative_edge_error(self, model):
+        '''
+        L(A-) - (Section 3.2.2)
+        '''
+        # number of ones modeled
+        num_modeled = len(model.tensor)
+        # num ones
+        num_unexplained = self.graph.m - num_modeled
+        # ones
+        length = self.length_binomial((self.graph.n ** 2) * self.graph.num_edge_labels - num_modeled, num_unexplained)
+        return length
+    def length_negative_label_error(self, model):
+        '''
+        L(L-) - (Section 3.2.2)
+        '''
+        # number of ones modeled
+        num_modeled = len(model.label_matrix)
+        # num ones
+        num_unexplained = self.graph.total_num_labels - num_modeled
+        # ones
+        length = self.length_binomial(self.graph.num_node_labels * self.graph.n - num_modeled, num_unexplained)
+        return length

src/research/COINs-KGGeneration/graph_analysis/KGist/graph.py ADDED Viewed

	@@ -0,0 +1,188 @@

+import numpy as np
+from collections import defaultdict
+import os
+from itertools import chain, combinations
+from scipy.sparse import lil_matrix
+import itertools
+import sys
+import _pickle as pickle
+class Graph:
+    '''
+    A Knowledge Graph representation which seeks to make the operations needed
+    for this method as efficient as possible.
+    '''
+    def __init__(self, name, ext='txt', delimiter=' ', idify=False, verbose=True, load_candidates_from_disk=False):
+        '''
+        :name: the name of the graph to load.
+        :ext: the file extension for the graph files.
+        :delimiter: the delimiter for the graph files.
+        :idify: if true, converts strings to ids
+        :verbose: whether or not to print simple stats (e.g., num nodes)
+        Assumes:
+        1) ../data/{name}.{ext} edgelist.
+        2) ../data/{name}_labels.{ext} node label mapping.
+        e.g., ../data/dbpedia.txt and ../data/dbpedia_labels.txt
+        '''
+        edgelist = f"graph_analysis/KGist/data/{name}.txt"
+        label_map = f"graph_analysis/KGist/data/{name}_labels.txt"
+        self.name = name
+        self.node_list = set()
+        self.node_to_labels = dict()
+        self.node_label_counts = defaultdict(int)
+        self.edge_label_counts = defaultdict(int)
+        self.idify = idify
+        # load the graph
+        self.load(edgelist, label_map, delimiter, verbose, load_candidates_from_disk)
+        self.num_node_labels = len(self.label_matrix)
+        if verbose:
+            print('|V| = {}'.format(self.n))
+            print('|E| = {}'.format(self.m))
+            print('|L_V| = {}'.format(self.num_node_labels))
+            print('|L_E| = {}'.format(self.num_edge_labels))
+    def load(self, edgelist, label_map, delimiter, verbose, load_candidates_from_disk):
+        '''
+        Loads a knowledge graph.
+        :edgelist: a path to the graph edgelist.
+        :label_map: a path to the node label mapping.
+        :delimiter: the delimiter used in the above files.
+        '''
+        self.label_matrix = dict()
+        if self.idify:
+            nid = 0
+            eid = 0
+            lid = 0
+            self.id_to_pred = dict()
+            self.pred_to_id = dict()
+            self.id_to_node = dict()
+            self.node_to_id = dict()
+            self.label_to_id = dict()
+            self.id_to_label = dict()
+        # read node label mapping
+        with open(label_map, mode="r", encoding="utf-8") as f:
+            for line in f.read().split("\n"):
+                line = line.strip().split(delimiter)
+                node = line[0]
+                if self.idify:
+                    self.id_to_node[nid] = node
+                    self.node_to_id[node] = nid
+                    node = nid
+                    nid += 1
+                self.node_list.add(node)
+                labels = tuple(line[1:])
+                self.node_to_labels[node] = labels
+                for label in labels:
+                    if self.idify:
+                        if label not in self.label_to_id:
+                            self.label_to_id[label] = lid
+                            self.id_to_label[lid] = label
+                            lid += 1
+                        label = self.label_to_id[label]
+                    self.node_label_counts[label] += 1
+                    if label not in self.label_matrix:
+                        self.label_matrix[label] = set()
+                    self.label_matrix[label].add(node)
+                if self.idify:
+                    self.node_to_labels[node] = tuple(self.label_to_id[label] for label in line[1:])
+        if verbose:
+            print('Node labels loaded.')
+        # candidates map to matches
+        self.candidates = dict()
+        self.m = 0
+        self.id_to_edge = dict()
+        # read edgelist
+        edge_labels = set()
+        with open(edgelist, mode="r", encoding="utf-8") as f:
+            self.tensor = set()
+            for line in f.read().split("\n"):
+                if len(line.strip().split(delimiter)) != 3:
+                    continue
+                sub, pred, obj = line.strip().split(delimiter)
+                edge_labels.add(pred)
+                if self.idify:
+                    if sub not in self.node_to_id:
+                        self.node_to_id[sub] = nid
+                        self.id_to_node[nid] = sub
+                        nid += 1
+                    sub = self.node_to_id[sub]
+                    if obj not in self.node_to_id:
+                        self.node_to_id[obj] = nid
+                        self.id_to_node[nid] = obj
+                        nid += 1
+                    obj = self.node_to_id[obj]
+                    if pred not in self.pred_to_id:
+                        self.pred_to_id[pred] = eid
+                        self.id_to_pred[eid] = pred
+                        eid += 1
+                    pred = self.pred_to_id[pred]
+                self.edge_label_counts[pred] += 1
+                self.node_list.add(sub)
+                self.node_list.add(obj)
+                self.tensor.add(self.m)
+                if not load_candidates_from_disk:
+                    # candidates
+                    sub_labels = self.labels(sub)
+                    # obj labels
+                    obj_labels = self.labels(obj)
+                    sls_ols = list(itertools.product(sub_labels, obj_labels))
+                    for sl, ol in sls_ols:
+                        if ((sl,), ((pred, 'out', ((ol,), ())),)) not in self.candidates:
+                            self.candidates[((sl,), ((pred, 'out', ((ol,), ())),))] = {'label_coverage': set(),
+                                                                                'edges': set(),
+                                                                                'ca_to_size': defaultdict(int)}
+                        self.candidates[((sl,), ((pred, 'out', ((ol,), ())),))]['label_coverage'].add((ol, obj))
+                        self.candidates[((sl,), ((pred, 'out', ((ol,), ())),))]['edges'].add(self.m)
+                        self.candidates[((sl,), ((pred, 'out', ((ol,), ())),))]['ca_to_size'][sub] += 1
+                        if ((ol,), ((pred, 'in', ((sl,), ())),)) not in self.candidates:
+                            self.candidates[((ol,), ((pred, 'in', ((sl,), ())),))] = {'label_coverage': set(),
+                                                                               'edges': set(),
+                                                                               'ca_to_size': defaultdict(int)}
+                        self.candidates[((ol,), ((pred, 'in', ((sl,), ())),))]['label_coverage'].add((sl, sub))
+                        self.candidates[((ol,), ((pred, 'in', ((sl,), ())),))]['edges'].add(self.m)
+                        self.candidates[((ol,), ((pred, 'in', ((sl,), ())),))]['ca_to_size'][obj] += 1
+                self.id_to_edge[self.m] = (sub, pred, obj)
+                self.m += 1
+                if verbose and self.m % 10000 == 0:
+                    sys.stdout.write('\r{}'.format(self.m))
+                    sys.stdout.flush()
+        typ_to_nodes = dict()
+        for candidate, info in self.candidates.items():
+            if candidate[0] not in typ_to_nodes:
+                typ_to_nodes[candidate[0]] = set((candidate[0][0], node) for node in self.nodes_with_type(candidate[0], num_only=False))
+            # info['label_coverage'].update(typ_to_nodes[candidate[0]])
+        self.node_list = list(self.node_list)
+        self.num_edge_labels = len(edge_labels)
+        self.total_num_labels = 0
+        for label, nodes in self.label_matrix.items():
+            self.total_num_labels += len(nodes)
+        self.n = len(self.node_list)
+        if load_candidates_from_disk:
+            with open('../data/{}_candidates.pickle'.format(self.name), 'rb') as f:
+                self.candidates = pickle.load(f)
+        if verbose:
+            print()
+    def nodes(self):
+        return self.node_list
+    def labels(self, node):
+        return self.node_to_labels[node]
+    def nodes_with_type(self, typ, num_only=True):
+        if len(typ) == 1:
+            return len(self.label_matrix[typ[0]]) if num_only else self.label_matrix[typ[0]]
+        return len(set.intersection(*list(self.label_matrix[label] for label in typ))) if num_only else set.intersection(*list(self.label_matrix[label] for label in typ))
+    def tuplify(self, rule):
+        if self.idify:
+            return (tuple(self.id_to_label[label] for label in rule[0]), tuple((self.id_to_pred[child[0]], child[1], self.tuplify(child[2])) for child in rule[1]))
+        return (rule[0], tuple((child[0], child[1], self.tuplify(child[2])) for child in rule[1]))

src/research/COINs-KGGeneration/graph_analysis/KGist/main.py ADDED Viewed

	@@ -0,0 +1,86 @@

+import argparse
+from graph_analysis.KGist.graph import Graph
+from graph_analysis.KGist.model import Model
+from graph_analysis.KGist.searcher import Searcher
+from graph_analysis.KGist.evaluator import Evaluator
+def parse_args():
+    def str2bool(v):
+        if v.lower() in ('yes', 'true', 't', 'y', '1'):
+            return True
+        elif v.lower() in ('no', 'false', 'f', 'n', '0'):
+            return False
+        else:
+            raise argparse.ArgumentTypeError('Boolean value expected.')
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--graph',
+                        '-g',
+                        type=str,
+                        required=True,
+                        help='The name of graph: nell or dbpedia if using our parsed data.')
+    parser.add_argument('--rule_merging',
+                        '-Rm',
+                        type=str2bool,
+                        default=False,
+                        required=False,
+                        help='If True, then run merging refinement (Section 4.2.2).')
+    parser.add_argument('--rule_nesting',
+                        '-Rn',
+                        type=str2bool,
+                        default=False,
+                        required=False,
+                        help='If True, then run nesting refinement (Section 4.2.2).')
+    parser.add_argument('--idify',
+                        '-i',
+                        type=str2bool,
+                        default=True,
+                        required=False,
+                        help='If True, then convert entities to integer ids for faster processing.')
+    parser.add_argument('--verbosity',
+                        '-v',
+                        type=int,
+                        default=1000000,
+                        required=False,
+                        help='How often to print output. If 0, then silence output.')
+    parser.add_argument('--output_path',
+                        '-o',
+                        type=str,
+                        default='../output/',
+                        help='path for output and log files')
+    return parser.parse_args()
+def main(args):
+    graph = Graph(args.graph, idify=args.idify, verbose=args.verbosity > 0)
+    if args.verbosity > 0:
+        print('Graph loaded.')
+    searcher = Searcher(graph)
+    if args.verbosity > 0:
+        print('Creating model.')
+    model = searcher.build_model(verbosity=args.verbosity,
+                                 passes=2,
+                                 label_qualify=True,
+                                 order=['mdl_err', 'coverage', 'lex'])
+    if args.verbosity > 0:
+        print('***** Initial model *****')
+        model.print_stats()
+        model.save('{}{}_model'.format(args.output_path, args.graph))
+    if args.rule_merging:
+        model = model.merge_rules(verbosity=args.verbosity)
+        if args.verbosity > 0:
+            print('***** Model refined with Rm *****')
+            model.print_stats()
+            model.save('{}{}_model_Rm'.format(args.output_path, args.graph))
+    if args.rule_nesting:
+        model = model.nest_rules(verbosity=args.verbosity)
+        if args.verbosity:
+            print('***** Model refined with Rn *****')
+            model.print_stats()
+            model.save('{}{}_model_Rm_Rn'.format(args.output_path, args.graph))
+if __name__ == "__main__":
+    args = parse_args()
+    main(args)

src/research/COINs-KGGeneration/graph_analysis/KGist/model.py ADDED Viewed

	@@ -0,0 +1,388 @@

+from collections import defaultdict
+from graph_analysis.KGist.evaluator import Evaluator
+from itertools import chain, combinations
+import os
+import sys
+import json
+import networkx as nx
+from graph_analysis.KGist.rule import Rule
+from graph_analysis.KGist.correct_assertion import CorrectAssertion
+from copy import copy, deepcopy
+import _pickle as pickle
+import random
+class Model:
+    '''
+    A model consisting of rules which explain a knowledge graph.
+    '''
+    def __init__(self, graph):
+        '''
+        :graph: the knowledge graph being modeled
+        '''
+        self.graph = graph
+        # rules stored mapping to matches
+        self.rules = dict()
+        self.node_label_counts = defaultdict(int)
+        self.tensor = set()
+        self.label_matrix = set()
+        self.cache = {'last_updated_rule': None}
+        self.rule_graph = None
+        self.shared_root_rule_dependency_graph = None
+        self.subject_to_rules = dict()
+    def add_rule(self, rule, ghost=False):
+        '''
+        :rule: the rule to be added
+        :ghost: (**ONLY MEANT FOR TESTING**) parameter that can be set to true during testing to not add the assertions of a rule.
+        '''
+        # store rule
+        if rule in self.rules:
+            print('Already added')
+            return
+        if ghost:
+            self.rules[rule] = 'ghost'
+            return
+        if type(rule) is not tuple:
+            self.rules[rule] = 'level1'
+            if rule.root not in self.subject_to_rules:
+                self.subject_to_rules[rule.root] = set()
+            self.subject_to_rules[rule.root].add(rule)
+        else:
+            self.rules[rule] = list(self.graph.candidates[rule]['ca_to_size'].values())
+            if rule[0] not in self.subject_to_rules:
+                self.subject_to_rules[rule[0]] = set()
+            self.subject_to_rules[rule[0]].add(rule)
+        self.make_assertions(rule)
+    def remove_rule(self, rule):
+        '''
+        :rule: the rule to be removed
+        '''
+        if rule != self.cache['last_updated_rule']:
+            print('We can only remove the last added rule.')
+            return
+        if rule not in self.rules: # make sure the rule is actually there
+            return
+        # remove rule
+        del self.rules[rule]
+        if type(rule) is not tuple:
+            self.subject_to_rules[rule.root].remove(rule)
+            if len(self.subject_to_rules[rule.root]) == 0:
+                del self.subject_to_rules[rule.root]
+        else:
+            self.subject_to_rules[rule[0]].remove(rule)
+            if len(self.subject_to_rules[rule[0]]) == 0:
+                del self.subject_to_rules[rule[0]]
+        self.undo_assertions(rule)
+    def make_assertions(self, rule):
+        '''
+        Fills in model's tensor and node label map with assertions of a rule
+        :rule: a rule
+        '''
+        # reset cache
+        self.cache = {'last_updated_rule': rule}
+        # update cache
+        if type(rule) is not tuple:
+            self.cache['new_edges'] = rule.get_edges_covered().difference(self.tensor)
+            self.cache['new_labels'] = rule.get_labels_covered().difference(self.label_matrix)
+        else:
+            self.cache['new_edges'] = self.graph.candidates[rule]['edges'].difference(self.tensor)
+            self.cache['new_labels'] = self.graph.candidates[rule]['label_coverage'].difference(self.label_matrix)
+        self.tensor.update(self.cache['new_edges'])
+        self.label_matrix.update(self.cache['new_labels'])
+    def undo_assertions(self, rule):
+        '''
+        Removes things from model's tensor and node label map
+        :rule: a rule
+        '''
+        self.tensor.difference_update(self.cache['new_edges'])
+        self.label_matrix.difference_update(self.cache['new_labels'])
+    def build_rule_graph(self, build_prime=False, force=False):
+        '''
+        Builds a dependency graph.
+            - Nodes are rules
+            - (r1, r2) means that r1's tail is r2's head and denotes a possible composition
+        :return: a dependency graph of rules connected by their possible compositions
+        '''
+        if not force and self.rule_graph:
+            return
+        root_to_rule = defaultdict(list)
+        # find dependencies and store as edges for a dependency graph
+        edges = list()
+        tree_rules = list()
+        for rule in self.rules:
+            if type(rule) is tuple:
+                parent, children = rule
+                rule = Rule(parent, children)
+            self.plant_forest(rule)
+            root_to_rule[rule.root].append(rule)
+            tree_rules.append(rule)
+        if not build_prime:
+            # build graph
+            for rule in tree_rules:
+                # if leaf matches other rules' roots
+                if len(set(rule.get_leaves()).intersection(set(root_to_rule.keys()))) > 0:
+                    # get the matching rules
+                    matching_rules = set()
+                    for leaf in rule.get_leaves():
+                        if leaf in root_to_rule:
+                            matching_rules.update(root_to_rule[leaf])
+                    for other_rule in matching_rules:
+                        if rule.root in other_rule.get_leaves(): # don't allow loops
+                            continue
+                        edges.append((rule, other_rule))
+            self.rule_graph = nx.DiGraph(edges)
+        if build_prime:
+            def jaccard_sim(r1, r2):
+                a = set(real.root for real in r1.correct_assertions)
+                b = set(real.root for real in r2.correct_assertions)
+                return len(a.intersection(b)) / len(a.union(b)) if len(a.union(b)) > 0 else 0
+            # build rule graph prime, which encodes dependencies of shared root types
+            edges = list()
+            for rule in tree_rules:
+                for other_rule in root_to_rule[rule.root]:
+                    if rule != other_rule: #(other_rule, rule) not in edges:
+                        self.plant_forest(rule)
+                        self.plant_forest(other_rule)
+                        if jaccard_sim(rule, other_rule) == 1.0:
+                            edges.append((rule, other_rule))
+            self.shared_root_rule_dependency_graph = nx.Graph(edges)
+    def pickle_copy(self, obj):
+        name = '{}'.format(random.randint(0, 10000000))
+        with open('{}.pickle'.format(name), 'wb') as handle:
+            pickle.dump(obj, handle)
+        with open('{}.pickle'.format(name), 'rb') as handle:
+            new_obj = pickle.load(handle)
+        os.remove('{}.pickle'.format(name))
+        return new_obj
+    def merge_rules(self, verbosity=1000000):
+        '''
+        Perfrom merging refinement (Rm).
+        :return: A new Model object with Rm applied to self.
+        '''
+        self.build_rule_graph(build_prime=True)
+        merged = set()
+        merged_model = Model(self.graph)
+        # find sets of rules that all have the same root (i.e., cliques in the dependency graph)
+        cliques = list(nx.find_cliques(self.shared_root_rule_dependency_graph))
+        n = len(cliques)
+        i = 0
+        for i, clique in enumerate(cliques):
+            # create a new rule composed that will consist of those in the clique merged
+            new_rule = self.pickle_copy(clique[0])
+            for r2 in clique[1:]:
+                new_rule.merge(r2)
+            merged_model.add_rule(new_rule)
+            if verbosity > 0 and i > 0 and i % 100 == 0:
+                print('{}% of candidate merges tested.'.format(round(i / n * 100, 2)))
+        if verbosity > 0:
+            print('{}% of candidate merges tested.'.format(round(i / n * 100, 2) if n > 0 else 100))
+        covered = set(node.tuplify() for node in self.shared_root_rule_dependency_graph.nodes())
+        for rule in self.rules.keys():
+            if rule not in covered:
+                merged_model.add_rule(rule)
+        return merged_model
+    def plant_forest(self, rule):
+        '''
+        Grow a forest of correct_assertions for a level-0 rule
+        '''
+        if rule.instantiated():
+            return
+        leaves = list()
+        roots = set(self.graph.candidates[rule.tuplify()]['ca_to_size'].keys())
+        # maps roots to the correct_assertions where they are the root
+        roots_to_correct_assertions = dict()
+        # iterate over edges explained by rule and construct stars
+        for eid in self.graph.candidates[rule.tuplify()]['edges']:
+            sub, pred, obj = self.graph.id_to_edge[eid]
+            dir = rule.children[0][1]
+            # decide which is root and spoke
+            root = sub if dir == 'out' else obj
+            spoke = obj if dir == 'out' else sub
+            u, v = root, spoke
+            # create a rule correct_assertion for the root
+            if root not in roots_to_correct_assertions:
+                roots_to_correct_assertions[root] = CorrectAssertion(root, label=rule.root)
+            root = roots_to_correct_assertions[root]
+            # add edge (u, u_typ, pred, dir, v, v_typ)
+            edge = (u, rule.root, pred, dir, v, rule.children[0][2].root)
+            root.add_edge(edge, eid=eid, labels=True)
+        # iterate over each star (rule correct_assertion), and add to the rule (at the root level)
+        for root, correct_assertion in roots_to_correct_assertions.items():
+            rule.insert_correct_assertion(correct_assertion)
+    def nest_rules(self, verbosity=1000000):
+        '''
+        Perfrom nesting refinement (Rn).
+        :return: A new Model object with Rn applied to self.
+        '''
+        def powerset(iterable):
+            s = list(iterable)
+            return set(chain.from_iterable(combinations(s, r) for r in range(1, len(s) + 1)))
+        self.build_rule_graph()
+        def compose(r1, r2):
+            '''
+            :r1: a Rule
+            :r2: a Rule
+            '''
+            composed_rule = self.pickle_copy(r1)
+            assert(composed_rule.pin_to_leaf(r2))
+            composed_rule.filter_errant()
+            return composed_rule
+        evaluator = Evaluator(self.graph)
+        null_val = evaluator.evaluate(Model(self.graph)) # for log
+        best_model = self
+        last_model = None
+        best_val = evaluator.evaluate(best_model)
+        _graph = nx.DiGraph(self.rule_graph.edges())
+        seen_pairs = set()
+        def checkable(r):
+            return r if type(r) is tuple else r.tuplify()
+        edge_to_jaccard_sim = dict()
+        for edge in _graph.edges():
+            edge_to_jaccard_sim[edge] = edge[0].jaccard_sim(edge[1])
+        num_nested = 0
+        while last_model != best_model:
+            num_rules = len(best_model.rules)
+            rules = set(best_model.rules.keys())
+            last_model = best_model
+            pairs = _graph.edges()
+            pairs = sorted(pairs, key=lambda edge: [edge[0].tuplify()] + [edge[1].tuplify()])
+            pairs = sorted(pairs, reverse=True, key=lambda edge: edge_to_jaccard_sim[edge])
+            pair_num = 0
+            for r1, r2 in pairs:
+                pair_num += 1
+                if (r1, r2) in seen_pairs or checkable(r1) == checkable(r2):
+                    continue
+                seen_pairs.add((r1, r2))
+                # pair is two rules that are candidates for composition
+                model_to_test = Model(self.graph)
+                candidate = compose(r1, r2)
+                if len(candidate.correct_assertions) == 0:
+                    continue
+                approx_L = best_val - evaluator.length_rule(r1) - evaluator.length_rule_assertions(r1, best_model) - evaluator.length_rule(r2) - evaluator.length_rule_assertions(r2, best_model) + evaluator.length_rule(candidate) + evaluator.length_rule_assertions(candidate, best_model)
+                if approx_L >= best_val:
+                    continue
+                for rule in rules:
+                    if checkable(rule) != checkable(r1) and checkable(rule) != checkable(r2):
+                        model_to_test.add_rule(rule)
+                model_to_test.add_rule(candidate)
+                # evaluate the possible new model
+                new_val = evaluator.evaluate(model_to_test)
+                if new_val < best_val:
+                    num_nested += 1
+                    best_model = model_to_test
+                    best_val = new_val
+                    before_n = _graph.number_of_nodes()
+                    before_m = _graph.number_of_edges()
+                    _graph = nx.algorithms.minors.contracted_nodes(_graph, r1, r2, self_loops=False)
+                    assert(_graph.number_of_nodes() == before_n - 1)
+                    _graph = nx.relabel_nodes(_graph, {r1: candidate})
+                    for edge in list(_graph.out_edges(candidate)) + list(_graph.in_edges(candidate)):
+                        if edge not in edge_to_jaccard_sim:
+                            edge_to_jaccard_sim[edge] = edge[0].jaccard_sim(edge[1])
+                    to_remove = list()
+                    # not all are legal
+                    for comp, y in _graph.out_edges(candidate): # edges where comp's inner match y's root
+                        assert(comp == candidate)
+                        if y.root not in comp.get_inner_nodes():
+                            to_remove.append((comp, y))
+                    for y, comp in _graph.in_edges(candidate): # edges where y's inner match comp's root
+                        assert(comp == candidate)
+                        if comp.root not in y.get_inner_nodes():
+                            to_remove.append((y, comp))
+                    _graph.remove_edges_from(to_remove)
+                    assert(_graph.number_of_nodes() == before_n - 1)
+                    break
+                if verbosity > 0 and pair_num % 100 == 0:
+                    print('{}% of candidate compositions processed.'.format(round((pair_num / len(pairs)) * 100, 2)))
+        if verbosity > 0:
+            print('{}% of candidate compositions processed.'.format(round((pair_num / len(pairs)) * 100, 2)
+                                                                    if len(pairs) > 0 else 100))
+            print('{} nestings performed.'.format(num_nested))
+        return best_model
+    def save(self, fname):
+        '''
+        Saves the model in json format.
+        :path: path to a json file where model should be saved
+        '''
+        rules = list()
+        for rule in self.rules:
+            if type(rule) is tuple:
+                parent, children = rule
+                rule = Rule(parent, children)
+            self.plant_forest(rule)
+            rules.append(rule)
+        ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
+        pickle_outfile = os.path.join(ROOT_DIR, fname + '.pickle')
+        with open(pickle_outfile, 'wb') as handle:
+            pickle.dump(self, handle)
+        rule_outfile = os.path.join(ROOT_DIR, fname + '.rules')
+        with open(rule_outfile, 'w') as f:
+            for rule in rules:
+                if self.graph.idify:
+                    f.write('{}\n'.format(rule.tuplify(id_to_node=self.graph.id_to_node, id_to_pred=self.graph.id_to_pred)))
+                else:
+                    f.write('{}\n'.format(rule.tuplify()))
+    def percent_improved(self):
+        evaluator = Evaluator(self.graph)
+        null_val = evaluator.evaluate(Model(self.graph))
+        val = evaluator.evaluate(self)
+        # the difference is what percent of the original?
+        return ((null_val - val) / null_val) * 100
+    def print_stats(self):
+        evaluator = Evaluator(self.graph)
+        val = evaluator.evaluate(self)
+        print('----- Model stats -----')
+        print('L(G,M) = {}'.format(round(val, 2)))
+        null_val = evaluator.evaluate(Model(self.graph))
+        print('% Bits needed: {}'.format(round((val / null_val) * 100, 2)))
+        print('# Rules: {}'.format(len(self.rules)))
+        print('% Edges Explained: {}'.format(round(len(self.tensor) / self.graph.m * 100, 2)))
+        print('-----------------------')
+    def get_stats(self):
+        evaluator = Evaluator(self.graph)
+        val = evaluator.evaluate(self)
+        null_val = evaluator.evaluate(Model(self.graph))
+        return val, val / null_val, len(self.rules), len(self.tensor) / self.graph.m

src/research/COINs-KGGeneration/graph_analysis/KGist/rule.py ADDED Viewed

	@@ -0,0 +1,263 @@

+from copy import deepcopy
+class Rule:
+    '''
+    A class which represents rules as trees.
+    Best used for rules after level 1.
+    '''
+    def __init__(self, root, children=None):
+        self.root = root
+        self.children = list()
+        # a list of trees that match the rule structure all the way down
+        #   - if empty, then the rule is not a root
+        self.correct_assertions = list()
+        if children:
+            for child in children:
+                self.insert(child)
+        # store the tuple form or rules for easy (ehh) debugging. Yay!
+        self.tuple_rules = [self.tuplify()]
+        self.correct_at_depth = dict()
+        self.assertions_at_depth = dict()
+    def has_children(self):
+        return len(self.children) > 0
+    def instantiated(self):
+        return len(self.correct_assertions) > 0
+    def insert(self, branch):
+        '''
+        Insert a branch into the rule.
+        :branch: (pred, dir, leaf)
+            - :pred: the edge type
+            - :dir: the direction of the edge
+            - :leaf: the node id
+        '''
+        if type(branch[2]) is tuple:
+            pred, dir, leaf = branch
+            leaf, _ = leaf
+            leaf = Rule(leaf)
+        else:
+            pred, dir, leaf = branch
+        self.children.append((pred, dir, leaf))
+    def insert_correct_assertion(self, correct_assertion):
+        '''
+        Adds a correct_assertion, which should only be used for roots.
+        :correct_assertion: a CorrectAssertion
+        '''
+        self.correct_assertions.append(correct_assertion)
+    def get_edges_covered(self):
+        covered = set()
+        for real in self.correct_assertions:
+            covered.update(real.edge_ids)
+        return covered
+    def get_labels_covered(self):
+        covered = set()
+        for real in self.correct_assertions:
+            covered.update(real.labels)
+        return covered
+    def merge(self, other_rule):
+        '''
+        Merge two rules with the same roots. For now we assume the correct assertions are identical, which is true a shocking amount of the time.
+        '''
+        assert(other_rule.root == self.root)
+        self.children.extend(other_rule.children)
+        self.correct_assertions = sorted(self.correct_assertions, key=lambda real: real.root)
+        other_rule.correct_assertions = sorted(other_rule.correct_assertions, key=lambda real: real.root)
+        for real, other_real in zip(self.correct_assertions, other_rule.correct_assertions):
+            real.merge(other_real)
+        return True
+    def pin_to_leaf(self, leaf):
+        '''
+        Pins a Rule to a leaf, and does likewise for the rule correct_assertions.
+        :leaf: a Rule object whose root is the label of the leaf to which we should pin it.
+        :return: True if leaf successfully pinned, False otherwise
+        '''
+        found = False
+        # search children for a matching leaf
+        for i, child in enumerate(self.children):
+            pred, dir, child = child
+            if child.root == leaf.root: # pin here
+                if self.instantiated():
+                    for leaf_real in leaf.correct_assertions: # append each leaf correct_assertion
+                        for real in self.correct_assertions: # pin to each correct_assertion as well, if appropriate
+                            if leaf_real.root in real.nodes:
+                                real.compose(leaf_real)
+                self.children[i][2].children.extend(leaf.children)
+                found = True
+        if not found: # try children recursively
+            for child in self.children:
+                found = child[2].pin_to_leaf(leaf) or found
+            return found
+        else:
+            return found
+    def get_matching_nodes(self, label):
+        matches = set()
+        frontier = [self]
+        while len(frontier) > 0:
+            node = frontier.pop()
+            if node.root == label:
+                matches.add(node)
+            for _, _, child in node.children:
+                frontier.insert(0, child)
+        return matches
+    def get_leaves(self):
+        leaves = list()
+        def add_leaf(r):
+            if not r.has_children():
+                leaves.append(r.root)
+            else:
+                for edge_type, dir, child in r.children:
+                    add_leaf(child)
+        add_leaf(self)
+        return leaves
+    def get_inner_nodes(self):
+        inner_nodes = set()
+        for _, _, child in self.children:
+            inner_nodes.add(child.root)
+            inner_nodes.update(child.get_inner_nodes())
+        return inner_nodes
+    def jaccard_sim(self, other_rule):
+        leaves = set()
+        for real in self.correct_assertions:
+            leaves.update(set(real.nodes.keys()).difference({real.root}))
+        roots = set(real.root for real in other_rule.correct_assertions)
+        return len(leaves.intersection(roots)) / len(leaves.union(roots)) if len(leaves.union(roots)) > 0 else 0
+    def tuplify(self, id_to_node=None, id_to_pred=None):
+        if id_to_node and id_to_pred:
+            return (tuple(id_to_node[label] for label in self.root), tuple((id_to_pred[child[0]], child[1], child[2].tuplify(id_to_node, id_to_pred)) for child in self.children))
+        return (self.root, tuple((child[0], child[1], child[2].tuplify()) for child in self.children))
+    def get_atoms(self):
+        atoms = list()
+        for child in self.children:
+            atoms.append((self.root, tuple(((child[0], child[1], (child[2].root, ())),))))
+            atoms.extend(child[2].get_atoms())
+        return atoms
+    def get_preds(self):
+        preds = set()
+        for pred, dir, child in self.children:
+            preds.add(pred)
+            preds.update(child.get_preds())
+        return preds
+    def max_depth(self):
+        if not self.has_children():
+            return 0
+        return 1 + max(child[2].max_depth() for child in self.children)
+    def min_depth(self):
+        if not self.has_children():
+            return 0
+        return 1 + min(child[2].min_depth() for child in self.children)
+    def get_correct(self, with_removed=False):
+        '''
+        Correct means that at every level, the correct_assertions align with the rule structure.
+        :return: a list of relization trees that are correct all the way down.
+        '''
+        def is_correct(real, real_node, node):
+            '''
+            :real: a correct_assertion
+            :node: a rule node
+            :return: true if the correct_assertion is correct from this node on
+            '''
+            # the types of children expected of the correct_assertion
+            expected_children = list((child[0], child[1], child[2].root) for child in node.children)
+            correct = True # innocent until proven guilty
+            for pred, dir, child in expected_children:
+                if (pred, dir, child) not in real_node.neighbors_of_type:
+                    correct = False
+                    break
+            if correct:
+                # iterate over each child of the rule node
+                for pred, dir, child in node.children:
+                    for real_child in real_node.neighbors_of_type[(pred, dir, child.root)]:
+                        correct = is_correct(real, real.nodes[real_child], child)
+                        if not correct:
+                            break
+                    if not correct:
+                        break
+            return correct
+        return list(filter(lambda real: is_correct(real, real.nodes[real.root], self), self.correct_assertions))
+    def filter_errant(self):
+        '''
+        Filter out the correct_assertions that err at some level.
+        '''
+        # bfs
+        self.correct_assertions = self.get_correct()
+    def print(self):
+        frontier = [(None, None, self, 0)]
+        while len(frontier) > 0:
+            pred, dir, rule, depth = frontier.pop()
+            if depth == 0:
+                print('{}{}'.format(''.join(['\t'] * depth), rule.root))
+            else:
+                print('{}({},{},{})'.format(''.join(['\t'] * depth), pred, dir, rule.root))
+            for pred, dir, child in rule.children:
+                frontier.append((pred, dir, child, depth + 1))
+    def plot(self, save_path=None):
+        '''
+        Create a plot of a rule.
+        '''
+        from graphviz import Digraph
+        dot = Digraph()
+        root = self
+        frontier = [self]
+        edges = list()
+        while len(frontier) > 0:
+            node = frontier.pop()
+            if node == root:
+                dot.attr('node', shape='doublecircle')
+            else:
+                dot.attr('node', shape='circle')
+            node_name = node.root[0].split(':')[-1]
+            dot.node(node_name, node_name)
+            for pred, delta, child in node.children:
+                frontier.insert(0, child)
+                node_name = node.root[0].split(':')[-1]
+                child_name = child.root[0].split(':')[-1]
+                pred = pred.split(':')[-1]
+                if delta == 'out':
+                    edges.append((node_name, pred, child_name))
+                else:
+                    edges.append((child_name, pred, node_name))
+        for edge in edges:
+            dot.edge(edge[0], edge[2], label=' {}'.format(edge[1]))
+        save_path = save_path if save_path else '../data/temp/rule_fig'
+        dot.render(save_path, view=True)
+    def json_dump(self, outfile=None):
+        import json
+        data = {'root': self.root,
+                'children': list({'pred': pred, 'dir': dir, 'rule': child.json_dump()} for pred, dir, child in self.children)}
+        if outfile:
+            with open(outfile, 'w') as outfile:
+                    json.dump(data, outfile, indent=4)
+        return data

src/research/COINs-KGGeneration/graph_analysis/KGist/searcher.py ADDED Viewed

	@@ -0,0 +1,257 @@

+from graph_analysis.KGist.graph import Graph
+from graph_analysis.KGist.model import Model
+from graph_analysis.KGist.evaluator import Evaluator
+import argparse
+import numpy as np
+import itertools
+import functools
+import random
+from math import log2 as log
+import heapq
+from multiprocessing import Pool
+class Searcher:
+    '''
+    A class which searches for the optimal model.
+    '''
+    def __init__(self, graph):
+        self.graph = graph
+        self.evaluator = Evaluator(graph)
+        self.candidates = list(self.graph.candidates.keys())
+    def rank_rules(self):
+        '''
+        Rank the candidate rules - (Section 4.1.3)
+        '''
+        # L(G|M_0)
+        null_val = self.evaluator.length_graph_with_model(Model(self.graph))[0]
+        def reduction_in_error(g):
+            '''
+            :g: a rule
+            '''
+            _model = Model(self.graph)
+            _model.add_rule(g)
+            # compute L(G|M_0) - L(G|M \cup {g})
+            red_in_err = null_val - self.evaluator.length_graph_with_model(_model)[0]
+            return red_in_err
+        self.candidates = sorted(self.candidates,
+                                 reverse=True,
+                                 key=lambda g: (reduction_in_error(g), # reduction in error
+                                                len(self.graph.candidates[g]['ca_to_size']), # number of correct assertions
+                                                g[0])) # rule root labels
+    class BoundedMinHeap:
+        '''
+        A Min Heap that only allows :bound: items.
+        If :bound: is << len(items), then this can be more efficient for finding the top k than sorting the whole list.
+        '''
+        def __init__(self, bound, key=lambda it: it):
+            self.bound = bound
+            self.key = key
+            self._data = list()
+        def push(self, it):
+            if len(self._data) < self.bound:
+                heapq.heappush(self._data, (self.key(it), it))
+            else:
+                heapq.heappushpop(self._data, (self.key(it), it))
+        def get_reversed(self):
+            temp = list()
+            while len(self._data) > 0:
+                temp.append(heapq.heappop(self._data)[1])
+            return list(reversed(temp))
+    def build_model_top_k_freq(self, k):
+        '''
+        Build a model containing the k rules with the most correct assertions.
+        '''
+        model = Model(self.graph)
+        heap = Searcher.BoundedMinHeap(bound=k, key=lambda rule: len(self.graph.candidates[rule]['ca_to_size']))
+        for rule in self.candidates:
+            heap.push(rule)
+        for rule in heap.get_reversed():
+            model.add_rule(rule)
+        return model
+    def build_model_top_k_coverage(self, k):
+        '''
+        Build a model containing the k rules that explain the most edges.
+        '''
+        model = Model(self.graph)
+        heap = Searcher.BoundedMinHeap(bound=k, key=lambda rule: sum(list((self.graph.candidates[rule]['ca_to_size']).values())))
+        for rule in self.candidates:
+            heap.push(rule)
+        for rule in heap.get_reversed():
+            model.add_rule(rule)
+        return model
+    def check_qualify(self, evaluator, verbosity, rule_and_new_labels):
+        '''
+        Check whether adding more labels to a rule (qualifying it) leads to improvements in MDL terms.
+        :evaluator: an Evaluator object with which to compute MDL scores
+        :verbosity: how much to print to the log
+        :rule_and_new_labels: (rule, the rule's new labels if qualified)
+        :return: True if the rule is qualified, False if not.
+        '''
+        rule, new_labels = rule_and_new_labels
+        old_rule = rule
+        root, children = rule
+        rule = (new_labels, children)
+        # create M_0
+        model = Model(self.graph)
+        # create M_0 \cup {g without qualification}
+        model.add_rule(old_rule)
+        # compute L(G, M_0 \cup {g without qualification})
+        # (Note: this computation works because the newly added labels are contained by all correct assertion starting points, so L(G|M) does not change)
+        cost_without_qualification = evaluator.length_rule(old_rule) + evaluator.length_rule_assertions(old_rule, model)
+        # replace the rule with the qualified version to make M_0 \cup {g with qualification}
+        model.rules[rule] = model.rules[old_rule]
+        del model.rules[old_rule]
+        # compute L(G, M_0 \cup {g with qualification})
+        cost_with_qualification = evaluator.length_rule(rule) + evaluator.length_rule_assertions(rule, model)
+        qualified = False
+        # if the cost went down, then keep the qualifiers
+        if cost_with_qualification < cost_without_qualification:
+            # update the data structures so that the new, qualified rules has the same correct assertions as the old, unqualified rule
+            self.graph.candidates[rule] = self.graph.candidates[old_rule]
+            # delete the data on the old, unqualified rule
+            del self.graph.candidates[old_rule]
+            return True
+        return False
+    def label_qualify(self, verbosity):
+        '''
+        Qualify rules where appropriate (Section 4.1.1).
+        :verbosity: How often to print progress.
+        '''
+        num_qualified = 0
+        rule_to_new_labels = dict()
+        if verbosity > 0:
+            print('Qualifying candidate rules (Section 4.1.2).')
+        n = len(self.candidates)
+        for i, rule in enumerate(self.candidates):
+            root = rule[0][0]
+            heads = list(self.graph.candidates[rule]['ca_to_size'].keys())
+            # if all correct assertions share more labels than the current root label, then we can add this label
+            shared_by_all = set(self.graph.labels(heads[0]))
+            for head in heads[1:]:
+                shared_by_all = shared_by_all.intersection(self.graph.labels(head))
+                if shared_by_all == {root}:
+                    break
+            if shared_by_all != {root}:
+                rule_to_new_labels[rule] = tuple(sorted(shared_by_all))
+            if verbosity > 0 and i > 0 and i % verbosity == 0:
+                print('{}% of candidates processed.'.format(round(i / n * 100, 2)))
+        if verbosity > 0:
+            print('{}% of candidates processed.'.format(round(i / n * 100, 2)))
+        evaluator = Evaluator(self.graph)
+        n = len(rule_to_new_labels.items())
+        for i, rule_and_new_labels in enumerate(rule_to_new_labels.items()):
+            # check whether the qualifier leads to improvements
+            qualified = self.check_qualify(evaluator, verbosity, rule_and_new_labels)
+            if qualified:
+                num_qualified += 1
+            if verbosity > 0 and i > 0 and i % verbosity == 0:
+                print('{}'.format(i / n))
+        self.candidates = list(self.graph.candidates.keys())
+        if verbosity > 0:
+            print('{}% of candidates qualified.'.format(round(num_qualified / n * 100, 2) if n > 0 else 100))
+    def build_model(self, rank='metric', order=['mdl_err', 'coverage', 'lex'], passes=2, label_qualify=True, verbosity=1000000):
+        '''
+        The core of the algorithm. (Sections 4.1.2-4.2.1)
+        '''
+        if label_qualify:
+            self.label_qualify(verbosity=verbosity)
+        if rank == 'metric':
+            self.rank_rules()
+        elif rank == 'random':
+            print('Random order.')
+            random.shuffle(self.candidates)
+        # starts null
+        model = Model(self.graph)
+        best_val, best_model_length, best_neg_edge, best_neg_node = self.evaluator.evaluate(model, with_lengths=True)
+        null_cost = best_val
+        if verbosity > 0:
+            print('Null encoding cost: {}'.format(round(best_val, 4)))
+        num_cans = len(self.candidates)
+        tried = set()
+        for _pass in range(1, passes + 1):
+            if verbosity > 0:
+                print('Starting pass {}.'.format(_pass))
+            for i, rule in enumerate(self.candidates):
+                # build reverse
+                root, child = rule[0], rule[1][0]
+                reverse_rule = (child[2][0], ((child[0], 'in' if child[1] == 'out' else 'out', (root, ())),))
+                if reverse_rule in self.graph.candidates:
+                    acc_rule = len(self.graph.candidates[rule]['ca_to_size']) / self.graph.nodes_with_type(rule[0])
+                    acc_rev_rule = len(self.graph.candidates[reverse_rule]['ca_to_size']) / self.graph.nodes_with_type(reverse_rule[0])
+                    if rule in model.rules or reverse_rule in model.rules:
+                        continue
+                    model.add_rule(rule)
+                    val, model_length, neg_edge, neg_node = self.evaluator.evaluate_change(model, rule, best_model_length)
+                    model.remove_rule(rule)
+                    model.add_rule(reverse_rule)
+                    rev_val, rev_model_length, rev_neg_edge, rev_neg_node = self.evaluator.evaluate_change(model, reverse_rule, best_model_length)
+                    model.remove_rule(reverse_rule)
+                    # if the cost didn't go down, remove the rule
+                    if val <= rev_val < best_val:
+                        assert(val <= rev_val)
+                        assert(val < best_val)
+                        model.add_rule(rule)
+                        best_val = val
+                        best_model_length = model_length
+                    elif rev_val < best_val:
+                        assert(rev_val < best_val)
+                        assert(rev_val < val)
+                        model.add_rule(reverse_rule)
+                        best_val = rev_val
+                        best_model_length = rev_model_length
+                else:
+                    acc_rule = len(self.graph.candidates[rule]['ca_to_size']) / self.graph.nodes_with_type(rule[0])
+                    if rule in model.rules:
+                        continue
+                    model.add_rule(rule)
+                    val, model_length, neg_edge, neg_node = self.evaluator.evaluate_change(model, rule, best_model_length)
+                    if val < best_val:
+                        best_val = val
+                        best_model_length = model_length
+                    else: # if the cost didn't go down, remove the rule
+                        model.remove_rule(rule)
+                if verbosity > 0 and i > 0 and i % verbosity == 0:
+                    print('Percent tried: {}. Num rules: {}. New encoding cost: {}. Percent saved: {}.'.format((i / num_cans) * 100, len(model.rules), best_val, ((null_cost - best_val) / null_cost) * 100))
+            if verbosity > 0:
+                print('Final number of rules after pass {}: {}'.format(_pass, len(model.rules)))
+        # M*
+        return model
+if __name__ == '__main__':
+    '''
+    Entry point for the program.
+    '''
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--graph',
+                        '-g',
+                        type=str,
+                        required=True,
+                        help='the name of the graph to summarize')
+    args = parser.parse_args()
+    graph = Graph(args.graph)

src/research/COINs-KGGeneration/graph_analysis/METIS/libmetis.so ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2aa000b0481594c6b8c5e9cc2f4b751c5d23f62a66f8963210275f629539bbc7
+size 499408

src/research/COINs-KGGeneration/graph_analysis/METIS/metis.dll ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5226f4d336778185e1410f7bfb3a01f13298fc0711e1aa797d3487ba80ffa1f8
+size 186880

src/research/COINs-KGGeneration/graph_analysis/__init__.py ADDED Viewed

File without changes

src/research/COINs-KGGeneration/graph_analysis/main.py ADDED Viewed

	@@ -0,0 +1,566 @@

+import argparse
+import itertools
+import re
+from glob import glob
+from os import makedirs, remove
+from typing import Dict, List, Tuple
+import pandas as pd
+from pyarrow.fs import copy_files
+from tensorboardX import SummaryWriter
+from tqdm import tqdm
+from graph_analysis.metrics import \
+    Connectivity, Graph, Heterogeneity, Importance, MetricSetView, Subgraphing, Summarization, metric_set_to_dashboard
+from graph_analysis.stats import \
+    estimate_and_test_comparison, estimate_and_test_pareto, estimate_and_test_uniform, metric_set_histogram
+from graph_data.cluster import cluster_dir, cluster_dir_path, hdfs_glob, hdfs_mkdirs
+from graph_data.load_data import Dataset, FreeBase, HighEnergyPhysicsCitations, NELL, OGBBioKG, PatentCitations, \
+    RedditHyperlinks, Swisscom, SwisscomBig, SwisscomCommunity, SwisscomOverlappingRegion, SwisscomSubgraphOverlap, \
+    Transport, WordNet
+from graph_data.serialization import save_object
+def store_metric_sets(metric_set_dict: Dict[str, List[MetricSetView]], filename: str):
+    save_object(metric_set_dict, f"{filename}.gz")
+    copy_files(f"{filename}.gz", f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{filename}.gz")
+def store_results_table(table: pd.DataFrame, filename: str):
+    table.to_csv(
+        f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{filename}.csv",
+        sep=",", header=True, index=False, mode="w", encoding="utf-8"
+    )
+    with cluster_dir.open_output_stream(
+            f"{cluster_dir_path}/results-{subgraph_roots}/{filename}.md"
+    ) as results_file:
+        results_file.write(table.to_markdown(index=False).encode("utf-8"))
+def compute_metrics_from_datasets(datasets: List[Dataset], datasets_loading_args: List[Tuple],
+                                  collect_metric_sets: bool = False,
+                                  run_subgraphing_summarization: bool = True,
+                                  write_to_dashboard: bool = True) -> Tuple[pd.DataFrame,
+                                                                            pd.DataFrame,
+                                                                            Dict[str, List[MetricSetView]]]:
+    scalar_tables = []
+    dist_tables = []
+    collected_metric_sets = {}
+    for dataset, loading_args in tqdm(zip(datasets, datasets_loading_args),
+                                      desc="Processing datasets for metrics", total=len(datasets), leave=False):
+        if dataset.name == "swisscom-big" and hdfs_glob("data/swisscom", f"relations_{subgraph_roots}"):
+            dataset.node_names_map = pd.read_csv(
+                f"hdfs://sbd{cluster_dir_path}/data/swisscom/node_names_{subgraph_roots}.csv",
+                header=0, index_col=0, squeeze=True, encoding="utf-8"
+            )
+            dataset.node_data = pd.read_csv(
+                f"hdfs://sbd{cluster_dir_path}/data/swisscom/nodes_{subgraph_roots}.csv",
+                header=0, index_col=None, parse_dates=["time", ], encoding="utf-8"
+            )
+            dataset.edge_data = pd.read_csv(
+                f"hdfs://sbd{cluster_dir_path}/data/swisscom/relations_{subgraph_roots}.csv",
+                header=0, index_col=None, parse_dates=["time", ], encoding="utf-8"
+            )
+        else:
+            dataset.load_from_disk(*loading_args)
+            if dataset.name == "swisscom-big":
+                dataset.node_names_map.to_csv(
+                    f"hdfs://sbd{cluster_dir_path}/data/swisscom/node_names_{subgraph_roots}.csv",
+                    header=True, index=True, encoding="utf-8")
+                dataset.node_data.to_csv(f"hdfs://sbd{cluster_dir_path}/data/swisscom/nodes_{subgraph_roots}.csv",
+                                         header=True, index=False, encoding="utf-8")
+                dataset.edge_data.to_csv(f"hdfs://sbd{cluster_dir_path}/data/swisscom/relations_{subgraph_roots}.csv",
+                                         header=True, index=False, encoding="utf-8")
+        if all(s not in dataset.name for s in ["swisscom-big", "swisscom-community", "swisscom-region"]):
+            dataset.time_sort_and_numerize()
+        dashboard = None
+        if write_to_dashboard:
+            makedirs(f"results-{subgraph_roots}/{dataset.name}", exist_ok=True)
+            hdfs_mkdirs(f"results-{subgraph_roots}/{dataset.name}")
+            new_run_id = len(hdfs_glob(f"results-{subgraph_roots}/{dataset.name}", "")) \
+                         - len(hdfs_glob(f"results-{subgraph_roots}/{dataset.name}", ".csv")) + 1
+            dashboard = SummaryWriter(f"results-{subgraph_roots}/{dataset.name}/{new_run_id}")
+        num_nodes, num_node_types, num_relations = \
+            len(dataset.node_data), len(dataset.node_types_map), len(dataset.relation_names_map)
+        graph = Graph(dataset.name)
+        metric_sets = [Heterogeneity(graph, dashboard, num_node_types, num_relations),
+                       Connectivity(graph, dashboard, num_nodes),
+                       Importance(graph, dashboard, num_nodes)]
+        if run_subgraphing_summarization:
+            metric_sets.append(Subgraphing(graph, dashboard, num_nodes, 1 / num_nodes, 1, 250))
+            metric_sets.append(Summarization(graph, dashboard, num_nodes, True, False, 0.95, 250, 5,
+                                             f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}"))
+        last_batch_time = None
+        for batch_time, nodes, edges in dataset.get_graph_data_batch_generator(pd.Timedelta(150 * 365, "d")):
+            graph.update_graph(nodes, edges)
+            for metric_set in tqdm(metric_sets, desc="Computing metric sets", leave=False):
+                metric_set.recursive_updates(nodes, edges)
+                metric_set.compute_metrics()
+                if write_to_dashboard:
+                    metric_set.update_dashboard_visuals(batch_time)
+            last_batch_time = batch_time
+        for metric_set in metric_sets:
+            metric_set.update_dashboard_table(last_batch_time, update_dashboard=write_to_dashboard)
+            relabel_distribution_table_supports(metric_set.dist_table,
+                                                dataset.node_names_map,
+                                                dataset.node_types_map,
+                                                dataset.relation_names_map)
+            scalar_tables.append(metric_set.scalar_table)
+            dist_tables.append(metric_set.dist_table)
+            if collect_metric_sets:
+                collected_metric_sets.setdefault(metric_set.metric_type, [])
+                collected_metric_sets[metric_set.metric_type].append(metric_set.simplified_view())
+        dataset.unload_from_memory()
+        if run_subgraphing_summarization:
+            kgist_rule_graph = pd.read_csv(
+                f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{dataset.name}/kgist_rule_graph.csv",
+                header=0, index_col=None, encoding="utf-8"
+            )
+            relabel_kgist_rule_graph_edges(kgist_rule_graph, dataset.node_types_map, dataset.relation_names_map)
+            kgist_rule_graph.to_csv(
+                f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{dataset.name}/kgist_rule_graph.csv",
+                header=True, index=False, encoding="utf-8"
+            )
+            node_names_map_reversed = dataset.node_names_map.reset_index(name="index").set_index("index")["n"]
+            node_types_map_reversed = dataset.node_types_map.reset_index(name="index").set_index("index")["type"]
+            relation_names_map_reversed = dataset.relation_names_map.reset_index(name="index").set_index("index")["r"]
+            node_reader = pd.read_csv(
+                f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{dataset.name}/dedensified_graph_nodes.csv",
+                header=0, index_col=None, chunksize=10000, encoding="utf-8"
+            )
+            edge_reader = pd.read_csv(
+                f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{dataset.name}/dedensified_graph_relations.csv",
+                header=0, index_col=None, chunksize=10000, encoding="utf-8"
+            )
+            compressor_nodes_all = []
+            first_chunk = True
+            for dedensified_nodes in node_reader:
+                compressor_nodes_all.extend(
+                    list(dedensified_nodes[dedensified_nodes.type.astype(str).str.contains("compressor")].n)
+                )
+                relabel_dedensified_graph_nodes(dedensified_nodes, num_nodes,
+                                                node_names_map_reversed,
+                                                node_types_map_reversed,
+                                                relation_names_map_reversed)
+                dedensified_nodes.to_csv(
+                    f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{dataset.name}/"
+                    f"dedensified_graph_nodes_relabeled.csv",
+                    mode="w" if first_chunk else "a", header=first_chunk, index=False, encoding="utf-8"
+                )
+                first_chunk = False
+            node_reader.close()
+            first_chunk = True
+            for dedensified_edges in edge_reader:
+                relabel_dedensified_graph_edges(dedensified_edges, compressor_nodes_all, num_nodes,
+                                                node_names_map_reversed,
+                                                relation_names_map_reversed)
+                dedensified_edges.to_csv(
+                    f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{dataset.name}/"
+                    f"dedensified_graph_relations_relabeled.csv",
+                    mode="w" if first_chunk else "a", header=first_chunk, index=False, encoding="utf-8"
+                )
+                first_chunk = False
+            edge_reader.close()
+            sp_tree_path_dist = pd.read_csv(
+                f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{dataset.name}/spanning_tree_path_summary.csv",
+                header=0, index_col=None, encoding="utf-8"
+            )
+            relabel_spanning_tree_paths(sp_tree_path_dist, dataset.node_types_map, dataset.relation_names_map)
+            sp_tree_path_dist.to_csv(
+                f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{dataset.name}/spanning_tree_path_summary.csv",
+                header=True, index=False, encoding="utf-8"
+            )
+            if num_nodes < 100000:
+                harp_coarsened_nodes = pd.read_csv(
+                    f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{dataset.name}/harp_coarsened_nodes.csv",
+                    header=0, index_col=None, encoding="utf-8"
+                )
+                harp_coarsened_edges = pd.read_csv(
+                    f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{dataset.name}/harp_coarsened_edges.csv",
+                    header=0, index_col=None, encoding="utf-8"
+                )
+                harp_node_hierarchy = pd.read_csv(
+                    f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{dataset.name}/harp_node_hierarchy.csv",
+                    header=0, index_col=None, encoding="utf-8"
+                )
+                relabel_harp_coarsened_graph(harp_coarsened_nodes, harp_coarsened_edges, harp_node_hierarchy,
+                                             dataset.node_names_map, dataset.node_types_map, dataset.relation_names_map)
+                harp_coarsened_nodes.to_csv(
+                    f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{dataset.name}/harp_coarsened_nodes.csv",
+                    header=True, index=False, encoding="utf-8"
+                )
+                harp_coarsened_edges.to_csv(
+                    f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{dataset.name}/harp_coarsened_edges.csv",
+                    header=True, index=False, encoding="utf-8"
+                )
+                harp_node_hierarchy.to_csv(
+                    f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{dataset.name}/harp_node_hierarchy.csv",
+                    header=True, index=False, encoding="utf-8"
+                )
+        if write_to_dashboard:
+            dashboard.close()
+            copy_files(f"results-{subgraph_roots}/{dataset.name}",
+                       f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/{dataset.name}")
+            run_id = len(hdfs_glob(f"results-{subgraph_roots}/{dataset.name}", "")) \
+                     - len(hdfs_glob(f"results-{subgraph_roots}/{dataset.name}", ".csv"))
+            for tensorboard_file in glob(f"results-{subgraph_roots}/{dataset.name}/{run_id}/*"):
+                remove(tensorboard_file)
+    scalar_tables = pd.concat(scalar_tables, ignore_index=True).sort_values(["Metric", "Dataset", "Value"],
+                                                                            ascending=[True, True, False],
+                                                                            ignore_index=True)
+    dist_tables = pd.concat(dist_tables, ignore_index=True).sort_values(["Metric", "Dataset", "Value"],
+                                                                        ascending=[True, True, False],
+                                                                        ignore_index=True)
+    return scalar_tables, dist_tables, collected_metric_sets
+def relabel_distribution_table_supports(dist_table: pd.DataFrame,
+                                        node_names_map: pd.Series,
+                                        node_types_map: pd.Series,
+                                        relation_names_map: pd.Series):
+    node_names_map_reversed = node_names_map.reset_index(name="index").set_index("index")["n"]
+    node_types_map_reversed = node_types_map.reset_index(name="index").set_index("index")["type"]
+    relation_names_map_reversed = relation_names_map.reset_index(name="index").set_index("index")["r"]
+    node_type_dist = dist_table.loc[dist_table.Metric == "NodeTypeDist", "Support"]
+    node_type_dist = node_type_dist.map(node_types_map_reversed)
+    dist_table.loc[dist_table.Metric == "NodeTypeDist", "Support"] = node_type_dist
+    relation_type_dist = dist_table.loc[dist_table.Metric == "RelationDist", "Support"]
+    relation_type_dist = relation_type_dist.map(relation_names_map_reversed)
+    dist_table.loc[dist_table.Metric == "RelationDist", "Support"] = relation_type_dist
+    label_interaction_dist = dist_table.loc[dist_table.Metric == "LabelInteractionDist", "Support"]
+    label_interaction_dist = label_interaction_dist.map(
+        lambda triplet: f"{node_types_map_reversed[triplet[0]]} -> "
+                        f"{relation_names_map_reversed[triplet[1]]} -> "
+                        f"{node_types_map_reversed[triplet[2]]}"
+    )
+    dist_table.loc[dist_table.Metric == "LabelInteractionDist", "Support"] = label_interaction_dist
+    non_node_support_dists = ["NodeTypeDist", "RelationDist", "LabelInteractionDist", "WCC", "SCC",
+                              "DisjointCommunitySizeDist", "OverlappingCommunitySizeDist", "OverlappingRegionSizeDist",
+                              "3MotifsDist", "4MotifsDist", "SpanningTreePathDist"]
+    other_dists = dist_table.loc[~dist_table.Metric.isin(non_node_support_dists), "Support"]
+    other_dists = other_dists.map(node_names_map_reversed)
+    dist_table.loc[~dist_table.Metric.isin(non_node_support_dists), "Support"] = other_dists
+def relabel_kgist_rule_graph_edges(rule_graph_edge_table: pd.DataFrame,
+                                   node_types_map: pd.Series,
+                                   relation_names_map: pd.Series):
+    node_types_map_reversed = node_types_map.reset_index(name="index").set_index("index")["type"]
+    relation_names_map_reversed = relation_names_map.reset_index(name="index").set_index("index")["r"]
+    s_type = rule_graph_edge_table.loc[:, "s_type"].map(node_types_map_reversed)
+    rule_graph_edge_table.loc[:, "s_type"] = s_type
+    r = rule_graph_edge_table.loc[:, "r"].map(relation_names_map_reversed)
+    rule_graph_edge_table.loc[:, "r"] = r
+    t_type = rule_graph_edge_table.loc[:, "t_type"].map(node_types_map_reversed)
+    rule_graph_edge_table.loc[:, "t_type"] = t_type
+def relabel_dedensified_graph_nodes(node_data: pd.DataFrame, num_non_compressors: int,
+                                    node_names_map_reversed: pd.Series,
+                                    node_types_map_reversed: pd.Series,
+                                    relation_names_map_reversed: pd.Series):
+    compressor_nodes = node_data.type.astype(str).str.contains("compressor")
+    node_types = node_data.loc[~compressor_nodes, "type"].astype(int)
+    node_types = node_types.map(node_types_map_reversed)
+    node_data.loc[~compressor_nodes, "type"] = node_types
+    node_types_compressors = node_data.loc[compressor_nodes, "type"]
+    node_types_compressors = node_types_compressors.map(
+        lambda x: f"compressor-{relation_names_map_reversed[int(x.split('-')[1])]}"
+    )
+    node_data.loc[compressor_nodes, "type"] = node_types_compressors
+    node_names = node_data.loc[~compressor_nodes, "n"]
+    node_names = node_names.map(node_names_map_reversed)
+    node_data.loc[~compressor_nodes, "n"] = node_names
+    node_names_compressors = node_data.loc[compressor_nodes, "n"]
+    node_names_compressors = node_names_compressors.map(lambda x: f"compressor-{x - num_non_compressors}")
+    node_data.loc[compressor_nodes, "n"] = node_names_compressors
+def relabel_dedensified_graph_edges(edge_data: pd.DataFrame, compressor_nodes: list, num_non_compressors: int,
+                                    node_names_map_reversed: pd.Series,
+                                    relation_names_map_reversed: pd.Series):
+    edge_data.r = edge_data.r.map(relation_names_map_reversed)
+    edges_s_names = edge_data.loc[~edge_data.source.isin(compressor_nodes), "source"]
+    edges_s_names = edges_s_names.map(node_names_map_reversed)
+    edge_data.loc[~edge_data.source.isin(compressor_nodes), "source"] = edges_s_names
+    edges_s_names_compressors = edge_data.loc[edge_data.source.isin(compressor_nodes), "source"]
+    edges_s_names_compressors = edges_s_names_compressors.map(lambda x: f"compressor-{x - num_non_compressors}")
+    edge_data.loc[edge_data.source.isin(compressor_nodes), "source"] = edges_s_names_compressors
+    edges_t_names = edge_data.loc[~edge_data.target.isin(compressor_nodes), "target"]
+    edges_t_names = edges_t_names.map(node_names_map_reversed)
+    edge_data.loc[~edge_data.target.isin(compressor_nodes), "target"] = edges_t_names
+    edges_t_names_compressors = edge_data.loc[edge_data.target.isin(compressor_nodes), "target"]
+    edges_t_names_compressors = edges_t_names_compressors.map(lambda x: f"compressor-{x - num_non_compressors}")
+    edge_data.loc[edge_data.target.isin(compressor_nodes), "target"] = edges_t_names_compressors
+def map_neo4j_like_str(string, node_types_map_reversed, relation_names_map_reversed):
+    chars = re.split(r':\d+', string)
+    nums = re.findall(r':\d+', string)
+    types = [f":{node_types_map_reversed[int(num[1:])]}"
+             if i % 2 == 0 else f":{relation_names_map_reversed[int(num[1:])]}"
+             for i, num in enumerate(nums)]
+    mapped_path = ""
+    for char, t in zip(chars, types):
+        mapped_path += char
+        mapped_path += t
+    mapped_path += chars[-1]
+    return mapped_path
+def relabel_spanning_tree_paths(spanning_tree_path_table: pd.DataFrame,
+                                node_types_map: pd.Series,
+                                relation_names_map: pd.Series):
+    node_types_map_reversed = node_types_map.reset_index(name="index").set_index("index")["type"]
+    relation_names_map_reversed = relation_names_map.reset_index(name="index").set_index("index")["r"]
+    spanning_tree_path_table.Path = spanning_tree_path_table.Path.apply(map_neo4j_like_str,
+                                                                        args=(node_types_map_reversed,
+                                                                              relation_names_map_reversed))
+def relabel_harp_coarsened_graph(node_data: pd.DataFrame, edge_data: pd.DataFrame, node_hierarchy: pd.DataFrame,
+                                 node_names_map: pd.Series,
+                                 node_types_map: pd.Series,
+                                 relation_names_map: pd.Series):
+    node_names_map_reversed = node_names_map.reset_index(name="index").set_index("index")["n"]
+    node_types_map_reversed = node_types_map.reset_index(name="index").set_index("index")["type"]
+    relation_names_map_reversed = relation_names_map.reset_index(name="index").set_index("index")["r"]
+    map_super_node = lambda x: str(node_names_map_reversed[x]) if x in node_names_map_reversed else str(x)
+    node_data.n = node_data.n.apply(map_super_node)
+    node_data.type = node_data.type.apply(lambda types: "|".join([node_types_map_reversed[t]
+                                                                  for t in sorted(eval(types))]))
+    edge_data.s = edge_data.s.apply(map_super_node)
+    edge_data.r = edge_data.r.map(relation_names_map_reversed)
+    edge_data.t = edge_data.t.apply(map_super_node)
+    node_hierarchy.node = node_hierarchy.node.apply(map_super_node)
+def store_analyze_subgraph_results(subgraphs_name: str, dataset_names_map: pd.Series,
+                                   scalar_tables: pd.DataFrame,
+                                   dist_tables: pd.DataFrame,
+                                   metric_sets: Dict[str, List[MetricSetView]],
+                                   node_types_map: pd.Series, relation_names_map: pd.Series) -> List[MetricSetView]:
+    store_results_table(scalar_tables, f"scalars_{subgraphs_name.replace('-', '_')}")
+    store_results_table(dist_tables, f"distributions_{subgraphs_name.replace('-', '_')}")
+    store_metric_sets(metric_sets, f"metric_sets_{subgraphs_name.replace('-', '_')}")
+    subgraphs_histograms, subgraphs_metric_sums = metric_set_histogram(metric_sets, subgraphs_name)
+    makedirs(f"results-{subgraph_roots}/{subgraphs_name}", exist_ok=True)
+    hdfs_mkdirs(f"results-{subgraph_roots}/{subgraphs_name}")
+    new_run_id = len(hdfs_glob(f"results-{subgraph_roots}/{subgraphs_name}", "")) + 1
+    dashboard = SummaryWriter(f"results-{subgraph_roots}/{subgraphs_name}/{new_run_id}")
+    last_batch_time = pd.Timestamp.now()
+    subgraphs_histogram_table = []
+    for i in range(len(metric_sets)):
+        scalar_table_hist, dist_table_hist, _ = metric_set_to_dashboard(subgraphs_histograms[i])
+        dashboard.add_text(f"{subgraphs_histograms[i][1]}/Scalars", scalar_table_hist.to_markdown(index=False),
+                           0, walltime=last_batch_time.timestamp())
+        dashboard.add_text(f"{subgraphs_histograms[i][1]}/Distributions", dist_table_hist.to_markdown(index=False),
+                           0, walltime=last_batch_time.timestamp())
+        subgraphs_histogram_table.append(dist_table_hist)
+    subgraphs_histogram_table = pd.concat(subgraphs_histogram_table, ignore_index=True)
+    relabel_distribution_table_supports(subgraphs_histogram_table, dataset_names_map,
+                                        node_types_map, relation_names_map)
+    store_results_table(subgraphs_histogram_table, f"{subgraphs_name.replace('-', '_')}_histograms")
+    uniform_tables_subgraphs = estimate_and_test_uniform(subgraphs_histograms, subgraphs_metric_sums)
+    store_results_table(uniform_tables_subgraphs, f"{subgraphs_name.replace('-', '_')}_uniform_tests")
+    return subgraphs_histograms
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-s", "--sites", action="store_true",
+                        help="Whether to subgraph from customer sites instead of organizations.")
+    args = parser.parse_args()
+    global subgraph_roots
+    subgraph_roots = "sites" if args.sites else "orgs"
+    entrypoints = pd.concat([
+        pd.read_csv(f"hdfs://sbd{customer_orgs_file}", header=0, squeeze=True, encoding="utf-8")
+        for customer_orgs_file in hdfs_glob("data/swisscom", f"customer_{subgraph_roots}_")
+    ], ignore_index=True).drop_duplicates().sort_values(ignore_index=True)
+    swisscom_subgraphs = [Swisscom(entrypoint) for entrypoint in entrypoints]
+    dataset_names_map = {subgraph.name: i for i, subgraph in enumerate(swisscom_subgraphs)}
+    dataset_names_map = pd.Series(dataset_names_map, name="index").sort_values()
+    dataset_names_map.index.name = "n"
+    swisscom_subgraphs_histograms = store_analyze_subgraph_results(
+        "swisscom-subgraphs", dataset_names_map,
+        *compute_metrics_from_datasets(swisscom_subgraphs, [(), ] * len(swisscom_subgraphs), True, False, False),
+        swisscom_subgraphs[0].node_types_map, swisscom_subgraphs[0].relation_names_map
+    )
+    node_appearance = {}
+    for k in tqdm(range(len(swisscom_subgraphs)), desc="Processing subgraphs for overlaps", leave=False):
+        swisscom_subgraphs[k].load_from_disk()
+        for node in swisscom_subgraphs[k].node_data.n:
+            node_appearance.setdefault(node, [])
+            node_appearance[node].append(k)
+        swisscom_subgraphs[k].unload_from_memory()
+    subgraph_overlaps = {}
+    for node, appearances in tqdm(node_appearance.items(), desc="Processing nodes for overlaps", leave=False):
+        if len(appearances) > 1:
+            for overlap in itertools.combinations(appearances, 2):
+                subgraph_overlaps.setdefault(overlap, 0)
+                subgraph_overlaps[overlap] += 1
+    node_appearance = {n: list(map(lambda s: swisscom_subgraphs[s].name, subgraphs))
+                       for n, subgraphs in node_appearance.items()}
+    node_appearance = pd.Series(node_appearance, name="subgraphs")
+    node_appearance.index.name = "n"
+    node_appearance.to_csv(
+        f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/nodes_in_which_swisscom_subgraphs.csv",
+        sep=",", header=True, index=True, mode="w", encoding="utf-8"
+    )
+    node_appearance = {}
+    swisscom_subgraphs_overlaps = []
+    swisscom_subgraphs_overlaps_loading_args = []
+    dataset_names_map = {}
+    i = 0
+    for (j, k), overlap_size in tqdm(subgraph_overlaps.items(), desc="Processing subgraph overlaps", leave=False):
+        if overlap_size >= 100 and i < 1000:
+            subgraph_overlap = SwisscomSubgraphOverlap(swisscom_subgraphs[j].customer_org,
+                                                       swisscom_subgraphs[k].customer_org)
+            swisscom_subgraphs_overlaps.append(subgraph_overlap)
+            swisscom_subgraphs_overlaps_loading_args.append((swisscom_subgraphs[j], swisscom_subgraphs[k]))
+            dataset_names_map[subgraph_overlap.name] = i
+            i += 1
+    subgraph_overlaps = {}
+    if len(swisscom_subgraphs_overlaps) > 0:
+        dataset_names_map = pd.Series(dataset_names_map, name="index").sort_values()
+        dataset_names_map.index.name = "n"
+        swisscom_subgraphs_overlaps_histograms = store_analyze_subgraph_results(
+            "swisscom-subgraphs-overlaps", dataset_names_map,
+            *compute_metrics_from_datasets(swisscom_subgraphs_overlaps,
+                                           swisscom_subgraphs_overlaps_loading_args,
+                                           True, False, False),
+            swisscom_subgraphs_overlaps[0].node_types_map, swisscom_subgraphs_overlaps[0].relation_names_map
+        )
+    swisscom_big_graph = SwisscomBig()
+    datasets = [swisscom_big_graph, RedditHyperlinks(), HighEnergyPhysicsCitations(), PatentCitations(),
+                OGBBioKG(), FreeBase(), WordNet(), NELL()]
+    loading_args = [(swisscom_subgraphs,), (), (), (), (), (), (), ()]
+    transport_cities = [path.split("/")[-1] for path in hdfs_glob("data/transport", "") if "." not in path]
+    for city in transport_cities:
+        datasets.append(Transport(city))
+        loading_args.append(())
+    scalar_tables, dist_tables, metric_sets = compute_metrics_from_datasets(datasets, loading_args,
+                                                                            collect_metric_sets=True)
+    store_results_table(scalar_tables, "scalars")
+    store_results_table(dist_tables, "distributions")
+    store_metric_sets(metric_sets, "metric_sets")
+    pareto_tables = estimate_and_test_pareto(metric_sets)
+    store_results_table(pareto_tables, "pareto_tests")
+    if hdfs_glob("data/swisscom", f"relations_{subgraph_roots}"):
+        swisscom_big_graph.node_names_map = pd.read_csv(
+            f"hdfs://sbd{cluster_dir_path}/data/swisscom/node_names_{subgraph_roots}.csv",
+            header=0, index_col=0, squeeze=True, encoding="utf-8"
+        )
+        swisscom_big_graph.node_data = pd.read_csv(
+            f"hdfs://sbd{cluster_dir_path}/data/swisscom/nodes_{subgraph_roots}.csv",
+            header=0, index_col=None, parse_dates=["time", ], encoding="utf-8"
+        )
+        swisscom_big_graph.edge_data = pd.read_csv(
+            f"hdfs://sbd{cluster_dir_path}/data/swisscom/relations_{subgraph_roots}.csv",
+            header=0, index_col=None, parse_dates=["time", ], encoding="utf-8"
+        )
+    else:
+        swisscom_big_graph.load_from_disk(swisscom_subgraphs)
+    swisscom_community_membership = metric_sets["Subgraphing"][0][2]["DisjointCommunityMembership"][-1]
+    swisscom_num_communities = metric_sets["Subgraphing"][0][2]["DisjointCommunityNumber"][-1]
+    swisscom_communities = []
+    dataset_names_map = {}
+    i = 0
+    for community_tuple in tqdm(swisscom_big_graph.get_community_generator(swisscom_community_membership,
+                                                                           swisscom_num_communities),
+                                desc="Processing disjoint communities", total=swisscom_num_communities, leave=False):
+        community_id, community_node_names_map, community_nodes, community_edges = community_tuple
+        swisscom_community = SwisscomCommunity(community_id)
+        swisscom_community.node_names_map = community_node_names_map
+        swisscom_community.node_data = community_nodes
+        swisscom_community.edge_data = community_edges
+        if len(swisscom_community.edge_data) > 0:
+            swisscom_communities.append(swisscom_community)
+            dataset_names_map[swisscom_community.name] = i
+            i += 1
+    dataset_names_map = pd.Series(dataset_names_map, name="index").sort_values()
+    dataset_names_map.index.name = "n"
+    swisscom_communities_histograms = store_analyze_subgraph_results(
+        "swisscom-communities", dataset_names_map,
+        *compute_metrics_from_datasets(swisscom_communities, [(), ] * len(swisscom_communities), True, False, False),
+        swisscom_communities[0].node_types_map, swisscom_communities[0].relation_names_map
+    )
+    swisscom_overlapping_community_membership = metric_sets["Subgraphing"][0][2]["OverlappingCommunityMembership"][-1]
+    swisscom_overlapping_regions = []
+    dataset_names_map = {}
+    i = 0
+    for region_tuple in tqdm(
+            swisscom_big_graph.get_overlapping_region_generator(swisscom_overlapping_community_membership),
+            desc="Processing overlapping regions", leave=False
+    ):
+        community_id, community_id_2, region_node_names_map, region_nodes, region_edges = region_tuple
+        swisscom_region = SwisscomOverlappingRegion(community_id, community_id_2)
+        swisscom_region.node_names_map = region_node_names_map
+        swisscom_region.node_data = region_nodes
+        swisscom_region.edge_data = region_edges
+        if len(swisscom_region.edge_data) > 0:
+            swisscom_overlapping_regions.append(swisscom_region)
+            dataset_names_map[swisscom_region.name] = i
+            i += 1
+    swisscom_big_graph.unload_from_memory()
+    if len(swisscom_overlapping_regions) > 0:
+        dataset_names_map = pd.Series(dataset_names_map, name="index").sort_values()
+        dataset_names_map.index.name = "n"
+        swisscom_regions_histograms = store_analyze_subgraph_results(
+            "swisscom-overlapping-regions", dataset_names_map,
+            *compute_metrics_from_datasets(swisscom_overlapping_regions, [(), ] * len(swisscom_overlapping_regions),
+                                           True, False, False),
+            swisscom_overlapping_regions[0].node_types_map, swisscom_overlapping_regions[0].relation_names_map
+        )
+    comparison_tables = estimate_and_test_comparison(metric_sets, linear_comparison=True)
+    store_results_table(comparison_tables, "comparisons")
+    histogram_metric_sets = {}
+    for metric_set in (swisscom_subgraphs_histograms + swisscom_communities_histograms):
+        histogram_metric_sets.setdefault(metric_set[1], [])
+        histogram_metric_sets[metric_set[1]].append(metric_set)
+    comparison_tables_subgraphs_communities = estimate_and_test_comparison(histogram_metric_sets,
+                                                                           linear_comparison=False)
+    store_results_table(comparison_tables_subgraphs_communities, "comparisons_subgraphs_communities")
+    if len(swisscom_subgraphs_overlaps) > 0 and len(swisscom_overlapping_regions) > 0:
+        histogram_metric_sets = {}
+        for metric_set in (swisscom_subgraphs_overlaps_histograms + swisscom_regions_histograms):
+            histogram_metric_sets.setdefault(metric_set[1], [])
+            histogram_metric_sets[metric_set[1]].append(metric_set)
+        comparison_tables_overlaps = estimate_and_test_comparison(histogram_metric_sets,
+                                                                  linear_comparison=False)
+        store_results_table(comparison_tables_overlaps, "comparisons_overlaps")
+    copy_files(f"results-{subgraph_roots}/", f"hdfs://sbd{cluster_dir_path}/results-{subgraph_roots}/")

src/research/COINs-KGGeneration/graph_analysis/metrics.py ADDED Viewed

	@@ -0,0 +1,950 @@

+import itertools
+from abc import ABC, abstractmethod
+from collections import Counter
+from math import sqrt
+from os import makedirs
+from typing import Dict, List, Optional, Tuple, Union
+import igraph
+import networkx as nx
+import numpy as np
+import pandas as pd
+import seaborn as sns
+from karateclub.community_detection.overlapping import EgoNetSplitter
+from matplotlib import pyplot as plt
+from pymetis import part_graph
+from scipy.linalg import eig
+from scipy.sparse import csr_matrix, triu
+from scipy.sparse.linalg import eigs
+from tensorboardX import SummaryWriter
+from tqdm import tqdm
+import graph_analysis.KGist.graph
+import graph_analysis.KGist.searcher
+sns.set(context="paper", style="darkgrid", font_scale=3.5)
+MetricSetValues = Dict[str, List[Union[float, np.ndarray, csr_matrix]]]
+MetricSetView = Tuple[str, str, MetricSetValues]
+def metric_set_to_dashboard(metric_set: MetricSetView, top_k=5):
+    graph_dataset_name, metrics_type, metric_set_values = metric_set
+    scalar_table = []
+    dist_table = []
+    motif_figures = {}
+    for metric_name, values in metric_set_values.items():
+        last_value = values[-1]
+        if type(last_value) not in [np.ndarray, csr_matrix]:
+            scalar_table.append((metric_name, graph_dataset_name, last_value))
+        else:
+            if "Membership" in metric_name:
+                continue
+            top_k_probs = np.argsort(-last_value, axis=None)
+            if len(top_k_probs) > top_k:
+                top_k_probs = top_k_probs[:top_k]
+            top_k_probs = np.unravel_index(top_k_probs, last_value.shape)
+            for support_values, support_prob in zip(np.stack(top_k_probs, axis=1), last_value[top_k_probs]):
+                support_values = tuple(support_values)
+                if len(support_values) == 1:
+                    support_values = support_values[0]
+                dist_table.append((metric_name, graph_dataset_name, support_values, support_prob))
+            if "MotifsDist" in metric_name:
+                support_motifs = [
+                    all_3_motifs[motif] if "3" in metric_name else all_4_motifs[motif]
+                    for motif in np.stack(top_k_probs, axis=1)[:, 0]
+                ]
+                figure, ax = plt.subplots(1, top_k, figsize=(9 * top_k, 9))
+                figure.suptitle(metric_name)
+                for k in range(top_k):
+                    ax[k].set_title(f"{round(last_value[top_k_probs][k], 4)}")
+                    ax[k].set_xticks([])
+                    ax[k].set_yticks([])
+                    motif = support_motifs[k].to_networkx(nx.DiGraph)
+                    nx.draw(motif, ax=ax[k], pos=nx.circular_layout(motif),
+                            node_size=3000, arrowstyle="->", arrowsize=30, linewidths=5, width=5)
+                figure.tight_layout()
+                motif_figures[f"{metrics_type}/{'3Motifs' if '3' in metric_name else '4Motifs'}"] = figure
+    scalar_table = pd.DataFrame.from_records(scalar_table, columns=["Metric", "Dataset", "Value"])
+    dist_table = pd.DataFrame.from_records(dist_table, columns=["Metric", "Dataset", "Support", "Value"])
+    return scalar_table, dist_table, motif_figures
+class Graph:
+    def __init__(self, dataset_name: str):
+        self.dataset_name = dataset_name
+        self.graph: igraph.Graph = igraph.Graph(directed=True, vertex_attrs={"type": []}, edge_attrs={"r": []})
+    def update_graph(self, node_batch: pd.DataFrame, edge_batch: pd.DataFrame):
+        self.graph.add_vertices(len(node_batch), {"type": node_batch.type.values})
+        self.graph.add_edges(edge_batch[["s", "t"]].itertuples(index=False, name=None), {"r": edge_batch.r.values})
+class MetricSet(ABC):
+    @abstractmethod
+    def __init__(self, graph: Graph, dashboard: Optional[SummaryWriter], metric_type: str, *args, **kwargs):
+        self.graph = graph
+        self.dashboard = dashboard
+        self.metric_type = metric_type
+        self.metric_values: MetricSetValues = {}
+        self.metric_is_recursive: Dict[str, bool] = {}
+        self.current_step = 0
+        self.scalar_table: pd.DataFrame = None
+        self.dist_table: pd.DataFrame = None
+        pass
+    def simplified_view(self) -> MetricSetView:
+        return self.graph.dataset_name, self.metric_type, self.metric_values
+    def __repr__(self) -> str:
+        return repr(self.simplified_view())
+    @abstractmethod
+    def recursive_updates(self, node_batch: pd.DataFrame, edge_batch: pd.DataFrame):
+        pass
+    @abstractmethod
+    def compute_metric(self, metric_name) -> np.ndarray:
+        pass
+    def compute_metrics(self):
+        for metric_name, is_recursive in self.metric_is_recursive.items():
+            if not is_recursive:
+                self.metric_values[metric_name].append(self.compute_metric(metric_name))
+        self.current_step += 1
+    def update_dashboard_visuals(self, batch_time):
+        for metric_name, values in self.metric_values.items():
+            curr_value = values[self.current_step]
+            if type(curr_value) not in [np.ndarray, csr_matrix]:
+                self.dashboard.add_scalar(f"{self.metric_type}/{metric_name}", curr_value, self.current_step,
+                                          walltime=batch_time.timestamp())
+            elif curr_value.ndim == 1:
+                if "Membership" in metric_name:
+                    continue
+                curr_value = -np.sort(-curr_value)
+                bucket_limits = np.arange(1, len(curr_value) + 1)
+                if len(curr_value) > 10000:
+                    subsample_indices = np.linspace(0, len(curr_value) - 1, num=10000, dtype=int)
+                    curr_value = curr_value[subsample_indices]
+                    bucket_limits = np.array(list(subsample_indices[1:]) + [len(curr_value), ])
+                self.dashboard.add_histogram_raw(f"{self.metric_type}/{metric_name}",
+                                                 np.nanmin(curr_value), np.nanmax(curr_value), len(curr_value),
+                                                 np.nansum(curr_value), np.nansum(curr_value ** 2), bucket_limits,
+                                                 curr_value, self.current_step, walltime=batch_time.timestamp())
+    def update_dashboard_table(self, last_batch_time=None, top_k=5, update_dashboard=True):
+        if last_batch_time is None:
+            last_batch_time = pd.Timestamp.now()
+        scalar_table, dist_table, motif_figures = metric_set_to_dashboard(self.simplified_view(), top_k)
+        for dashboard_tag, motif_figure in motif_figures.items():
+            self.dashboard.add_figure(dashboard_tag, motif_figure,
+                                      self.current_step, walltime=last_batch_time.timestamp())
+        self.scalar_table = scalar_table
+        self.dist_table = dist_table
+        if update_dashboard:
+            self.dashboard.add_text(f"{self.metric_type}/Scalars", self.scalar_table.to_markdown(index=False),
+                                    self.current_step, walltime=last_batch_time.timestamp())
+            self.dashboard.add_text(f"{self.metric_type}/Distributions", self.dist_table.to_markdown(index=False),
+                                    self.current_step, walltime=last_batch_time.timestamp())
+class Heterogeneity(MetricSet):
+    def __init__(self, graph, dashboard, num_type, num_r):
+        super().__init__(graph, dashboard, "Heterogeneity")
+        self.metric_values = {
+            "NumNodes": [0, ],
+            "NumEdges": [0, ],
+            "EdgeDensity": [0, ],
+            "NodeTypeDist": [np.zeros(num_type), ],
+            "RelationDist": [np.zeros(num_r), ],
+            "LabelInteractionDist": [np.zeros((num_type, num_r, num_type)), ],
+            "NodeTypeAssortativity": [0, ],
+        }
+        self.metric_is_recursive = {
+            "NumNodes": True,
+            "NumEdges": True,
+            "EdgeDensity": True,
+            "NodeTypeDist": True,
+            "RelationDist": False,
+            "LabelInteractionDist": True,
+            "NodeTypeAssortativity": False,
+        }
+        self.type_index = pd.Index(range(num_type), name="type")
+        self.label_index = pd.MultiIndex.from_product((range(num_type), range(num_r), range(num_type)),
+                                                      names=("s_type", "r", "t_type"))
+    def recursive_updates(self, node_batch: pd.DataFrame, edge_batch: pd.DataFrame):
+        prev_num_nodes = self.metric_values["NumNodes"][self.current_step]
+        prev_num_edges = self.metric_values["NumEdges"][self.current_step]
+        prev_type_dist = self.metric_values["NodeTypeDist"][self.current_step]
+        prev_label_dist = self.metric_values["LabelInteractionDist"][self.current_step]
+        self.metric_values["NumNodes"].append(prev_num_nodes + len(node_batch))
+        self.metric_values["NumEdges"].append(prev_num_edges + len(edge_batch))
+        self.metric_values["EdgeDensity"].append((prev_num_edges + len(edge_batch))
+                                                 / (prev_num_nodes + len(node_batch)) ** 2)
+        type_counts = prev_type_dist * prev_num_nodes + node_batch.type.value_counts().reindex(self.type_index,
+                                                                                               fill_value=0).values
+        self.metric_values["NodeTypeDist"].append(type_counts / (prev_num_nodes + len(node_batch)))
+        label_batch = edge_batch.assign(
+            s_type=self.graph.graph.vs(edge_batch.s)["type"],
+            t_type=self.graph.graph.vs(edge_batch.t)["type"],
+        )[["s_type", "r", "t_type"]]
+        new_label_counts = label_batch.value_counts().reindex(self.label_index,
+                                                              fill_value=0).values.reshape(prev_label_dist.shape)
+        label_counts = prev_label_dist * prev_num_edges + new_label_counts
+        self.metric_values["LabelInteractionDist"].append(label_counts / (prev_num_edges + len(edge_batch)))
+    def compute_metric(self, metric_name):
+        if metric_name == "RelationDist":
+            return np.nansum(self.metric_values["LabelInteractionDist"][self.current_step + 1], axis=(0, 2))
+        elif metric_name == "NodeTypeAssortativity":
+            return self.graph.graph.assortativity_nominal(types="type",
+                                                          directed=True) if len(self.type_index) > 1 else 1
+class Connectivity(MetricSet):
+    def __init__(self, graph, dashboard, num_nodes):
+        super().__init__(graph, dashboard, "Connectivity")
+        self.metric_values = {
+            "OutDegree": [np.zeros(num_nodes), ],
+            "InDegree": [np.zeros(num_nodes), ],
+            "Degree": [np.zeros(num_nodes), ],
+            "WCC": [np.zeros(num_nodes), ],
+            "SCC": [np.zeros(num_nodes), ],
+            "GiantWCC": [0, ],
+            "GiantSCC": [0, ],
+            "AveragePathLength": [0, ],
+            "Diameter": [0, ],
+            "Clustering": [np.zeros(num_nodes), ],
+            "AverageClustering": [0, ],
+            "DegreeAssortativity": [0, ],
+            "AlgebraicConnectivity": [0, ],
+        }
+        self.metric_is_recursive = {
+            "OutDegree": True,
+            "InDegree": True,
+            "Degree": True,
+            "WCC": True,
+            "SCC": True,
+            "GiantWCC": False,
+            "GiantSCC": False,
+            "AveragePathLength": False,
+            "Diameter": False,
+            "Clustering": False,
+            "AverageClustering": False,
+            "DegreeAssortativity": False,
+            "AlgebraicConnectivity": False,
+        }
+        self.node_index = pd.Index(range(num_nodes), name="n")
+    def recursive_updates(self, node_batch: pd.DataFrame, edge_batch: pd.DataFrame):
+        prev_e_count = self.graph.graph.ecount()
+        prev_out_degree = self.metric_values["OutDegree"][self.current_step]
+        prev_in_degree = self.metric_values["InDegree"][self.current_step]
+        s_counts = edge_batch.s.value_counts().reindex(self.node_index, fill_value=0).values
+        new_s_counts = prev_e_count * prev_out_degree + s_counts
+        self.metric_values["OutDegree"].append(new_s_counts / self.graph.graph.ecount())
+        t_counts = edge_batch.t.value_counts().reindex(self.node_index, fill_value=0).values
+        new_t_counts = prev_e_count * prev_in_degree + t_counts
+        self.metric_values["InDegree"].append(new_t_counts / self.graph.graph.ecount())
+        self.metric_values["Degree"].append((new_s_counts + new_t_counts) / (2 * self.graph.graph.ecount()))
+        wcc = self.graph.graph.clusters(mode="weak").sizes()
+        wcc_padded = np.zeros(len(self.node_index))
+        wcc_padded[:len(wcc)] = wcc / np.nansum(wcc)
+        self.metric_values["WCC"].append(wcc_padded)
+        scc = self.graph.graph.clusters(mode="strong").sizes()
+        scc_padded = np.zeros(len(self.node_index))
+        scc_padded[:len(scc)] = scc / np.nansum(scc)
+        self.metric_values["SCC"].append(scc_padded)
+    def compute_metric(self, metric_name):
+        if metric_name == "GiantWCC":
+            return np.nanmax(self.metric_values["WCC"][self.current_step + 1])
+        elif metric_name == "GiantSCC":
+            return np.nanmax(self.metric_values["SCC"][self.current_step + 1])
+        elif metric_name == "AveragePathLength":
+            return self.graph.graph.average_path_length(directed=True, unconn=True)
+        elif metric_name == "Diameter":
+            return self.graph.graph.diameter(directed=True, unconn=True)
+        elif metric_name == "Clustering":
+            clustering = self.graph.graph.transitivity_local_undirected(mode="zero")
+            clustering_padded = np.zeros(len(self.node_index))
+            clustering_norm = np.nansum(clustering)
+            clustering_padded[:len(clustering)] = (clustering / clustering_norm) if clustering_norm > 0 else clustering
+            return clustering_padded
+        elif metric_name == "AverageClustering":
+            return self.graph.graph.transitivity_undirected(mode="zero")
+        elif metric_name == "DegreeAssortativity":
+            return self.graph.graph.assortativity_degree(directed=True)
+        elif metric_name == "AlgebraicConnectivity":
+            A = self.graph.graph.clusters("weak").giant()
+            num_nodes = A.vcount()
+            D_ind = np.arange(num_nodes)
+            D = A.degree()
+            D = csr_matrix((D, (D_ind, D_ind)))
+            A = A.get_adjacency_sparse()
+            L = D - A
+            if num_nodes > 3:
+                return abs(eigs(L.asfptype(), k=2, which="SM", return_eigenvectors=False, tol=1e-4)[0])
+            else:
+                eigvals = np.abs(eig(L.asfptype().toarray("C"))[0])
+                eigvals = np.sort(eigvals)
+                return eigvals[1] if num_nodes > 1 else 0
+class Importance(MetricSet):
+    def __init__(self, graph, dashboard, num_nodes):
+        super().__init__(graph, dashboard, "Importance")
+        self.metric_values = {
+            "ClosenessCentrality": [np.zeros(num_nodes), ],
+            "BetweennessCentrality": [np.zeros(num_nodes), ],
+            "HubScore": [np.zeros(num_nodes), ],
+            "AuthorityScore": [np.zeros(num_nodes), ],
+            "PageRank": [np.zeros(num_nodes), ],
+            "MaxCloseness": [0, ],
+            "MaxBetweenness": [0, ],
+            "MaxHubScore": [0, ],
+            "MaxAuthorityScore": [0, ],
+            "MaxPageRank": [0, ],
+        }
+        self.metric_is_recursive = {
+            "ClosenessCentrality": True,
+            "BetweennessCentrality": True,
+            "HubScore": True,
+            "AuthorityScore": True,
+            "PageRank": True,
+            "MaxCloseness": False,
+            "MaxBetweenness": False,
+            "MaxHubScore": False,
+            "MaxAuthorityScore": False,
+            "MaxPageRank": False,
+        }
+        self.num_nodes = num_nodes
+    def recursive_updates(self, node_batch: pd.DataFrame, edge_batch: pd.DataFrame):
+        closeness = self.graph.graph.harmonic_centrality(mode="all", cutoff=10)
+        closeness_padded = np.zeros(self.num_nodes)
+        closeness_norm = np.nansum(closeness)
+        closeness_padded[:len(closeness)] = \
+            (closeness / closeness_norm) if closeness_norm > 0 else closeness
+        self.metric_values["ClosenessCentrality"].append(closeness_padded)
+        betweenness = self.graph.graph.betweenness(directed=True, cutoff=10)
+        betweenness_padded = np.zeros(self.num_nodes)
+        betweenness_norm = np.nansum(betweenness)
+        betweenness_padded[:len(betweenness)] = \
+            (betweenness / betweenness_norm) if betweenness_norm > 0 else betweenness
+        self.metric_values["BetweennessCentrality"].append(betweenness_padded)
+        hub_score = self.graph.graph.hub_score()
+        hub_score_padded = np.zeros(self.num_nodes)
+        hub_score_padded[:len(hub_score)] = hub_score / np.nansum(hub_score)
+        self.metric_values["HubScore"].append(hub_score_padded)
+        authority_score = self.graph.graph.authority_score()
+        authority_score_padded = np.zeros(self.num_nodes)
+        authority_score_padded[:len(authority_score)] = authority_score / np.nansum(authority_score)
+        self.metric_values["AuthorityScore"].append(authority_score_padded)
+        pagerank = self.graph.graph.pagerank(directed=True)
+        pagerank_padded = np.zeros(self.num_nodes)
+        pagerank_padded[:len(pagerank)] = pagerank / np.nansum(pagerank)
+        self.metric_values["PageRank"].append(pagerank_padded)
+    def compute_metric(self, metric_name):
+        if metric_name == "MaxCloseness":
+            return np.nanmax(self.metric_values["ClosenessCentrality"][self.current_step + 1])
+        elif metric_name == "MaxBetweenness":
+            return np.nanmax(self.metric_values["BetweennessCentrality"][self.current_step + 1])
+        elif metric_name == "MaxHubScore":
+            return np.nanmax(self.metric_values["HubScore"][self.current_step + 1])
+        elif metric_name == "MaxAuthorityScore":
+            return np.nanmax(self.metric_values["AuthorityScore"][self.current_step + 1])
+        elif metric_name == "MaxPageRank":
+            return np.nanmax(self.metric_values["PageRank"][self.current_step + 1])
+def get_all_motifs(k: int) -> List[igraph.Graph]:
+    possible_graph_edges = [(s, t) for s in range(k) for t in range(k) if s != t]
+    all_subgraphs_edges = [np.array(combination)
+                           for num_edges in range(k - 1, len(possible_graph_edges) + 1)
+                           for combination in itertools.combinations(possible_graph_edges, num_edges)]
+    all_motifs = []
+    for possible_edges in all_subgraphs_edges:
+        possible_new_motif = igraph.Graph(k, directed=True)
+        possible_new_motif.add_edges(possible_edges)
+        if not possible_new_motif.is_connected("weak"):
+            continue
+        is_isomorphic = False
+        for motif in all_motifs:
+            is_isomorphic = is_isomorphic or possible_new_motif.isomorphic(motif)
+        if not is_isomorphic:
+            all_motifs.append(possible_new_motif)
+    return all_motifs
+all_3_motifs = get_all_motifs(3)
+all_4_motifs = get_all_motifs(4)
+def get_esa_frequencies(graph: igraph.Graph, num_samples: int = 10000):
+    def sampled_edges_to_boundary(sampled_edges):
+        return set(np.concatenate(sampled_edges))
+    def expand_boundary(boundary, sampled_edges, prob):
+        cut = [edge.tuple for edge in di_graph.es.select(_source_in=boundary, _target_notin=boundary)]
+        cut.extend([edge.tuple for edge in di_graph.es.select(_source_notin=boundary, _target_in=boundary)])
+        cut_size = len(cut)
+        if cut_size > 0:
+            prob *= 1 / cut_size
+            new_edge_id = np.random.choice(cut_size, size=1, replace=False)[0]
+            sampled_edges.append(cut[new_edge_id])
+        else:
+            prob = 0
+        return sampled_edges_to_boundary(sampled_edges), sampled_edges, prob
+    di_graph = graph.copy().simplify()
+    num_edges = di_graph.ecount()
+    scores_3_motifs = np.zeros(len(all_3_motifs))
+    scores_4_motifs = np.zeros(len(all_4_motifs))
+    initial_edge_ids = np.random.choice(num_edges, size=num_samples, replace=True)
+    for i in tqdm(range(num_samples), desc="Processing ESA samples", leave=False):
+        prob = 1 / num_edges
+        sampled_edges = [di_graph.es[initial_edge_ids[i]].tuple, ]
+        boundary = sampled_edges_to_boundary(sampled_edges)  # 2 nodes
+        boundary, sampled_edges, prob = expand_boundary(boundary, sampled_edges, prob)  # 3 nodes
+        subgraph = di_graph.subgraph(boundary)
+        motif = [j for j, motif in enumerate(all_3_motifs) if subgraph.isomorphic(motif)]
+        scores_3_motifs[motif] += (1 / prob if prob > 0 else 0)
+        boundary, sampled_edges, prob = expand_boundary(boundary, sampled_edges, prob)  # 4 nodes
+        subgraph = di_graph.subgraph(boundary)
+        motif = [j for j, motif in enumerate(all_4_motifs) if subgraph.isomorphic(motif)]
+        scores_4_motifs[motif] += (1 / prob if prob > 0 else 0)
+    total_score_3, total_score_4 = np.nansum(scores_3_motifs), np.nansum(scores_4_motifs)
+    return scores_3_motifs / total_score_3, scores_4_motifs / total_score_4
+def get_sp_tree(graph: igraph.Graph) -> igraph.Graph:
+    pageranks_nodes = np.array(graph.pagerank(directed=True))
+    edge_frame = graph.get_edge_dataframe()
+    graph.es["weight"] = -pageranks_nodes[edge_frame.source.values] - pageranks_nodes[edge_frame.target.values]
+    del edge_frame, pageranks_nodes
+    sp_tree = graph.spanning_tree()
+    del graph.es["weight"]
+    return sp_tree
+class Subgraphing(MetricSet):
+    def __init__(self, graph, dashboard, num_nodes, community_method,
+                 community_resolution_disjoint, community_resolution_overlapping, num_esa_samples):
+        super().__init__(graph, dashboard, "Subgraphing")
+        self.num_nodes = num_nodes
+        self.community_method = community_method
+        self.community_resolution_disjoint = \
+            community_resolution_disjoint if community_resolution_disjoint > 0 else 1 / num_nodes
+        self.community_resolution_overlapping = community_resolution_overlapping
+        self.num_esa_samples = num_esa_samples
+        self.node_index = pd.Index(range(num_nodes), name="n")
+        self.egonet_splitter = EgoNetSplitter(community_resolution_overlapping)
+        self.metric_values = {
+            "DisjointCommunityMembership": [np.zeros(num_nodes), ],
+            "DisjointCommunityNumber": [0, ],
+            "DisjointCommunitySizeDist": [np.zeros(num_nodes), ],
+            "GiantDisjointCommunity": [0, ],
+            "DisjointCommunityCutSize": [0, ],
+            "DisjointCommunityModularity": [0, ],
+            "OverlappingCommunityMembership": [csr_matrix((num_nodes, 0)), ],
+            "OverlappingCommunityNumber": [0, ],
+            "OverlappingCommunitySizeDist": [np.zeros(num_nodes), ],
+            "GiantOverlappingCommunity": [0, ],
+            "OverlappingRegionSizeDist": [np.zeros(1), ],
+            "GiantOverlappingRegion": [0, ],
+            "3MotifsDist": [np.zeros(len(all_3_motifs)), ],
+            "4MotifsDist": [np.zeros(len(all_4_motifs)), ],
+            "MostFrequent3Motif": [0, ],
+            "MostFrequent4Motif": [0, ],
+        }
+        self.metric_is_recursive = {
+            "DisjointCommunityMembership": True,
+            "DisjointCommunityNumber": False,
+            "DisjointCommunitySizeDist": True,
+            "GiantDisjointCommunity": False,
+            "DisjointCommunityCutSize": True,
+            "DisjointCommunityModularity": False,
+            "OverlappingCommunityMembership": True,
+            "OverlappingCommunityNumber": True,
+            "OverlappingCommunitySizeDist": True,
+            "GiantOverlappingCommunity": False,
+            "OverlappingRegionSizeDist": True,
+            "GiantOverlappingRegion": False,
+            "3MotifsDist": True,
+            "4MotifsDist": True,
+            "MostFrequent3Motif": False,
+            "MostFrequent4Motif": False
+        }
+    def recursive_updates(self, node_batch: pd.DataFrame, edge_batch: pd.DataFrame):
+        if self.community_method != "random":
+            disjoint_partitioning = np.zeros(self.num_nodes, dtype=int)
+            wcc = self.graph.graph.clusters("weak")
+            wcc_membership, wcc_sizes = np.array(wcc.membership), wcc.sizes()
+            nodes_in_larger_wcc = np.zeros(self.num_nodes, dtype=bool)
+            small_wcc_community, small_wcc_nodes = 0, 0
+            for i, wcc_size in enumerate(wcc_sizes):
+                nodes_in_wcc = wcc_membership == i
+                if wcc_size < sqrt(self.num_nodes):
+                    if small_wcc_nodes + wcc_size > sqrt(self.num_nodes):
+                        small_wcc_community += 1
+                        small_wcc_nodes = 0
+                    disjoint_partitioning[nodes_in_wcc] = small_wcc_community
+                    small_wcc_nodes += wcc_size
+                else:
+                    nodes_in_larger_wcc[nodes_in_wcc] = True
+            nodes_in_larger_wcc = nodes_in_larger_wcc.nonzero()[0]
+        if self.community_method == "leiden":
+            if len(nodes_in_larger_wcc) != self.num_nodes:
+                graph_larger_wcc = self.graph.graph.subgraph(nodes_in_larger_wcc)
+                larger_wcc_comms = graph_larger_wcc.as_undirected(mode="each").community_leiden(
+                    "CPM", resolution_parameter=self.community_resolution_disjoint,
+                    n_iterations=-1 if self.num_nodes <= 100000 else 2
+                )
+                disjoint_partitioning[nodes_in_larger_wcc] = (np.array(larger_wcc_comms.membership)
+                                                              + small_wcc_community + 1)
+                del graph_larger_wcc
+            else:
+                disjoint_partitioning = self.graph.graph.as_undirected(mode="each").community_leiden(
+                    "CPM", resolution_parameter=self.community_resolution_disjoint,
+                    n_iterations=-1 if self.num_nodes <= 100000 else 2
+                )
+                disjoint_partitioning = np.array(disjoint_partitioning.membership)
+            edges = self.graph.graph.get_edge_dataframe()
+            cut_size = np.sum(disjoint_partitioning[edges.source] != disjoint_partitioning[edges.target])
+            self.metric_values["DisjointCommunityCutSize"].append(cut_size)
+            del wcc, wcc_membership, wcc_sizes, nodes_in_larger_wcc
+        elif self.community_method == "metis":
+            if len(nodes_in_larger_wcc) != self.num_nodes:
+                graph_larger_wcc = self.graph.graph.subgraph(nodes_in_larger_wcc)
+                graph_metis = graph_larger_wcc.get_adjlist(mode="all")
+                cut_size, larger_wcc_comms = part_graph(adjacency=graph_metis,
+                                                        nparts=int(sqrt(self.num_nodes)) - small_wcc_community)
+                disjoint_partitioning[nodes_in_larger_wcc] = np.array(larger_wcc_comms) + small_wcc_community + 1
+                del graph_larger_wcc
+            else:
+                graph_metis = self.graph.graph.get_adjlist(mode="all")
+                cut_size, disjoint_partitioning = part_graph(adjacency=graph_metis, nparts=int(sqrt(self.num_nodes)))
+                disjoint_partitioning = np.array(disjoint_partitioning)
+            self.metric_values["DisjointCommunityCutSize"].append(cut_size)
+            del wcc, wcc_membership, wcc_sizes, nodes_in_larger_wcc
+        elif self.community_method == "random":
+            disjoint_partitioning = np.random.choice(int(sqrt(self.num_nodes)), size=self.num_nodes)
+            edges = self.graph.graph.get_edge_dataframe()
+            cut_size = np.sum(disjoint_partitioning[edges.source] != disjoint_partitioning[edges.target])
+            self.metric_values["DisjointCommunityCutSize"].append(cut_size)
+        else:
+            raise ValueError("Invalid community detection method!")
+        self.metric_values["DisjointCommunityMembership"].append(disjoint_partitioning)
+        disjoint_sizes = np.unique(disjoint_partitioning, return_counts=True)[1]
+        disjoint_sizes_padded = np.zeros(self.num_nodes)
+        disjoint_sizes_padded[:len(disjoint_sizes)] = disjoint_sizes / np.nansum(disjoint_sizes)
+        self.metric_values["DisjointCommunitySizeDist"].append(disjoint_sizes_padded)
+        self.egonet_splitter.fit(self.graph.graph.clusters(mode="weak").giant().as_undirected().to_networkx())
+        M_id = [(node, community)
+                for node, communities in self.egonet_splitter.get_memberships().items()
+                for community in communities]
+        M_id = np.array(M_id)
+        M = csr_matrix((np.ones(len(M_id)), (M_id[:, 0], M_id[:, 1])))
+        self.metric_values["OverlappingCommunityMembership"].append(M)
+        overlapping_sizes = np.array(M.sum(axis=0))[0]
+        self.metric_values["OverlappingCommunityNumber"].append(len(overlapping_sizes))
+        self.metric_values["OverlappingCommunitySizeDist"].append(overlapping_sizes / np.nansum(overlapping_sizes))
+        overlapping_regions = triu(M.T @ M, k=1).data
+        overlapping_regions = overlapping_regions[overlapping_regions > 1]
+        overlapping_regions /= np.nansum(overlapping_regions)
+        self.metric_values["OverlappingRegionSizeDist"].append(
+            overlapping_regions if len(overlapping_regions) > 0 else np.zeros(1)
+        )
+        motif_dists = get_esa_frequencies(self.graph.graph, self.num_esa_samples)
+        self.metric_values["3MotifsDist"].append(motif_dists[0])
+        self.metric_values["4MotifsDist"].append(motif_dists[1])
+    def compute_metric(self, metric_name):
+        if metric_name == "DisjointCommunityNumber":
+            return np.nanmax(self.metric_values["DisjointCommunityMembership"][self.current_step + 1]) + 1
+        elif metric_name == "GiantDisjointCommunity":
+            return np.nanmax(self.metric_values["DisjointCommunitySizeDist"][self.current_step + 1])
+        elif metric_name == "DisjointCommunityModularity":
+            return self.graph.graph.modularity(
+                self.metric_values["DisjointCommunityMembership"][self.current_step + 1]
+            )
+        elif metric_name == "GiantOverlappingCommunity":
+            return np.nanmax(self.metric_values["OverlappingCommunitySizeDist"][self.current_step + 1])
+        elif metric_name == "GiantOverlappingRegion":
+            return np.nanmax(self.metric_values["OverlappingRegionSizeDist"][self.current_step + 1])
+        elif metric_name == "MostFrequent3Motif":
+            return np.nanmax(self.metric_values["3MotifsDist"][self.current_step + 1])
+        elif metric_name == "MostFrequent4Motif":
+            return np.nanmax(self.metric_values["4MotifsDist"][self.current_step + 1])
+def rule_to_edge_list_r(rule, edge_list):
+    root, children = rule if type(rule) is tuple else (rule.root, rule.children)
+    if len(children) == 0:
+        return []
+    new_edges = []
+    for r, direction, child_rule in children:
+        child_root, _ = child_rule if type(child_rule) is tuple else (child_rule.root, child_rule.children)
+        new_edges.extend([(root_type, r, child_root_type) if direction == "out" else (child_root_type, r, root_type)
+                          for root_type in root for child_root_type in child_root])
+        new_edges.extend(rule_to_edge_list_r(child_rule, edge_list))
+    return edge_list + new_edges
+def rule_to_edge_list(rule):
+    return rule_to_edge_list_r(rule, [])
+def dedensify(graph: igraph.Graph, degree_threshold: float = 0.95):
+    num_nodes = graph.vcount()
+    in_degrees = np.sort(graph.indegree())
+    degree_percentile = np.percentile(in_degrees, 100 * degree_threshold)
+    high_degree_nodes = in_degrees >= degree_percentile
+    if np.sum(high_degree_nodes) > 100000:
+        high_degree_nodes[:-100000] = False
+    high_degree_nodes = set(np.nonzero(high_degree_nodes)[0])
+    high_degree_neighbourhoods = {}
+    edge_frame = graph.get_edge_dataframe()
+    neighbourhoods = edge_frame.groupby(["source", "r"])["target"].agg(list).to_dict()
+    for (n, r), neighbourhood in tqdm(neighbourhoods.items(), desc="Processing node neighbourhoods", leave=False):
+        h = tuple(sorted(high_degree_nodes.intersection(set(neighbourhood))))
+        if len(h) > 0:
+            high_degree_neighbourhoods.setdefault((r, h), [])
+            high_degree_neighbourhoods[(r, h)].append(n)
+    num_compressor_nodes = 0
+    compressor_nodes, edges_to_add, edges_to_delete = [], [], set()
+    for (r, h), m in tqdm(high_degree_neighbourhoods.items(),
+                          desc="Processing high-degree node neighbourhoods", leave=False):
+        edges_to_delete_batch = edge_frame.index[edge_frame.source.isin(m)
+                                                 & edge_frame.target.isin(h)
+                                                 & (edge_frame.r == r)].values.tolist()
+        num_edges_to_delete = len(edges_to_delete_batch)
+        edges_to_add_batch = [(source, num_nodes + num_compressor_nodes, r) for source in m]
+        edges_to_add_batch.extend([(num_nodes + num_compressor_nodes, target, r) for target in h])
+        num_edges_to_add = len(edges_to_add_batch)
+        if num_edges_to_add < num_edges_to_delete:
+            compressor_nodes.append(f"compressor-{r}")
+            edges_to_delete.update(edges_to_delete_batch)
+            edges_to_add.extend(edges_to_add_batch)
+            num_compressor_nodes += 1
+    edge_frame.drop(index=edges_to_delete, inplace=True)
+    edge_frame = pd.concat((edge_frame,
+                            pd.DataFrame(edges_to_add, columns=["source", "target", "r"])), ignore_index=True)
+    node_frame = pd.concat((graph.get_vertex_dataframe(),
+                            pd.DataFrame(compressor_nodes, columns=["type", ])), ignore_index=True)
+    return (num_compressor_nodes / (num_nodes + num_compressor_nodes),
+            (graph.ecount() - len(edges_to_delete)) / graph.ecount()), \
+        node_frame.reset_index().rename(columns={"index": "n"}), edge_frame
+def path_summary(graph):
+    def path_summary_r(root, summaries, path, visited):
+        out_children = []
+        if root in out_neighbourhoods:
+            out_children = out_neighbourhoods[root].to_dict()
+            out_children = [(r, child)
+                            for r, children in out_children.items()
+                            for child in children if not visited[child]]
+        in_children = []
+        if root in in_neighbourhoods:
+            in_children = in_neighbourhoods[root].to_dict()
+            in_children = [(r, child)
+                           for r, children in in_children.items()
+                           for child in children if not visited[child]]
+        if len(out_children) + len(in_children) == 0:
+            return Counter([path, ])
+        new_summaries = Counter()
+        new_visited = visited
+        for r_child, child in out_children:
+            t_type_child = node_types.loc[child, "type"]
+            new_visited[child] = 1
+            progress_bar.update()
+            out_stub = f"-[:{r_child}]->(:{t_type_child})"
+            out_stub_repeated = f"-[:{r_child}*1..]->(:{t_type_child})"
+            repeated = path.endswith(f"{out_stub}{out_stub}")
+            repeated_again = path.endswith(f"{out_stub}{out_stub_repeated}")
+            new_path = path[
+                       :-len(out_stub if repeated else out_stub_repeated)
+                       ] + out_stub_repeated if repeated or repeated_again else f"{path}{out_stub}"
+            new_summaries += path_summary_r(child, summaries, new_path, new_visited)
+        for r_child, child in in_children:
+            s_type_child = node_types.loc[child, "type"]
+            new_visited[child] = 1
+            progress_bar.update()
+            in_stub = f"<-[:{r_child}]-(:{s_type_child})"
+            in_stub_repeated = f"<-[:{r_child}*1..]-(:{s_type_child})"
+            repeated = path.endswith(f"{in_stub}{in_stub}")
+            repeated_again = path.endswith(f"{in_stub}{in_stub_repeated}")
+            new_path = path[
+                       :-len(in_stub if repeated else in_stub_repeated)
+                       ] + in_stub_repeated if repeated or repeated_again else f"{path}{in_stub}"
+            new_summaries += path_summary_r(child, summaries, new_path, new_visited)
+        return summaries + new_summaries
+    sp_tree = get_sp_tree(graph.clusters("weak").giant())
+    num_sp_nodes = sp_tree.vcount()
+    node_types = sp_tree.get_vertex_dataframe()
+    sp_tree = sp_tree.get_edge_dataframe()
+    out_neighbourhoods = sp_tree.groupby(["source", "r"])["target"].agg(list)
+    in_neighbourhoods = sp_tree.groupby(["target", "r"])["source"].agg(list)
+    del sp_tree
+    root_visited = np.zeros(num_sp_nodes)
+    root_visited[0] = 1
+    progress_bar = tqdm(desc="Processing nodes for spanning tree paths", total=num_sp_nodes, leave=False)
+    progress_bar.update()
+    summaries = path_summary_r(0, Counter(), f"(:{node_types.loc[0, 'type']})", root_visited)
+    progress_bar.close()
+    return summaries
+def edge_collapsing(node_data, edge_data, node_hierarchy: dict, total_nodes):
+    node_collapsing_map = {}
+    all_edge_nodes = []
+    edge_collapsing_subset = []
+    curr_node_id = total_nodes
+    for i, edge in tqdm(edge_data.iterrows(), total=len(edge_data), desc="Collapsing edges", leave=False):
+        if edge.s not in all_edge_nodes and edge.t not in all_edge_nodes:
+            edge_collapsing_subset.append(i)
+            all_edge_nodes.extend([edge.s, edge.t])
+            node_collapsing_map[edge.s] = curr_node_id
+            node_collapsing_map[edge.t] = curr_node_id
+            curr_node_id += 1
+    edge_collapsing_subset = edge_data.loc[edge_collapsing_subset].reset_index(drop=True)
+    new_node_data = pd.DataFrame(columns=["n", "type"])
+    new_edge_data = edge_data.copy()
+    node_s_types: pd.Series = edge_collapsing_subset.merge(node_data, left_on="s", right_on="n", how="left").type
+    node_t_types = edge_collapsing_subset.merge(node_data, left_on="t", right_on="n", how="left").type
+    new_node_data = new_node_data.assign(
+        n=np.arange(total_nodes, curr_node_id),
+        type=node_s_types.combine(node_t_types, lambda s, t: s | t),
+    )
+    node_hierarchy.update(node_collapsing_map)
+    uncollapsed_nodes = node_data[~node_data.n.isin(all_edge_nodes)]
+    for node in uncollapsed_nodes.n:
+        node_collapsing_map[node] = node
+    node_collapsing_map = pd.Series(node_collapsing_map, name="n")
+    new_node_data = pd.concat((new_node_data, uncollapsed_nodes), ignore_index=True)
+    new_edge_data.s = new_edge_data.s.map(node_collapsing_map.rename("s"))
+    new_edge_data.t = new_edge_data.t.map(node_collapsing_map.rename("t"))
+    new_edge_data = new_edge_data.drop_duplicates(ignore_index=True)
+    new_edge_data = new_edge_data[new_edge_data.s != new_edge_data.t]
+    return new_node_data, new_edge_data, node_hierarchy, curr_node_id
+def star_collapsing(node_data, edge_data, node_hierarchy, total_nodes, min_star_degree=5):
+    degrees = edge_data.s.value_counts().add(edge_data.t.value_counts(), fill_value=0)
+    star_nodes = degrees[degrees >= min_star_degree].index
+    del degrees
+    node_collapsing_map = {}
+    new_node_data = []
+    curr_node_id = total_nodes
+    for node in tqdm(star_nodes, desc="Collapsing stars", leave=False):
+        for _, star_neighbours in enumerate((
+                edge_data.loc[edge_data.s == node, ["t", "r"]].rename(columns={"t": "n"}),
+                edge_data.loc[edge_data.t == node, ["s", "r"]].rename(columns={"s": "n"})
+        )):
+            for i in range(1, len(star_neighbours), 2):
+                node_1, node_2 = star_neighbours.iloc[i - 1], star_neighbours.iloc[i]
+                if node_1.n not in node_collapsing_map and node_2.n not in node_collapsing_map:
+                    node_collapsing_map[node_1.n] = curr_node_id
+                    node_collapsing_map[node_2.n] = curr_node_id
+                    node_1_type = node_data.loc[node_data.n == node_1.n, "type"].iloc[0]
+                    node_2_type = node_data.loc[node_data.n == node_2.n, "type"].iloc[0]
+                    new_node_data.append((
+                        curr_node_id,
+                        node_1_type | node_2_type
+                    ))
+                    curr_node_id += 1
+    node_hierarchy.update(node_collapsing_map)
+    uncollapsed_nodes = node_data[~node_data.n.isin(node_collapsing_map.keys())]
+    for node in uncollapsed_nodes.n:
+        node_collapsing_map[node] = node
+    node_collapsing_map = pd.Series(node_collapsing_map, name="n")
+    new_node_data = pd.DataFrame(new_node_data, columns=["n", "type"])
+    new_node_data = pd.concat((new_node_data, uncollapsed_nodes), ignore_index=True)
+    new_edge_data = edge_data.copy()
+    new_edge_data.s = new_edge_data.s.map(node_collapsing_map.rename("s"))
+    new_edge_data.t = new_edge_data.t.map(node_collapsing_map.rename("t"))
+    new_edge_data = new_edge_data.drop_duplicates(ignore_index=True)
+    new_edge_data = new_edge_data[new_edge_data.s != new_edge_data.t]
+    return new_node_data, new_edge_data, node_hierarchy, curr_node_id
+class Summarization(MetricSet):
+    def __init__(self, graph, dashboard, num_nodes, kgist_merge_rules, kgist_nest_rules,
+                 dedensification_degree_threshold, harp_node_threshold, harp_min_star_degree, results_path):
+        super().__init__(graph, dashboard, "Summarization")
+        self.num_nodes = num_nodes
+        self.kgist_merge_rules = kgist_merge_rules
+        self.kgist_nest_rules = kgist_nest_rules
+        self.dedensification_degree_threshold = dedensification_degree_threshold
+        self.harp_node_threshold = harp_node_threshold
+        self.harp_min_star_degree = harp_min_star_degree
+        self.results_path = results_path
+        self.metric_values = {
+            "KGistMDL": [0, ],
+            "KGistCompressionRatio": [0, ],
+            "KGistNumRules": [0, ],
+            "KGistEdgesExplained": [0, ],
+            "DedensificationNodeDecompressionRatio": [0, ],
+            "DedensificationEdgeCompressionRatio": [0, ],
+            "SpanningTreeNumPaths": [0, ],
+            "SpanningTreePathDist": [np.zeros(num_nodes), ],
+            "SpanningTreeTopPath": [0, ],
+            "HARPNodeCompressionRatio": [0, ],
+            "HARPEdgeCompressionRatio": [0, ],
+        }
+        self.metric_is_recursive = {
+            "KGistMDL": False,
+            "KGistCompressionRatio": False,
+            "KGistNumRules": False,
+            "KGistEdgesExplained": False,
+            "DedensificationNodeDecompressionRatio": False,
+            "DedensificationEdgeCompressionRatio": False,
+            "SpanningTreeNumPaths": True,
+            "SpanningTreePathDist": True,
+            "SpanningTreeTopPath": False,
+            "HARPNodeCompressionRatio": True,
+            "HARPEdgeCompressionRatio": True,
+        }
+        self.kgist_stats = None
+        self.dedensification_stats = None
+    def recursive_updates(self, node_batch: pd.DataFrame, edge_batch: pd.DataFrame):
+        makedirs("graph_analysis/KGist/data", exist_ok=True)
+        self.graph.graph.get_vertex_dataframe().reset_index().to_csv(
+            f"graph_analysis/KGist/data/{self.graph.dataset_name}_labels.txt",
+            sep=" ", header=False, index=False, encoding="utf-8"
+        )
+        self.graph.graph.get_edge_dataframe()[["source", "r", "target"]].to_csv(
+            f"graph_analysis/KGist/data/{self.graph.dataset_name}.txt",
+            sep=" ", header=False, index=False, encoding="utf-8"
+        )
+        kgist_graph = graph_analysis.KGist.graph.Graph(self.graph.dataset_name, idify=False, verbose=False)
+        kgist_searcher = graph_analysis.KGist.searcher.Searcher(kgist_graph)
+        kgist_model = kgist_searcher.build_model(verbosity=0)
+        if self.kgist_merge_rules:
+            kgist_model = kgist_model.merge_rules(verbosity=0)
+        if self.kgist_nest_rules:
+            kgist_model = kgist_model.nest_rules(verbosity=0)
+        self.kgist_stats = kgist_model.get_stats()
+        kgist_rule_graph_edge_list = []
+        for rule in kgist_model.rules:
+            kgist_rule_graph_edge_list.extend(rule_to_edge_list(rule))
+        kgist_rule_graph_edge_list = pd.DataFrame(kgist_rule_graph_edge_list, columns=["s_type", "r", "t_type"])
+        kgist_rule_graph_edge_list.apply(pd.to_numeric)
+        kgist_rule_graph_edge_list.to_csv(
+            f"{self.results_path}/{self.graph.dataset_name}/kgist_rule_graph.csv",
+            header=True, index=False, encoding="utf-8"
+        )
+        self.dedensification_stats, dedensified_node_data, dedensified_edge_data = \
+            dedensify(self.graph.graph, self.dedensification_degree_threshold)
+        dedensified_node_data.to_csv(
+            f"{self.results_path}/{self.graph.dataset_name}/dedensified_graph_nodes.csv",
+            header=True, index=False, encoding="utf-8"
+        )
+        dedensified_edge_data.to_csv(
+            f"{self.results_path}/{self.graph.dataset_name}/dedensified_graph_relations.csv",
+            header=True, index=False, encoding="utf-8"
+        )
+        sp_tree_path_dist = path_summary(self.graph.graph)
+        sp_tree_path_dist = pd.DataFrame(sp_tree_path_dist.items(),
+                                         columns=["Path", "Probability"]).set_index("Path").squeeze("columns")
+        sp_tree_path_dist = sp_tree_path_dist.sort_values(ascending=False) / np.nansum(sp_tree_path_dist)
+        sp_tree_path_dist.to_csv(
+            f"{self.results_path}/{self.graph.dataset_name}/spanning_tree_path_summary.csv",
+            header=True, index=True, encoding="utf-8"
+        )
+        sp_tree_path_dist_padded = np.zeros(self.num_nodes)
+        sp_tree_path_dist_padded[:len(sp_tree_path_dist)] = sp_tree_path_dist.values
+        self.metric_values["SpanningTreePathDist"].append(sp_tree_path_dist_padded)
+        self.metric_values["SpanningTreeNumPaths"].append(len(sp_tree_path_dist))
+        if self.num_nodes < 100000:
+            coarsened_graph = (
+                self.graph.graph.get_vertex_dataframe().reset_index().rename(columns={"vertex ID": "n"}),
+                self.graph.graph.get_edge_dataframe().rename(columns={"source": "s", "target": "t"}),
+                {},
+                self.num_nodes
+            )
+            coarsened_graph[0].type = coarsened_graph[0].type.apply(lambda t: {t})
+            num_edges_before = len(coarsened_graph[1])
+            while len(coarsened_graph[0]) > self.harp_node_threshold:
+                num_coarsened_nodes_before = len(coarsened_graph[0])
+                coarsened_graph = star_collapsing(*coarsened_graph, self.harp_min_star_degree)
+                coarsened_graph = edge_collapsing(*coarsened_graph)
+                num_coarsened_nodes_after = len(coarsened_graph[0])
+                if num_coarsened_nodes_before - num_coarsened_nodes_after < 10:
+                    break
+            self.metric_values["HARPNodeCompressionRatio"].append(len(coarsened_graph[0]) / self.num_nodes)
+            self.metric_values["HARPEdgeCompressionRatio"].append(len(coarsened_graph[1]) / num_edges_before)
+            coarsened_graph[0].sort_values(["n", ], ignore_index=True).to_csv(
+                f"{self.results_path}/{self.graph.dataset_name}/harp_coarsened_nodes.csv",
+                header=True, index=False, encoding="utf-8"
+            )
+            coarsened_graph[1].sort_values(["s", "t"], ignore_index=True).to_csv(
+                f"{self.results_path}/{self.graph.dataset_name}/harp_coarsened_edges.csv",
+                header=True, index=False, encoding="utf-8"
+            )
+            node_hierarchy = pd.DataFrame(list(coarsened_graph[2].items()), columns=["node", "supernode"])
+            node_hierarchy = node_hierarchy.sort_values(["node", ], ignore_index=True)
+            node_hierarchy.to_csv(
+                f"{self.results_path}/{self.graph.dataset_name}/harp_node_hierarchy.csv",
+                header=True, index=False, encoding="utf-8"
+            )
+        else:
+            self.metric_values["HARPNodeCompressionRatio"].append(0)
+            self.metric_values["HARPEdgeCompressionRatio"].append(0)
+    def compute_metric(self, metric_name) -> np.ndarray:
+        if metric_name == "KGistMDL":
+            return self.kgist_stats[0]
+        elif metric_name == "KGistCompressionRatio":
+            return self.kgist_stats[1]
+        elif metric_name == "KGistNumRules":
+            return self.kgist_stats[2]
+        elif metric_name == "KGistEdgesExplained":
+            return self.kgist_stats[3]
+        elif metric_name == "DedensificationNodeDecompressionRatio":
+            return self.dedensification_stats[0]
+        elif metric_name == "DedensificationEdgeCompressionRatio":
+            return self.dedensification_stats[1]
+        elif metric_name == "SpanningTreeTopPath":
+            return np.nanmax(self.metric_values["SpanningTreePathDist"][self.current_step + 1])

src/research/COINs-KGGeneration/graph_analysis/stats.py ADDED Viewed

	@@ -0,0 +1,158 @@

+from typing import Dict, List, Tuple
+import numpy as np
+import pandas as pd
+from scipy.sparse import csr_matrix
+from scipy.stats import chisquare, kstest, ttest_ind_from_stats
+from tqdm import tqdm
+from graph_analysis.metrics import MetricSetView
+def metric_set_histogram(metric_sets: Dict[str, List[MetricSetView]], aggregate_name: str) -> Tuple[List[MetricSetView],
+                                                                                                    List[Dict[str,
+                                                                                                              float]]]:
+    metric_sets_histogram = []
+    metric_sets_sum = []
+    for metrics_type, metric_set_list in metric_sets.items():
+        metric_set_histogram = [aggregate_name, f"{metrics_type}Histogram"]
+        metric_names = metric_set_list[0][2].keys()
+        metric_set_collected_scalars = {metric_name: [metric_set[2][metric_name][-1]
+                                                      for metric_set in metric_set_list
+                                                      if type(metric_set[2][metric_name][-1])
+                                                      not in [np.ndarray, csr_matrix]]
+                                        for metric_name in metric_names}
+        metric_set_histogram.append({f"{metric_name}Dist": [np.array(metric_values) / sum(metric_values), ]
+                                     for metric_name, metric_values in metric_set_collected_scalars.items()
+                                     if len(metric_values) > 0})
+        metric_sets_histogram.append(tuple(metric_set_histogram))
+        metric_sets_sum.append({f"{metric_name}Dist": sum(metric_values)
+                                for metric_name, metric_values in metric_set_collected_scalars.items()
+                                if len(metric_values) > 0})
+    return metric_sets_histogram, metric_sets_sum
+def estimate_and_test_pareto(metric_sets: Dict[str, List[MetricSetView]]):
+    pareto_test_table = []
+    for metrics_type, metric_set_list in metric_sets.items():
+        for graph_dataset_name, _, metric_set_values in metric_set_list:
+            for metric_name, metric_values in metric_set_values.items():
+                metric_values = metric_values[-1]
+                if "Membership" in metric_name:
+                    continue
+                if type(metric_values) is not np.ndarray:
+                    continue
+                x = metric_values[metric_values > 0]
+                n = len(x)
+                if n == 0:
+                    continue
+                pareto_min_hat = np.min(x)
+                pareto_alpha_hat = n / np.sum(np.log(x / pareto_min_hat)) if (np.max(x) != pareto_min_hat
+                                                                              and n > 1) else 0
+                pareto_cdf = 1 - ((pareto_min_hat / x) ** pareto_alpha_hat)
+                pareto_p_value = kstest(x, pareto_cdf, alternative="two-sided").pvalue
+                pareto_test_table.append((metric_name, graph_dataset_name, pareto_alpha_hat, pareto_p_value))
+    pareto_test_table = pd.DataFrame(pareto_test_table, columns=["Metric", "Dataset", "ParetoAlpha", "ParetoPValue"])
+    return pareto_test_table
+def estimate_and_test_uniform(metric_set_histograms: List[MetricSetView], metric_sets_sums: List[Dict[str, float]]):
+    uniform_test_table = []
+    for metric_set, metric_set_sums in zip(metric_set_histograms, metric_sets_sums):
+        for metric_name, metric_values in metric_set[2].items():
+            metric_values = metric_values[-1]
+            if "Membership" in metric_name:
+                continue
+            if type(metric_values) is not np.ndarray:
+                continue
+            x = metric_values[metric_values > 0]
+            n = len(x)
+            if n == 0:
+                continue
+            rel_entropy = (-np.sum(x * np.log(x)) / np.log(n)) if n > 1 else 1
+            uniform_p_value = chisquare(x * metric_set_sums[metric_name]).pvalue if n > 1 else 1
+            uniform_test_table.append((metric_name, metric_set[0], rel_entropy, uniform_p_value))
+    uniform_test_table = pd.DataFrame(uniform_test_table,
+                                      columns=["Metric", "Dataset", "RelativeEntropy", "UniformPValue"])
+    return uniform_test_table
+def estimate_and_test_comparison(metric_sets: Dict[str, List[MetricSetView]],
+                                 linear_comparison: bool,
+                                 num_bootstrap_samples: int = 100):
+    def prepare_for_kde(x, y, sample_size=10000):
+        n, m = len(x), len(y)
+        if n > sample_size:
+            subsample_indices = np.linspace(0, n - 1, num=sample_size, dtype=int)
+            x = x[subsample_indices]
+            x /= np.sum(x)
+            n = sample_size
+        if m > sample_size:
+            subsample_indices = np.linspace(0, m - 1, num=sample_size, dtype=int)
+            y = y[subsample_indices]
+            y /= np.sum(y)
+            m = sample_size
+        xy = np.concatenate((x, y), axis=None).reshape((-1, 1))
+        dists = np.abs(xy.reshape(-1, 1) - xy.reshape(1, -1))
+        sigma = 0.5 * np.nanmedian(dists)
+        sigma = 1e-8 if sigma == 0 else sigma
+        return n, m, xy, sigma
+    def gaussian_kernel(x, y, sigma):
+        dists = np.abs(x.reshape(-1, 1) - y.reshape(1, -1))
+        return np.exp(-(dists ** 2) / (2 * sigma ** 2))
+    def compute_mmd(n, m, xy, sigma):
+        k = gaussian_kernel(xy, xy, sigma)
+        k_x, k_y, k_xy = np.mean(k[:n, :n]), np.mean(k[-m:, -m:]), np.mean(k[:n, -m:])
+        mmd = k_x + k_y - 2 * k_xy
+        return mmd
+    comparison_matrices = []
+    for metrics_type, metric_set_list in metric_sets.items():
+        num_datasets = len(metric_set_list)
+        if linear_comparison:
+            dataset_indices = [(0, j) for j in range(1, num_datasets)]
+        else:
+            dataset_indices = [(i, j) for i in range(num_datasets) for j in range(i + 1, num_datasets)]
+        for i, j in dataset_indices:
+            metric_set, metric_set_2 = metric_set_list[i], metric_set_list[j]
+            if metric_set[0] == metric_set_2[0]:
+                continue
+            for metric_name in metric_set[2].keys():
+                metric_values = metric_set[2][metric_name][-1]
+                metric_values_2 = metric_set_2[2][metric_name][-1]
+                if "Membership" in metric_name:
+                    continue
+                if type(metric_values) is not np.ndarray:
+                    continue
+                metric_values, metric_values_2 = metric_values.flatten(), metric_values_2.flatten()
+                metric_values = metric_values[metric_values > 0]
+                metric_values_2 = metric_values_2[metric_values_2 > 0]
+                n_1, n_2 = len(metric_values), len(metric_values_2)
+                if n_1 == 0 or n_2 == 0:
+                    continue
+                if np.max(metric_values) == np.min(metric_values) \
+                        and np.max(metric_values_2) == np.min(metric_values_2):
+                    comparison_matrices.append((metric_set[0], metric_set_2[0],
+                                                metric_name, 0, 1, 0, 1))
+                    continue
+                t_test = ttest_ind_from_stats(np.nanmean(metric_values), np.nanstd(metric_values), n_1,
+                                              np.nanmean(metric_values_2), np.nanstd(metric_values_2), n_2,
+                                              equal_var=False)
+                n, m, xy, sigma = prepare_for_kde(metric_values, metric_values_2)
+                mmd_estimate = compute_mmd(n, m, xy, sigma)
+                xys = [np.random.permutation(xy) for _ in range(num_bootstrap_samples)]
+                mmds_bootstrap_samples = [compute_mmd(n, m, xy, sigma)
+                                          for xy in tqdm(xys, desc=f"Bootstrapping {metric_name} MMD", leave=False)]
+                mmds_bootstrap_samples = np.array(mmds_bootstrap_samples)
+                mmd_p_value = np.mean(mmd_estimate <= mmds_bootstrap_samples)
+                comparison_matrices.append((metric_set[0], metric_set_2[0],
+                                            metric_name, t_test.statistic, t_test.pvalue,
+                                            mmd_estimate, mmd_p_value))
+    comparison_matrices = pd.DataFrame(comparison_matrices,
+                                       columns=["Dataset1", "Dataset2", "Metric",
+                                                "TStat", "TTestPValue", "MMD", "MMDPValue"])
+    return comparison_matrices

src/research/COINs-KGGeneration/graph_completion/README.md ADDED Viewed

	@@ -0,0 +1,89 @@

+# Graph Embedding Methods for Graph Completion
+*Repository containing the implementation of the second version of the Digital Lab model for graph completion and link prediction on the Monty data*
+---
+## Abstract
+<div style="text-align: justify"> Knowledge graphs have recently attracted significant attention from both industry and
+academia in scenarios that require exploiting large-scale heterogeneous data collections.
+Knowledge graphs are a type of database where the general structure is a network of
+entities, their semantic types, properties, and relationships. They support reasoning over
+the integrated information as the main application. This process is very closely linked
+to solving the problems of link prediction and query answering for the knowledge graph.
+The most common approach in tackling these tasks is to compute suitable numeric
+representations for each element of the graph, called graph embeddings. </div>
+## File structure
+```
+.
+├── Results - Graph Preprocessing.ipynb: Jupyter notebook containing the code to generate the network science and graph preprocessing results.
+├── Results - Training & Evaluation.ipynb: Jupyter notebook containing the code to analyse the model training and evaluation results.
+├── database.py: Module implementing the interface for connecting to the Neo4j database and executing the required Cypher queries.
+├── extract_data.py: Script for extracting the Swisscom graph data from the database and preprocessing the obtained files into cleaner formats. Contains also two methods used in the preprocessing of the other data sources as well.
+├── graphs
+│   ├── load_graph.py: Module containing methods for creating graph objects, from an edge dataset or using the algorithm for simulation.
+│   ├── preprocess.py: Module containing the implementation of the context extraction process.
+│   └── random_walks.py: Module containing the implementation of the context extraction process.
+├── load_data.py: Module containing methods for loading graph from disk.
+├── main.py: Model training and evaluation script.
+├── models
+│   ├── autoencoders.py: Module containing the implementation of general autoencoders.
+│   ├── embedding_aggregation_model.py: Module containing the implementation of the new embedding model and its loss function.
+│   ├── evaluate.py: Module containing the implementation of the evaluation procedure.
+│   ├── graph_embedders.py: Module containing the implementation of the embedding models from related work.
+│   ├── helper_layers.py: Module containing implementations of helper layers to the embedding models from previous work.
+│   ├── link_rankers.py: Module containing the implementation of the link prediction models from related work.
+│   ├── loss_terms.py: Module containing the implementation of the specific loss functions utilized by embedding models from related work.
+│   ├── timestamp_prediction.py: Module containing the implementation of timestamp prediction.
+│   └── utils.py: Module containing the implementation of important utility methods required for model training and evaluation.
+├── preprocess.py: Module implementing general pipelines for iteratively cleaning data obtained from the database.
+├── serialization.py: Module containing helper functions for (de)serializing and (de)compressing Python objects from/to local storage.
+└── test.py: Model testing script.
+```
+## Notes
+### Prerequisites
+- Python >=3.7 is installed
+- Run ```python -m pip install -r requirements.txt``` from the root folder to install the dependencies
+### Execution
+- To execute model training and evaluation run ```python main.py --config_file <PATH_TO_YAML_CONFIG_FILE>```
+- Example: ```python main.py --config_file ./config_wordnet.yml```
+- The YAML config file should contain the following model configuration settings:
+    - ```dataset_folder_path```: path on disk of the folder where the execution files should be stored, ```str```
+    - ```nodes_filepath```: path on disk of the node list file, ```str```
+    - ```relations_filepath```: path on disk of the edge list file, ```str```
+    - ```simulated_data```: whether a fake graph should be generated instead, ```bool```
+    - ```train```: whether a model should be trained or one is already available, ```bool```
+    - ```num_nodes```: number of nodes in the graph, or desired number of nodes for the simulated graph, positive ```int```
+    - ```num_node_types```: number of node types in the graph, positive ```int```
+    - ```num_relation_types```: number of edge types in the graph, positive ```int```
+    - ```num_higher_communities```: desired number of higher communities, positive ```int```
+    - ```num_lower_communities```: desired number of lower communities, positive ```int```
+    - ```random_walk_length```: chosen maximum random walk length, positive ```int```
+    - ```context_radius```: chosen context set radius, positive ```int```
+    - ```num_epochs```: chosen maximum number of model training epochs, positive ```int```
+    - ```batch_size```: chosen number of samples per batch, positive ```int```
+    - ```validation_patience```: number of epochs to wait before early stopping, positive ```int```
+    - ```verbosity_frequency```: after how many epochs to report training progress, positive ```int```
+    - ```validation_size```: fraction of dataset samples that should be dedicated to validation, ```float``` in [0.0, 1.0]
+    - ```test_size```: fraction of dataset samples that should be dedicated to testing, ```float``` in [0.0, 1.0]
+    - ```use_gpus```: whether to transfer all execution to a GPU if available,  ```bool```
+    - ```train_coverage```: fraction of train set size that should be covered by the context triplet set, ```float``` in [0.0, 1.0]
+    - ```pagerank_importances```: whether to use PageRank or uniform node importances for random walk generation, ```bool```
+    - ```walks_relation_specific```: whether to generate relation-specific random walks, ```bool```
+    - ```embedding_dim```: chosen embedding dimension, positive ```int```
+    - ```community_model```: chosen community and type embedding model from related work, ```str``` in {"transe"<sup>1</sup>, "rotate"<sup>2</sup>, "mlp"<sup>3</sup>, "gatne"<sup>3</sup>, "sacn"<sup>1</sup>}, superscripts denote compatibility with other model settings
+    - ```node_models```: chosen model from related work for intra- and inter-community embedding, ```str``` in {"transe"<sup>1</sup>, "rotate"<sup>2</sup>, "mlp"<sup>3</sup>, "gatne"<sup>3</sup>, "sacn"<sup>1</sup>}, superscripts denote compatibility with other model settings
+    - ```link_ranker```: chosen edge scoring model from related work, ```str``` in {"transe"<sup>1</sup>, "rotate"<sup>2</sup>, "mlp"<sup>3</sup>, "gatne"<sup>3</sup>, "sacn"<sup>1</sup>}, superscripts denote compatibility with other model settings
+- To simply test a trained model for query answering through an interactive command line, run ```python test.py --config_file <PATH_TO_YAML_CONFIG_FILE>```
+    - Type ```query <HEAD_NODE> <RELATION> [<TAIL_NODE_TYPE>]``` to get the model's response to the given relation query;
+    - Type ```quit``` to exit the interactive programme.
+### Diagram of the execution pipeline for main.py
+![Execution pipeline](execution_pipeline.svg)
+## Authors
+- Andrej Janchevski, INI-DNA-NB

src/research/COINs-KGGeneration/graph_completion/__init__.py ADDED Viewed

File without changes

src/research/COINs-KGGeneration/graph_completion/configs/freebase.yml ADDED Viewed

	@@ -0,0 +1,29 @@

+seed: 4089853924
+device: cuda:0
+val_size: 0.01
+test_size: 0.02
+mini_batch_size: 256
+lr: 1.0e-3
+weight_decay: 1.0e-6
+val_patience: 50
+val_tolerance: 1.0e-4
+algorithm: transe
+leiden_resolution: 5.0e-3
+loader_hpars:
+  dataset_name: freebase
+  simulated: false
+  sample_source: smore
+  sampler_hpars:
+    num_negative_samples: 128
+    num_neighbours: 10
+    random_walk_length: 10
+    context_radius: 2
+    pagerank_importances: true
+    walks_relation_specific: true
+embedder_hpars:
+  embedding_dim: 100
+link_ranker_hpars:
+  algorithm: transe
+embedding_loss_hpars:
+  margin: 1.0
+  rotate_alpha: 1.0

src/research/COINs-KGGeneration/graph_completion/configs/freebase_betae.yml ADDED Viewed

	@@ -0,0 +1,44 @@

+seed: 123456789
+device: cuda:0
+val_size: 0.01
+test_size: 0.02
+mini_batch_size: 512
+lr: 1.0e-4
+weight_decay: 1.0e-6
+val_patience: 50
+val_tolerance: 1.0e-4
+max_epochs: 60
+validation_freq: 12
+checkpoint_freq: 12000
+algorithm: betae
+leiden_resolution: 5.0e-3
+loader_hpars:
+  dataset_name: freebase
+  simulated: false
+  sample_source: smore
+  sampler_hpars:
+    query_structure:
+      - 1p
+      - 2p
+      - 3p
+      - 2i
+      - 3i
+      - 2i1p
+      - 1p2i
+      - 2d
+      - 3d
+      - 2d1p
+      - 1p2b
+      - 1p2d
+    num_negative_samples: 128
+    num_neighbours: 10
+    random_walk_length: 10
+    context_radius: 2
+    pagerank_importances: true
+    walks_relation_specific: true
+embedder_hpars:
+  embedding_dim: 400
+link_ranker_hpars:
+  algorithm: betae
+embedding_loss_hpars:
+  margin: 60.0

src/research/COINs-KGGeneration/graph_completion/configs/freebase_complex.yml ADDED Viewed

	@@ -0,0 +1,31 @@

+seed: 4089853924
+device: cuda:0
+val_size: 0.01
+test_size: 0.02
+mini_batch_size: 256
+lr: 1.0e-3
+weight_decay: 1.0e-6
+val_patience: 50
+val_tolerance: 1.0e-4
+max_epochs: 50
+validation_freq: 10
+checkpoint_freq: 10000
+algorithm: complex
+leiden_resolution: 5.0e-3
+loader_hpars:
+  dataset_name: freebase
+  sample_source: smore
+  sampler_hpars:
+    num_negative_samples: 128
+    num_neighbours: 10
+    random_walk_length: 10
+    context_radius: 2
+    pagerank_importances: true
+    walks_relation_specific: true
+embedder_hpars:
+  embedding_dim: 100
+link_ranker_hpars:
+  algorithm: complex
+embedding_loss_hpars:
+  margin: 1.0
+  rotate_alpha: 1.0

src/research/COINs-KGGeneration/graph_completion/configs/freebase_distmult.yml ADDED Viewed

	@@ -0,0 +1,31 @@

+seed: 4089853924
+device: cuda:0
+val_size: 0.01
+test_size: 0.02
+mini_batch_size: 256
+lr: 1.0e-3
+weight_decay: 1.0e-6
+val_patience: 50
+val_tolerance: 1.0e-4
+max_epochs: 50
+validation_freq: 10
+checkpoint_freq: 10000
+algorithm: distmult
+leiden_resolution: 5.0e-3
+loader_hpars:
+  dataset_name: freebase
+  sample_source: smore
+  sampler_hpars:
+    num_negative_samples: 128
+    num_neighbours: 10
+    random_walk_length: 10
+    context_radius: 2
+    pagerank_importances: true
+    walks_relation_specific: true
+embedder_hpars:
+  embedding_dim: 100
+link_ranker_hpars:
+  algorithm: distmult
+embedding_loss_hpars:
+  margin: 1.0
+  rotate_alpha: 1.0

src/research/COINs-KGGeneration/graph_completion/configs/freebase_gnn.yml ADDED Viewed

	@@ -0,0 +1,34 @@

+seed: 123456789
+device: cuda:0
+val_size: 0.01
+test_size: 0.02
+mini_batch_size: 64
+lr: 1.0e-3
+weight_decay: 1.0e-6
+val_patience: 50
+val_tolerance: 1.0e-4
+max_epochs: 50
+validation_freq: 100
+checkpoint_freq: 10000
+algorithm: kbgat
+leiden_resolution: 5.0e-3
+transe_initialize: true
+loader_hpars:
+  dataset_name: freebase
+  simulated: false
+  sample_source: smore
+  sampler_hpars:
+    num_negative_samples: 64
+    num_neighbours: 10
+    random_walk_length: 10
+    context_radius: 2
+    pagerank_importances: true
+    walks_relation_specific: true
+embedder_hpars:
+  embedding_dim: 100
+link_ranker_hpars:
+  algorithm: kbgat
+  sacn_num_conv_channels: 300
+embedding_loss_hpars:
+  margin: 9.0
+  rotate_alpha: 1.0

src/research/COINs-KGGeneration/graph_completion/configs/freebase_gqe.yml ADDED Viewed

	@@ -0,0 +1,39 @@

+seed: 123456789
+device: cuda:0
+val_size: 0.01
+test_size: 0.02
+mini_batch_size: 512
+lr: 1.0e-4
+weight_decay: 1.0e-6
+val_patience: 50
+val_tolerance: 1.0e-4
+max_epochs: 35
+validation_freq: 7
+checkpoint_freq: 7000
+algorithm: gqe
+leiden_resolution: 5.0e-3
+loader_hpars:
+  dataset_name: freebase
+  simulated: false
+  sample_source: smore
+  sampler_hpars:
+    query_structure:
+      - 1p
+      - 2p
+      - 3p
+      - 2i
+      - 3i
+      - 2i1p
+      - 1p2i
+    num_negative_samples: 128
+    num_neighbours: 10
+    random_walk_length: 10
+    context_radius: 2
+    pagerank_importances: true
+    walks_relation_specific: true
+embedder_hpars:
+  embedding_dim: 800
+link_ranker_hpars:
+  algorithm: gqe
+embedding_loss_hpars:
+  margin: 24.0

src/research/COINs-KGGeneration/graph_completion/configs/freebase_q2b.yml ADDED Viewed

	@@ -0,0 +1,40 @@

+seed: 1503136574
+device: cuda:0
+val_size: 0.01
+test_size: 0.02
+mini_batch_size: 256
+lr: 1.0e-4
+weight_decay: 1.0e-6
+val_patience: 50
+val_tolerance: 1.0e-4
+max_epochs: 35
+validation_freq: 7
+checkpoint_freq: 7000
+algorithm: q2b
+leiden_resolution: 5.0e-3
+coins_shared_relation_embedding: true
+loader_hpars:
+  dataset_name: freebase
+  simulated: false
+  sample_source: smore
+  sampler_hpars:
+    query_structure:
+      - 1p
+      - 2p
+      - 3p
+      - 2i
+      - 3i
+      - 2i1p
+      - 1p2i
+    num_negative_samples: 128
+    num_neighbours: 10
+    random_walk_length: 10
+    context_radius: 2
+    pagerank_importances: true
+    walks_relation_specific: true
+embedder_hpars:
+  embedding_dim: 400
+link_ranker_hpars:
+  algorithm: q2b
+embedding_loss_hpars:
+  margin: 24.0

src/research/COINs-KGGeneration/graph_completion/configs/freebase_rotate.yml ADDED Viewed

	@@ -0,0 +1,29 @@

+seed: 4089853924
+device: cuda:0
+val_size: 0.01
+test_size: 0.02
+mini_batch_size: 256
+lr: 1.0e-3
+weight_decay: 1.0e-6
+val_patience: 50
+val_tolerance: 1.0e-4
+algorithm: rotate
+leiden_resolution: 5.0e-3
+loader_hpars:
+  dataset_name: freebase
+  simulated: false
+  sample_source: smore
+  sampler_hpars:
+    num_negative_samples: 128
+    num_neighbours: 10
+    random_walk_length: 10
+    context_radius: 2
+    pagerank_importances: true
+    walks_relation_specific: true
+embedder_hpars:
+  embedding_dim: 100
+link_ranker_hpars:
+  algorithm: rotate
+embedding_loss_hpars:
+  margin: 9.0
+  rotate_alpha: 1.0

src/research/COINs-KGGeneration/graph_completion/configs/hep.yml ADDED Viewed

	@@ -0,0 +1,28 @@

+seed: 123456789
+device: cuda:0
+val_size: 0.01
+test_size: 0.02
+mini_batch_size: 25
+lr: 1.0e-4
+weight_decay: 1.0e-6
+val_patience: 50
+val_tolerance: 1.0e-4
+algorithm: transe
+coins_alpha: 0.01
+loader_hpars:
+  dataset_name: hep
+  simulated: false
+  sample_source: smore
+  sampler_hpars:
+    num_negative_samples: 5
+    num_neighbours: 10
+    random_walk_length: 10
+    context_radius: 2
+    pagerank_importances: true
+    walks_relation_specific: true
+embedder_hpars:
+  embedding_dim: 25
+link_ranker_hpars:
+  algorithm: transe
+embedding_loss_hpars:
+  margin: 1.0

src/research/COINs-KGGeneration/graph_completion/configs/nell.yml ADDED Viewed

	@@ -0,0 +1,30 @@

+seed: 3192206669
+device: cuda:0
+val_size: 0.01
+test_size: 0.02
+mini_batch_size: 256
+lr: 1.0e-3
+weight_decay: 1.0e-6
+val_patience: 50
+val_tolerance: 1.0e-4
+max_epochs: 10
+algorithm: transe
+leiden_resolution: 2.0e-5
+loader_hpars:
+  dataset_name: nell
+  simulated: false
+  sample_source: smore
+  sampler_hpars:
+    num_negative_samples: 128
+    num_neighbours: 10
+    random_walk_length: 10
+    context_radius: 2
+    pagerank_importances: true
+    walks_relation_specific: true
+embedder_hpars:
+  embedding_dim: 100
+link_ranker_hpars:
+  algorithm: transe
+embedding_loss_hpars:
+  margin: 1.0
+  rotate_alpha: 0.5

src/research/COINs-KGGeneration/graph_completion/configs/nell_betae.yml ADDED Viewed

	@@ -0,0 +1,44 @@

+seed: 123456789
+device: cuda:0
+val_size: 0.01
+test_size: 0.02
+mini_batch_size: 512
+lr: 1.0e-4
+weight_decay: 1.0e-6
+val_patience: 50
+val_tolerance: 1.0e-4
+max_epochs: 120
+validation_freq: 12
+checkpoint_freq: 12000
+algorithm: betae
+leiden_resolution: 2.0e-5
+loader_hpars:
+  dataset_name: nell
+  simulated: false
+  sample_source: smore
+  sampler_hpars:
+    query_structure:
+      - 1p
+      - 2p
+      - 3p
+      - 2i
+      - 3i
+      - 2i1p
+      - 1p2i
+      - 2d
+      - 3d
+      - 2d1p
+      - 1p2b
+      - 1p2d
+    num_negative_samples: 128
+    num_neighbours: 10
+    random_walk_length: 10
+    context_radius: 2
+    pagerank_importances: true
+    walks_relation_specific: true
+embedder_hpars:
+  embedding_dim: 400
+link_ranker_hpars:
+  algorithm: betae
+embedding_loss_hpars:
+  margin: 60.0