notebooks/1_biokg_training_inference.ipynb

{
    "cells": [
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# KGE Training and Inference on OGBL-BioKG\n",
                "\n",
                "<em>Copyright (c) 2023 Graphcore Ltd. All rights reserved.</em>\n",
                "\n",
                "BESS-KGE (`besskge`) is a PyTorch library for knowledge graph embedding (KGE) models on IPUs implementing the distribution framework [BESS](https://arxiv.org/abs/2211.12281), with embedding tables stored in the IPU SRAM.\n",
                "\n",
                "In this notebook we will learn how to use the BESS-KGE library to train KGE models and perform link prediction inference, using the biomedical dataset [ogbl-biokg](https://ogb.stanford.edu/docs/linkprop/#ogbl-biokg)."
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Environment setup\n",
                "\n",
                "To run the demo using IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to do this. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine."
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Dependencies"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "We recommend that you install `besskge` directly from the GitHub sources:"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 1,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Found existing installation: besskge 0.1\n",
                        "Uninstalling besskge-0.1:\n",
                        "  Successfully uninstalled besskge-0.1\n"
                    ]
                }
            ],
            "source": [
                "import sys\n",
                "!{sys.executable} -m pip uninstall -y besskge\n",
                "!pip install -q git+https://github.com/graphcore-research/bess-kge.git\n",
                "\n",
                "!pip install -q matplotlib"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Next, import the necessary dependencies. "
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 3,
            "metadata": {},
            "outputs": [],
            "source": [
                "import os\n",
                "import pathlib\n",
                "import time\n",
                "\n",
                "import matplotlib.pyplot as plt\n",
                "import numpy as np\n",
                "import poptorch\n",
                "import torch\n",
                "\n",
                "from besskge.batch_sampler import RigidShardedBatchSampler\n",
                "from besskge.bess import EmbeddingMovingBessKGE, ScoreMovingBessKGE\n",
                "from besskge.dataset import KGDataset\n",
                "from besskge.loss import LogSigmoidLoss\n",
                "from besskge.metric import Evaluation\n",
                "from besskge.negative_sampler import (\n",
                "    RandomShardedNegativeSampler,\n",
                "    TripleBasedShardedNegativeSampler,\n",
                ")\n",
                "from besskge.scoring import RotatE\n",
                "from besskge.sharding import PartitionedTripleSet, Sharding\n",
                "\n",
                "dataset_directory = os.getenv(\"DATASET_DIR\", \"../datasets/\") + \"/biokg/\""
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Sharding entities and triples\n",
                "\n",
                "The OGBL-BioKG dataset can be downloaded and preprocessed with the built-in `KGDataset` method, `build_ogbl_biokg`. `KGDataset` is the standard class which holds data from the knowledge graph dataset, such as head-relation-tail triples (with entities and relation types suitably converted to their integer IDs), triple-specific data (for example negative heads/tails to be used to corrupt the triple), ID -> label lists for entities and relation types, and so on."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Number of entities: 93,773\n",
                        "\n",
                        "Number of relation types: 51\n",
                        "\n",
                        "Number of triples: \n",
                        " training: 4,762,678 \n",
                        " validation/test: 162,886\n",
                        "\n",
                        "Number of negative heads/tails for validation/test triples: 500\n"
                    ]
                }
            ],
            "source": [
                "biokg = KGDataset.build_ogbl_biokg(root=pathlib.Path(dataset_directory))\n",
                "\n",
                "print(f\"Number of entities: {biokg.n_entity:,}\\n\")\n",
                "print(f\"Number of relation types: {biokg.n_relation_type}\\n\")\n",
                "print(\n",
                "    f\"Number of triples: \\n training: {biokg.triples['train'].shape[0]:,} \\n validation/test: {biokg.triples['valid'].shape[0]:,}\\n\"\n",
                ")\n",
                "print(\n",
                "    f\"Number of negative heads/tails for validation/test triples: {biokg.neg_heads['valid'].shape[-1]}\"\n",
                ")"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Entities in the OGBL-BioKG dataset have different types. Entity IDs need to be always assigned so that entities of the same type have contiguous IDs. Then, the ID offsets corresponding to different types are stored in `KGDataset.type_offsets`:"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 5,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "{'disease': 0,\n",
                            " 'drug': 10687,\n",
                            " 'function': 21220,\n",
                            " 'protein': 66305,\n",
                            " 'sideeffect': 83804}"
                        ]
                    },
                    "execution_count": 5,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "biokg.type_offsets"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "For example, entities with ID in the range [0, 10686] are of type 'disease'."
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "To train on 4 IPUs, we shard the entity set into four parts of equal size. This is done using the `Sharding` class. To use a different number of IPUs, just change the value of the `n_shard` variable."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 6,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Number of shards: 4\n",
                        "\n",
                        "Number of entities in each shard: 23,444\n",
                        "\n",
                        "Global entity IDs on 4 shards:\n",
                        " [[    0    10    23 ... 93768 93770 93774]\n",
                        " [    2     3     4 ... 93760 93769 93772]\n",
                        " [    7     8    15 ... 93771 93773 93775]\n",
                        " [    1     5     9 ... 93757 93763 93767]]\n",
                        "\n",
                        "Number of actual (=non-padding) entities per shard:\n",
                        " [23443 23444 23442 23444]\n",
                        "\n",
                        "Type offsets per shard: \n",
                        " [[    0  2582  5203 16455 20861]\n",
                        " [    0  2715  5354 16661 20954]\n",
                        " [    0  2755  5422 16569 21008]\n",
                        " [    0  2635  5241 16620 20981]]\n"
                    ]
                }
            ],
            "source": [
                "seed = 1234\n",
                "n_shard = 4\n",
                "\n",
                "sharding = Sharding.create(\n",
                "    n_entity=biokg.n_entity,\n",
                "    n_shard=n_shard,\n",
                "    seed=seed,\n",
                "    type_offsets=np.fromiter(biokg.type_offsets.values(), dtype=np.int32),\n",
                ")\n",
                "\n",
                "print(f\"Number of shards: {sharding.n_shard}\\n\")\n",
                "\n",
                "print(f\"Number of entities in each shard: {sharding.max_entity_per_shard:,}\\n\")\n",
                "\n",
                "print(f\"Global entity IDs on {n_shard} shards:\\n {sharding.shard_and_idx_to_entity}\\n\")\n",
                "\n",
                "# If the number of entities is not divisible by n_shard,\n",
                "# some shards will have one trailing padding entity (ID >= n_entity)\n",
                "print(\n",
                "    f\"Number of actual (=non-padding) entities per shard:\\n {sharding.shard_counts}\\n\"\n",
                ")\n",
                "\n",
                "# Entities of the same type maintain contiguous local IDs in each shard\n",
                "print(f\"Type offsets per shard: \\n\", sharding.entity_type_offsets)"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Since the total number of entities (93,773) is not divisible by `n_shard` (4), some padding entities are created. These entities have global entity IDs with values greater than or equal to 93,773 and can be seen printed above.\n",
                "\n",
                "The entity sharding induces a partitioning of the set of training triples into `n_shard ** 2 = 16` **shard-pairs**, based on the entity shard of the head and of the tail. Triple partitioning is performed using the `PartitionedTripleSet` class."
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "<p align=\"left\">\n",
                "<img src=\"https://raw.githubusercontent.com/graphcore-research/bess-kge/main/notebooks/img/batch.png\"  width=\"500\" >\n",
                "</p>"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 7,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Number of triples per (h,t) shard-pair:\n",
                        " [[304061 299922 282942 298834]\n",
                        " [309129 305430 290318 303298]\n",
                        " [300612 296719 277542 294614]\n",
                        " [308192 302322 286517 302226]]\n"
                    ]
                }
            ],
            "source": [
                "train_triples = PartitionedTripleSet.create_from_dataset(\n",
                "    dataset=biokg, part=\"train\", sharding=sharding, partition_mode=\"ht_shardpair\"\n",
                ")\n",
                "\n",
                "# train_triples.triple_counts[i,j] is the number of triples with head entity in shard i and tail entity in shard j\n",
                "print(f\"Number of triples per (h,t) shard-pair:\\n {train_triples.triple_counts}\")"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Triples in `train_triples.triples` have been sorted based on the sequence of (h,t) shard-pairs: `(0,0), (0,1), ..., (n_shard-1, n_shard-1)`. You can use `train_triples.triple_sort_idx` to recover the original ordering.\n",
                "\n",
                "Moreover, the global entity IDs for heads and tails have been replaced with the local indices on the corresponding shard."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 8,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": [
                            "True"
                        ]
                    },
                    "execution_count": 8,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "# Put original triples in the same order as train_triples.triples\n",
                "triple_sorted = biokg.triples[\"train\"][train_triples.triple_sort_idx]\n",
                "# Pass from global IDs to local IDs with sharding.entity_to_idx\n",
                "triple_sorted[:, 0] = sharding.entity_to_idx[triple_sorted[:, 0]]\n",
                "triple_sorted[:, 2] = sharding.entity_to_idx[triple_sorted[:, 2]]\n",
                "# Compare with the content of train_triples.triples\n",
                "np.all(triple_sorted == train_triples.triples)"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Negative and batch samplers\n",
                "\n",
                "A key component in training KGE models is the selection of negative samples to contrast against positive triples. A standard strategy to construct negative samples is to replace (corrupt) either the head or the tail of a positive triple with another entity.\n",
                "\n",
                "To sample the negative entities used to corrupt positive triples, we use a **negative sampler**. You can find different types of negative samplers implemented in `besskge.negative_sampler`. Two suitable samplers are:\n",
                "\n",
                "* `RandomShardedNegativeSampler`, which randomly picks the negative entity among all the entities in the knowledge graph;\n",
                "* `TypeBasedShardedNegativeSampler`, which selects the corrupted entity only among entities of the same type as the original one.\n",
                "\n",
                "By default, BESS samples the same number of negative entities from each entity shard, to minimize selection bias. For each pair of devices, the same amount of negative entities is exchanged in both directions, through collective operators.\n",
                "\n",
                "* If `flat_negative_format=False`, negative entities are sampled on a triple basis. For each positive triple in a micro-batch, `n_negative` corrupted entities are received from each device, for a total of `shard_bs * n_negative * n_shard` negatives used in the micro-batch. We can then decide whether to score each of the `shard_bs` triples in the micro-batch only against the corresponding `n_negative * n_shard` negative entities, or against all negatives seen in the micro-batch (**negative sample sharing**).\n",
                "\n",
                "* If `flat_negative_format=True`, negative entities are sampled on a shard-pair basis. Each device receives `n_negative` negatives from each shard, for a total of `n_negative * n_shard` negatives used in the micro-batch. As the negatives are not sampled on a triple basis, this requires the use of negative sample sharing.\n",
                "\n",
                "In order to reduce inter-device communication, we have the option of sampling all negatives just from the device where the triple loss will be computed, by setting `local_sampling=True`. This, however, introduces a bias in the construction of negative samples.\n",
                "\n",
                "When instantiating the negative sampler, we must also specify the corruption scheme:\n",
                "* `corruption_scheme='h'` if negative samples are to be constructed by corrupting the head entity;\n",
                "* `corruption_scheme='t'` if negative samples are to be constructed by corrupting the tail entity;\n",
                "* `corruption_scheme='ht'` if negative samples are to be constructed by corrupting the head entity for half the triples in the micro-batch and the tail entity for the other half (in this case, when using negative sample sharing, negatives are shared only among the triples in the same half)."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 9,
            "metadata": {},
            "outputs": [],
            "source": [
                "neg_sampler = RandomShardedNegativeSampler(\n",
                "    n_negative=1,\n",
                "    sharding=sharding,\n",
                "    seed=seed,\n",
                "    corruption_scheme=\"ht\",\n",
                "    local_sampling=False,\n",
                "    flat_negative_format=False,\n",
                ")"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "In order to start training, we are missing only one component: a **batch sampler**. This class is responsible for cooking up the batches to pass to each device. This is not a trivial task, since at each step each device needs to know which embeddings stored in its local memory are needed by itself and by all other devices.\n",
                "Different types of batch samplers are implemented in `besskge.batch_sampler`. All of them, at each step, sample the same number of triples from each of the 16 shard-pair buckets. Here we use `RigidShardedBatchSampler`, where each bucket is consumed sequentially. The length of an epoch is then dictated by the length of the largest bucket."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 10,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "# triples per shard-pair per step: 60 \n",
                        "\n",
                        "head         torch.Size([48, 4, 4, 60])     torch.int32;\n",
                        "relation     torch.Size([48, 4, 4, 60])     torch.int32;\n",
                        "tail         torch.Size([48, 4, 4, 60])     torch.int32;\n",
                        "triple_mask  torch.Size([48, 4, 4, 60])     torch.bool;\n",
                        "negative     torch.Size([48, 4, 4, 240, 1]) torch.int32;\n"
                    ]
                }
            ],
            "source": [
                "device_iterations = 8\n",
                "accum_factor = 6\n",
                "# Micro-batch size, which means the number of positive triples processed on each device at each step\n",
                "shard_bs = 240\n",
                "\n",
                "batch_sampler = RigidShardedBatchSampler(\n",
                "    partitioned_triple_set=train_triples,\n",
                "    negative_sampler=neg_sampler,\n",
                "    shard_bs=shard_bs,\n",
                "    batches_per_step=device_iterations * accum_factor,\n",
                "    seed=seed,\n",
                ")\n",
                "\n",
                "\n",
                "print(f\"# triples per shard-pair per step: {batch_sampler.positive_per_partition} \\n\")\n",
                "\n",
                "# Example batch\n",
                "idx_sampler = iter(batch_sampler.get_dataloader_sampler(shuffle=True))\n",
                "for k, v in batch_sampler[next(idx_sampler)].items():\n",
                "    print(f\"{k:<12} {str(v.shape):<30} {v.dtype};\")"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "We see that each call returns `device_iterations * accum_factor = 48` batches. Each of them is composed of 60 triples from each of the 16 shard-pair buckets. In particular, IPU `i` will process 240 triples, 60 from each of the four (h, t) shard-pairs `(i,0), (i,1), (i,2), (i,3)`. \n",
                "\n",
                "`head[:,i,:,:], tail[:,i,:,:]` are the entity IDs of the embeddings that need to be gathered from the SRAM of IPU `i`.\n",
                "\n",
                "`negative[:,i,j,t,:]` are the negative entities sampled on IPU `i` to be used for triple `t` on IPU `j` (where the trailing 1 is the `n_negative` specified in the negative sampler).\n",
                "\n",
                "`triple_mask` is a boolean mask indicating which of the positive triples in the batch are non-padding (since, as mentioned above, `RigidShardedBatchSampler` will repeat triples in smaller shard-pair buckets during an epoch). We can ignore it during training, but it is important at inference time, when we want to visit each triple only once. We can then use `triple_mask` to discard any duplicates.\n"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "<p align=\"left\">\n",
                "<img src=\"https://raw.githubusercontent.com/graphcore-research/bess-kge/main/notebooks/img/negsplit.png\"  width=\"880\" >\n",
                "</p>"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Distributed training\n",
                "\n",
                "The method `get_dataloader` of the batch sampler returns the PopTorch dataloader which we iterate over during training."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 11,
            "metadata": {},
            "outputs": [],
            "source": [
                "options = poptorch.Options()\n",
                "options.replication_factor = sharding.n_shard\n",
                "options.deviceIterations(device_iterations)\n",
                "options.Training.gradientAccumulation(accum_factor)\n",
                "# Add a memory saving optimisation pattern. This removes an unnecessary\n",
                "# entity_embedding gradient all-reduce, which is a no-op since it is fully\n",
                "# sharded across replicas.\n",
                "options._popart.setPatterns(dict(RemoveAllReducePattern=True))\n",
                "\n",
                "# Construction similar to PyTorch dataloader\n",
                "train_dl = batch_sampler.get_dataloader(\n",
                "    options=options, shuffle=True, num_workers=5, persistent_workers=True\n",
                ")"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "We are now ready to define the model and train it. We'll use **RotatE** with (complex) 64-dimensional embeddings and the **logsigmoid** loss function with **negative adversarial sampling**. We will also use **negative sample sharing** within the micro-batches to increase the effective number of negative samples without the need for sampling and exchanging more negative entities.\n",
                "\n",
                "The **BESS distribution scheme** is implemented in `besskge.bess` and it works as a wrapper around the KGE scoring function. It comes in different flavours: here we use the basic `EmbeddingMovingBessKGE` class, where entity embeddings are exchanged between IPUs (from the one where the embedding is stored to the one where it is needed for computation) through AllToAll collectives (for details on the embedding sharing, see the [BESS-KGE documentation](https://graphcore-research.github.io/bess-kge/API_ref/bess.html))."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 12,
            "metadata": {},
            "outputs": [
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "Graph compilation: 100%|██████████| 100/100 [00:43<00:00]\n"
                    ]
                }
            ],
            "source": [
                "# Loss function\n",
                "logsigmoid_loss_fn = LogSigmoidLoss(margin=12.0, negative_adversarial_sampling=True)\n",
                "# KGE model\n",
                "rotate_score_fn = RotatE(\n",
                "    negative_sample_sharing=True,\n",
                "    scoring_norm=1,\n",
                "    sharding=sharding,\n",
                "    n_relation_type=biokg.n_relation_type,\n",
                "    embedding_size=64,\n",
                ")\n",
                "# BESS wrapper\n",
                "model = EmbeddingMovingBessKGE(\n",
                "    negative_sampler=neg_sampler, score_fn=rotate_score_fn, loss_fn=logsigmoid_loss_fn\n",
                ")\n",
                "\n",
                "# Optimizer\n",
                "opt = poptorch.optim.AdamW(\n",
                "    model.parameters(),\n",
                "    lr=0.001,\n",
                ")\n",
                "\n",
                "# PopTorch wrapper\n",
                "poptorch_model = poptorch.trainingModel(model, options=options, optimizer=opt)\n",
                "\n",
                "# The variable entity_embedding needs to hold different values on each replica,\n",
                "# corresponding to the distinct shards of the entity embedding table\n",
                "poptorch_model.entity_embedding.replicaGrouping(\n",
                "    poptorch.CommGroupType.NoGrouping,\n",
                "    0,\n",
                "    poptorch.VariableRetrievalMode.OnePerGroup,\n",
                ")\n",
                "\n",
                "# Compile model\n",
                "batch = next(iter(train_dl))\n",
                "_ = batch.pop(\"triple_mask\")\n",
                "#  PopTorch requires to flatten device_iterations and shard into a single dimension\n",
                "res = poptorch_model(**{k: v.flatten(end_dim=1) for k, v in batch.items()})"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 13,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Epoch 1 loss: 1.146490 --- positive triples processed: 4.98e+06\n",
                        "Epoch duration (sec): 1.83244 (average step time: 0.01512)\n",
                        "Epoch 2 loss: 0.503822 --- positive triples processed: 9.95e+06\n",
                        "Epoch duration (sec): 1.64367 (average step time: 0.01516)\n",
                        "Epoch 3 loss: 0.445583 --- positive triples processed: 1.49e+07\n",
                        "Epoch duration (sec): 1.65151 (average step time: 0.01519)\n",
                        "Epoch 4 loss: 0.411330 --- positive triples processed: 1.99e+07\n",
                        "Epoch duration (sec): 1.64069 (average step time: 0.01513)\n",
                        "Epoch 5 loss: 0.389104 --- positive triples processed: 2.49e+07\n",
                        "Epoch duration (sec): 1.63915 (average step time: 0.01511)\n",
                        "Epoch 6 loss: 0.378068 --- positive triples processed: 2.99e+07\n",
                        "Epoch duration (sec): 1.64212 (average step time: 0.01514)\n",
                        "Epoch 7 loss: 0.369016 --- positive triples processed: 3.48e+07\n",
                        "Epoch duration (sec): 1.63842 (average step time: 0.01511)\n",
                        "Epoch 8 loss: 0.367234 --- positive triples processed: 3.98e+07\n",
                        "Epoch duration (sec): 1.63810 (average step time: 0.01511)\n",
                        "Epoch 9 loss: 0.359143 --- positive triples processed: 4.48e+07\n",
                        "Epoch duration (sec): 1.65234 (average step time: 0.01523)\n",
                        "Epoch 10 loss: 0.359279 --- positive triples processed: 4.98e+07\n",
                        "Epoch duration (sec): 1.63927 (average step time: 0.01511)\n",
                        "Epoch 11 loss: 0.357416 --- positive triples processed: 5.47e+07\n",
                        "Epoch duration (sec): 1.63432 (average step time: 0.01507)\n",
                        "Epoch 12 loss: 0.356108 --- positive triples processed: 5.97e+07\n",
                        "Epoch duration (sec): 1.65299 (average step time: 0.01524)\n",
                        "Epoch 13 loss: 0.355720 --- positive triples processed: 6.47e+07\n",
                        "Epoch duration (sec): 1.63936 (average step time: 0.01512)\n",
                        "Epoch 14 loss: 0.355667 --- positive triples processed: 6.97e+07\n",
                        "Epoch duration (sec): 1.64642 (average step time: 0.01512)\n",
                        "Epoch 15 loss: 0.355123 --- positive triples processed: 7.46e+07\n",
                        "Epoch duration (sec): 1.64033 (average step time: 0.01512)\n",
                        "Epoch 16 loss: 0.351044 --- positive triples processed: 7.96e+07\n",
                        "Epoch duration (sec): 1.63891 (average step time: 0.01511)\n",
                        "Epoch 17 loss: 0.350440 --- positive triples processed: 8.46e+07\n",
                        "Epoch duration (sec): 1.63386 (average step time: 0.01507)\n",
                        "Epoch 18 loss: 0.351676 --- positive triples processed: 8.96e+07\n",
                        "Epoch duration (sec): 1.63382 (average step time: 0.01507)\n",
                        "Epoch 19 loss: 0.351343 --- positive triples processed: 9.46e+07\n",
                        "Epoch duration (sec): 1.63209 (average step time: 0.01505)\n",
                        "Epoch 20 loss: 0.351067 --- positive triples processed: 9.95e+07\n",
                        "Epoch duration (sec): 1.64723 (average step time: 0.01519)\n",
                        "Epoch 21 loss: 0.352264 --- positive triples processed: 1.05e+08\n",
                        "Epoch duration (sec): 1.64162 (average step time: 0.01512)\n",
                        "Epoch 22 loss: 0.351381 --- positive triples processed: 1.09e+08\n",
                        "Epoch duration (sec): 1.63871 (average step time: 0.01511)\n",
                        "Epoch 23 loss: 0.350780 --- positive triples processed: 1.14e+08\n",
                        "Epoch duration (sec): 1.63756 (average step time: 0.01509)\n",
                        "Epoch 24 loss: 0.349812 --- positive triples processed: 1.19e+08\n",
                        "Epoch duration (sec): 1.63480 (average step time: 0.01505)\n",
                        "Epoch 25 loss: 0.350110 --- positive triples processed: 1.24e+08\n",
                        "Epoch duration (sec): 1.64080 (average step time: 0.01512)\n"
                    ]
                },
                {
                    "data": {
                        "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAG1CAYAAADX6N+4AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABDdklEQVR4nO3dd3RUZeLG8e/MpFdISAghgdBLgNCr9CKgiLgKAi7qLlYs6LK7sq6KWEB+9u7qroANEBVRQESQJiC9hY6kAIEkQDqZJDPz+yMSl6WYfmcmz+ecOUdm7sw8Ey+Zh/d9770mh8PhQERERMQJmY0OICIiInIlKioiIiLitFRURERExGmpqIiIiIjTUlERERERp6WiIiIiIk5LRUVEREScloqKiIiIOC0PowNUhN1u5+TJkwQGBmIymYyOIyIiIqXgcDjIzs4mMjISs/nqYyYuXVROnjxJdHS00TFERESkHJKTk4mKirrqNi5dVAIDA4HiDxoUFGRwGhERESmNrKwsoqOjS77Hr8ali8qF6Z6goCAVFRERERdTmmUbWkwrIiIiTktFRURERJyWioqIiIg4LRUVERERcVoqKiIiIuK0VFRERETEaamoiIiIiNNSURERERGnpaIiIiIiTktFRURERJyWioqIiIg4LRWVKyiy2Y2OICIiUuOpqFxGWraVfi+uZu7GBBUWERERA6moXMZHGxM4fu48T34dz3Wvr2fDkXSjI4mIiNRIKiqX8dDAZjwzMpZafp4cPJ3NuA9+5t6PtpF8Ns/oaCIiIjWKoUVl2rRpmEymi24tW7Y0MhIAHhYzf+wRw+op/bi9R0MsZhPfxZ9i4MtreOn7g+QVFBkdUUREpEYwfEQlNjaWlJSUktv69euNjlSilp8XT49sw9KHetOzSSgFRXbeWHWEgS+t4eudJ3A4HEZHFBERcWuGFxUPDw8iIiJKbnXq1DE60iVaRATyycRuvHtbR6Jq+5KSmc/D83Zyy7sb2Xsi0+h4IiIibsvwonL48GEiIyNp3Lgx48ePJykp6YrbWq1WsrKyLrpVF5PJxNA29fjh0b78ZXBzfD0tbE08x4g31/PYF7tJz7FWWxYREZGawuQwcP5i2bJl5OTk0KJFC1JSUnj66ac5ceIEe/fuJTAw8JLtp02bxtNPP33J/ZmZmQQFBVVH5BIpmeeZuewAX+88CUCgjwcPD2zG7T1j8LQY3v9EREScVlZWFsHBwaX6/ja0qPyvjIwMGjZsyMsvv8yf//znSx63Wq1Yrb+NXGRlZREdHW1IUblga8JZpn0Tz94TxaM7TcL8eXJELH2bhxmSR0RExNmVpag41T/9a9WqRfPmzTly5MhlH/f29iYoKOiim9E6x4Tw9aRrmHlTW0L9vTialsvt/9nMxDlbSEjPNTqeiIiIS3OqopKTk8PRo0epV6+e0VHKxGI2cWvXBqya0o+J1zTCw2zih/2pDH5lDTOW7SfHqsOZRUREysPQojJlyhTWrFlDQkICGzZsYNSoUVgsFsaOHWtkrHIL9vXkn9e35rvJfejbPIxCm4P31vxC/xdXs3Dbcex2p5llExERcQmGFpXjx48zduxYWrRowejRowkNDWXTpk2Ehbn2+o6m4QHMvrML/769MzGhfqRlW5ny+S5GvbOBHUnnjI4nIiLiMpxqMW1ZlWUxjlGsRTY+/CmBN1YeJrfABsAfOkbx96EtCA/yMTidiIhI9XPZo37KyhWKygWp2fnM+u4gC7cdB8Dfy8IDA5rxp2ti8PawGJxORESk+qioOLGdyRlMWxzPzuQMAGJC/fjnda0Z2Cock8lkbDgREZFqoKLi5Ox2B1/tOMHM7w6Qll18Xpg+zcN48vpWNA2/9ER3IiIi7kRFxUXkWIt4c9UR/rP+GAU2Ox5mExN6xPDwoGYE+3oaHU9ERKRKqKi4mIT0XJ5dsp8f9p8GINTfiynXtmB052gsZk0HiYiIe1FRcVFrDqUx/Zt4jqYVn9E2NjKIaTfE0iUmxOBkIiIilUdFxYUV2uzM3ZjIqz8cIju/+Iy2I+IimTqsJZG1fA1OJyIiUnEqKm7gTI6VF78/yLwtyTgc4Otp4b5+Tbi7T2N8PHU4s4iIuC4VFTey90QmT38Tz5aE4jPaRtX25fHhrRjaJkKHM4uIiEtSUXEzDoeDb3anMGPpflIy8wHo2SSU18d2oE6At8HpREREyqYs399OdfVkuTyTycQNcZGs/EtfHhrQFG8PMxuOnmHMexs5nZVvdDwREZEqo6LiQvy8PHh0SAuWPtybyGAfjqblMvq9jRw/l2d0NBERkSqhouKCmoQFMP+eHkSH+JJ4Jo8x720i8Uyu0bFEREQqnYqKi4oO8ePze3rSuI4/JzLOM/q9jRxJzTE6loiISKVSUXFhEcE+zLunO83rBnA6y8qt/9rIgVNZRscSERGpNCoqLi480Id5d/cgNjKI9JwCbv3XJvYczzQ6loiISKVQUXEDIf5efHpXd9pH1yIjr5BxH2xiW+I5o2OJiIhUmIqKmwj29eTjid3oGhNCdn4Rf/z3z2z65YzRsURERCpERcWNBHh7MPtPXbimaR3yCmzc8eFm1h5KMzqWiIhIuamouBk/Lw8+uL0zA1qGk19oZ+Kcrfyw77TRsURERMpFRcUN+XhaePe2TgyNjaDAZufej7exdE+K0bFERETKTEXFTXl5mHlzXAdGto+kyO7ggU+3s2jHCaNjiYiIlImKihvzsJh5eXR7RneOwu6ARxbsZP6WJKNjiYiIlJqKipuzmE3MvKkdt3VvgMMBf/9iD3M3JhgdS0REpFRUVGoAs9nEMyPbMPGaRgA8+XU8/1p71OBUIiIiv09FpYYwmUw8fl0rHhzQFIDnlx7g9ZWHcTgcBicTERG5MhWVGsRkMvGXIS2YMqQ5AC+vOMT/LT+osiIiIk5LRaUGemBAM/55XSsA3l59lGe+3a+yIiIiTklFpYaa2Lsxz9zYBoD//HSMfy7ai92usiIiIs5FRaUG+2P3hsz6QztMJvjk5yT+9sVubCorIiLiRFRUarjRXaJ5dUx7LGYTC7cdZ/L8nRTa7EbHEhERAVRUBBjZvj5vjeuAp8XEN7tO8sCn2ykoUlkRERHjqagIAEPb1OO9P3bCy8PM8vjTvPDdAaMjiYiIqKjIbwa0rMubYzsAxQtsNx87a3AiERGp6VRU5CJDYiMY3TkKhwOmfL6LXGuR0ZFERKQGU1GRSzxxfWvq1/Il6WweM5dpCkhERIyjoiKXCPTxZNbN7QD4aFMi6w+nG5xIRERqKhUVuaxeTevwx+4NAfjbwl1k5RcanEhERGoiFRW5oseGtaRBiB8nM/N59tt9RscREZEaSEVFrsjf24MXb4nDZIIFW4+z6sBpoyOJiEgNo6IiV9W1UQh/7tUIgMe+2ENGXoHBiUREpCZRUZHfNeXaFjQJ8yc128pTi+ONjiMiIjWIior8Lh9PCy/eEofZBF/vPMl3e1OMjiQiIjWEioqUSocGtbm3bxMAHv9qL2dyrAYnEhGRmkBFRUrt4UHNaBkRyJncAh7/ai8Oh8PoSCIi4uZUVKTUvD2Kp4A8zCa+iz/F4l0njY4kIiJuTkVFyqRN/WAeHNAMgCe/jud0Vr7BiURExJ2pqEiZ3d+/CW3rB5N5vpCpX+7RFJCIiFQZFRUpM0+LmZdGx+FlMbPqQCqfbztudCQREXFTKipSLs3rBvLI4OYAPPPNPk5knDc4kYiIuCMVFSm3u/s0pkODWmRbi/j7wt2aAhIRkUqnoiLlZjGbeOmWOHw8zaw/ks7HPycZHUlERNyMiopUSOOwAP52bUsAZizdT9KZPIMTiYiIO1FRkQq7o2cM3RqFkFdgY8rCXdjtmgISEZHKoaIiFWY2m/i/m+Pw87Kw+dhZPtyQYHQkERFxEyoqUikahPrxj+GtAJj13QGOpuUYnEhERNyBiopUmvHdGtC7WR2sRXb++rmmgEREpOJUVKTSmEwmXvhDO/y8LGxPyuDLHSeMjiQiIi5ORUUqVWQtXx4aWHwtoJnL9pOVX2hwIhERcWVOU1RmzpyJyWRi8uTJRkeRCvpTr0Y0ruNPek4Br644bHQcERFxYU5RVLZs2cJ7771Hu3btjI4ilcDLw8y0G2IBmLMxgUOnsw1OJCIirsrwopKTk8P48eN5//33qV27ttFxpJL0aR7GkNZ1sdkdPPV1vE6vLyIi5WJ4UZk0aRLXXXcdgwYN+t1trVYrWVlZF93EeT1xfWu8Pcxs/OUMS/akGB1HRERckKFFZd68eWzfvp0ZM2aUavsZM2YQHBxccouOjq7ihFIR0SF+3NevCQDPLdlPXkGRwYlERMTVGFZUkpOTefjhh/nkk0/w8fEp1XOmTp1KZmZmyS05ObmKU0pF3du3CVG1fUnJzOetH48YHUdERFyMyWHQ4oFFixYxatQoLBZLyX02mw2TyYTZbMZqtV702OVkZWURHBxMZmYmQUFBVR1Zyml5/Cnu+WgbXhYzyx/pQ6M6/kZHEhERA5Xl+9uwEZWBAweyZ88edu7cWXLr3Lkz48ePZ+fOnb9bUsR1DGldlz7Nwyiw2Zn+TbzRcURExIV4GPXGgYGBtGnT5qL7/P39CQ0NveR+cW0mk4mnRrRm6Ktr+fFgGiv3n2Zgq7pGxxIRERdg+FE/UjM0CQvgT9c0AuDpb/aRX2gzOJGIiLgCw0ZULmf16tVGR5Aq9OCAZizacYKks3m8v/YXHvz1VPsiIiJXohEVqTYB3h78Y3grAN5afYQTGecNTiQiIs5ORUWq1Q1xkXRtFEJ+oZ3nluwzOo6IiDg5FRWpViaTiadviMVsgqV7TrH+cLrRkURExImpqEi1a1UviAk9YgCY9k08hTa7sYFERMRpqaiIIR4Z1JwQfy+OpOYwZ0OC0XFERMRJqaiIIYL9PPn70BYAvPrDYVKz8g1OJCIizkhFRQxzS6do4qKCybEWMXPZAaPjiIiIE1JREcOYzSamj2yDyQRf7jjB1oSzRkcSEREno6IihoqLrsXoTtEAurqyiIhcQkVFDHdP38YArDmUppPAiYjIRVRUxHCNwwLo0TgUuwMWbEk2Oo6IiDgRFRVxCrd2LZ7+WbA1GZvdYXAaERFxFioq4hSujY2gtp8nKZn5rDmUanQcERFxEioq4hR8PC3c1DEKgM82a/pHRESKqaiI0xj76/TPqgOpnMrUCeBERERFRZxI0/BAusTUxmZ38PlWjaqIiIiKijiZsV0bADBvSzJ2LaoVEanxVFTEqQxvW48gHw9OZJxn3ZF0o+OIiIjBVFTEqfz3otp5m5MMTiMiIkZTURGnc+GcKiv2nSYt22pwGhERMZKKijidlhFBdGhQiyK7g4XbjhsdR0REDKSiIk7pt0W1SVpUKyJSg6moiFO6vl09Ar09SDyTx8ZfzhgdR0REDKKiIk7Jz8uDkR0iAfhMi2pFRGosFRVxWhemf5bHn+JMjhbViojURCoq4rRiI4NpFxVMoc3Bl9tPGB1HREQMoKIiTu3CqMpnW5JwOLSoVkSkplFREac2Ii4SPy8Lv6TlsvnYWaPjiIhINVNREacW4O3ByPZaVCsiUlOpqIjTu7VL8fTP0r2nyMgrMDiNiIhUJxUVcXrtooJpXS+IgiK7FtWKiNQwKiri9EwmE2O7/bqodrMW1YqI1CQqKuISRraPxNfTwuHUHLYnnTM6joiIVBMVFXEJQT6eXN+uHgCf/pxscBoREakuKiriMm799Zwq3+4+yblcLaoVEakJVFTEZXRsUIvYyCCsRXY+1aHKIiI1goqKuAyTycSfejUCYO7GBAptdoMTiYhIVVNREZdyfVw9wgK9OZ1lZemeFKPjiIhIFVNREZfi7WHhj90bAvDv9cd0qLKIiJtTURGXM75bA7w8zOw+nqlDlUVE3JyKiric0ABvRrWvDxSPqoiIiPtSURGXdOc1MQB8t/cUx8/lGRtGRESqjIqKuKSWEUFc07QOdgfM2ZBgdBwREakiKirisv7066jKvC3J5FiLjA0jIiJVQkVFXFa/5uE0ruNPdn4RX2w7bnQcERGpAioq4rLMZhN39ooB4MOfjmG361BlERF3o6IiLu2mjlEE+XiQcCaPVQdSjY4jIiKVTEVFXJq/twdjuxVfrFCHKouIuB8VFXF5t/eIwWI2sfGXM+w7mWV0HBERqUQqKuLyImv5MqxNBFC8VkVERNyHioq4hT9dU3xV5a93niQt22pwGhERqSwqKuIWOjaoTfvoWhTY7Hzyc6LRcUREpJKoqIjb+POvoyofb0okv9BmcBoREakMKiriNoa2iaBesA/pOQV8s+uk0XFERKQSqKiI2/C0mLm9ZwwAs3X9HxERt6CiIm5lTOdovCxm4k9msed4ptFxRESkglRUxK3U9vdi6K+HKn+2JcngNCIiUlEqKuJ2bu0aDcDinSfJ1VWVRURcmoqKuJ0ejUOJCfUjx1rEkt0pRscREZEKUFERt2MymRjTpfj6P5r+ERFxbYYWlXfeeYd27doRFBREUFAQPXr0YNmyZUZGEjdxc6coPMwmdiRlcPBUttFxRESknAwtKlFRUcycOZNt27axdetWBgwYwMiRI4mPjzcylriBsEBvBrWqC8BnmzWqIiLiqgwtKiNGjGD48OE0a9aM5s2b89xzzxEQEMCmTZuMjCVu4sKi2i+3H9eZakVEXJTTrFGx2WzMmzeP3NxcevTocdltrFYrWVlZF91ErqR3szDq1/IlK7+IZXu1qFZExBUZXlT27NlDQEAA3t7e3HvvvXz11Ve0bt36stvOmDGD4ODgklt0dHQ1pxVXYjGbGN25eB/5bHOywWlERKQ8TA6Hw2FkgIKCApKSksjMzGThwoV88MEHrFmz5rJlxWq1YrVaS/6clZVFdHQ0mZmZBAUFVWdscREpmefpNXMVdges/EtfmoQFGB1JRKTGy8rKIjg4uFTf34aPqHh5edG0aVM6derEjBkziIuL47XXXrvstt7e3iVHCF24iVxNvWBf+rUIB2D+Fo2qiIi4GsOLyv+y2+0XjZqIVNStXYqnf77YdpyCIrvBaUREpCw8jHzzqVOnMmzYMBo0aEB2djaffvopq1evZvny5UbGEjczoGU44YHepGZbWbHvNNe1q2d0JBERKSVDR1RSU1OZMGECLVq0YODAgWzZsoXly5czePBgI2OJm/GwmEsW1c7TmWpFRFyKoSMq//73v418e6lBxnSJ5s0fj7DucDrJZ/OIDvEzOpKIiJSC061REakK0SF+9G5WB9CiWhERV6KiIjXGrb9eqPDzbckU2bSoVkTEFaioSI0xuHVdQv29OJ1lZfXBNKPjiIhIKZSrqCQnJ3P8+PGSP2/evJnJkyfzr3/9q9KCiVQ2Lw8zI9vXB2DJHp1SX0TEFZSrqIwbN44ff/wRgFOnTjF48GA2b97M448/zvTp0ys1oEhlGt42AoAf9p/GWqQLFYqIOLtyFZW9e/fStWtXABYsWECbNm3YsGEDn3zyCbNnz67MfCKVqmOD2oQHepOdX8SGI2eMjiMiIr+jXEWlsLAQb29vAH744QduuOEGAFq2bElKiobUxXmZzSaujS0eVdEVlUVEnF+5ikpsbCzvvvsu69atY8WKFQwdOhSAkydPEhoaWqkBRSrbsDbFRWXFvtM6+kdExMmVq6i88MILvPfee/Tr14+xY8cSFxcHwOLFi0umhEScVddGIdT28+RcXiE/HztrdBwREbmKcp2Ztl+/fqSnp5OVlUXt2rVL7r/77rvx89MZP8W5eVjMDGkdwfytySzbm0KvpnWMjiQiIldQrhGV8+fPY7VaS0pKYmIir776KgcPHiQ8PLxSA4pUhaG/Hv2zPP40drvD4DQiInIl5SoqI0eOZO7cuQBkZGTQrVs3XnrpJW688UbeeeedSg0oUhV6NalDoI8HadlWtiWdMzqOiIhcQbmKyvbt2+nduzcACxcupG7duiQmJjJ37lxef/31Sg0oUhW8PMwMalUXgGV7ThmcRkRErqRcRSUvL4/AwEAAvv/+e2666SbMZjPdu3cnMTGxUgOKVJULR/8sjz+Fw6HpHxERZ1SuotK0aVMWLVpEcnIyy5cvZ8iQIQCkpqYSFBRUqQFFqkqf5mH4eVk4kXGe3cczjY4jIiKXUa6i8uSTTzJlyhRiYmLo2rUrPXr0AIpHVzp06FCpAUWqio+nhf4tixd/L9XJ30REnFK5isrNN99MUlISW7duZfny5SX3Dxw4kFdeeaXSwolUtQvTP9/t1fSPiIgzKtd5VAAiIiKIiIgouYpyVFSUTvYmLqd/i3C8Pcwknsljf0o2rSM1dSki4kzKNaJit9uZPn06wcHBNGzYkIYNG1KrVi2eeeYZ7Hadklxch7+3B32ahwHwnaZ/REScTrmKyuOPP86bb77JzJkz2bFjBzt27OD555/njTfe4IknnqjsjCJV6sL0z7K9OkxZRMTZlGvqZ86cOXzwwQclV00GaNeuHfXr1+f+++/nueeeq7SAIlVtYKu6eFpMHE7N4UhqDk3DA4yOJCIivyrXiMrZs2dp2bLlJfe3bNmSs2d1kTdxLcG+nvRsUny9H03/iIg4l3IVlbi4ON58881L7n/zzTdp165dhUOJVDdN/4iIOKdyTf3MmjWL6667jh9++KHkHCobN24kOTmZpUuXVmpAkeowJDaCxxftJf5kFvO3JDGmSwOjI4mICOUcUenbty+HDh1i1KhRZGRkkJGRwU033UR8fDwfffRRZWcUqXIh/l7c3acxAI99uYfPtyYbnEhERABMjko8y9WuXbvo2LEjNputsl7yqrKysggODiYzM1On7pcKczgcPP3NPmZvSMBkgpduieOmjlFGxxIRcTtl+f4u14iKiDsymUw8NaI1t3VvgMMBUz7fxdc7TxgdS0SkRlNREfkvJpOJ6Te0YWzXBtgd8Mj8nXyz66TRsUREaiwVFZH/YTabeO7GNozuHIXdAZPn72TJbh22LCJihDId9XPTTTdd9fGMjIyKZBFxGmaziZk3tcNmhy+2H2fy/B20b1CL+rV8jY4mIlKjlKmoBAcH/+7jEyZMqFAgEWdhNpuYdXM7Es7ksi3xHIt2nGBS/6ZGxxIRqVHKVFQ+/PDDqsoh4pQsZhNjOkezLfEcX24/zv39mmAymYyOJSJSY2iNisjvGNY2Am8PM0fTctl7IsvoOCIiNYqKisjvCPTxZHDrugB8ueO4wWlERGoWFRWRUhjVoT4A3+w6SZHNbnAaEZGaQ0VFpBT6NA8j1N+L9JwC1h1JNzqOiEiNoaIiUgqeFjMj4iIB+Gq7zlYrIlJdVFRESunGX6d/vt93ihxrkcFpRERqBhUVkVKKiwqmcR1/8gvtfLf3lNFxRERqBBUVkVIymUwloypf6egfEZFqoaIiUgYXjv7ZcPQMpzLzDU4jIuL+VFREyiA6xI8uMbVxOODrnVpUKyJS1VRURMrot+kfFRURkaqmoiJSRte3jcTLYubAqWz2p+iU+iIiVUlFRaSMgv08GdAyHIBPfk40OI2IiHtTUREphwk9GwLw2eZkjqRmG5xGRMR9qaiIlEPPJnUY1KouNruD55bsNzqOiIjbUlERKafHr2uFp8XEjwfTWH0w1eg4IiJuSUVFpJwa1fHn9h4xADy7ZL+uqiwiUgVUVEQq4MGBzajt58mR1Bw+3ZxkdBwREbejoiJSAcG+njw6uDkAL684RGZeocGJRETci4qKSAWN7dqAZuEBZOQV8trKw0bHERFxKyoqIhXkYTHzxPWtAZi7MYFf0nIMTiQi4j5UVEQqQZ/mYQxoGU6R3aFRFRGRSqSiIlJJLqxV+XZ3CiczzhucRkTEPaioiFSSNvWD6dE4FJvdwewNCUbHERFxCyoqIpXorj6NAPjs5ySy83UEkIhIRamoiFSifs3DaRoeQLa1iPlbko2OIyLi8lRURCqR2Wxi4jXFoyr/WX+MQp2tVkSkQgwtKjNmzKBLly4EBgYSHh7OjTfeyMGDB42MJFJhN3aoT50AL05m5rN0T4rRcUREXJqhRWXNmjVMmjSJTZs2sWLFCgoLCxkyZAi5ublGxhKpEB9PCxN+vQbQ++t+weFwGBtIRMSFmRxO9Fs0LS2N8PBw1qxZQ58+fX53+6ysLIKDg8nMzCQoKKgaEoqUztncAnrOXEl+oZ3P7upOjyahRkcSEXEaZfn+dqo1KpmZmQCEhIRc9nGr1UpWVtZFNxFnFOLvxc2dogD4YN0vBqcREXFdTlNU7HY7kydPplevXrRp0+ay28yYMYPg4OCSW3R0dDWnFCm9P1/TGJMJVh5I5Vi6pjNFRMrDaYrKpEmT2Lt3L/PmzbviNlOnTiUzM7Pklpyswz/FeTWq40+/5mEAzNuSZHAaERHX5BRF5YEHHuDbb7/lxx9/JCoq6orbeXt7ExQUdNFNxJnd2rUBAF9sO05BkQ5VFhEpK0OLisPh4IEHHuCrr75i1apVNGrUyMg4IpVuQMtwwgO9Sc8p4If9p42OIyLicgwtKpMmTeLjjz/m008/JTAwkFOnTnHq1CnOn9cF3cQ9eFrM3NK5eJTws82a/hERKStDi8o777xDZmYm/fr1o169eiW3+fPnGxlLpFKN6Vw8/bPucDrJZ/MMTiMi4loMn/q53O2OO+4wMpZIpWoQ6kfvZnUAdP0fEZEycorFtCLu7tYuxaMqC7YmU6Tr/4iIlJqKikg1GNy6LqH+XqRmW1l1INXoOCIiLkNFRaQaeHmYS85UO0/TPyIipaaiIlJNxnQpPpPy6oOpHDyVbXAaERHXoKIiUk0ahwXQu1kd7A4Y9fZPLNiarCsri4j8DhUVkWr00i1x9GwSSl6Bjb8t3M0Dn+0g83yh0bFERJyWiopINQoP8uGjP3fjb0Nb4GE2sWR3Cte9vo7U7Hyjo4mIOCUVFZFqZjGbuL9fUxbe15P6tXw5fu48/15/zOhYIiJOSUVFxCDto2sx7YZYAD77OYlca5HBiUREnI+KioiBBrYMJybUj6z8Ir7YftzoOCIiTkdFRcRAZrOJO3sVXzX8P+uPYbfrKCARkf+moiJisJs7RRHk40HCmTxW6qy1IiIXUVERMZi/twdjuxVfC+jf638xOI2IiHNRURFxAnf0jMHDbGLTL2fZeyLT6DgiIk5DRUXECdQL9mV423oAfLBOoyoiIheoqIg4iYm9ixfVLtp5krdXHzE4jYiIc1BREXES7aJq8fDAZgDM+u4gs747oGsBiUiNp6Ii4kQeGdycqcNaAvD26qM8tThehyyLSI2moiLiZO7p24TnRrXBZIK5GxN5dsl+oyOJiBhGRUXECY3v1pAXb44D4D8/HdO1gESkxlJREXFSf+gUxWO/TgM9u2QfS/ekGJxIRKT6eRgdQESu7J4+jTmZcZ65GxOZPH8n6w6nYTaZMJtMjGwfSeeYEKMjiohUKRUVESdmMpl4akQsKZn5rNh3ms82J5c8tnjXSVY80ofwIB8DE4qIVC2Tw4WPf8zKyiI4OJjMzEyCgoKMjiNSZfILbXy5/QRp2VYAlu5J4eDpbAa1Cuf9CZ0xmUwGJxQRKb2yfH9rREXEBfh4Whj36/WAAIa2iWDEG+v5YX8qX2w/wc2dogxMJyJSdbSYVsQFtYgIZPLg4pPDPf1NPCmZ5w1OJCJSNVRURFzU3b0bExddi+z8Ih7/aq/OYisibklFRcRFeVjMvHRLOzwtJlYdSGV5/GmjI4mIVDoVFREX1jQ8kLv7NAaKp4ByrUUGJxIRqVwqKiIu7oH+zYiq7UtKZj6vrTxsdBwRkUqloiLi4ny9LEwfGQvAv9cf44N1v3D8XJ7BqUREKofOoyLiJu79aBvfxZ8q+XPb+sHc1LE+I9vXJ8Tfy8BkIiIXK8v3t4qKiJvIL7Tx8aZEvo8/zdbEs9h//ZvtaTHxwh/acVNHnWtFRJyDiopIDXcmx8q3u1NYsDWZ+JNZhAd689NjA/C0aLZXRIxXlu9v/dYScUOhAd7c3jOGr+7vRZ0AL1Kzrazcn2p0LBGRMlNREXFjXh5mbukcDcCnm5MMTiMiUnYqKiJu7tYuxUVl3eE0ks/qaCARcS0qKiJurmGoP72b1cHhgHlbNKoiIq5FRUWkBhjXtfjKy/O3JHM0LcfgNCIipaeiIlIDDGpdlwYhfqTnFHDd6+uY/dMxXcRQRFyCiopIDeBpMTP/nu70blaH/EI7077ZxztrjhodS0Tkd6moiNQQ9YJ9mXNnV/42tAUAr6w4xP6ULINTiYhcnYqKSA1iNpu4r28TBreuS6HNwV8W7KKgyG50LBGRK1JREalhTCYTz49qS20/T/alZPH80v1aryIiTktFRaQGCgv05vlRbQGYvSGBWcsPqqyIiFNSURGpoYa1rcfTN8QC8M7qo/x14W6OpecanEpE5GIqKiI12O09Y3hqRGsAFm47zoCXVvPYF7spsmndiog4BxUVkRruzl6NmHd3dwa0DP/17LXJPLpgFza7poJExHgqKiJC98ah/OeOLrw/oTMeZhOLd53ksS92a92KiBhORUVESgxuXZc3x3XAYjbx+bbjfLwp0ehIIlLDqaiIyEWGtqnHY0NbAjD9233sSDpncCIRqclUVETkEhN7N2JobASFNge3ffAz8zYnkZFXQHZ+odHRRKSGMTlceBI6KyuL4OBgMjMzCQoKMjqOiFvJzi/kz3O2svnY2ZL7zCa4o2cj/jG8JR4W/TtHRMqnLN/f+k0jIpcV6OPJZ3d15/HhrfD3sgBgd8B/fjrGn+dsJb/QZnBCEakJNKIiIr/LZnfgcDj4ft9pHl2wk/xCO3++phFPXN/a6Ggi4oI0oiIilcpiNuFhMTO8bT3eHt8RKB5Z+fmXMwYnExF3p6IiImUyoGVdxnSOxuGAv32xm0KdxVZEqpCKioiU2T+vb0WovxeJZ/JYuieF6d/sY+irazmbW2B0NBFxMyoqIlJmgT6e3NEzBoDp3+zjPz8d48CpbJbHnzI2mIi4HRUVESmX27o3xNfTwpn/GkXZcFRrVkSkchlaVNauXcuIESOIjIzEZDKxaNEiI+OISBnU9vdiTJdoAAJ9PADYeDRd1wcSkUplaFHJzc0lLi6Ot956y8gYIlJOfxnSnIcGNuOL+3ri42kmPaeAQ6dzjI4lIm7Ew8g3HzZsGMOGDTMygohUQKCPJ48Obg5Al5gQ1h1OZ8PRdFpEBBqcTETchUutUbFarWRlZV10ExHn0KtpHQDeWX2Up7+J13WBRKRSuFRRmTFjBsHBwSW36OhooyOJyK8Gt66Ll8VMaraVD39KYNKnOyjSOVZEpIJcqqhMnTqVzMzMkltycrLRkUTkV03CAvjpsQG8dmt7fD0trD2Uxv8tP8iZHCtTPt/FtsRzRkcUERdk6BqVsvL29sbb29voGCJyBWGB3oxsXx9Pi5n7P9nOnI0JZOUXsnDbcY6k5rBoUi+jI4qIi3GpERURcQ3D2kTQqI4/+YV2PttcPPK563gGadlWg5OJiKsxtKjk5OSwc+dOdu7cCcCxY8fYuXMnSUlJRsYSkQoymUyM6lD/ovscDvjxQKpBiUTEVRlaVLZu3UqHDh3o0KEDAI8++igdOnTgySefNDKWiFSC/y4qgd7Fs8wr9p82Ko6IuChDi0q/fv1wOByX3GbPnm1kLBGpBNEhfgxuXRcfTzPPjmoDwNpDaexP0WkFRKT0tEZFRKrMW+M68vPUQdwQF0mPxqFYi+xM+M9mjp/LMzqaiLgIFRURqTJeHmaC/TwxmUy8+8dOtIwIJC3bykOf6RwrIlI6KioiUi2CfT15f0JnAr092J6Uwf8tP2h0JBFxASoqIlJtokP8mPmHdgC8t/YXPtqUaHAiEXF2KioiUq2ua1ePhwc2A+CJRXt56LMdui6QiFyRS52ZVkTcw+RBzbA7HLz14xEW7zrJ3hOZDG0TwbhuDYiq7ceCLcmk5Vi5v18TTCaT0XFFxEAmh8PhMDpEeWVlZREcHExmZiZBQUFGxxGRMtqRdI77Pt7Oqax8ACKCfJh2Qyz3frwNgM/u6k6PJqFGRhSRKlCW729N/YiIYTo0qM2Sh67hqRGtaRLmz6ms/JKSAvDl9uMGphMRZ6CiIiKGCg3w5s5ejfh4Yjda1bv4X1ZL96SQV1BkUDIRcQYqKiLiFOoF+7LkwWv49K5ufDe5Nw1C/MgtsPHjgTSjo4mIgVRURMRpmM0mejapQ8uIIK6NrQvAyl+vD3QkNYdpi+NJz9EVmEVqEh31IyJOaVCrury/7hhf7jhBi4hA5m9N5pe0XI6k5vDxxG5Yi2wU2hwEeOvXmIg7099wEXFKnRrWppafJxl5hcxYdqDk/vVH0nlz1WFeX3kEm8PBA/2bMnlQMx3GLOKmdHiyiDitD9b9wrNL9v/udsPbRrD3RBZmEwT4eNC9USiPX9dK5UXESZXl+1tFRUScmsPhYNDLazialsu1sXVZHl+8ZqVF3UAGtArnndVHL/u8mzrUJ8jXk9Gdo2kdqd8PIs5ERUVE3EpqVj5L96Rwa9cGrDuczvvrfuHpG2KpF+xD++krfvf5c//UlT7Nw7AW2Xhi0V5iI4OZ0KMhRXYH/1r7C1sSzvLWuI74a72LSLVQURGRGuOrHcdZeyidFhGBzPyvtSz/LcDbg1fHtOdf635h87GzADQLD+Bwak7JNq+Oac+NHepXS2aRmk5nphWRGmNUhyheGdOeYW0iSu67r18TAO7u05hujULIsRYxce7WkpICXFRSAHYmZwBwvsB22fdJSM8tOfnc/pQsvtx+HBf+d56Iy9CIioi4jfWH0wnw8aBt/WB2JJ2jQ4PaZOcXlmp66H8FenswoFU4L/yhHcfScxn22jo6N6zNwvt6EvPYEgA+vKML/VuGV/bHEHF7GlERkRrpmmZ1aB9dC4vZROeYECxmE7X8vHhzXAcsZhPXNK3DxGsaXfScd8Z3vOxrZVuL+HrnSdpP/55hr60DYGviOWb/dKxkm+1J56ruw4gIoPOoiEgNcH27SIa1qYfFXHy4cniQN7N/SmDun7vRNDyAMZ2jmb81+bLPzS+0X/Tnad/sK/nvHUkZVZZZRIpp6kdEarwcaxGf/ZxEl0YhRNf2ZUvCOSZ9uh2b/fd/PTYJ8+eePk04diaXu3o35sCpLJ5fup+nRsTSJSakGtKLuB4d9SMiUgnO5RbwxfbjpTrpHMC1sXXZmZzB6azi6xH98vxwzGaddE7kf2mNiohIJajt78XE3o0Z3TkKKD6aqG6Q9xW3Xx5/uqSkADT+x1KmLY4n83whhbbfppAcDgepWfnY7Q4e+2I3T329l6JfH88vtPF9/ClyrUUXvbbD4aCg6OJpKJGaQCMqIiJl8Oy3+/hgffGC2u8m92boq+tK9bwJPRoydVgr/v7FbpLP5V2yvqV3szo0Cw/kSFoOaw+lAbDrqSEE+3oCMHHOVnYdz2DVX/oS6ONZeR9IxABl+f7WYloRkTL4y5AWhAZ4c21sXRqHBVx2m5hQPxLO5F1039yNiRxIyWZzwtnLPmfd4XTWHU6/6L7nluzjwQHNeOG7A/ywv/jSASv2neamjlGlzptfaMPH01Lq7UWcjaZ+RETKwNfLwn39mpSUlGdGxl6yzT19m1z2uVcqKVeyYOtxes/6kW93p5Tc9+iCXWTnF5JjLSL5bB7f7j5JzGNL6Pb8D6Rknr/o+fO3JNHmqeUsjz9VpvcVcSaa+hERqQCHw4G1yM79n2xn1YFUoHgR7fRv91Fgs/Ppz0mV/p79W4Sx8Zczlxw6DTDr5naM7hzNyYzz9Jy5CoBQfy+2PTG4Qu95NreAdYfTuDY2otQjNNYiG9YiO0GaqpL/oakfEZFqYjKZ8PG08Niwlvh6Wnh4UDPMZhPTbigeafH1tPDVjhPc1q0Br686AoCnpfg5T1zXmtFdorHZHRw6nc3LKw6xYt/p333PHw+mXfGxvy3cjY+nhTdXHS6570xuAX+avYVVB1J5ZUwc+1Oy+WP3hkQE+zBnQwJx0bUuOZR6zoYETmfl079lOD/sP8338ac5lp5LeKA3mx8fVKqfzZBX1pJ4Jo9dTw4h2E9lRcpHIyoiItXkx4Op1Pbzon10rcs+nl9o46ONiRTY7Pzf8oPVmm1s12ja1q/FubwCXl5x6KrnkOkaE8K5vAK+uL/nVUdLLlxq4N6+TXhsWMtKzyyuS+dRERFxcVn5haRlW/l863Hu79+EjzYm0rFBbT7bnMTiXSdpUz+Ihff2ZM2hNO75aJthOT+8swsOh4NCm4NrY4svDLlgSzKLd51k/ZHfFgcnzLwOgCKbnSNpObSoG4jJpHPM1FQqKiIiNYTD4WDVgVROZubzxKK9hmbZ+eRg0rKtDH5l7SWPjevWAA+ziez8Ir7acYJ7+jQmspYvg1vXJbKWrwFpxUgqKiIiNZDD4cBkMrE8/hSJZ3IZ06UBcU9/b3SsqwoL9GbL44NIPpvHqax84k9kckP7+gT5eLBs7yn6tghjzcE0diRlcCbXyku3xOFhKT5gdWdyBj//coaBreqydE8K3RqFsDXxHNe1rUeDEL9ynRX4x4OpzP4pgRf+0I6IYJ9K+5w2u4OdyRm0qR+Et0fpFiOvP5zO9/tO8Y/hrdzuEHMVFRERAX5bJ3JtbF2Wx1+8ULdZeACHU3Muec6Hd3bhz7O3UIpLHVWKP3SM4ovtx0u9/W3dG7D+cPol56r5XweeGUqutYi1h9OwFtp57Ms91Av2YcE9PYgO8bvscy78vGJC/bijZwzjujXEy+O3M3nY7Q4enLeDVhGBPDCgGQVFdgptdvy9r35sykvfH+SNVUe4IS6S18d2uOw225POsf5wOvf1a4KnxVyS5c5eMfRpHkbfZmGVekkGm93BD/tP06FBLcIDi0tZoc2Oh9mEyWQir6CIbYnn6N44FE9L5Z7NREVFREQA2HA0nTUH0/jLkBbkWIsotNnZmnCOHw+m8uyNbci1FvHHf29mX0oW6/7Wv+QLvNBmJy3bWnKIM0CnhrWZPjKW+VuSmbsx0aiPVKn+c0dnTpw7z6ebk3E4HBw4lX3R47d2iebQ6WyOpOYw7+4eDH/9tzMRv3RLHH/5fBcAHRvU4pOJ3TlfaGP2hgRGdaiPv7cFX08Lx9JzufmdjRT8epmETVMH4utpYf2RdLYknOX2njGsO5zGk1/HA/DXa1tcdjG1j6eZ+Xf3YNMvZ5izIYGXx7Sne+NQtiedo3Edf2r5eXG+wEbS2TxaRASy4Ug6HhYznRvWJjXbSn6hjU9+TmRCjxhWH0zlxe8PkXm+kNp+niy4pwff7k7hvbVH6d8inHdu68To9zay+dhZ7uvXhL8PrdzF0CoqIiJSavmFNjLPF1I36NKpju/2pvD3L/bw+tgO9G0eVnL/VzuO88j8XZdsf0NcJCv2neZ8oY1/DG/J80sPVGl2+c3R54fT6snvKv2aUIE+HuyZdm2lvqaKioiIVJoLa1/+m7XIRot/fgdAdIgvN3WIYkhsXWIjg8mxFnG+wIa3p5l2035bIzO+WwM+qYIT4EnVu3DUVmVRURERkSp3IuM86dlW4q5wXhiAzLxCDp7OpmW9QIJ8PDmRcZ7ks3mczsrn4Xk7qy2rVIyRRUXX+hERkXKpX8v3qiUFINjPk66NQkpODFe/li/dG4cysn39i7b729AWJf+9/u/9mTyoGQADWoZzS6co7u3bhOmXua6SuD+dQl9ERAzx8ug4nl2yn/cndKJTwxCua1uPc3mFRNX2Y/Kg5kwe1PyS52w+drbkIo3b/jmIrPwi+r+4mkBvDxY/eA0xoX48umAXX+04Ud0fR6qIpn5ERMQwl1v/8nvyC23kWosIDfC+4jbWIhuZeYW8vOIQTcMDuKNnDGk5VuoFF59cLvFMLgdOZTN53k7+c0cXpn+7j+vaRvDi94cAmHFTW6Z+uafk9VpGBBJZy5dxXRswce5WAB4f3ornlu4HYFCruvyw/+LDv/u1CGP1Va7L5Cpeu7X9JSNgFaU1KiIiIuWQlm0l8UwunWNC2Hcyi53JGdzaJfqS85cU2ux4Wsw4HA6SzuYRXdsPB5CTX0SAjwcnM86XHOpdUGTnfIGNIF+PklL2xbbj/OXzXbw+tgODWoVzJqeAvyzYRc+modzdpzFQfEFLk8mE3e6g0G5nZ1IGY/61iYEtw1n565W6AXY9OYRjZ3IpstlZvOskhTY7n21O5uXRcfx05Aw/HzvDiLhI3ll99LKfee/T11JYZGf6t/vIKygiLdtKozoBnM21UmR3MPdPXSv9cgcqKiIiIm7MWmRja8I5OjWsXeaz1qZlWzl4KpuGoX74eVmuOjJVVcry/a01KiIiIi7G28NCr6Z1yvXcsEBvwgKrv5yUl476EREREaeloiIiIiJOS0VFREREnJaKioiIiDgtFRURERFxWioqIiIi4rRUVERERMRpqaiIiIiI01JREREREaeloiIiIiJOS0VFREREnJaKioiIiDgtFRURERFxWi599WSHwwEUXy5aREREXMOF7+0L3+NX49JFJTs7G4Do6GiDk4iIiEhZZWdnExwcfNVtTI7S1BknZbfbOXnyJIGBgZhMpjI/v0uXLmzZsqVKti/NtlfbpqyPZWVlER0dTXJyMkFBQaXKWJ3K+rOu7tcvz/NL+5yK7gtXe9wV9wWo2v3BmfeF0mxbkcddcX9w5n2hvK9RXb8bXHlfcDgcZGdnExkZidl89VUoLj2iYjabiYqKKvfzLRZLmf5nlWX70mx7tW3K+1hQUJDhO+DllPVnXd2vX57nl/Y5Fd0Xrva4K+4LULX7gzPvC6XZtiKPu+L+4Mz7Qnlfo7p+N7j6vvB7IykX1OjFtJMmTaqy7Uuz7dW2Ke9jzqqqM1f09cvz/NI+p6L7wtUed8V9Aao2tzPvC6XZtiKPu+L+4Mz7Qnlfo7p+N7jbvnAlLj31I7/JysoiODiYzMxMp2jKYhztC/LftD/IBa66L9ToERV34u3tzVNPPYW3t7fRUcRg2hfkv2l/kAtcdV/QiIqIiIg4LY2oiIiIiNNSURERERGnpaIiIiIiTktFRURERJyWioqIiIg4LRWVGiAmJoZ27drRvn17+vfvb3QcMdixY8fo378/rVu3pm3btuTm5hodSQxw8OBB2rdvX3Lz9fVl0aJFRscSg7zyyivExsbSunVrHnrooVJdLLC66PDkGiAmJoa9e/cSEBBgdBRxAn379uXZZ5+ld+/enD17lqCgIDw8XPpqGlJBOTk5xMTEkJiYiL+/v9FxpJqlpaXRvXt34uPj8fT0pE+fPrz44ov06NHD6GiAi1/rR0TK5sIvot69ewMQEhJicCJxBosXL2bgwIEqKTVYUVER+fn5ABQWFhIeHm5wot9o6sfJrV27lhEjRhAZGYnJZLrs0Oxbb71FTEwMPj4+dOvWjc2bN1/0uMlkom/fvnTp0oVPPvmkmpJLVajo/nD48GECAgIYMWIEHTt25Pnnn6/G9FKZKuN3wwULFixgzJgxVZxYqkpF94WwsDCmTJlCgwYNiIyMZNCgQTRp0qQaP8HVqag4udzcXOLi4njrrbcu+/j8+fN59NFHeeqpp9i+fTtxcXFce+21pKamlmyzfv16tm3bxuLFi3n++efZvXt3dcWXSlbR/aGoqIh169bx9ttvs3HjRlasWMGKFSuq8yNIJamM3w1QfP2XDRs2MHz48OqILVWgovvCuXPn+Pbbb0lISODEiRNs2LCBtWvXVudHuDqHuAzA8dVXX110X9euXR2TJk0q+bPNZnNERkY6ZsyYcdnXmDJliuPDDz+swpRSXcqzP2zYsMExZMiQksdnzZrlmDVrVrXklapTkd8Nc+fOdYwfP746Yko1KM++sGDBAsf9999f8visWbMcL7zwQrXkLQ2NqLiwgoICtm3bxqBBg0ruM5vNDBo0iI0bNwLFTTs7OxsoXjC3atUqYmNjDckrVas0+0OXLl1ITU3l3Llz2O121q5dS6tWrYyKLFWkNPvCBZr2cW+l2Reio6PZsGED+fn52Gw2Vq9eTYsWLYyKfAktpnVh6enp2Gw26tate9H9devW5cCBAwCcPn2aUaNGAWCz2bjrrrvo0qVLtWeVqlea/cHDw4Pnn3+ePn364HA4GDJkCNdff70RcaUKlWZfAMjMzGTz5s188cUX1R1Rqklp9oXu3bszfPhwOnTogNlsZuDAgdxwww1GxL0sFRU317hxY3bt2mV0DHEiw4YNY9iwYUbHECcQHBzM6dOnjY4hTuC5557jueeeMzrGZWnqx4XVqVMHi8VyyS+a06dPExERYVAqMYr2B7lA+4Jc4A77goqKC/Py8qJTp06sXLmy5D673c7KlSud5kQ9Un20P8gF2hfkAnfYFzT14+RycnI4cuRIyZ+PHTvGzp07CQkJoUGDBjz66KPcfvvtdO7cma5du/Lqq6+Sm5vLnXfeaWBqqSraH+QC7QtygdvvC0YfdiRX9+OPPzqAS2633357yTZvvPGGo0GDBg4vLy9H165dHZs2bTIusFQp7Q9ygfYFucDd9wVd60dEREScltaoiIiIiNNSURERERGnpaIiIiIiTktFRURERJyWioqIiIg4LRUVERERcVoqKiIiIuK0VFRERETEaamoiEiZrF69GpPJREZGxlW3i4mJ4dVXX62WTKVxxx13cOONN5bpOc72GURqIhUVETd0xx13YDKZMJlMeHl50bRpU6ZPn05RUVGFX7tnz56kpKQQHBwMwOzZs6lVq9Yl223ZsoW77767wu93NWUpEq+99hqzZ8+u0jwiUvl0UUIRNzV06FA+/PBDrFYrS5cuZdKkSXh6ejJ16tQKva6Xl1epLg8fFhZWofepLDabDZPJVFKsRMS1aERFxE15e3sTERFBw4YNue+++xg0aBCLFy8G4Ny5c0yYMIHatWvj5+fHsGHDOHz4cMlzExMTGTFiBLVr18bf35/Y2FiWLl0KXDz1s3r1au68804yMzNLRnCmTZsGXDzaMW7cOMaMGXNRvsLCQurUqcPcuXOB4kvPz5gxg0aNGuHr60tcXBwLFy684ufr168fiYmJPPLIIyXvDb+N8CxevJjWrVvj7e1NUlLSJVM//fr144EHHuCBBx4gODiYOnXq8MQTT3C1y59lZGQwceJEwsLCCAoKYsCAAezatavk8V27dtG/f38CAwMJCgqiU6dObN269Xf+T4nI1aioiNQQvr6+FBQUAMVTQ1u3bmXx4sVs3LgRh8PB8OHDKSwsBGDSpElYrVbWrl3Lnj17eOGFFwgICLjkNXv27Mmrr75KUFAQKSkppKSkMGXKlEu2Gz9+PN988w05OTkl9y1fvpy8vDxGjRoFwIwZM5g7dy7vvvsu8fHxPPLII9x2222sWbPmsp/nyy+/JCoqiunTp5e89wV5eXm88MILfPDBB8THxxMeHn7Z15gzZw4eHh5s3ryZ1157jZdffpkPPvjgij/DW265hdTUVJYtW8a2bdvo2LEjAwcO5OzZsyWfMyoqii1btrBt2zYee+wxPD09r/h6IvL7NPUj4uYcDgcrV65k+fLlPPjggxw+fJjFixfz008/0bNnTwA++eQToqOjWbRoEbfccgtJSUn84Q9/oG3btgA0btz4sq/t5eVFcHAwJpPpqtNB1157Lf7+/nz11Vf88Y9/BODTTz/lhhtuIDAwEKvVyvPPP88PP/xAjx49St5z/fr1vPfee/Tt2/eS1wwJCcFisRAYGHjJexcWFvL2228TFxd31Z9NdHQ0r7zyCiaTiRYtWrBnzx5eeeUV7rrrrku2Xb9+PZs3byY1NRVvb28AXnzxRRYtWsTChQu5++67SUpK4q9//SstW7YEoFmzZld9fxH5fRpREXFT3377LQEBAfj4+DBs2DDGjBnDtGnT2L9/Px4eHnTr1q1k29DQUFq0aMH+/fsBeOihh3j22Wfp1asXTz31FLt3765QFg8PD0aPHs0nn3wCQG5uLl9//TXjx48H4MiRI+Tl5TF48GACAgJKbnPnzuXo0aNlfj8vLy/atWv3u9t17969ZMoIoEePHhw+fBibzXbJtrt27SInJ4fQ0NCLMh47dqwk46OPPsrEiRMZNGgQM2fOLFd2EbmYRlRE3FT//v1555138PLyIjIyEg+P0v91nzhxItdeey1Llizh+++/Z8aMGbz00ks8+OCD5c4zfvx4+vbtS2pqKitWrMDX15ehQ4cClEwJLVmyhPr161/0vAujF2Xh6+t7UQGpDDk5OdSrV4/Vq1df8tiFo56mTZvGuHHjWLJkCcuWLeOpp55i3rx5JdNbIlJ2Kioibsrf35+mTZtecn+rVq0oKiri559/Lpn6OXPmDAcPHqR169Yl20VHR3Pvvfdy7733MnXqVN5///3LFhUvL6/LjkD8r549exIdHc38+fNZtmwZt9xyS8n6jf9e9Hq5aZ4rKe17X8nPP/980Z83bdpEs2bNsFgsl2zbsWNHTp06hYeHBzExMVd8zebNm9O8eXMeeeQRxo4dy4cffqiiIlIBmvoRqWGaNWvGyJEjueuuu1i/fj27du3itttuo379+owcORKAyZMns3z5co4dO8b27dv58ccfadWq1WVfLyYmhpycHFauXEl6ejp5eXlXfO9x48bx7rvvsmLFipJpH4DAwECmTJnCI488wpw5czh69Cjbt2/njTfeYM6cOVd8vZiYGNauXcuJEydIT08v888iKSmJRx99lIMHD/LZZ5/xxhtv8PDDD19220GDBtGjRw9uvPFGvv/+exISEtiwYQOPP/44W7du5fz58zzwwAOsXr2axMREfvrpJ7Zs2XLFn5uIlI6KikgN9OGHH9KpUyeuv/56evTogcPhYOnSpSUjHDabjUmTJtGqVSuGDh1K8+bNefvtty/7Wj179uTee+9lzJgxhIWFMWvWrCu+7/jx49m3bx/169enV69eFz32zDPP8MQTTzBjxoyS912yZAmNGjW64utNnz6dhIQEmjRpUq7ztkyYMIHz58/TtWtXJk2axMMPP3zFk9SZTCaWLl1Knz59uPPOO2nevDm33noriYmJ1K1bF4vFwpkzZ5gwYQLNmzdn9OjRDBs2jKeffrrMuUTkNybH1U4aICLipvr160f79u11inwRJ6cRFREREXFaKioiIiLitDT1IyIiIk5LIyoiIiLitFRURERExGmpqIiIiIjTUlERERERp6WiIiIiIk5LRUVEREScloqKiIiIOC0VFREREXFaKioiIiLitP4fYBbpbfCn3v8AAAAASUVORK5CYII=",
                        "text/plain": [
                            "<Figure size 640x480 with 1 Axes>"
                        ]
                    },
                    "metadata": {},
                    "output_type": "display_data"
                }
            ],
            "source": [
                "# Train for 25 epochs\n",
                "n_epochs = 25\n",
                "\n",
                "# Positive triples seen until now\n",
                "cumulative_triples = 0\n",
                "training_loss = []\n",
                "for ep in range(n_epochs):\n",
                "    ep_start_time = time.time()\n",
                "    ep_log = []\n",
                "    for batch in train_dl:\n",
                "        step_start_time = time.time()\n",
                "        # triple_mask is not used during training\n",
                "        triple_mask = batch.pop(\"triple_mask\")\n",
                "        cumulative_triples += triple_mask.numel()\n",
                "        res = poptorch_model(**{k: v.flatten(end_dim=1) for k, v in batch.items()})\n",
                "        # res[\"loss\"] contains the summed loss of elements in the last batch, for each IPU\n",
                "        ep_log.append(\n",
                "            dict(\n",
                "                loss=float(torch.sum(res[\"loss\"])) / triple_mask[-1].numel(),\n",
                "                step_time=(time.time() - step_start_time),\n",
                "            )\n",
                "        )\n",
                "    ep_loss = [v[\"loss\"] for v in ep_log]\n",
                "    training_loss.extend([v[\"loss\"] for v in ep_log])\n",
                "    print(\n",
                "        f\"Epoch {ep+1} loss: {np.mean(ep_loss):.6f} --- positive triples processed: {cumulative_triples:.2e}\"\n",
                "    )\n",
                "    print(\n",
                "        f\"Epoch duration (sec): {(time.time() - ep_start_time):.5f} (average step time: {np.mean([v['step_time'] for v in ep_log]):.5f})\"\n",
                "    )\n",
                "\n",
                "# Plot loss as a function of the number of positive triples processed\n",
                "total_triples = np.cumsum(n_epochs * len(train_dl) * [triple_mask.numel()])\n",
                "ax = plt.gca()\n",
                "ax.plot(total_triples, training_loss)\n",
                "ax.set_xscale(\"log\")\n",
                "ax.set_xlabel(\"Positive triples\")\n",
                "ax.set_ylabel(\"Loss\")\n",
                "\n",
                "poptorch_model.detachFromDevice()\n",
                "del train_dl"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Inference\n",
                "\n",
                "Let's see now how the trained model performs on the **validation** set. \n",
                "\n",
                "We create a new `PartitionedTripleSet` holding the validation triples. Differently from the training set, each validation triple has a specific set of 500 negative heads/tails to be scored against, hence we use the `TripleBasedShardedNegativeSampler` negative sampler class to sample exactly the negative entities needed by each triple. This means that we will not be using negative sample sharing any more.\n",
                "\n",
                "For the batch sampler, we again use the `RigidShardedBatchSampler` class, but we now set the option `duplicate_batch=True`. This means that the two halves of the micro-batch (where we corrupt heads and tails respectively) contain the same positive triples, so that we can score negative heads and negative tails with a single model."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 14,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "head            torch.Size([10, 4, 4, 60])          torch.int32;\n",
                        "relation        torch.Size([10, 4, 4, 60])          torch.int32;\n",
                        "tail            torch.Size([10, 4, 4, 60])          torch.int32;\n",
                        "triple_mask     torch.Size([10, 4, 4, 60])          torch.bool;\n",
                        "negative        torch.Size([10, 4, 4, 240, 172])    torch.int32;\n",
                        "negative_mask   torch.Size([10, 4, 240, 4, 172])    torch.bool;\n"
                    ]
                }
            ],
            "source": [
                "valid_triples = PartitionedTripleSet.create_from_dataset(\n",
                "    dataset=biokg, part=\"valid\", sharding=sharding, partition_mode=\"ht_shardpair\"\n",
                ")\n",
                "ns_valid = TripleBasedShardedNegativeSampler(\n",
                "    negative_heads=valid_triples.neg_heads,\n",
                "    negative_tails=valid_triples.neg_tails,\n",
                "    sharding=sharding,\n",
                "    corruption_scheme=\"ht\",\n",
                "    seed=seed,\n",
                ")\n",
                "bs_valid = RigidShardedBatchSampler(\n",
                "    partitioned_triple_set=valid_triples,\n",
                "    negative_sampler=ns_valid,\n",
                "    shard_bs=shard_bs,\n",
                "    batches_per_step=10,\n",
                "    seed=seed,\n",
                "    duplicate_batch=True,\n",
                ")\n",
                "\n",
                "# Example batch\n",
                "idx_sampler = iter(bs_valid.get_dataloader_sampler(shuffle=False))\n",
                "for k, v in bs_valid[next(idx_sampler)].items():\n",
                "    print(f\"{k:<15} {str(v.shape):<35} {v.dtype};\")"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "We see that the `negative` tensor returned now by the dataloader has a trailing dimension of 172, meaning that each validation query is scored against `4*172` negative heads/tails, which is larger than 500. This is due to the fact that the triple-specific negatives are, in general, not equally split between the 4 shards, therefore some padding needs to be applied. `negative_mask` returned by the negative sampler is used to identify the padding negative entities, so that the corresponding scores can be filtered out when computing the metrics.\n",
                "\n",
                "We can now instantiate the inference model. We use the `besskge.metric.Evaluation` class to specify which **metrics** we want to compute and pass it to the BESS module. Here we look at Hits@K for K=1,5,10, giving us the percentage of the validation triples where the ground truth head/tail is among the K most-likely predictions made by the model, and the mean reciprocal rank (MRR). For all these metrics, **higher is better**. By specifying `reduction=\"sum\"` we reduce communication between host and device, by returning the summed values, over the elements in the same micro-batch, for each metric.\n",
                "\n",
                "We now use a different flavour of BESS compared to training, namely `ScoreMovingBessKGE`. This is recommended when the number of negative entities to be fetched from other devices is large, as it is typically the case when using `TripleBasedShardedNegativeSampler`. While `EmbeddingMovingBessKGE` sends the negative embeddings to the device where the positive triple is scored, `ScoreMovingBessKGE` fetches the queries with an AllGather and computes negative scores on the shard where the negative entities are stored, and then sends the scores back to the original device. This allows for communicating of scores instead of embeddings, which is usually cheaper, although it requires additional collective communications between devices. We encourage you to play with different configurations to see which one gives the shorter overall validation time."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 15,
            "metadata": {},
            "outputs": [
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "[09:54:14.815] [poptorch:cpp] [warning] [DISPATCHER] Type coerced from Long to Int for tensor id 435\n",
                        "[09:54:14.817] [poptorch:cpp] [warning] [DISPATCHER] Type coerced from Long to Int for tensor id 441\n",
                        "Graph compilation: 100%|██████████| 100/100 [00:53<00:00]\n"
                    ]
                }
            ],
            "source": [
                "# Each triple is now to be scored against a specific set of negatives, so we turn off negative sample sharing\n",
                "rotate_score_fn.negative_sample_sharing = False\n",
                "\n",
                "val_options = poptorch.Options()\n",
                "val_options.replication_factor = sharding.n_shard\n",
                "val_options.deviceIterations(bs_valid.batches_per_step)\n",
                "val_options.outputMode(poptorch.OutputMode.All)\n",
                "\n",
                "# Validation dataloader\n",
                "valid_dl = bs_valid.get_dataloader(options=val_options, shuffle=False)\n",
                "\n",
                "# With reduction=\"sum\" the returned res[\"metrics\"] has shape (batches_per_step * n_shard, n_metrics)\n",
                "evaluator = Evaluation([\"mrr\", \"hits@1\", \"hits@5\", \"hits@10\"], reduction=\"sum\")\n",
                "# BESS wrapper\n",
                "model_inf = ScoreMovingBessKGE(\n",
                "    negative_sampler=ns_valid, score_fn=rotate_score_fn, evaluation=evaluator\n",
                ")\n",
                "\n",
                "# PopTorch wrapper\n",
                "poptorch_model_inf = poptorch.inferenceModel(model_inf, options=val_options)\n",
                "poptorch_model_inf.entity_embedding.replicaGrouping(\n",
                "    poptorch.CommGroupType.NoGrouping,\n",
                "    0,\n",
                "    poptorch.VariableRetrievalMode.OnePerGroup,\n",
                ")\n",
                "\n",
                "# Compile model\n",
                "batch = next(iter(valid_dl))\n",
                "res = poptorch_model_inf(**{k: v.flatten(end_dim=1) for k, v in batch.items()})"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 16,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "hits@1 : 0.746998\n",
                        "hits@5 : 0.905811\n",
                        "hits@10 : 0.941649\n",
                        "mrr : 0.818204\n"
                    ]
                }
            ],
            "source": [
                "# Perform validation and print metrics\n",
                "\n",
                "val_log = []\n",
                "# The final value of n_val_queries will be twice the number of triples in the validation set\n",
                "# as each triple is scored against negative heads and negative tails separately\n",
                "n_val_queries = 0\n",
                "for batch_val in valid_dl:\n",
                "    res = poptorch_model_inf(**{k: v.flatten(end_dim=1) for k, v in batch_val.items()})\n",
                "\n",
                "    n_val_queries += batch_val[\"triple_mask\"].sum()\n",
                "    # By transposing res[\"metrics\"] we separate the outputs for the different metrics\n",
                "    val_log.append(\n",
                "        {\n",
                "            k: v.sum()\n",
                "            for k, v in zip(\n",
                "                evaluator.metrics.keys(),\n",
                "                res[\"metrics\"].T,\n",
                "            )\n",
                "        }\n",
                "    )\n",
                "\n",
                "for metric in val_log[0].keys():\n",
                "    reduced_metric = sum([l[metric] for l in val_log]) / n_val_queries\n",
                "    print(\"%s : %f\" % (metric, reduced_metric))\n",
                "\n",
                "poptorch_model_inf.detachFromDevice()\n",
                "del valid_dl"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## How many parameters does our model have?\n",
                "\n",
                "We can see how many parameters the model has:"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 17,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "# model parameters: 12,006,592\n"
                    ]
                }
            ],
            "source": [
                "# Equivalent to sum(p.numel() for p in model_inf.parameters()), as embeddings are the only trainable parameters\n",
                "print(f\"# model parameters: {model_inf.n_embedding_parameters:,}\")"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Looking at the [OGB leaderboard](https://ogb.stanford.edu/docs/leader_linkprop/#ogbl-biokg) for the OGBL-BioKG dataset, the validation MRR we got is **very competitive even when compared against other classical shallow KGE models with 15 times more parameters** than we used. Not bad for a model we trained so quickly!"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Scaling analysis\n",
                "\n",
                "The OGBL-BioKG knowledge graph is small enough that we can actually train the model above (with same embedding size) on a single IPU. This allows us to get a better grasp on the advantages of distribution.\n",
                "\n",
                "In order to train on one IPU, we just need to set `n_shard = 1` when defining the `Sharding` of the entity table."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 18,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Number of shards: 1\n",
                        "\n",
                        "Number of entities in each shard: 93773\n",
                        "\n",
                        "Number of triples per (h,t) shard-pair:\n",
                        " [[4762678]]\n"
                    ]
                }
            ],
            "source": [
                "n_shard = 1\n",
                "\n",
                "sharding = Sharding.create(\n",
                "    n_entity=biokg.n_entity,\n",
                "    n_shard=n_shard,\n",
                "    seed=seed,\n",
                "    type_offsets=np.fromiter(biokg.type_offsets.values(), dtype=np.int32),\n",
                ")\n",
                "\n",
                "print(f\"Number of shards: {sharding.n_shard}\\n\")\n",
                "\n",
                "# All entities in the KG are now on a single shard\n",
                "print(f\"Number of entities in each shard: {sharding.max_entity_per_shard}\\n\")\n",
                "\n",
                "train_triples = PartitionedTripleSet.create_from_dataset(\n",
                "    dataset=biokg, part=\"train\", sharding=sharding, partition_mode=\"ht_shardpair\"\n",
                ")\n",
                "\n",
                "# There is now a single (h,t) shard-pair, containing all training triples\n",
                "print(f\"Number of triples per (h,t) shard-pair:\\n {train_triples.triple_counts}\")"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "As explained in the section [Negative and batch samplers](#negative-and-batch-samplers), in order to score each triple against the same number of negatives as before, we need to multiply `n_negative` by 4."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 19,
            "metadata": {},
            "outputs": [],
            "source": [
                "# When reducing the number of shards by a factor of 4,\n",
                "# the number of negatives per triple per shard (i.e. n_negative) needs to increase by the same factor\n",
                "neg_sampler = RandomShardedNegativeSampler(\n",
                "    n_negative=4,\n",
                "    sharding=sharding,\n",
                "    seed=seed,\n",
                "    corruption_scheme=\"ht\",\n",
                "    local_sampling=False,\n",
                "    flat_negative_format=False,\n",
                ")"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Similarly, since we maintain the same micro-batch size used before (to get a fair comparison using negative sample sharing), we multiply the accumulation factor by 4 so that the global batch size is the same."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 20,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "# triples per shard-pair per step: 240 \n",
                        "\n",
                        "head         torch.Size([192, 1, 1, 240])   torch.int32;\n",
                        "relation     torch.Size([192, 1, 1, 240])   torch.int32;\n",
                        "tail         torch.Size([192, 1, 1, 240])   torch.int32;\n",
                        "triple_mask  torch.Size([192, 1, 1, 240])   torch.bool;\n",
                        "negative     torch.Size([192, 1, 1, 240, 4]) torch.int32;\n"
                    ]
                }
            ],
            "source": [
                "device_iterations = 8\n",
                "accum_factor = 24\n",
                "shard_bs = 240\n",
                "\n",
                "batch_sampler = RigidShardedBatchSampler(\n",
                "    partitioned_triple_set=train_triples,\n",
                "    negative_sampler=neg_sampler,\n",
                "    shard_bs=shard_bs,\n",
                "    batches_per_step=device_iterations * accum_factor,\n",
                "    seed=seed,\n",
                ")\n",
                "\n",
                "\n",
                "print(f\"# triples per shard-pair per step: {batch_sampler.positive_per_partition} \\n\")\n",
                "\n",
                "# Example batch\n",
                "idx_sampler = iter(batch_sampler.get_dataloader_sampler(shuffle=True))\n",
                "for k, v in batch_sampler[next(idx_sampler)].items():\n",
                "    print(f\"{k:<12} {str(v.shape):<30} {v.dtype};\")"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "The dataloader and model are created exactly as before."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 21,
            "metadata": {},
            "outputs": [
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "Graph compilation: 100%|██████████| 100/100 [00:48<00:00]\n"
                    ]
                }
            ],
            "source": [
                "options = poptorch.Options()\n",
                "options.deviceIterations(device_iterations)\n",
                "options.Training.gradientAccumulation(accum_factor)\n",
                "\n",
                "train_dl = batch_sampler.get_dataloader(\n",
                "    options=options, shuffle=True, num_workers=5, persistent_workers=True\n",
                ")\n",
                "\n",
                "logsigmoid_loss_fn = LogSigmoidLoss(margin=12.0, negative_adversarial_sampling=True)\n",
                "rotate_score_fn = RotatE(\n",
                "    negative_sample_sharing=True,\n",
                "    scoring_norm=1,\n",
                "    sharding=sharding,\n",
                "    n_relation_type=biokg.n_relation_type,\n",
                "    embedding_size=64,\n",
                ")\n",
                "\n",
                "model = EmbeddingMovingBessKGE(\n",
                "    negative_sampler=neg_sampler, score_fn=rotate_score_fn, loss_fn=logsigmoid_loss_fn\n",
                ")\n",
                "\n",
                "opt = poptorch.optim.AdamW(\n",
                "    model.parameters(),\n",
                "    lr=0.001,\n",
                ")\n",
                "\n",
                "poptorch_model = poptorch.trainingModel(model, options=options, optimizer=opt)\n",
                "poptorch_model.entity_embedding.replicaGrouping(\n",
                "    poptorch.CommGroupType.NoGrouping,\n",
                "    0,\n",
                "    poptorch.VariableRetrievalMode.OnePerGroup,\n",
                ")\n",
                "\n",
                "\n",
                "# Compile model\n",
                "batch = next(iter(train_dl))\n",
                "_ = batch.pop(\"triple_mask\")\n",
                "res = poptorch_model(**{k: v.flatten(end_dim=1) for k, v in batch.items()})"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 22,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Epoch 1 loss: 1.163840 --- positive triples processed: 4.79e+06\n",
                        "Epoch duration (sec): 5.82170 (average step time: 0.05055)\n",
                        "Epoch 2 loss: 0.507319 --- positive triples processed: 9.58e+06\n",
                        "Epoch duration (sec): 5.24277 (average step time: 0.05031)\n",
                        "Epoch 3 loss: 0.447497 --- positive triples processed: 1.44e+07\n",
                        "Epoch duration (sec): 5.24105 (average step time: 0.05033)\n",
                        "Epoch 4 loss: 0.407904 --- positive triples processed: 1.92e+07\n",
                        "Epoch duration (sec): 5.24417 (average step time: 0.05033)\n",
                        "Epoch 5 loss: 0.395906 --- positive triples processed: 2.40e+07\n",
                        "Epoch duration (sec): 5.23227 (average step time: 0.05025)\n",
                        "Epoch 6 loss: 0.372093 --- positive triples processed: 2.88e+07\n",
                        "Epoch duration (sec): 5.23252 (average step time: 0.05025)\n",
                        "Epoch 7 loss: 0.367683 --- positive triples processed: 3.35e+07\n",
                        "Epoch duration (sec): 5.25688 (average step time: 0.05048)\n",
                        "Epoch 8 loss: 0.370355 --- positive triples processed: 3.83e+07\n",
                        "Epoch duration (sec): 5.24375 (average step time: 0.05035)\n",
                        "Epoch 9 loss: 0.357917 --- positive triples processed: 4.31e+07\n",
                        "Epoch duration (sec): 5.24873 (average step time: 0.05040)\n",
                        "Epoch 10 loss: 0.362484 --- positive triples processed: 4.79e+07\n",
                        "Epoch duration (sec): 5.24854 (average step time: 0.05032)\n"
                    ]
                }
            ],
            "source": [
                "# Train for the first 10 epochs\n",
                "n_epochs = 10\n",
                "\n",
                "cumulative_triples = 0\n",
                "training_loss = []\n",
                "for ep in range(n_epochs):\n",
                "    ep_start_time = time.time()\n",
                "    ep_log = []\n",
                "    for batch in train_dl:\n",
                "        step_start_time = time.time()\n",
                "        triple_mask = batch.pop(\"triple_mask\")\n",
                "        cumulative_triples += triple_mask.numel()\n",
                "        res = poptorch_model(**{k: v.flatten(end_dim=1) for k, v in batch.items()})\n",
                "        ep_log.append(\n",
                "            dict(\n",
                "                loss=float(torch.sum(res[\"loss\"])) / triple_mask[-1].numel(),\n",
                "                step_time=(time.time() - step_start_time),\n",
                "            )\n",
                "        )\n",
                "    ep_loss = [v[\"loss\"] for v in ep_log]\n",
                "    training_loss.extend([v[\"loss\"] for v in ep_log])\n",
                "    print(\n",
                "        f\"Epoch {ep+1} loss: {np.mean(ep_loss):.6f} --- positive triples processed: {cumulative_triples:.2e}\"\n",
                "    )\n",
                "    print(\n",
                "        f\"Epoch duration (sec): {(time.time() - ep_start_time):.5f} (average step time: {np.mean([v['step_time'] for v in ep_log]):.5f})\"\n",
                "    )\n",
                "\n",
                "poptorch_model.detachFromDevice()\n",
                "del train_dl"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "By comparing the epoch durations, we see that **scaling from 1 to 4 IPUs speeds up training by a factor of ~3.2X**. This is actually a conservative estimate, because - as one can notice from the printouts - the number of triples processed at each epoch is larger when running on 4 IPUs, as a consequence of using `RigidShardedBatchSampler` (the number of positive triples seen during an epoch is determined by the number of triples in the largest shard-pair bucket)."
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Conclusions and next steps\n",
                "\n",
                "To recap, these are the basic steps to run distributed training/inference with BESS-KGE:\n",
                "* wrap your knowledge graph dataset with `besskge.dataset.KGDataset` (see also [Using BESS-KGE with your own data](0_custom_KG_dataset.ipynb));\n",
                "* shard entities in the graph based on the number of IPUs you want to use, by using `besskge.sharding.Sharding`, and partition triples accordingly with `besskge.sharding.PartitionedTripleSet`;\n",
                "* select a negative sampler from `besskge.negative_sampler` to sample the entities used to corrupt positive triples;\n",
                "* use a batch sampler from `besskge.batch_sampler` to create the batch dataloader; \n",
                "* specify the scoring and loss function (see `besskge.scoring` and `besskge.loss`);\n",
                "* instantiate the distributed model with one of the subclasses of `besskge.bess.BessKGE`.\n",
                "\n",
                "You can easily modify this notebook to use different combinations of KGE models, embedding sizes and loss functions, change the number of negative samples or the sampling scheme (for instance, try training with `TypeBasedShardedNegativeSampler`).\n",
                "\n",
                "What next? If you are interested in using these models to complete queries when no candidate heads/tails are provided, have a look at the the [Knowledge Graph Completion on YAGO3-10](2_yago_topk_prediction.ipynb) notebook. For using FP16 embeddings to deal with larger graphs, look at the [FP16 embeddings on OGBL-WikiKG2](3_wikikg2_fp16.ipynb) notebook."
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": ".venv_3.2",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.8.10"
        },
        "orig_nbformat": 4
    },
    "nbformat": 4,
    "nbformat_minor": 2
}