☄️ Update Comet integration to include LogCompletionsCallback and Trainer.evaluation_loop()

[yaricom]

What does this PR do?

Updated Comet integration to include the following:

• Integration with LogCompletionsCallback
• Integration with CPOTrainer.evaluation_loop()
• Integration with DPOTrainer.evaluation_loop()
• Integration with BCOTrainer.evaluation_loop()
• Integration with KTOTrainer.evaluation_loop()
• Integration with ORPOTrainer.evaluation_loop()

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[qgallouedec]

can you save it in the output_dir instead?

[yaricom]

Comet SDK post submitted files to the server in the background. The intermediate copy of the file is stored in the temporary directory until upload to the Comet server completes.

Could you please elaborate on your idea so I can understand it better?

[qgallouedec]

When you run experiment.log_table(tabular_data=table, filename=name), does it save locally something in a filename name?

[yaricom]

It is saves indeed the table data into temporary file in temporary directory. This file lives until its upload to the Comet server is complete. After that it is automatically cleaned either by Comet SDK or by OS if something goes bad during Python script execution.

[qgallouedec]

Can you screenshot a result?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yaricom]

Sure, here is s screenshot from my account at Comet.

[yaricom]

And this is a DataFrame encoded as CSV.
game_log.csv

[yaricom]

The script I was using to test DPO trainer integration.

import os

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer

from trl import DPOConfig, DPOTrainer

os.environ["TOKENIZERS_PARALLELISM"] = "false"


def main():
    output_dir = "models/minimal/dpo_my"

    model_id = "trl-internal-testing/tiny-Qwen2ForCausalLM-2.5"
    # model_id = "Qwen/Qwen2-0.5B-Instruct"
    model = AutoModelForCausalLM.from_pretrained(model_id)
    ref_model = AutoModelForCausalLM.from_pretrained(model_id)
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    tokenizer.pad_token = tokenizer.eos_token


    training_args = DPOConfig(
        output_dir=output_dir,
        per_device_train_batch_size=2,
        max_steps=1,
        remove_unused_columns=False,
        gradient_accumulation_steps=8,
        precompute_ref_log_probs=False,
        learning_rate=5.0e-7,
        eval_strategy="steps",
        eval_steps=1,
        report_to="all",
        generate_during_eval=True,
        max_length=1024,
    )

    # dummy_dataset = load_dataset("trl-internal-testing/zen", "standard_preference")
    dummy_dataset = load_dataset("trl-lib/ultrafeedback_binarized", "default")

    dummy_dataset["train"] = dummy_dataset["train"].select(range(20))
    dummy_dataset["test"] = dummy_dataset["test"].select(range(40))

    trainer = DPOTrainer(
        model=model,
        ref_model=ref_model,
        args=training_args,
        processing_class=tokenizer,
        train_dataset=dummy_dataset["train"],
        eval_dataset=dummy_dataset["test"],
    )

    trainer.train()

    trainer.evaluate()


if __name__ == "__main__":
    main()

Do not forget to set COMET_APY_KEY environment variable while executing it.

[qgallouedec]

pandas is indeed installed.

trl-> datasets->pandas

[qgallouedec]

for the record, this doesn't work, because the pol can be over-truncated. See

can be reproduced with

from datasets import load_dataset
from trl import CPOConfig, CPOTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-0.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B-Instruct")
train_dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train[:1%]")
eval_dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="test[:1%]")

training_args = CPOConfig(output_dir="Qwen2-0.5B-CPO", logging_steps=10,  generate_during_eval=True, eval_steps=2, eval_strategy="steps")
trainer = CPOTrainer(model=model, args=training_args, processing_class=tokenizer, train_dataset=train_dataset, eval_dataset=eval_dataset)
trainer.train()

But this is out of the scope of this PR.

[qgallouedec]

very nice, thanks @yaricom!