How can I implement partial output in the react demo?

[DikkooXie]

Question

Hello! I am reading the Transformers.js documentation for "Building a react application", but I encountered an issue at step 4.
I don't know how to implement the partial output of the translation results, even though the documentation provides the following instructions:

  let output = await translator(event.data.text, {
      tgt_lang: event.data.tgt_lang,
      src_lang: event.data.src_lang,

      // Allows for partial output
      callback_function: x => {
          self.postMessage({
              status: 'update',
              output: translator.tokenizer.decode(x[0].output_token_ids, { skip_special_tokens: true })
          });
      }
  });

I have completed all the steps in the tutorial documentation, but I still cannot get the output to work properly. I tried using console.log for debugging and found that the callback_function is not working, and the main thread is not receiving any messages with the status update. I have also not found any information about the callback_function in the transformers.js documentation. I apologize for taking up your time, but I sincerely need your help. 🙏

[xenova]

Hi there 👋 Apologies for the delayed response. This is possible with the TextStreamer class, which was introduced in Transformers.js v3. Example code:

import { pipeline, TextStreamer } from "@huggingface/transformers";

// Create a text generation pipeline
const generator = await pipeline(
  "text-generation",
  "onnx-community/Qwen2.5-Coder-0.5B-Instruct",
  { dtype: "q4" },
);

// Define the list of messages
const messages = [
  { role: "system", content: "You are a helpful assistant." },
  { role: "user", content:  "Write a quick sort algorithm." },
];

// Create text streamer
const streamer = new TextStreamer(generator.tokenizer, {
  skip_prompt: true,
  callback_function: (text) => console.log(text), // Optional callback function
})

// Generate a response
const output = await generator(messages, { max_new_tokens: 512, do_sample: false, streamer });
console.log(output[0].generated_text.at(-1).content);

---

However, this has not yet been updated in the documentation. If anyone would like to make a PR for this, I would greatly appreciate it!