[Samples] merge LLM samples to "text_generation" folder #1411

olpipi · 2024-12-19T11:17:59Z

No description provided.

samples/cpp/text_generation/CMakeLists.txt

samples/cpp/text_generation/README.md

Wovchena · 2024-12-19T11:34:58Z

samples/cpp/text_generation/README.md

+- **Main Feature:** Demonstrates simple text continuation.
+- **Run Command:**
+  ```bash
+  ./text_generation -m <model> -i "Hello, how are you?" -d CPU


If it's an example command, it must be specific, replace <model> with TinyLlama. If it's a help message, align it with what the sample would print.

There's no text_generation target.

The samples don't take device as an arg. They also don't have named args

If it's a help message, align it with what the sample would print.

Align them to match exactly. For example greedy would print greedy_causal_lm <MODEL_DIR> "<PROMPT>" (remove Usage: because you have *Run Command:).

samples/cpp/text_generation/README.md

Wovchena · 2024-12-19T11:43:58Z

samples/cpp/text_generation/README.md

Same must be done for python

I suggest to apply all comments first. And then make similar changes to python samples (or in the next PR)

I suggest to apply all comments first. And then make similar changes to python samples (or in the next PR)

Please change the structure of cpp sample and python in the same PR. It will be more convenient to update OpenVINO documentation and other public materials, which reference these samples.

samples/cpp/text_generation/README.md

ilya-lavrenov · 2024-12-20T10:58:25Z

samples/cpp/text_generation/CMakeLists.txt

+    COMPILE_PDB_NAME prompt_lookup_decoding_lm
+    # Ensure out of box LC_RPATH on macOS with SIP
+    INSTALL_RPATH_USE_LINK_PATH ON)
+# Don't install prompt_lookup_decoding_lm because it doesn't use openvino_genai library and is not verified yet.


it's obsolete information and now prompt_lookup_decoding_lm uses GenAI.
Please, uncomment this code.

ilya-lavrenov · 2024-12-20T10:58:47Z

samples/CMakeLists.txt

@@ -2,14 +2,8 @@
 # SPDX-License-Identifier: Apache-2.0
 #

-add_subdirectory(cpp/beam_search_causal_lm)
 add_subdirectory(cpp/benchmark_genai)


benchmark_genai is also LLM sample

See VLM https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp/visual_language_chat as example

ilya-lavrenov · 2024-12-20T11:01:02Z

samples/cpp/text_generation/README.md


-This example showcases inference of text-generation Large Language Models (LLMs): `chatglm`, `LLaMA`, `Qwen` and other models with the same signature. The application doesn't have many configuration options to encourage the reader to explore and modify the source code. For example, change the device for inference to GPU. The sample fearures `ov::genai::LLMPipeline` and configures it to run the simplest deterministic greedy sampling algorithm. There is also a Jupyter [notebook](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/llm-chatbot) which provides an example of LLM-powered Chatbot in Python.
+# OpenVINO AI Text Generation Samples


Suggested change

# OpenVINO AI Text Generation Samples

# OpenVINO GenAI Text Generation Samples

ilya-lavrenov · 2024-12-20T11:02:29Z

samples/cpp/text_generation/README.md

+
+These samples showcase the use of OpenVINO's inference capabilities for text generation tasks, including different decoding strategies such as beam search, multinomial sampling, and speculative decoding. Each sample has a specific focus and demonstrates a unique aspect of text generation.
+The applications don't have many configuration options to encourage the reader to explore and modify the source code. For example, change the device for inference to GPU.
+There is also a Jupyter [notebook](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/llm-chatbot) that provides an example of LLM-powered text generation in Python.


do we have multiple text generation notebooks in openvino_notebooks?
If yes, we need per-sample reference to notebook

CC @eaidova @sbalandi

ilya-lavrenov · 2024-12-20T11:05:28Z

samples/cpp/text_generation/README.md

+## Table of Contents
+1. [Download and Convert the Model and Tokenizers](#download-and-convert-the-model-and-tokenizers)
+2. [Running the Samples](#running-the-samples)
+3. [Using encrypted models](#using-encrypted-models)


this section is missed.

BTW, it not a dedicated section, it's also just a sample as others.

ilya-lavrenov · 2024-12-20T11:06:24Z

samples/cpp/text_generation/README.md

@@ -13,32 +24,99 @@ pip install --upgrade-strategy eager -r ../../requirements.txt
 optimum-cli export openvino --trust-remote-code --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 TinyLlama-1.1B-Chat-v1.0
 ```

-## Run
+## Running the Samples


Run Command in each sample already demonstrates how to run

ilya-lavrenov · 2024-12-20T11:07:46Z

samples/cpp/text_generation/README.md

@@ -13,32 +24,99 @@ pip install --upgrade-strategy eager -r ../../requirements.txt
 optimum-cli export openvino --trust-remote-code --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 TinyLlama-1.1B-Chat-v1.0


I suppose we cannot use just one model for all use cases.

For chat sample - chat model
For typical generation - either instruct or typical model
For speculative decoding we need 2 models and it's not shown at all what models to use

Move cpp text generation samples to one folder

92c3d94

github-actions bot added category: GHA CI based on Github actions category: cmake / build Cmake scripts category: samples GenAI samples labels Dec 19, 2024

olpipi force-pushed the samples_movement branch from 490743c to 9e7f861 Compare December 19, 2024 11:21

Update readme.md

3901fbb

olpipi force-pushed the samples_movement branch from 9e7f861 to 3901fbb Compare December 19, 2024 11:22

Wovchena reviewed Dec 19, 2024

View reviewed changes

Apply comments

b270500

ilya-lavrenov assigned ilya-lavrenov, Wovchena and DimaPastushenkov Dec 20, 2024

ilya-lavrenov changed the title ~~Samples movement~~ [Samples] merge LLM samples to "text_generation" folder Dec 20, 2024

ilya-lavrenov requested changes Dec 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Samples] merge LLM samples to "text_generation" folder #1411

[Samples] merge LLM samples to "text_generation" folder #1411

olpipi commented Dec 19, 2024

Wovchena Dec 19, 2024

olpipi Dec 19, 2024

Wovchena Dec 19, 2024

Wovchena Dec 19, 2024

olpipi Dec 19, 2024

DimaPastushenkov Dec 20, 2024

ilya-lavrenov Dec 20, 2024

ilya-lavrenov Dec 20, 2024

ilya-lavrenov Dec 20, 2024

ilya-lavrenov Dec 20, 2024

ilya-lavrenov Dec 20, 2024

ilya-lavrenov Dec 20, 2024

ilya-lavrenov Dec 20, 2024


		This example showcases inference of text-generation Large Language Models (LLMs): `chatglm`, `LLaMA`, `Qwen` and other models with the same signature. The application doesn't have many configuration options to encourage the reader to explore and modify the source code. For example, change the device for inference to GPU. The sample fearures `ov::genai::LLMPipeline` and configures it to run the simplest deterministic greedy sampling algorithm. There is also a Jupyter [notebook](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/llm-chatbot) which provides an example of LLM-powered Chatbot in Python.
		# OpenVINO AI Text Generation Samples

	# OpenVINO AI Text Generation Samples
	# OpenVINO GenAI Text Generation Samples

		@@ -13,32 +24,99 @@ pip install --upgrade-strategy eager -r ../../requirements.txt
		optimum-cli export openvino --trust-remote-code --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 TinyLlama-1.1B-Chat-v1.0

[Samples] merge LLM samples to "text_generation" folder #1411

Are you sure you want to change the base?

[Samples] merge LLM samples to "text_generation" folder #1411

Conversation

olpipi commented Dec 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment