Update mixtral.md

mixtral.md

@@ -285,8 +285,30 @@ output = model.generate(**inputs, max_new_tokens=50)
 print(tokenizer.decode(output[0], skip_special_tokens=True))
 ```
 
-Note that for both QLoRA and GPTQ you need at least 30 GB of GPU VRAM to fit the model. You can make it work with 24 GB if you use `device_map="auto"`, like in the example above, so some layers are offloaded to CPU.
+You could also just load the model using a GPTQ configuration setting the desired parameters , as usual when working with transformers .
+For faster inference and production load we want to leverage the [exllama kernels](https://github.com/turboderp/exllama) (Achieving the same latency as fp16 model, but 4x less memory usage) .
 
+```python
+import torch
+from transformers
+
+model_id = "TheBloke/Mixtral-8x7B-v0.1-GPTQ"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+gptq_config = GPTQConfig(bits=4, use_exllama=True)
+model = AutoModelForCausalLM.from_pretrained(model_id,quantization_config=gptq_config,
+                                             device_map="auto")
+
+prompt = "[INST] Explain what a Mixture of Experts is in less than 100 words. [/INST]"
+inputs = tokenizer(prompt, return_tensors="pt").to(0)
+
+output = model.generate(**inputs, max_new_tokens=50)
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+If left unset , the "use_exllama" parameter defaults to True , enabling the exllama backend functionality, specifically designed to work with the "bits" value of 4.
+
+Note that for both QLoRA and GPTQ you need at least 30 GB of GPU VRAM to fit the model. You can make it work with 24 GB if you use `device_map="auto"`, like in the example above, so some layers are offloaded to CPU.
 
 ## Disclaimers and ongoing work