Having issue with Gemma model tokenizer

[ivyas21]

I'm having an issue with Gemma model. I used swift playground. Here is sample code to reproduce below error:

import UIKit
import Tokenizers

func testTokenizer() async throws -> Tokenizer {
    let tokenizer = try await AutoTokenizer.from(pretrained: "google/gemma-2b")
    let inputIds = tokenizer("Today she took a train to the West")
    return tokenizer
}

var greeting = "Hello, playground"

let t = try await testTokenizer()
t

error: Execution was interrupted, reason: .
The process has been left at the point where it was interrupted, use "thread return -x" to return to the state before expression evaluation.

Please advice me. Thank you so much!

[vinu-vanjari]

@ivyas21 , Google gemma model is gated and you need authentication to access it.

refer these documentations:
https://huggingface.co/docs/hub/en/models-gated
https://huggingface.co/google/gemma-2b/discussions/28

You can do following steps to get this working:

1. Go to google/gemma-2b page and request access
2. Go to Hugging face access tokens page in settings. (https://huggingface.co/settings/tokens)
3. Generate new token if not already created and copy it
4. Modify your code to pass hfToken to HubApi

import UIKit
import Tokenizers

func testTokenizer() async throws -> Tokenizer {
    let tokenizer = try await AutoTokenizer.from(
        pretrained: "google/gemma-2b",
        hubApi: HubApi(hfToken: MY_ACCESS_TOKEN)
    )
    let inputIds = tokenizer("Today she took a train to the West")
    return tokenizer
}

var greeting = "Hello, playground"

let t = try await testTokenizer()
t