Improvements on Exposed ORT support

[kallebysantos]

Hey there 🙏,
Coming from #947, I did test the new Exposed ORT feature in alpha-v20 and alpha-21 releases. I'd notice that to make it work we must specify 'auto' as inference device:

Current behaviour:

Without specify a device:

import { pipeline } from '@huggingface/transformers';

let pipe = await pipeline('sentiment-analysis');
// The exposed runtime is loaded 
// but when trying to inference:
let out = await pipe('I love transformers!');
//Error: Unsupported device: "wasm". Should be one of: 

Explicit using auto as device:

import { pipeline } from '@huggingface/transformers';

let pipe = await pipeline('sentiment-analysis', null, { device: 'auto' });

let out = await pipe('I love transformers!');
// [{'label': 'POSITIVE', 'score': 0.999817686}]

---

To solve this annoying config, the PR improves the environment support for Exposed ORT by implicitly selecting 'auto' as the default fallback instead of 'wasm'.

import { pipeline } from '@huggingface/transformers';

let pipe = await pipeline('sentiment-analysis');

let out = await pipe('I love transformers!');
// [{'label': 'POSITIVE', 'score': 0.999817686}]

[xenova]

Hi again 👋 Apologies for the late reply.

For this PR, could you explain the meaning/origin of the "auto" device? Is this an execution provider defined by the custom ORT runner?

[kallebysantos]

Hi again 👋 Apologies for the late reply.

That's ok, saw that you had a loot of work to do!
Btw congratulations v3 is finally launched 🎉

For this PR, could you explain the meaning/origin of the "auto" device? Is this an execution provider defined by the custom ORT runner?

Currently the runner will not look on it to execute, it does based on the environment. So the device property should do nothing in this case.

---

But the problem is that transformers.js tries to explicit use some device, like "wasm", and that is not possible to execute it without specify "auto".

Example, if we don't specify a device, or anything instead of "auto" it will throw:

const pipe = await pipeline('feature-extraction', 'supabase/gte-small')
// Error: Unsupported device "wasm", Should be one of:    .

const pipe = await pipeline('feature-extraction', 'supabase/gte-small', { device: 'cpu' })
// Error: Unsupported device "cpu", Should be one of:    .

const pipe = await pipeline('feature-extraction', 'supabase/gte-small', { device: 'webgpu' })
// Error: Unsupported device "webgpu", Should be one of:    .

But if I do "auto" it works:

const pipe = await pipeline('feature-extraction', 'supabase/gte-small',  { device: 'auto' })

// [.... embeddings ]

Custom ORT exposed:

console.log(globalThis[Symbol.for("onnxruntime"])
/*
{
  Tensor: [class Tensor],
  env: {},
  InferenceSession: { create: [AsyncFunction: fromBuffer] }
}
*/

So the purpose of this PR is to automatically set "auto" when its running from some Custom ORT. To avoid this annoying config.
Also I did add some global flags to detect if is or not running from Custom ORT.

---

I invite you to try run transformers.js from Supa stack, Custom ORT rust backend its available from [email protected]^.
You can use it from supabase cli:

npx supabase functions new "ort-test"

npx supabase functions serve

[xenova]

Thanks for the additional context! 👍 Just on that note, what if an exposed ORT library does allow for different devices to be specified, and the user could choose the version they would like?

For Supabase's runner, specifically, I imagine it is running on CPU, right? So "cpu" could be mapped to the default device on their side perhaps?

[kallebysantos]

Thanks for the additional context! 👍 Just on that note, what if an exposed ORT library does allow for different devices to be specified, and the user could choose the version they would like?

For Supabase's runner, specifically, I imagine it is running on CPU, right? So "cpu" could be mapped to the default device on their side perhaps?

Sure, it makes sense. But this way they still need to manually specify a device?

-I need to study more about how transformers.js handle the available devices.

At this moment, could you give me any suggestion? about how can we achieve: the following, without it throws an error?

const pipe = await pipeline('feature-extraction', 'supabase/gte-small')

Do I need to export some property that says "Hey trasnformers.js I'm using "cpu" as default so pls map to it" ?

---

EDIT: I'd look in both transfomers.js and onnxruntime-common and I think that I got your point, but I still thinking that custom providers should fallback to "auto" as default.

[xenova]

Note that there is no "auto" execution provider in onnxruntime-web/onnxruntime-node (and this is a layer added on top by Transformers.js).

That said, I think I see what the problem is: for the custom runtime, we don't specify supportedDevices or defaultDevices, which will be sent to the executionProviders option in createInferenceSession

transformers.js/src/backends/onnx.js

Lines 91 to 92 in 2c92943

	supportedDevices.push('cpu');
	defaultDevices = ['cpu'];

device -> execution provider mapping:

transformers.js/src/backends/onnx.js

Lines 117 to 136 in 2c92943

	export function deviceToExecutionProviders(device = null) {
	// Use the default execution providers if the user hasn't specified anything
	if (!device) return defaultDevices;
	// Handle overloaded cases
	switch (device) {
	case "auto":
	return supportedDevices;
	case "gpu":
	return supportedDevices.filter(x =>
	["webgpu", "cuda", "dml", "webnn-gpu"].includes(x),
	);
	}
	if (supportedDevices.includes(device)) {
	return [DEVICE_TO_EXECUTION_PROVIDER_MAPPING[device] ?? device];
	}
	throw new Error(`Unsupported device: "${device}". Should be one of: ${supportedDevices.join(', ')}.`)
	}

---

So, I think the solution here would be to set them both to undefined by default (instead of [] as it is currently), since then the custom ORT environment will use its defaults: https://onnxruntime.ai/docs/api/js/interfaces/InferenceSession.SessionOptions.html#executionProviders

[kallebysantos]

Hey Joshua, thanks for your suggestion

So, I think the solution here would be to set them both to undefined by default (instead of [] as it is currently), since then the custom ORT environment will use its defaults..

Yes it worked, 42ae1fe

[xenova]

Thanks for iterating! ✅ Looks good!

[xenova]

Thanks again for the modifications! I was talking to a colleague about this PR and one thing which was brought up was the possibility of accidental namespace clashes (if onnxruntime is detected globally).

For that reason, I think this should be "opt-in" behaviour based on the presence or absence of an environment variable, and should not be the default for all users.

In the supabase runtime, are you able to set default environment variables. e.g., HF_TRANSFORMERS_USE_EXPOSED_RUNTIME (name subject to change, ofc).

We can then use this to detect when to use the exposed runtime or not.

[xenova]

Using an array map would be great here! (minimal code duplication).