Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add StripNormalizer #133

Merged
merged 1 commit into from
Oct 2, 2024
Merged

add StripNormalizer #133

merged 1 commit into from
Oct 2, 2024

Conversation

mzbac
Copy link
Contributor

@mzbac mzbac commented Sep 30, 2024

try to resolve #127

@@ -1,6 +1,6 @@
//
// Normalizer.swift
//
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if there is any formatter I should use, but I noticed the current codebase seems to have the inconsistent format.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's not. We'll get there eventually. For now, removing trailing spaces like these is totally fine (thank you!), but I'd rather not reformat existing blocks of code until we agree on a style.

@@ -1,6 +1,6 @@
//
// Normalizer.swift
//
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's not. We'll get there eventually. For now, removing trailing spaces like these is totally fine (thank you!), but I'd rather not reformat existing blocks of code until we agree on a style.

case .Precompiled : return PrecompiledNormalizer(config: config)
case .StripAccents : return StripAccentsNormalizer(config: config)
default : fatalError("Unsupported Normalizer type: \(typeName)")
case .Prepend: return PrependNormalizer(config: config)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these spaces are fine to remove too

Copy link
Member

@pcuenca pcuenca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution @mzbac!

@pcuenca pcuenca merged commit a7a61a2 into huggingface:main Oct 2, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unable to load Flux.1's T5 tokenizer
2 participants