[fix] fix token healing tests and usage errors #33931
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
cc @gante! 🤗 |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
gante
approved these changes
Oct 7, 2024
There was a problem hiding this comment.
@alpertunga-bile thank you for pinning the issue, finding the cause and problems with the test setup, and opening a PR with the fix -- contributors like you make transformers shine 💛
LGTM 👍
Note for our future selves, in case we need more context about PR: check this comment
|
The red CI should get fixed after #33950 is merged. At that point, I'd like to kindly ask you to rebase and force-push, to re-run CI 🤗 |
|
@e can see token healing tests being run in the CI 👍 from https://app.circleci.com/pipelines/github/huggingface/transformers/106787/workflows/081efefd-ecd1-4eed-8496-b4d03d6e492e/jobs/1417470/parallel-runs/7?filterBy=ALL : 
|
alpertunga-bile
force-pushed
the
fix_token_healing
branch
from
e4dac42 to
47a04f7
Compare
|
cc @ArthurZucker as we just discussed it in person |
alpertunga-bile
force-pushed
the
fix_token_healing
branch
2 times, most recently
from
81ced1c to
641b200
Compare
|
@alpertunga-bile sorry about that :( #34067 -- this would be the PR |
alpertunga-bile
force-pushed
the
fix_token_healing
branch
from
ea06e24 to
4a476ba
Compare
Thanks, it is passed through the tests now. |
ArthurZucker
approved these changes
Oct 16, 2024
NielsRogge
pushed a commit
to NielsRogge/transformers
that referenced
this pull request
Oct 21, 2024
* auto-gptq requirement is removed & model is changed & tokenizer pad token is assigned * values func is changed with extensions & sequence key value bug is fixed * map key value check is added in ExtensionsTree * empty trimmed_ids bug is fixed * tail_id IndexError is fixed * empty trimmed_ids bug fix is updated for failed test * too much specific case for specific tokenizer is removed * input_ids check is updated * require auto-gptq import is removed * key error check is changed with empty list check * empty input_ids check is added * empty trimmed_ids fix is checked with numel function * usage change comments are added * test changes are commented * comment style and quality bugs are fixed * test comment style and quality bug is fixed
BernardZach
pushed a commit
to BernardZach/transformers
that referenced
this pull request
Dec 5, 2024
* auto-gptq requirement is removed & model is changed & tokenizer pad token is assigned * values func is changed with extensions & sequence key value bug is fixed * map key value check is added in ExtensionsTree * empty trimmed_ids bug is fixed * tail_id IndexError is fixed * empty trimmed_ids bug fix is updated for failed test * too much specific case for specific tokenizer is removed * input_ids check is updated * require auto-gptq import is removed * key error check is changed with empty list check * empty input_ids check is added * empty trimmed_ids fix is checked with numel function * usage change comments are added * test changes are commented * comment style and quality bugs are fixed * test comment style and quality bug is fixed
BernardZach
pushed a commit
to innovationcore/transformers
that referenced
this pull request
Dec 6, 2024
* auto-gptq requirement is removed & model is changed & tokenizer pad token is assigned * values func is changed with extensions & sequence key value bug is fixed * map key value check is added in ExtensionsTree * empty trimmed_ids bug is fixed * tail_id IndexError is fixed * empty trimmed_ids bug fix is updated for failed test * too much specific case for specific tokenizer is removed * input_ids check is updated * require auto-gptq import is removed * key error check is changed with empty list check * empty input_ids check is added * empty trimmed_ids fix is checked with numel function * usage change comments are added * test changes are commented * comment style and quality bugs are fixed * test comment style and quality bug is fixed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Hello! As I described in the #33827; token healing functionality is failing in usage and tests.
This PR is fixing the token healing tests and usage bugs.
Fixes #33827
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@gante