mirror of
https://github.com/jellyfin/jellyfin.git
synced 2026-01-23 23:20:51 +01:00
Transliteration does not work with Japanese characters #5544
Labels
No labels
area:database
awaiting-feedback
backend
blocked
breaking change: web api
bug
build
ci
confirmed
discussion needed
dotnet future
downstream
duplicate
EFjellyfin.db
enhancement
feature
future
github-actions
good first issue
hdr
help wanted
invalid
investigation
librarydb
live-tv
lyrics
media playback
music
needs testing
nuget
performance
platform
pull-request
question
regression
release critical
requires-web
roadmap
security
security
stale
support
syncplay
ui & ux
upstream
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: starred/jellyfin#5544
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @scampower3 on GitHub (Mar 25, 2024).
Please describe your bug
Cannot access the any library if one of the card's title has japanese characters in it. [includes japanese punctuation].
Reproduction Steps
or
Jellyfin Version
Unstable (master branch)
if other:
No response
Environment
Jellyfin logs
FFmpeg logs
Please attach any browser or client logs here
n/a
Please attach any screenshots here
No response
Code of Conduct
@jellyfin-bot commented on GitHub (Mar 25, 2024):
Hi, it seems like your issue report has the following item(s) that need to be addressed:
This is an automated message, currently under testing. Please file an issue here if you encounter any problems.
@crobibero commented on GitHub (Mar 25, 2024):
https://github.com/dotnet/runtime/blob/main/docs/design/features/globalization-invariant-mode.md
So either we need to disable globalization invariant mode or find a different way to transliterate
@gnattu commented on GitHub (Mar 25, 2024):
Sorry, but I'm not able to reproduce this issue on my local environment with master.
Mind share what title are you using for the library?
For pure Katakana it returns correct results:
For mixes of hiragana and kanji, kanji characters are transliterated as Chinese characters, which is a known issue with ICU. It needs to be fixed by a plugin because a simple transliterator is not sufficient, and it will require an actual tokenizer with a dictionary to work, which is too heavy to be included in the server.
But either way, it should not throw exceptions like you posted.
@scampower3 commented on GitHub (Mar 25, 2024):
The titles are アイドル and Pablo Honey (Special Collector’s Edition). Also, I tested both titles when running master on a windows machine and they worked fine. But they break when I run them in a docker on a linux machine.
@gnattu commented on GitHub (Mar 25, 2024):
Can't reproduce on my side. Probably the docker image does not have required ICU libs. Need investigation and to add a fallback path.
@crobibero commented on GitHub (Mar 25, 2024):
Yes, this is what I commented
@gnattu commented on GitHub (Mar 25, 2024):
Can you try this branch to see if it fixed your issue?
https://github.com/jellyfin/jellyfin/tree/jpn-transliteration
The ICU4N's Transliterator should use its own metadata and not using the system ICU data per docs, so this seems to be an initialization issue. I've noticed that the default culture is set to something like
en-USwhich may confuse the Transliterator's initializer.@b-rad15 commented on GitHub (Mar 25, 2024):
Having the same issue so I'd be able to test. What Dockerfile does the official docker image use to build? Just to make sure I match the environment. The site says there should be a Dockerfile in the main repo but I don't see it
@felix920506 commented on GitHub (Mar 25, 2024):
@b-rad15 https://github.com/jellyfin/jellyfin-packaging
@b-rad15 commented on GitHub (Mar 26, 2024):
I'm like 99% I built it right (followed the steps there, on debian arm64, just adding a cd into jellyfin-server and a
git checkout jpn-transliterationafter doing the./checkout.py masterthen later after being not sure it was changing I edited an error string to make sure) and am still getting the same error@gnattu commented on GitHub (Mar 26, 2024):
Confirmed as a packaging issue. This only happens to our docker image and you should be fine with the deb installation. Fixing incoming.
@felix920506 commented on GitHub (Mar 26, 2024):
Should be transferred to packaging repo
@caesarm commented on GitHub (Apr 8, 2024):
I'm still having this issue but the exception is a slightly different error now. I am still unable to directly access my libraries.
Build Version 2024040805
@gnattu commented on GitHub (Apr 9, 2024):
This really should not happen now and only providing this log is not helpful. We need to know the environment information.
@caesarm commented on GitHub (Apr 9, 2024):