mirror of
https://github.com/jellyfin/jellyfin.git
synced 2026-01-23 23:20:51 +01:00
10.11.0 GetLatestMediaLegacy, GetItemsByUserIdLegacy 100x slower #7462
Labels
No labels
area:database
awaiting-feedback
backend
blocked
breaking change: web api
bug
build
ci
confirmed
discussion needed
dotnet future
downstream
duplicate
EFjellyfin.db
enhancement
feature
future
github-actions
good first issue
hdr
help wanted
invalid
investigation
librarydb
live-tv
lyrics
media playback
music
needs testing
nuget
performance
platform
pull-request
question
regression
release critical
requires-web
roadmap
security
security
stale
support
syncplay
ui & ux
upstream
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: starred/jellyfin#7462
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @monik3r on GitHub (Oct 21, 2025).
Description of the bug
I followed the steps to upgrade to 10.11.0 and everything seemed alright migration-wise, no errors. I have a large movies library of ~20k YouTube videos that I have re scanned a few times now.
When loading the main page it will take about 30-50 seconds to load recent additions, the library, etc on the page. When attempting to browse to the problem library it has not loaded after 50 minutes. Previously with 10.10.7 these would load in <500ms. It also pins a core now whenever someone attempts to load one of these pages.
To debug I have attempted to remove the problem library, upgrade from 10.10.7 to 10.11.0, then re-add a new library manually. However this doesn't help with the issue. In this library embedded image extractor, screen grabber, real time monitoring and Prefer embedded titles over filenames are enabled.
Reproduction steps
What is the current bug behavior?
Recent videos and other elements for the problem library are 100x slower to load, the library page never finishes loading to display the library,
What is the expected correct behavior?
Main page and library load as approximately fast as previous versions.
Jellyfin Server version
10.10.0+
Specify commit id
No response
Specify unstable release number
No response
Specify version number
No response
Specify the build version
10.11.0
Environment
Jellyfin logs
FFmpeg logs
Client / Browser logs
No response
Relevant screenshots or videos
No response
Additional information
No response
@kiawizard commented on GitHub (Oct 27, 2025):
I confirm this bug in 10.11.1 for libraries of type "other" (Mixed Movies and Shows) and "Home Videos and Photos". It takes 32 seconds to display a single level-1 subfolder in library. Ten level-2 subfolders won't ever show, will keep spinning and waiting for GetItemLegacy to finish.
Recreating library doesn't make it any better. There are only 200 videos in problematic library in 2 levels of subfolders. There are 130_000 BaseItems total in the database.
Have to revert to 10.10.7 for now
@kandykarter commented on GitHub (Oct 27, 2025):
I don't have any mixed or home video libraries, but this seems like it's probably the same issue I'm having with Collections #15090
@monik3r commented on GitHub (Oct 29, 2025):
Would it be possible for you to see what is happening under the hood via the logs to confirm if it is the same? When initially checking I used
docker logs jellyfin | grep "Executed DbCommand" -a5to get the timings and some context for each.To turn up logging you can use https://jellyfin.org/docs/general/administration/troubleshooting/#debug-logging
@kandykarter commented on GitHub (Oct 31, 2025):
Guess it was a false alarm, that command outputs nothing. Thanks!
@monik3r commented on GitHub (Oct 31, 2025):
If it doesn't output anything, then you aren't getting logs with details about DbCommand, not necessarily that it isn't the same issue. I also don't get the log line when the DbCommand doesn't finish.
@kandykarter commented on GitHub (Oct 31, 2025):
Fair enough! I updated my issue with some weird stuff I did see in the debug logs, but definitely nothing with DbCommand.
@pelluch commented on GitHub (Nov 11, 2025):
I posted a comment in https://github.com/jellyfin/jellyfin/issues/15090 where I attached a log and some context from a single collections call. It made about 1,391 queries (as counted by "Executed DbCommand").
@monik3r commented on GitHub (Nov 17, 2025):
Tested with a fresh install of 10.11.3 with just the single library and user, no improvement/change.
@monik3r commented on GitHub (Nov 17, 2025):
You seem to be having the same/a similar issue:
Note the 2244.3337ms. It is a bit faster than my library (mine doesn't finish after hours!), but that is still extremely slow WRT the 10.11 vs 10.10.
Sadly reinstalling 10.11.0 or 10.11 from scratch did not help here.
@pelluch commented on GitHub (Nov 25, 2025):
@monik3r While I intend on investigating the underlying issue, since this is a problem massively hindering a lot of people, I spent some time making a small plugin, Disable User Data, which is meant to serve as a mitigation in the meantime. The README in that repo explains what it does in much more detail, but in essence it allows one to disable user data on a few endpoints, which greatly speeds up some cases using
GetLatestMediaLegacyorGetItemsByUserIdLegacy(and several others).To test it, I created a collection with my entire library. Going to "Collections" without the plugin takes ~13s-18s to load, but with it, ~1s. You should hopefully see similar speed ups if your home section is what's causing the issue (just configure the plugin in its settings). I should note that while loading the general collections view was much faster, going into the giant collection itself didn't show visible improvement, mostly because Jellyfin tries to get the information about thousands of items all at once.
If you or anyone are curious to give it a try, I'd be very curious to hear what sort of improvements people are getting and from what - it would help steer optimization efforts, though the user data part seems an obvious place to try.
@kiawizard commented on GitHub (Nov 25, 2025):
@pelluch Thanks for your work! I have tested your plugin (all checkboxes checked in settings) and found that there is still an api call which takes enormous time:
/Users/bb233bd50fb94798b49763fff4c25640/Items?StartIndex=0&Limit=100&Fields=PrimaryImageAspectRatio%2CSortName%2CPath%2CChildCount%2CMediaSourceCount%2CPrimaryImageAspectRatio&ImageTypeLimit=1&ParentId=956b68e9adc2a5b1b856cbf288fdb1a4&SortBy=IsFolder%2CSortName&SortOrder=Ascending
Example reply:
This is called on entering a folder inside a library. Could you update the plugin to disable UserData for this api endpoint too?
@pelluch commented on GitHub (Nov 25, 2025):
@kiawizard Yeah, it should be pretty easy to add now that the framework is established - I'll get a quick update published during the day with the option to do so for general folders. It's not a guarantee that it will massively speed it up, but it certainly can't hurt to try, and from your JSON
UserDatais there so it's likely a factor anyway. I'll give an update here when that's done.@kiawizard commented on GitHub (Nov 26, 2025):
@pelluch Thanks a lot! Finally with version 0.2.1.1 of your Disable User Data plugin Jellyfin 10.11 started to work
@pelluch commented on GitHub (Nov 27, 2025):
@kiawizard I'm happy to hear! I included a more encompassing option that allows disabling for all /Items APIs, as there are many types of media (collections, photos, movies, shows, live TV, music etc). Did it make a noticeable change for you?
I am thinking about improving it by allowing users to directly select or type all of the places they want to apply it to, while keeping the "catch-all" option I added.
@kiawizard commented on GitHub (Nov 27, 2025):
Sure it did, libraries of type "Home Videos and Photos" which I use for all media without metadata from the internet started to open with this option. Before there was an endless waiting on entering such libraries and folders inside them
@kiawizard commented on GitHub (Dec 1, 2025):
Finally my issue with folders opening forever returned. Without any changes to the content, with pelluch's plugin running. I downloaded the source code, edited MediaBrowser.Controller\Entities\Folder.cs , turned off GetRecursiveChildCount:
then built release, copied MediaBrowser.Controller.dll into 10.11.4 folder, and it works again
@rushiiMachine commented on GitHub (Dec 5, 2025):
@pelluch Thank you very much for the workaround! I have a Music library containing 680 albums/8900 tracks, and this massively improved the speed for listing items, and searching! With my Raspi 3B+ already being so resource-constrained, searching previously completely froze Jellyfin for multiple minutes, but now it's down to a few seconds, with listing items down from over a minute to just 2.5s!
@cvium commented on GitHub (Dec 6, 2025):
Are the collections also slow when they only contain movies (no boxsets or other groupings)?
@kandykarter commented on GitHub (Dec 6, 2025):
@cvium In my case, yes. The Collections library on my server is movie-exclusive. However, there is an odd behaviour I've recently observed. Clicking into "Collections" from the main homepage takes ~10 seconds longer to load than it does if I click into it from the topbar inside the Movies library, even though no Collections contain any TV shows (and there's no other types of media on my server).
@monik3r commented on GitHub (Dec 6, 2025):
There are no boxsets or groupings that I have set up. The folder structure for my problem library is:
Where channel_id is the YouTube channel ID, and the video IDs are the YouTube video IDs.
@cvium commented on GitHub (Dec 7, 2025):
Which database locking strategy are you using?
@kandykarter commented on GitHub (Dec 7, 2025):
Optimistic here.
@cvium commented on GitHub (Dec 7, 2025):
How many of your collections are empty?
@kandykarter commented on GitHub (Dec 7, 2025):
None as far as I know.
@monik3r commented on GitHub (Dec 7, 2025):
Also no empty collections, I did not change or manually set the locking strategy.
@cvium commented on GitHub (Dec 7, 2025):
My 96 collections take about 800ms to load, which isn't fast, but not unreasonably slow. Removing the empty ones takes it down to 500-600 ms (different code path for empty ones -- seems unintentional). I think my DB isn't big enough to really reproduce it.
If you can (and want to, it's privacy concern for some), send me your DBs in a DM on Matrix (or if you have other suitable ways).
@kandykarter commented on GitHub (Dec 7, 2025):
I don't have matrix (and it doesn't look like I can do it through discord) and my db is around 500mb, not sure about an easy way to get it to you
edit: I'm going to send a link in a PM to you on the Jellyfin forums, let me know if that doesn't work for you
@FrankJeager commented on GitHub (Dec 13, 2025):
Thanks to the OP queries in the logs I've been profiling the GetLatestItems SQL generation and noticed a couple of things really killing performance on large libraries.
First, the query is slow because of the multiple one-to-many Include calls (Images, Trailers, Providers) happening. For a request of 80 items, the DB is returning thousands of rows that EF core has to deduplicate in memory. This spikes both network I/O and RAM usage unnecessarily.
One of the possible fix is in BaseItemRepository.cs. We just need to force split queries:
Another fix could be in TranslateQuery around the IsPlayed filter. Currently, it generates a correlated subquery in the WHERE clause (the COALESCE) which runs for every single row scanned.
Refactoring this to use .Any() allows EF core to generate a proper EXISTS or LEFT JOIN, which is much friendlier on the execution plan:
These two changes should drastically drop the execution time and memory footprint if i'm correct...
But I've not the time to test them. So if someone with time and good will would accept to test it I will be glad.