dt-bindings: gpu: mali-valhall-csf: Add support for Arm Mali CSF GPUs
Arm has introduced a new v10 GPU architecture that replaces the Job Manager interface with a new Command Stream Frontend. It adds firmware driven command stream queues that can be used by kernel and user space to submit jobs to the GPU. Add the initial schema for the device tree that is based on support for RK3588 SoC. The minimum number of clocks is one for the IP, but on Rockchip platforms they will tend to expose the semi-independent clocks for better power management. v5: - Move the opp-table node under the gpu node v4: - Fix formatting issue v3: - Cleanup commit message to remove redundant text - Added opp-table property and re-ordered entries - Clarified power-domains and power-domain-names requirements for RK3588. - Cleaned up example Note: power-domains and power-domain-names requirements for other platforms are still work in progress, hence the bindings are left incomplete here. v2: - New commit Signed-off-by: Liviu Dudau <liviu.dudau@arm.com> Cc: Krzysztof Kozlowski <krzysztof.kozlowski+dt@linaro.org> Cc: Rob Herring <robh+dt@kernel.org> Cc: Conor Dooley <conor+dt@kernel.org> Cc: devicetree@vger.kernel.org Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Rob Herring <robh@kernel.org> drm: execution context for GEM buffers v7 This adds the infrastructure for an execution context for GEM buffers which is similar to the existing TTMs execbuf util and intended to replace it in the long term. The basic functionality is that we abstracts the necessary loop to lock many different GEM buffers with automated deadlock and duplicate handling. v2: drop xarray and use dynamic resized array instead, the locking overhead is unnecessary and measurable. v3: drop duplicate tracking, radeon is really the only one needing that. v4: fixes issues pointed out by Danilo, some typos in comments and a helper for lock arrays of GEM objects. v5: some suggestions by Boris Brezillon, especially just use one retry macro, drop loop in prepare_array, use flags instead of bool v6: minor changes suggested by Thomas, Boris and Danilo v7: minor typos pointed out by checkpatch.pl fixed Signed-off-by: Christian König <christian.koenig@amd.com> Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Danilo Krummrich <dakr@redhat.com> Tested-by: Danilo Krummrich <dakr@redhat.com> Acked-by: Alex Deucher <alexander.deucher@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20230711133122.3710-2-christian.koenig@amd.com drm: manager to keep track of GPUs VA mappings Add infrastructure to keep track of GPU virtual address (VA) mappings with a decicated VA space manager implementation. New UAPIs, motivated by Vulkan sparse memory bindings graphics drivers start implementing, allow userspace applications to request multiple and arbitrary GPU VA mappings of buffer objects. The DRM GPU VA manager is intended to serve the following purposes in this context. 1) Provide infrastructure to track GPU VA allocations and mappings, using an interval tree (RB-tree). 2) Generically connect GPU VA mappings to their backing buffers, in particular DRM GEM objects. 3) Provide a common implementation to perform more complex mapping operations on the GPU VA space. In particular splitting and merging of GPU VA mappings, e.g. for intersecting mapping requests or partial unmap requests. Acked-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Acked-by: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com> Tested-by: Matthew Brost <matthew.brost@intel.com> Tested-by: Donald Robson <donald.robson@imgtec.com> Suggested-by: Dave Airlie <airlied@redhat.com> Signed-off-by: Danilo Krummrich <dakr@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20230720001443.2380-2-dakr@redhat.com drm: manager: Fix printk format for size_t sizeof() returns a size_t which may be different to an unsigned long. Use the correct format specifier of '%zu' to prevent compiler warnings. Fixes: e6303f323b1a ("drm: manager to keep track of GPUs VA mappings") Reviewed-by: Danilo Krummrich <dakr@redhat.com> Signed-off-by: Steven Price <steven.price@arm.com> Signed-off-by: Danilo Krummrich <dakr@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/2bf64010-c40a-8b84-144c-5387412b579e@arm.com drm/gpuva_mgr: remove unused prev pointer in __drm_gpuva_sm_map() The prev pointer in __drm_gpuva_sm_map() was used to implement automatic merging of mappings. Since automatic merging did not make its way upstream, remove this leftover. Fixes: e6303f323b1a ("drm: manager to keep track of GPUs VA mappings") Signed-off-by: Danilo Krummrich <dakr@redhat.com> Reviewed-by: Dave Airlie <airlied@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20230823233119.2891-1-dakr@redhat.com drm/gpuvm: rename struct drm_gpuva_manager to struct drm_gpuvm Rename struct drm_gpuva_manager to struct drm_gpuvm including corresponding functions. This way the GPUVA manager's structures align very well with the documentation of VM_BIND [1] and VM_BIND locking [2]. It also provides a better foundation for the naming of data structures and functions introduced for implementing a common dma-resv per GPU-VM including tracking of external and evicted objects in subsequent patches. [1] Documentation/gpu/drm-vm-bind-async.rst [2] Documentation/gpu/drm-vm-bind-locking.rst Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Acked-by: Dave Airlie <airlied@redhat.com> Acked-by: Christian König <christian.koenig@amd.com> Signed-off-by: Danilo Krummrich <dakr@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20230920144343.64830-2-dakr@redhat.com drm/gpuvm: allow building as module HB: drivers/gpu/drm/nouveau/Kconfig skipped because there is no gpuvm support of nouveau in 6.1 Currently, the DRM GPUVM does not have any core dependencies preventing a module build. Also, new features from subsequent patches require helpers (namely drm_exec) which can be built as module. Reviewed-by: Christian König <christian.koenig@amd.com> Reviewed-by: Dave Airlie <airlied@redhat.com> Signed-off-by: Danilo Krummrich <dakr@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20230920144343.64830-3-dakr@redhat.com drm/gpuvm: convert WARN() to drm_WARN() variants HB: drivers/gpu/drm/nouveau/nouveau_uvmm.c skipped since 6.1 does not support gpuvm on nv Use drm_WARN() and drm_WARN_ON() variants to indicate drivers the context the failing VM resides in. Acked-by: Christian König <christian.koenig@amd.com> Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Signed-off-by: Danilo Krummrich <dakr@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231108001259.15123-2-dakr@redhat.com drm/gpuvm: don't always WARN in drm_gpuvm_check_overflow() Don't always WARN in drm_gpuvm_check_overflow() and separate it into a drm_gpuvm_check_overflow() and a dedicated drm_gpuvm_warn_check_overflow() variant. This avoids printing warnings due to invalid userspace requests. Acked-by: Christian König <christian.koenig@amd.com> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com> Signed-off-by: Danilo Krummrich <dakr@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231108001259.15123-3-dakr@redhat.com drm/gpuvm: export drm_gpuvm_range_valid() Drivers may use this function to validate userspace requests in advance, hence export it. Acked-by: Christian König <christian.koenig@amd.com> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com> Signed-off-by: Danilo Krummrich <dakr@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231108001259.15123-4-dakr@redhat.com drm/gpuvm: add common dma-resv per struct drm_gpuvm hb: drivers/gpu/drm/nouveau/nouveau_uvmm.c skipped Provide a common dma-resv for GEM objects not being used outside of this GPU-VM. This is used in a subsequent patch to generalize dma-resv, external and evicted object handling and GEM validation. Acked-by: Christian König <christian.koenig@amd.com> Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Signed-off-by: Danilo Krummrich <dakr@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231108001259.15123-6-dakr@redhat.com drm/gpuvm: add drm_gpuvm_flags to drm_gpuvm HB: drivers/gpu/drm/nouveau/nouveau_uvmm.c skipped Introduce flags for struct drm_gpuvm, this required by subsequent commits. Acked-by: Christian König <christian.koenig@amd.com> Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Signed-off-by: Danilo Krummrich <dakr@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231108001259.15123-8-dakr@redhat.com drm/gpuvm: reference count drm_gpuvm structures HB: drivers/gpu/drm/nouveau/nouveau_uvmm.c skipped Implement reference counting for struct drm_gpuvm. Acked-by: Christian König <christian.koenig@amd.com> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com> Signed-off-by: Danilo Krummrich <dakr@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231108001259.15123-10-dakr@redhat.com drm/gpuvm: add an abstraction for a VM / BO combination HB: drivers/gpu/drm/nouveau/nouveau_uvmm.c skipped Add an abstraction layer between the drm_gpuva mappings of a particular drm_gem_object and this GEM object itself. The abstraction represents a combination of a drm_gem_object and drm_gpuvm. The drm_gem_object holds a list of drm_gpuvm_bo structures (the structure representing this abstraction), while each drm_gpuvm_bo contains list of mappings of this GEM object. This has multiple advantages: 1) We can use the drm_gpuvm_bo structure to attach it to various lists of the drm_gpuvm. This is useful for tracking external and evicted objects per VM, which is introduced in subsequent patches. 2) Finding mappings of a certain drm_gem_object mapped in a certain drm_gpuvm becomes much cheaper. 3) Drivers can derive and extend the structure to easily represent driver specific states of a BO for a certain GPUVM. The idea of this abstraction was taken from amdgpu, hence the credit for this idea goes to the developers of amdgpu. Cc: Christian König <christian.koenig@amd.com> Acked-by: Christian König <christian.koenig@amd.com> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com> Signed-off-by: Danilo Krummrich <dakr@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231108001259.15123-11-dakr@redhat.com drm/gpuvm: track/lock/validate external/evicted objects Currently the DRM GPUVM offers common infrastructure to track GPU VA allocations and mappings, generically connect GPU VA mappings to their backing buffers and perform more complex mapping operations on the GPU VA space. However, there are more design patterns commonly used by drivers, which can potentially be generalized in order to make the DRM GPUVM represent a basis for GPU-VM implementations. In this context, this patch aims at generalizing the following elements. 1) Provide a common dma-resv for GEM objects not being used outside of this GPU-VM. 2) Provide tracking of external GEM objects (GEM objects which are shared with other GPU-VMs). 3) Provide functions to efficiently lock all GEM objects dma-resv the GPU-VM contains mappings of. 4) Provide tracking of evicted GEM objects the GPU-VM contains mappings of, such that validation of evicted GEM objects is accelerated. 5) Provide some convinience functions for common patterns. Big thanks to Boris Brezillon for his help to figure out locking for drivers updating the GPU VA space within the fence signalling path. Acked-by: Christian König <christian.koenig@amd.com> Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Suggested-by: Matthew Brost <matthew.brost@intel.com> Signed-off-by: Danilo Krummrich <dakr@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231108001259.15123-12-dakr@redhat.com drm/gpuvm: fall back to drm_exec_lock_obj() Fall back to drm_exec_lock_obj() if num_fences is zero for the drm_gpuvm_prepare_* function family. Otherwise dma_resv_reserve_fences() would actually allocate slots even though num_fences is zero. Cc: Christian König <christian.koenig@amd.com> Acked-by: Donald Robson <donald.robson@imgtec.com> Signed-off-by: Danilo Krummrich <dakr@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231129220835.297885-2-dakr@redhat.com drm/gpuvm: Let drm_gpuvm_bo_put() report when the vm_bo object is destroyed Some users need to release resources attached to the vm_bo object when it's destroyed. In Panthor's case, we need to release the pin ref so BO pages can be returned to the system when all GPU mappings are gone. This could be done through a custom drm_gpuvm::vm_bo_free() hook, but this has all sort of locking implications that would force us to expose a drm_gem_shmem_unpin_locked() helper, not to mention the fact that having a ::vm_bo_free() implementation without a ::vm_bo_alloc() one seems odd. So let's keep things simple, and extend drm_gpuvm_bo_put() to report when the object is destroyed. Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Danilo Krummrich <dakr@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231204151406.1977285-1-boris.brezillon@collabora.com drm/exec: Pass in initial # of objects HB: skipped drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c drivers/gpu/drm/amd/amdkfd/kfd_svm.c drivers/gpu/drm/imagination/pvr_job.c drivers/gpu/drm/nouveau/nouveau_uvmm.c In cases where the # is known ahead of time, it is silly to do the table resize dance. Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Christian König <christian.koenig@amd.com> Patchwork: https://patchwork.freedesktop.org/patch/568338/ drm/gem-shmem: When drm_gem_object_init failed, should release object when goto err_free, the object had init, so it should be release when fail. Signed-off-by: ChunyouTang <tangchunyou@163.com> Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de> Link: https://patchwork.freedesktop.org/patch/msgid/20221119064131.364-1-tangchunyou@163.com drm: Remove usage of deprecated DRM_DEBUG_PRIME drm_print.h says DRM_DEBUG_PRIME is deprecated in favor of drm_dbg_prime(). Signed-off-by: Siddh Raman Pant <code@siddh.me> Reviewed-by: Simon Ser <contact@emersion.fr> Signed-off-by: Simon Ser <contact@emersion.fr> Link: https://patchwork.freedesktop.org/patch/msgid/cd663b1bc42189e55898cddecdb3b73c591b341a.1673269059.git.code@siddh.me drm/shmem: Cleanup drm_gem_shmem_create_with_handle() Once we create the handle, the handle owns the reference. Currently nothing was doing anything with the shmem ptr after the handle was created, but let's change drm_gem_shmem_create_with_handle() to not return the pointer, so-as to not encourage problematic use of this function in the future. As a bonus, it makes the code a bit cleaner. Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Steven Price <steven.price@arm.com> Signed-off-by: Steven Price <steven.price@arm.com> Link: https://patchwork.freedesktop.org/patch/msgid/20230123154831.3191821-1-robdclark@gmail.com drm/shmem-helper: Fix locking for drm_gem_shmem_get_pages_sgt() Other functions touching shmem->sgt take the pages lock, so do that here too. drm_gem_shmem_get_pages() & co take the same lock, so move to the _locked() variants to avoid recursive locking. Discovered while auditing locking to write the Rust abstractions. Fixes:2194a63a81("drm: Add library for shmem backed GEM objects") Fixes:4fa3d66f13("drm/shmem: Do dma_unmap_sg before purging pages") Signed-off-by: Asahi Lina <lina@asahilina.net> Reviewed-by: Javier Martinez Canillas <javierm@redhat.com> Signed-off-by: Javier Martinez Canillas <javierm@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20230205125124.2260-1-lina@asahilina.net drm/shmem-helper: Switch to use drm_* debug helpers Ease debugging of a multi-GPU system by using drm_WARN_*() and drm_dbg_kms() helpers that print out DRM device name corresponding to shmem GEM. Reviewed-by: Thomas Zimmermann <tzimmermann@suse.de> Suggested-by: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Link: https://lore.kernel.org/all/20230108210445.3948344-6-dmitry.osipenko@collabora.com/ drm/shmem-helper: Don't use vmap_use_count for dma-bufs DMA-buf core has its own refcounting of vmaps, use it instead of drm-shmem counting. This change prepares drm-shmem for addition of memory shrinker support where drm-shmem will use a single dma-buf reservation lock for all operations performed over dma-bufs. Reviewed-by: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Link: https://lore.kernel.org/all/20230108210445.3948344-7-dmitry.osipenko@collabora.com/ drm/shmem-helper: Switch to reservation lock Replace all drm-shmem locks with a GEM reservation lock. This makes locks consistent with dma-buf locking convention where importers are responsible for holding reservation lock for all operations performed over dma-bufs, preventing deadlock between dma-buf importers and exporters. Suggested-by: Daniel Vetter <daniel@ffwll.ch> Acked-by: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Link: https://lore.kernel.org/all/20230108210445.3948344-8-dmitry.osipenko@collabora.com/ drm/shmem-helper: Revert accidental non-GPL export The referenced commit added a wrapper for drm_gem_shmem_get_pages_sgt(), but in the process it accidentally changed the export type from GPL to non-GPL. Switch it back to GPL. Reported-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Fixes: ddddedaa0db9 ("drm/shmem-helper: Fix locking for drm_gem_shmem_get_pages_sgt()") Signed-off-by: Asahi Lina <lina@asahilina.net> Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de> Link: https://patchwork.freedesktop.org/patch/msgid/20230227-shmem-export-fix-v1-1-8880b2c25e81@asahilina.net Revert "drm/shmem-helper: Switch to reservation lock" This reverts commit 67b7836d4458790f1261e31fe0ce3250989784f0. The locking appears incomplete. A caller of SHMEM helper's pin function never acquires the dma-buf reservation lock. So we get WARNING: CPU: 3 PID: 967 at drivers/gpu/drm/drm_gem_shmem_helper.c:243 drm_gem_shmem_pin+0x42/0x90 [drm_shmem_helper] Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de> Acked-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Link: https://patchwork.freedesktop.org/patch/msgid/20230228152612.19971-1-tzimmermann@suse.de drm/shmem-helper: Switch to reservation lock Replace all drm-shmem locks with a GEM reservation lock. This makes locks consistent with dma-buf locking convention where importers are responsible for holding reservation lock for all operations performed over dma-bufs, preventing deadlock between dma-buf importers and exporters. Suggested-by: Daniel Vetter <daniel@ffwll.ch> Acked-by: Thomas Zimmermann <tzimmermann@suse.de> Reviewed-by: Emil Velikov <emil.l.velikov@gmail.com> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Link: https://patchwork.freedesktop.org/patch/msgid/20230529223935.2672495-7-dmitry.osipenko@collabora.com drm/shmem-helper: Reset vma->vm_ops before calling dma_buf_mmap() The dma-buf backend is supposed to provide its own vm_ops, but some implementation just have nothing special to do and leave vm_ops untouched, probably expecting this field to be zero initialized (this is the case with the system_heap implementation for instance). Let's reset vma->vm_ops to NULL to keep things working with these implementations. Fixes:26d3ac3cb0("drm/shmem-helpers: Redirect mmap for imported dma-buf") Cc: <stable@vger.kernel.org> Cc: Daniel Vetter <daniel.vetter@ffwll.ch> Reported-by: Roman Stratiienko <r.stratiienko@gmail.com> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Tested-by: Roman Stratiienko <r.stratiienko@gmail.com> Reviewed-by: Thomas Zimmermann <tzimmermann@suse.de> Link: https://patchwork.freedesktop.org/patch/msgid/20230724112610.60974-1-boris.brezillon@collabora.com iommu: Allow passing custom allocators to pgtable drivers This will be useful for GPU drivers who want to keep page tables in a pool so they can: - keep freed page tables in a free pool and speed-up upcoming page table allocations - batch page table allocation instead of allocating one page at a time - pre-reserve pages for page tables needed for map/unmap operations, to ensure map/unmap operations don't try to allocate memory in paths they're allowed to block or fail It might also be valuable for other aspects of GPU and similar use-cases, like fine-grained memory accounting and resource limiting. We will extend the Arm LPAE format to support custom allocators in a separate commit. Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Steven Price <steven.price@arm.com> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/20231124142434.1577550-2-boris.brezillon@collabora.com Signed-off-by: Joerg Roedel <jroedel@suse.de> iommu: Extend LPAE page table format to support custom allocators We need that in order to implement the VM_BIND ioctl in the GPU driver targeting new Mali GPUs. VM_BIND is about executing MMU map/unmap requests asynchronously, possibly after waiting for external dependencies encoded as dma_fences. We intend to use the drm_sched framework to automate the dependency tracking and VM job dequeuing logic, but this comes with its own set of constraints, one of them being the fact we are not allowed to allocate memory in the drm_gpu_scheduler_ops::run_job() to avoid this sort of deadlocks: - VM_BIND map job needs to allocate a page table to map some memory to the VM. No memory available, so kswapd is kicked - GPU driver shrinker backend ends up waiting on the fence attached to the VM map job or any other job fence depending on this VM operation. With custom allocators, we will be able to pre-reserve enough pages to guarantee the map/unmap operations we queued will take place without going through the system allocator. But we can also optimize allocation/reservation by not free-ing pages immediately, so any upcoming page table allocation requests can be serviced by some free page table pool kept at the driver level. I might also be valuable for other aspects of GPU and similar use-cases, like fine-grained memory accounting and resource limiting. Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Steven Price <steven.price@arm.com> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/20231124142434.1577550-3-boris.brezillon@collabora.com Signed-off-by: Joerg Roedel <jroedel@suse.de> drm/sched: Add FIFO sched policy to run queue When many entities are competing for the same run queue on the same scheduler, we observe an unusually long wait times and some jobs get starved. This has been observed on GPUVis. The issue is due to the Round Robin policy used by schedulers to pick up the next entity's job queue for execution. Under stress of many entities and long job queues within entity some jobs could be stuck for very long time in it's entity's queue before being popped from the queue and executed while for other entities with smaller job queues a job might execute earlier even though that job arrived later then the job in the long queue. Fix: Add FIFO selection policy to entities in run queue, chose next entity on run queue in such order that if job on one entity arrived earlier then job on another entity the first job will start executing earlier regardless of the length of the entity's job queue. v2: Switch to rb tree structure for entities based on TS of oldest job waiting in the job queue of an entity. Improves next entity extraction to O(1). Entity TS update O(log N) where N is the number of entities in the run-queue Drop default option in module control parameter. v3: Various cosmetical fixes and minor refactoring of fifo update function. (Luben) v4: Switch drm_sched_rq_select_entity_fifo to in order search (Luben) v5: Fix up drm_sched_rq_select_entity_fifo loop (Luben) v6: Add missing drm_sched_rq_remove_fifo_locked v7: Fix ts sampling bug and more cosmetic stuff (Luben) v8: Fix module parameter string (Luben) Cc: Luben Tuikov <luben.tuikov@amd.com> Cc: Christian König <christian.koenig@amd.com> Cc: Direct Rendering Infrastructure - Development <dri-devel@lists.freedesktop.org> Cc: AMD Graphics <amd-gfx@lists.freedesktop.org> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com> Tested-by: Yunxiang Li (Teddy) <Yunxiang.Li@amd.com> Signed-off-by: Luben Tuikov <luben.tuikov@amd.com> Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20220930041258.1050247-1-luben.tuikov@amd.com drm/scheduler: Set the FIFO scheduling policy as the default The currently default Round-Robin GPU scheduling can result in starvation of entities which have a large number of jobs, over entities which have a very small number of jobs (single digit). This can be illustrated in the following diagram, where jobs are alphabetized to show their chronological order of arrival, where job A is the oldest, B is the second oldest, and so on, to J, the most recent job to arrive. ---> entities j | H-F-----A--E--I-- o | --G-----B-----J-- b | --------C-------- s\/ --------D-------- WLOG, assuming all jobs are "ready", then a R-R scheduling will execute them in the following order (a slice off of the top of the entities' list), H, F, A, E, I, G, B, J, C, D. However, to mitigate job starvation, we'd rather execute C and D before E, and so on, given, of course, that they're all ready to be executed. So, if all jobs are ready at this instant, the order of execution for this and the next 9 instances of picking the next job to execute, should really be, A, B, C, D, E, F, G, H, I, J, which is their chronological order. The only reason for this order to be broken, is if an older job is not yet ready, but a younger job is ready, at an instant of picking a new job to execute. For instance if job C wasn't ready at time 2, but job D was ready, then we'd pick job D, like this: 0 +1 +2 ... A, B, D, ... And from then on, C would be preferred before all other jobs, if it is ready at the time when a new job for execution is picked. So, if C became ready two steps later, the execution order would look like this: ......0 +1 +2 ... A, B, D, E, C, F, G, H, I, J This is what the FIFO GPU scheduling algorithm achieves. It uses a Red-Black tree to keep jobs sorted in chronological order, where picking the oldest job is O(1) (we use the "cached" structure), and balancing the tree is O(log n). IOW, it picks the *oldest ready* job to execute now. The implementation is already in the kernel, and this commit only changes the default GPU scheduling algorithm to use. This was tested and achieves about 1% faster performance over the Round Robin algorithm. Cc: Christian König <christian.koenig@amd.com> Cc: Alex Deucher <Alexander.Deucher@amd.com> Cc: Direct Rendering Infrastructure - Development <dri-devel@lists.freedesktop.org> Signed-off-by: Luben Tuikov <luben.tuikov@amd.com> Reviewed-by: Christian König <christian.koenig@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20221024212634.27230-1-luben.tuikov@amd.com Signed-off-by: Christian König <christian.koenig@amd.com> drm/scheduler: add drm_sched_job_add_resv_dependencies Add a new function to update job dependencies from a resv obj. Signed-off-by: Christian König <christian.koenig@amd.com> Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20221014084641.128280-3-christian.koenig@amd.com drm/scheduler: remove drm_sched_dependency_optimized Not used any more. Signed-off-by: Christian König <christian.koenig@amd.com> Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20221014084641.128280-12-christian.koenig@amd.com drm/scheduler: rework entity flush, kill and fini This was buggy because when we had to wait for entities which were killed as well we would just deadlock. Instead move all the dependency handling into the callbacks so that will all happen asynchronously. Signed-off-by: Christian König <christian.koenig@amd.com> Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20221014084641.128280-13-christian.koenig@amd.com drm/scheduler: rename dependency callback into prepare_job This now matches much better what this is doing. Signed-off-by: Christian König <christian.koenig@amd.com> Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20221014084641.128280-14-christian.koenig@amd.com drm/amdgpu: revert "implement tdr advanced mode" This reverts commite6c6338f39. This feature basically re-submits one job after another to figure out which one was the one causing a hang. This is obviously incompatible with gang-submit which requires that multiple jobs run at the same time. It's also absolutely not helpful to crash the hardware multiple times if a clean recovery is desired. For testing and debugging environments we should rather disable recovery alltogether to be able to inspect the state with a hw debugger. Additional to that the sw implementation is clearly buggy and causes reference count issues for the hardware fence. Signed-off-by: Christian König <christian.koenig@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com> Signed-off-by: Alex Deucher <alexander.deucher@amd.com> drm/scheduler: Fix lockup in drm_sched_entity_kill() The drm_sched_entity_kill() is invoked twice by drm_sched_entity_destroy() while userspace process is exiting or being killed. First time it's invoked when sched entity is flushed and second time when entity is released. This causes a lockup within wait_for_completion(entity_idle) due to how completion API works. Calling wait_for_completion() more times than complete() was invoked is a error condition that causes lockup because completion internally uses counter for complete/wait calls. The complete_all() must be used instead in such cases. This patch fixes lockup of Panfrost driver that is reproducible by killing any application in a middle of 3d drawing operation. Fixes: 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini") Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Reviewed-by: Christian König <christian.koenig@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20221123001303.533968-1-dmitry.osipenko@collabora.com drm/scheduler: deprecate drm_sched_resubmit_jobs This interface is not working as it should. Signed-off-by: Christian König <christian.koenig@amd.com> Reviewed-by: Alex Deucher <alexander.deucher@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20221109095010.141189-5-christian.koenig@amd.com drm/scheduler: track GPU active time per entity Track the accumulated time that jobs from this entity were active on the GPU. This allows drivers using the scheduler to trivially implement the DRM fdinfo when the hardware doesn't provide more specific information than signalling job completion anyways. [Bagas: Append missing colon to @elapsed_ns] Signed-off-by: Bagas Sanjaya <bagasdotme@gmail.com> Signed-off-by: Lucas Stach <l.stach@pengutronix.de> Reviewed-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com> drm/sched: Create wrapper to add a syncobj dependency to job In order to add a syncobj's fence as a dependency to a job, it is necessary to call drm_syncobj_find_fence() to find the fence and then add the dependency with drm_sched_job_add_dependency(). So, wrap these steps in one single function, drm_sched_job_add_syncobj_dependency(). Reviewed-by: Christian König <christian.koenig@amd.com> Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Signed-off-by: Maíra Canal <mcanal@igalia.com> Signed-off-by: Maíra Canal <mairacanal@riseup.net> Link: https://patchwork.freedesktop.org/patch/msgid/20230209124447.467867-2-mcanal@igalia.com drm/scheduler: Fix variable name in function description Compiling AMD GPU drivers displays two warnings: drivers/gpu/drm/scheduler/sched_main.c:738: warning: Function parameter or member 'file' not described in 'drm_sched_job_add_syncobj_dependency' drivers/gpu/drm/scheduler/sched_main.c:738: warning: Excess function parameter 'file_private' description in 'drm_sched_job_add_syncobj_dependency' Get rid of them by renaming the variable name on the function description Signed-off-by: Caio Novais <caionovais@usp.br> Link: https://lore.kernel.org/r/20230325131532.6356-1-caionovais@usp.br Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Signed-off-by: Luben Tuikov <luben.tuikov@amd.com> drm/scheduler: Add fence deadline support As the finished fence is the one that is exposed to userspace, and therefore the one that other operations, like atomic update, would block on, we need to propagate the deadline from from the finished fence to the actual hw fence. v2: Split into drm_sched_fence_set_parent() (ckoenig) v3: Ensure a thread calling drm_sched_fence_set_deadline_finished() sees fence->parent set before drm_sched_fence_set_parent() does this test_bit(DMA_FENCE_FLAG_HAS_DEADLINE_BIT). Signed-off-by: Rob Clark <robdclark@chromium.org> Acked-by: Luben Tuikov <luben.tuikov@amd.com> Revert "drm/scheduler: track GPU active time per entity" This reverts commit df622729ddbf as it introduces a use-after-free, which isn't easy to fix without going back to the design drawing board. Reported-by: Danilo Krummrich <dakr@redhat.com> Signed-off-by: Lucas Stach <l.stach@pengutronix.de> drm/scheduler: Fix UAF race in drm_sched_entity_push_job() After a job is pushed into the queue, it is owned by the scheduler core and may be freed at any time, so we can't write nor read the submit timestamp after that point. Fixes oopses observed with the drm/asahi driver, found with kASAN. Signed-off-by: Asahi Lina <lina@asahilina.net> Link: https://lore.kernel.org/r/20230406-scheduler-uaf-2-v1-1-972531cf0a81@asahilina.net Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Signed-off-by: Luben Tuikov <luben.tuikov@amd.com> drm/sched: Check scheduler ready before calling timeout handling During an IGT GPU reset test we see the following oops, [ +0.000003] ------------[ cut here ]------------ [ +0.000000] WARNING: CPU: 9 PID: 0 at kernel/workqueue.c:1656 __queue_delayed_work+0x6d/0xa0 [ +0.000004] Modules linked in: iptable_filter bpfilter amdgpu(OE) nls_iso8859_1 snd_hda_codec_realtek snd_hda_codec_generic intel_rapl_msr ledtrig_audio snd_hda_codec_hdmi intel_rapl_common snd_hda_intel edac_mce_amd snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_hda_core iommu_v2 gpu_sched(OE) kvm_amd drm_buddy snd_hwdep kvm video drm_ttm_helper snd_pcm ttm snd_seq_midi drm_display_helper snd_seq_midi_event snd_rawmidi cec crct10dif_pclmul ghash_clmulni_intel sha512_ssse3 snd_seq aesni_intel rc_core crypto_simd cryptd binfmt_misc drm_kms_helper rapl snd_seq_device input_leds joydev snd_timer i2c_algo_bit syscopyarea snd ccp sysfillrect sysimgblt wmi_bmof k10temp soundcore mac_hid sch_fq_codel msr parport_pc ppdev drm lp parport ramoops reed_solomon pstore_blk pstore_zone efi_pstore ip_tables x_tables autofs4 hid_generic usbhid hid r8169 ahci xhci_pci gpio_amdpt realtek i2c_piix4 wmi crc32_pclmul xhci_pci_renesas libahci gpio_generic [ +0.000070] CPU: 9 PID: 0 Comm: swapper/9 Tainted: G W OE 6.1.11+ #2 [ +0.000003] Hardware name: Gigabyte Technology Co., Ltd. AB350-Gaming 3/AB350-Gaming 3-CF, BIOS F7 06/16/2017 [ +0.000001] RIP: 0010:__queue_delayed_work+0x6d/0xa0 [ +0.000003] Code: 7a 50 48 01 c1 48 89 4a 30 81 ff 00 20 00 00 75 38 4c 89 cf e8 64 3e 0a 00 5d e9 1e c5 11 01 e8 99 f7 ff ff 5d e9 13 c5 11 01 <0f> 0b eb c1 0f 0b 48 81 7a 38 70 5c 0e 81 74 9f 0f 0b 48 8b 42 28 [ +0.000002] RSP: 0018:ffffc90000398d60 EFLAGS: 00010007 [ +0.000002] RAX: ffff88810d589c60 RBX: 0000000000000000 RCX: 0000000000000000 [ +0.000002] RDX: ffff88810d589c58 RSI: 0000000000000000 RDI: 0000000000002000 [ +0.000001] RBP: ffffc90000398d60 R08: 0000000000000000 R09: ffff88810d589c78 [ +0.000002] R10: 72705f305f39765f R11: 7866673a6d72645b R12: ffff88810d589c58 [ +0.000001] R13: 0000000000002000 R14: 0000000000000000 R15: 0000000000000000 [ +0.000002] FS: 0000000000000000(0000) GS:ffff8887fee40000(0000) knlGS:0000000000000000 [ +0.000001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ +0.000002] CR2: 00005562c4797fa0 CR3: 0000000110da0000 CR4: 00000000003506e0 [ +0.000002] Call Trace: [ +0.000001] <IRQ> [ +0.000001] mod_delayed_work_on+0x5e/0xa0 [ +0.000004] drm_sched_fault+0x23/0x30 [gpu_sched] [ +0.000007] gfx_v9_0_fault.isra.0+0xa6/0xd0 [amdgpu] [ +0.000258] gfx_v9_0_priv_reg_irq+0x29/0x40 [amdgpu] [ +0.000254] amdgpu_irq_dispatch+0x1ac/0x2b0 [amdgpu] [ +0.000243] amdgpu_ih_process+0x89/0x130 [amdgpu] [ +0.000245] amdgpu_irq_handler+0x24/0x60 [amdgpu] [ +0.000165] __handle_irq_event_percpu+0x4f/0x1a0 [ +0.000003] handle_irq_event_percpu+0x15/0x50 [ +0.000001] handle_irq_event+0x39/0x60 [ +0.000002] handle_edge_irq+0xa8/0x250 [ +0.000003] __common_interrupt+0x7b/0x150 [ +0.000002] common_interrupt+0xc1/0xe0 [ +0.000003] </IRQ> [ +0.000000] <TASK> [ +0.000001] asm_common_interrupt+0x27/0x40 [ +0.000002] RIP: 0010:native_safe_halt+0xb/0x10 [ +0.000003] Code: 46 ff ff ff cc cc cc cc cc cc cc cc cc cc cc eb 07 0f 00 2d 69 f2 5e 00 f4 e9 f1 3b 3e 00 90 eb 07 0f 00 2d 59 f2 5e 00 fb f4 <e9> e0 3b 3e 00 0f 1f 44 00 00 55 48 89 e5 53 e8 b1 d4 fe ff 66 90 [ +0.000002] RSP: 0018:ffffc9000018fdc8 EFLAGS: 00000246 [ +0.000002] RAX: 0000000000004000 RBX: 000000000002e5a8 RCX: 000000000000001f [ +0.000001] RDX: 0000000000000001 RSI: ffff888101298800 RDI: ffff888101298864 [ +0.000001] RBP: ffffc9000018fdd0 R08: 000000527f64bd8b R09: 000000000001dc90 [ +0.000001] R10: 000000000001dc90 R11: 0000000000000003 R12: 0000000000000001 [ +0.000001] R13: ffff888101298864 R14: ffffffff832d9e20 R15: ffff888193aa8c00 [ +0.000003] ? acpi_idle_do_entry+0x5e/0x70 [ +0.000002] acpi_idle_enter+0xd1/0x160 [ +0.000003] cpuidle_enter_state+0x9a/0x6e0 [ +0.000003] cpuidle_enter+0x2e/0x50 [ +0.000003] call_cpuidle+0x23/0x50 [ +0.000002] do_idle+0x1de/0x260 [ +0.000002] cpu_startup_entry+0x20/0x30 [ +0.000002] start_secondary+0x120/0x150 [ +0.000003] secondary_startup_64_no_verify+0xe5/0xeb [ +0.000004] </TASK> [ +0.000000] ---[ end trace 0000000000000000 ]--- [ +0.000003] BUG: kernel NULL pointer dereference, address: 0000000000000102 [ +0.006233] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_low timeout, signaled seq=3, emitted seq=4 [ +0.000734] #PF: supervisor read access in kernel mode [ +0.009670] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process amd_deadlock pid 2002 thread amd_deadlock pid 2002 [ +0.005135] #PF: error_code(0x0000) - not-present page [ +0.000002] PGD 0 P4D 0 [ +0.000002] Oops: 0000 [#1] PREEMPT SMP NOPTI [ +0.000002] CPU: 9 PID: 0 Comm: swapper/9 Tainted: G W OE 6.1.11+ #2 [ +0.000002] Hardware name: Gigabyte Technology Co., Ltd. AB350-Gaming 3/AB350-Gaming 3-CF, BIOS F7 06/16/2017 [ +0.012101] amdgpu 0000:0c:00.0: amdgpu: GPU reset begin! [ +0.005136] RIP: 0010:__queue_work+0x1f/0x4e0 [ +0.000004] Code: 87 cd 11 01 0f 1f 80 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 49 89 d5 41 54 49 89 f4 53 48 83 ec 10 89 7d d4 <f6> 86 02 01 00 00 01 0f 85 6c 03 00 00 e8 7f 36 08 00 8b 45 d4 48 For gfx_rings the schedulers may not be initialized by amdgpu_device_init_schedulers() due to ring->no_scheduler flag being set to true and thus the timeout_wq is NULL. As a result, since all ASICs call drm_sched_fault() unconditionally even for schedulers which have not been initialized, it is simpler to use the ready condition which indicates whether the given scheduler worker thread runs and whether the timeout_wq of the reset domain has been initialized. Signed-off-by: Vitaly Prosyak <vitaly.prosyak@amd.com> Cc: Christian König <christian.koenig@amd.com> Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Signed-off-by: Luben Tuikov <luben.tuikov@amd.com> Link: https://lore.kernel.org/r/20230406200054.633379-1-luben.tuikov@amd.com drm/scheduler: set entity to NULL in drm_sched_entity_pop_job() It already happend a few times that patches slipped through which implemented access to an entity through a job that was already removed from the entities queue. Since jobs and entities might have different lifecycles, this can potentially cause UAF bugs. In order to make it obvious that a jobs entity pointer shouldn't be accessed after drm_sched_entity_pop_job() was called successfully, set the jobs entity pointer to NULL once the job is removed from the entity queue. Moreover, debugging a potential NULL pointer dereference is way easier than potentially corrupted memory through a UAF. Signed-off-by: Danilo Krummrich <dakr@redhat.com> Link: https://lore.kernel.org/r/20230418100453.4433-1-dakr@redhat.com Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Signed-off-by: Luben Tuikov <luben.tuikov@amd.com> drm/scheduler: properly forward fence errors When a hw fence is signaled with an error properly forward that to the finished fence. Signed-off-by: Christian König <christian.koenig@amd.com> Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20230420115752.31470-1-christian.koenig@amd.com drm/scheduler: add drm_sched_entity_error and use rcu for last_scheduled Switch to using RCU handling for the last scheduled job and add a function to return the error code of it. Signed-off-by: Christian König <christian.koenig@amd.com> Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20230420115752.31470-2-christian.koenig@amd.com drm/scheduler: mark jobs without fence as canceled When no hw fence is provided for a job that means that the job didn't executed. Signed-off-by: Christian König <christian.koenig@amd.com> Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20230427122726.1290170-1-christian.koenig@amd.com drm/sched: Check scheduler work queue before calling timeout handling During an IGT GPU reset test we see again oops despite of commit 0c8c901aaaebc9 (drm/sched: Check scheduler ready before calling timeout handling). It uses ready condition whether to call drm_sched_fault which unwind the TDR leads to GPU reset. However it looks the ready condition is overloaded with other meanings, for example, for the following stack is related GPU reset : 0 gfx_v9_0_cp_gfx_start 1 gfx_v9_0_cp_gfx_resume 2 gfx_v9_0_cp_resume 3 gfx_v9_0_hw_init 4 gfx_v9_0_resume 5 amdgpu_device_ip_resume_phase2 does the following: /* start the ring */ gfx_v9_0_cp_gfx_start(adev); ring->sched.ready = true; The same approach is for other ASICs as well : gfx_v8_0_cp_gfx_resume gfx_v10_0_kiq_resume, etc... As a result, our GPU reset test causes GPU fault which calls unconditionally gfx_v9_0_fault and then drm_sched_fault. However now it depends on whether the interrupt service routine drm_sched_fault is executed after gfx_v9_0_cp_gfx_start is completed which sets the ready field of the scheduler to true even for uninitialized schedulers and causes oops vs no fault or when ISR drm_sched_fault is completed prior gfx_v9_0_cp_gfx_start and NULL pointer dereference does not occur. Use the field timeout_wq to prevent oops for uninitialized schedulers. The field could be initialized by the work queue of resetting the domain. v1: Corrections to commit message (Luben) Fixes: 11b3b9f461c5c4 ("drm/sched: Check scheduler ready before calling timeout handling") Signed-off-by: Vitaly Prosyak <vitaly.prosyak@amd.com> Link: https://lore.kernel.org/r/20230510135111.58631-1-vitaly.prosyak@amd.com Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Signed-off-by: Luben Tuikov <luben.tuikov@amd.com> drm/sched: Remove redundant check The rq pointer points inside the drm_gpu_scheduler structure. Thus it can't be NULL. Found by Linux Verification Center (linuxtesting.org) with SVACE. Fixes:c61cdbdbff("drm/scheduler: Fix hang when sched_entity released") Signed-off-by: Vladislav Efanov <VEfanov@ispras.ru> Link: https://lore.kernel.org/r/20230517125247.434103-1-VEfanov@ispras.ru Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Signed-off-by: Luben Tuikov <luben.tuikov@amd.com> drm/sched: Rename to drm_sched_can_queue() Rename drm_sched_ready() to drm_sched_can_queue(). "ready" can mean many things and is thus meaningless in this context. Instead, rename to a name which precisely conveys what is being checked. Cc: Christian König <christian.koenig@amd.com> Cc: Alex Deucher <Alexander.Deucher@amd.com> Signed-off-by: Luben Tuikov <luben.tuikov@amd.com> Reviewed-by: Alex Deucher <Alexander.Deucher@amd.com> Link: https://lore.kernel.org/r/20230517233550.377847-1-luben.tuikov@amd.com drm/sched: Rename to drm_sched_wakeup_if_can_queue() Rename drm_sched_wakeup() to drm_sched_wakeup_if_canqueue() since the former is misleading, as it wakes up the GPU scheduler _only if_ more jobs can be queued to the underlying hardware. This distinction is important to make, since the wake conditional in the GPU scheduler thread wakes up when other conditions are also true, e.g. when there are jobs to be cleaned. For instance, a user might want to wake up the scheduler only because there are more jobs to clean, but whether we can queue more jobs is irrelevant. v2: Separate "canqueue" to "can_queue". (Alex D.) Cc: Christian König <christian.koenig@amd.com> Cc: Alex Deucher <Alexander.Deucher@amd.com> Signed-off-by: Luben Tuikov <luben.tuikov@amd.com> Link: https://lore.kernel.org/r/20230517233550.377847-2-luben.tuikov@amd.com Reviewed-by: Alex Deucher <Alexander.Deucher@amd.com> drm/scheduler: avoid infinite loop if entity's dependency is a scheduled error fence [Why] drm_sched_entity_add_dependency_cb ignores the scheduled fence and return false. If entity's dependency is a scheduler error fence and drm_sched_stop is called due to TDR, drm_sched_entity_pop_job will wait for the dependency infinitely. [How] Do not wait or ignore the scheduled error fence, add drm_sched_entity_wakeup callback for the dependency with scheduled error fence. Signed-off-by: ZhenGuo Yin <zhenguo.yin@amd.com> Acked-by: Alex Deucher <alexander.deucher@amd.com> Reviewed-by: Christian König <christian.koenig@amd.com> Signed-off-by: Alex Deucher <alexander.deucher@amd.com> drm/sched: Make sure we wait for all dependencies in kill_jobs_cb() drm_sched_entity_kill_jobs_cb() logic is omitting the last fence popped from the dependency array that was waited upon before drm_sched_entity_kill() was called (drm_sched_entity::dependency field), so we're basically waiting for all dependencies except one. In theory, this wait shouldn't be needed because resources should have their users registered to the dma_resv object, thus guaranteeing that future jobs wanting to access these resources wait on all the previous users (depending on the access type, of course). But we want to keep these explicit waits in the kill entity path just in case. Let's make sure we keep all dependencies in the array in drm_sched_job_dependency(), so we can iterate over the array and wait in drm_sched_entity_kill_jobs_cb(). We also make sure we wait on drm_sched_fence::finished if we were originally asked to wait on drm_sched_fence::scheduled. In that case, we assume the intent was to delegate the wait to the firmware/GPU or rely on the pipelining done at the entity/scheduler level, but when killing jobs, we really want to wait for completion not just scheduling. v2: - Don't evict deps in drm_sched_job_dependency() v3: - Always wait for drm_sched_fence::finished fences in drm_sched_entity_kill_jobs_cb() when we see a sched_fence v4: - Fix commit message - Fix a use-after-free bug v5: - Flag deps on which we should only wait for the scheduled event at insertion time v6: - Back to v4 implementation - Add Christian's R-b Cc: Frank Binns <frank.binns@imgtec.com> Cc: Sarah Walker <sarah.walker@imgtec.com> Cc: Donald Robson <donald.robson@imgtec.com> Cc: Luben Tuikov <luben.tuikov@amd.com> Cc: David Airlie <airlied@gmail.com> Cc: Daniel Vetter <daniel@ffwll.ch> Cc: Sumit Semwal <sumit.semwal@linaro.org> Cc: "Christian König" <christian.koenig@amd.com> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Suggested-by: "Christian König" <christian.koenig@amd.com> Reviewed-by: "Christian König" <christian.koenig@amd.com> Acked-by: Luben Tuikov <luben.tuikov@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20230619071921.3465992-1-boris.brezillon@collabora.com drm/sched: Call drm_sched_fence_set_parent() from drm_sched_fence_scheduled() Drivers that can delegate waits to the firmware/GPU pass the scheduled fence to drm_sched_job_add_dependency(), and issue wait commands to the firmware/GPU at job submission time. For this to be possible, they need all their 'native' dependencies to have a valid parent since this is where the actual HW fence information are encoded. In drm_sched_main(), we currently call drm_sched_fence_set_parent() after drm_sched_fence_scheduled(), leaving a short period of time during which the job depending on this fence can be submitted. Since setting parent and signaling the fence are two things that are kinda related (you can't have a parent if the job hasn't been scheduled), it probably makes sense to pass the parent fence to drm_sched_fence_scheduled() and let it call drm_sched_fence_set_parent() before it signals the scheduled fence. Here is a detailed description of the race we are fixing here: Thread A Thread B - calls drm_sched_fence_scheduled() - signals s_fence->scheduled which wakes up thread B - entity dep signaled, checking the next dep - no more deps waiting - entity is picked for job submission by drm_gpu_scheduler - run_job() is called - run_job() tries to collect native fence info from s_fence->parent, but it's NULL => BOOM, we can't do our native wait - calls drm_sched_fence_set_parent() v2: * Fix commit message v3: * Add a detailed description of the race to the commit message * Add Luben's R-b Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Cc: Frank Binns <frank.binns@imgtec.com> Cc: Sarah Walker <sarah.walker@imgtec.com> Cc: Donald Robson <donald.robson@imgtec.com> Cc: Luben Tuikov <luben.tuikov@amd.com> Cc: David Airlie <airlied@gmail.com> Cc: Daniel Vetter <daniel@ffwll.ch> Cc: Sumit Semwal <sumit.semwal@linaro.org> Cc: "Christian König" <christian.koenig@amd.com> Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20230623075204.382350-1-boris.brezillon@collabora.com dma-buf: add dma_fence_timestamp helper When a fence signals there is a very small race window where the timestamp isn't updated yet. sync_file solves this by busy waiting for the timestamp to appear, but on other ocassions didn't handled this correctly. Provide a dma_fence_timestamp() helper function for this and use it in all appropriate cases. Another alternative would be to grab the spinlock when that happens. v2 by teddy: add a wait parameter to wait for the timestamp to show up, in case the accurate timestamp is needed and/or the timestamp is not based on ktime (e.g. hw timestamp) v3 chk: drop the parameter again for unified handling Signed-off-by: Yunxiang Li <Yunxiang.Li@amd.com> Signed-off-by: Christian König <christian.koenig@amd.com> Fixes:1774baa64f("drm/scheduler: Change scheduled fence track v2") Reviewed-by: Alex Deucher <alexander.deucher@amd.com> CC: stable@vger.kernel.org Link: https://patchwork.freedesktop.org/patch/msgid/20230929104725.2358-1-christian.koenig@amd.com drm/sched: Convert the GPU scheduler to variable number of run-queues The GPU scheduler has now a variable number of run-queues, which are set up at drm_sched_init() time. This way, each driver announces how many run-queues it requires (supports) per each GPU scheduler it creates. Note, that run-queues correspond to scheduler "priorities", thus if the number of run-queues is set to 1 at drm_sched_init(), then that scheduler supports a single run-queue, i.e. single "priority". If a driver further sets a single entity per run-queue, then this creates a 1-to-1 correspondence between a scheduler and a scheduled entity. Cc: Lucas Stach <l.stach@pengutronix.de> Cc: Russell King <linux+etnaviv@armlinux.org.uk> Cc: Qiang Yu <yuq825@gmail.com> Cc: Rob Clark <robdclark@gmail.com> Cc: Abhinav Kumar <quic_abhinavk@quicinc.com> Cc: Dmitry Baryshkov <dmitry.baryshkov@linaro.org> Cc: Danilo Krummrich <dakr@redhat.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Alex Deucher <alexander.deucher@amd.com> Cc: Christian König <christian.koenig@amd.com> Cc: Emma Anholt <emma@anholt.net> Cc: etnaviv@lists.freedesktop.org Cc: lima@lists.freedesktop.org Cc: linux-arm-msm@vger.kernel.org Cc: freedreno@lists.freedesktop.org Cc: nouveau@lists.freedesktop.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: Luben Tuikov <luben.tuikov@amd.com> Acked-by: Christian König <christian.koenig@amd.com> Link: https://lore.kernel.org/r/20231023032251.164775-1-luben.tuikov@amd.com drm/sched: Add drm_sched_wqueue_* helpers Add scheduler wqueue ready, stop, and start helpers to hide the implementation details of the scheduler from the drivers. v2: - s/sched_wqueue/sched_wqueue (Luben) - Remove the extra white line after the return-statement (Luben) - update drm_sched_wqueue_ready comment (Luben) Cc: Luben Tuikov <luben.tuikov@amd.com> Signed-off-by: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Link: https://lore.kernel.org/r/20231031032439.1558703-2-matthew.brost@intel.com Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> drm/sched: Convert drm scheduler to use a work queue rather than kthread In Xe, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first this seems a bit odd but let us explain the reasoning below. 1. In Xe the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same hardware engine. This is because in Xe we have a firmware scheduler, the GuC, which allowed to reorder, timeslice, and preempt submissions. If a using shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR falls apart as the TDR expects submission order == completion order. Using a dedicated drm_gpu_scheduler per drm_sched_entity solve this problem. 2. In Xe submissions are done via programming a ring buffer (circular buffer), a drm_gpu_scheduler provides a limit on number of jobs, if the limit of number jobs is set to RING_SIZE / MAX_SIZE_PER_JOB we get flow control on the ring for free. A problem with this design is currently a drm_gpu_scheduler uses a kthread for submission / job cleanup. This doesn't scale if a large number of drm_gpu_scheduler are used. To work around the scaling issue, use a worker rather than kthread for submission / job cleanup. v2: - (Rob Clark) Fix msm build - Pass in run work queue v3: - (Boris) don't have loop in worker v4: - (Tvrtko) break out submit ready, stop, start helpers into own patch v5: - (Boris) default to ordered work queue v6: - (Luben / checkpatch) fix alignment in msm_ringbuffer.c - (Luben) s/drm_sched_submit_queue/drm_sched_wqueue_enqueue - (Luben) Update comment for drm_sched_wqueue_enqueue - (Luben) Positive check for submit_wq in drm_sched_init - (Luben) s/alloc_submit_wq/own_submit_wq v7: - (Luben) s/drm_sched_wqueue_enqueue/drm_sched_run_job_queue v8: - (Luben) Adjust var names / comments Signed-off-by: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Link: https://lore.kernel.org/r/20231031032439.1558703-3-matthew.brost@intel.com Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> drm/sched: Split free_job into own work item Rather than call free_job and run_job in same work item have a dedicated work item for each. This aligns with the design and intended use of work queues. v2: - Test for DMA_FENCE_FLAG_TIMESTAMP_BIT before setting timestamp in free_job() work item (Danilo) v3: - Drop forward dec of drm_sched_select_entity (Boris) - Return in drm_sched_run_job_work if entity NULL (Boris) v4: - Replace dequeue with peek and invert logic (Luben) - Wrap to 100 lines (Luben) - Update comments for *_queue / *_queue_if_ready functions (Luben) v5: - Drop peek argument, blindly reinit idle (Luben) - s/drm_sched_free_job_queue_if_ready/drm_sched_free_job_queue_if_done (Luben) - Update work_run_job & work_free_job kernel doc (Luben) v6: - Do not move drm_sched_select_entity in file (Luben) Signed-off-by: Matthew Brost <matthew.brost@intel.com> Link: https://lore.kernel.org/r/20231031032439.1558703-4-matthew.brost@intel.com Reviewed-by: Luben Tuikov <ltuikov89@gmail.com> Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> drm/sched: Add drm_sched_start_timeout_unlocked helper Also add a lockdep assert to drm_sched_start_timeout. Signed-off-by: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Link: https://lore.kernel.org/r/20231031032439.1558703-5-matthew.brost@intel.com Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> drm/sched: Add a helper to queue TDR immediately Add a helper whereby a driver can invoke TDR immediately. v2: - Drop timeout args, rename function, use mod delayed work (Luben) v3: - s/XE/Xe (Luben) - present tense in commit message (Luben) - Adjust comment for drm_sched_tdr_queue_imm (Luben) v4: - Adjust commit message (Luben) Cc: Luben Tuikov <luben.tuikov@amd.com> Signed-off-by: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Luben Tuikov <luben.tuikov@amd.com> Link: https://lore.kernel.org/r/20231031032439.1558703-6-matthew.brost@intel.com Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> drm/sched: Rename drm_sched_get_cleanup_job to be more descriptive "Get cleanup job" makes it sound like helper is returning a job which will execute some cleanup, or something, while the kerneldoc itself accurately says "fetch the next _finished_ job". So lets rename the helper to be self documenting. Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Luben Tuikov <ltuikov89@gmail.com> Cc: Matthew Brost <matthew.brost@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231102105538.391648-2-tvrtko.ursulin@linux.intel.com Reviewed-by: Luben Tuikov <ltuikov89@gmail.com> Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> drm/sched: Move free worker re-queuing out of the if block Whether or not there are more jobs to clean up does not depend on the existance of the current job, given both drm_sched_get_finished_job and drm_sched_free_job_queue_if_done take and drop the job list lock. Therefore it is confusing to make it read like there is a dependency. Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Luben Tuikov <ltuikov89@gmail.com> Cc: Matthew Brost <matthew.brost@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231102105538.391648-3-tvrtko.ursulin@linux.intel.com Reviewed-by: Luben Tuikov <ltuikov89@gmail.com> Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> drm/sched: Rename drm_sched_free_job_queue to be more descriptive The current name makes it sound like helper will free a queue, while what it does is it enqueues the free job worker. Rename it to drm_sched_run_free_queue to align with existing drm_sched_run_job_queue. Despite that creating an illusion there are two queues, while in reality there is only one, at least it creates a consistent naming for the two enqueuing helpers. At the same time simplify the "if done" helper by dropping the suffix and adding a double underscore prefix to the one which just enqueues. Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Luben Tuikov <ltuikov89@gmail.com> Cc: Matthew Brost <matthew.brost@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231102105538.391648-4-tvrtko.ursulin@linux.intel.com Reviewed-by: Luben Tuikov <ltuikov89@gmail.com> Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> drm/sched: Rename drm_sched_run_job_queue_if_ready and clarify kerneldoc "If ready" is not immediately clear what it means - is the scheduler ready or something else? Drop the suffix, clarify kerneldoc, and employ the same naming scheme as in drm_sched_run_free_queue: - drm_sched_run_job_queue - enqueues if there is something to enqueue *and* scheduler is ready (can queue) - __drm_sched_run_job_queue - low-level helper to simply queue the job Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Luben Tuikov <ltuikov89@gmail.com> Cc: Matthew Brost <matthew.brost@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231102105538.391648-5-tvrtko.ursulin@linux.intel.com Reviewed-by: Luben Tuikov <ltuikov89@gmail.com> Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> drm/sched: Drop suffix from drm_sched_wakeup_if_can_queue Because a) helper is exported to other parts of the scheduler and b) there isn't a plain drm_sched_wakeup to begin with, I think we can drop the suffix and by doing so separate the intimiate knowledge between the scheduler components a bit better. Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com> Cc: Luben Tuikov <ltuikov89@gmail.com> Cc: Matthew Brost <matthew.brost@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231102105538.391648-6-tvrtko.ursulin@linux.intel.com Reviewed-by: Luben Tuikov <ltuikov89@gmail.com> Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> drm/sched: Don't disturb the entity when in RR-mode scheduling Don't call drm_sched_select_entity() in drm_sched_run_job_queue(). In fact, rename __drm_sched_run_job_queue() to just drm_sched_run_job_queue(), and let it do just that, schedule the work item for execution. The problem is that drm_sched_run_job_queue() calls drm_sched_select_entity() to determine if the scheduler has an entity ready in one of its run-queues, and in the case of the Round-Robin (RR) scheduling, the function drm_sched_rq_select_entity_rr() does just that, selects the _next_ entity which is ready, sets up the run-queue and completion and returns that entity. The FIFO scheduling algorithm is unaffected. Now, since drm_sched_run_job_work() also calls drm_sched_select_entity(), then in the case of RR scheduling, that would result in drm_sched_select_entity() having been called twice, which may result in skipping a ready entity if more than one entity is ready. This commit fixes this by eliminating the call to drm_sched_select_entity() from drm_sched_run_job_queue(), and leaves it only in drm_sched_run_job_work(). v2: Rebased on top of Tvrtko's renames series of patches. (Luben) Add fixes-tag. (Tvrtko) Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> Fixes: f7fe64ad0f22ff ("drm/sched: Split free_job into own work item") Reviewed-by: Matthew Brost <matthew.brost@intel.com> Reviewed-by: Danilo Krummrich <dakr@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231107041020.10035-2-ltuikov89@gmail.com drm/sched: Qualify drm_sched_wakeup() by drm_sched_entity_is_ready() Don't "wake up" the GPU scheduler unless the entity is ready, as well as we can queue to the scheduler, i.e. there is no point in waking up the scheduler for the entity unless the entity is ready. Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> Fixes: bc8d6a9df99038 ("drm/sched: Don't disturb the entity when in RR-mode scheduling") Reviewed-by: Danilo Krummrich <dakr@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231110000123.72565-2-ltuikov89@gmail.com drm/sched: implement dynamic job-flow control Currently, job flow control is implemented simply by limiting the number of jobs in flight. Therefore, a scheduler is initialized with a credit limit that corresponds to the number of jobs which can be sent to the hardware. This implies that for each job, drivers need to account for the maximum job size possible in order to not overflow the ring buffer. However, there are drivers, such as Nouveau, where the job size has a rather large range. For such drivers it can easily happen that job submissions not even filling the ring by 1% can block subsequent submissions, which, in the worst case, can lead to the ring run dry. In order to overcome this issue, allow for tracking the actual job size instead of the number of jobs. Therefore, add a field to track a job's credit count, which represents the number of credits a job contributes to the scheduler's credit limit. Signed-off-by: Danilo Krummrich <dakr@redhat.com> Reviewed-by: Luben Tuikov <ltuikov89@gmail.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231110001638.71750-1-dakr@redhat.com drm/sched: Fix bounds limiting when given a malformed entity If we're given a malformed entity in drm_sched_entity_init()--shouldn't happen, but we verify--with out-of-bounds priority value, we set it to an allowed value. Fix the expression which sets this limit. Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> Fixes: 56e449603f0ac5 ("drm/sched: Convert the GPU scheduler to variable number of run-queues") Link: https://patchwork.freedesktop.org/patch/msgid/20231123122422.167832-2-ltuikov89@gmail.com Reviewed-by: Christian König <christian.koenig@amd.com> Link: https://lore.kernel.org/r/dbb91dbe-ef77-4d79-aaf9-2adb171c1d7a@amd.com drm/sched: Rename priority MIN to LOW Rename DRM_SCHED_PRIORITY_MIN to DRM_SCHED_PRIORITY_LOW. This mirrors DRM_SCHED_PRIORITY_HIGH, for a list of DRM scheduler priorities in ascending order, DRM_SCHED_PRIORITY_LOW, DRM_SCHED_PRIORITY_NORMAL, DRM_SCHED_PRIORITY_HIGH, DRM_SCHED_PRIORITY_KERNEL. Cc: Rob Clark <robdclark@gmail.com> Cc: Abhinav Kumar <quic_abhinavk@quicinc.com> Cc: Dmitry Baryshkov <dmitry.baryshkov@linaro.org> Cc: Danilo Krummrich <dakr@redhat.com> Cc: Alex Deucher <alexander.deucher@amd.com> Cc: Christian König <christian.koenig@amd.com> Cc: linux-arm-msm@vger.kernel.org Cc: freedreno@lists.freedesktop.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> Reviewed-by: Christian König <christian.koenig@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231124052752.6915-5-ltuikov89@gmail.com drm/sched: Reverse run-queue priority enumeration Reverse run-queue priority enumeration such that the higest priority is now 0, and for each consecutive integer the prioirty diminishes. Run-queues correspond to priorities. To an external observer a scheduler created with a single run-queue, and another created with DRM_SCHED_PRIORITY_COUNT number of run-queues, should always schedule sched->sched_rq[0] with the same "priority", as that index run-queue exists in both schedulers, i.e. a scheduler with one run-queue or many. This patch makes it so. In other words, the "priority" of sched->sched_rq[n], n >= 0, is the same for any scheduler created with any allowable number of run-queues (priorities), 0 to DRM_SCHED_PRIORITY_COUNT. Cc: Rob Clark <robdclark@gmail.com> Cc: Abhinav Kumar <quic_abhinavk@quicinc.com> Cc: Dmitry Baryshkov <dmitry.baryshkov@linaro.org> Cc: Danilo Krummrich <dakr@redhat.com> Cc: Alex Deucher <alexander.deucher@amd.com> Cc: Christian König <christian.koenig@amd.com> Cc: linux-arm-msm@vger.kernel.org Cc: freedreno@lists.freedesktop.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> Reviewed-by: Christian König <christian.koenig@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231124052752.6915-6-ltuikov89@gmail.com drm/sched: Partial revert of "Qualify drm_sched_wakeup() by drm_sched_entity_is_ready()" Commit f3123c2590005c, in combination with the use of work queues by the GPU scheduler, leads to random lock-ups of the GUI. This is a partial revert of of commit f3123c2590005c since drm_sched_wakeup() still needs its entity argument to pass it to drm_sched_can_queue(). Link: https://gitlab.freedesktop.org/drm/amd/-/issues/2994 Link: https://lists.freedesktop.org/archives/dri-devel/2023-November/431606.html Signed-off-by: Bert Karwatzki <spasswolf@web.de> Link: https://patchwork.freedesktop.org/patch/msgid/20231127160955.87879-1-spasswolf@web.de Link: https://lore.kernel.org/r/36bece178ff5dc705065e53d1e5e41f6db6d87e4.camel@web.de Fixes: f3123c2590005c ("drm/sched: Qualify drm_sched_wakeup() by drm_sched_entity_is_ready()") Reviewed-by: Luben Tuikov <ltuikov89@gmail.com> Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> drm/sched: One function call less in drm_sched_init() after error detection The kfree() function was called in one case by the drm_sched_init() function during error handling even if the passed data structure member contained a null pointer. This issue was detected by using the Coccinelle software. Thus adjust a jump target. Signed-off-by: Markus Elfring <elfring@users.sourceforge.net> Link: https://patchwork.freedesktop.org/patch/msgid/85066512-983d-480c-a44d-32405ab1b80e@web.de Reviewed-by: Luben Tuikov <ltuikov89@gmail.com> Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> drm/sched: Return an error code only as a constant in drm_sched_init() Return an error code without storing it in an intermediate variable. Signed-off-by: Markus Elfring <elfring@users.sourceforge.net> Link: https://patchwork.freedesktop.org/patch/msgid/85f8004e-f0c9-42d9-8c59-30f1b4e0b89e@web.de Reviewed-by: Luben Tuikov <ltuikov89@gmail.com> Signed-off-by: Luben Tuikov <ltuikov89@gmail.com> drm/sched: Drain all entities in DRM sched run job worker All entities must be drained in the DRM scheduler run job worker to avoid the following case. An entity found that is ready, no job found ready on entity, and run job worker goes idle with other entities + jobs ready. Draining all ready entities (i.e. loop over all ready entities) in the run job worker ensures all job that are ready will be scheduled. Cc: Thorsten Leemhuis <regressions@leemhuis.info> Reported-by: Mikhail Gavrilov <mikhail.v.gavrilov@gmail.com> Closes: https://lore.kernel.org/all/CABXGCsM2VLs489CH-vF-1539-s3in37=bwuOWtoeeE+q26zE+Q@mail.gmail.com/ Reported-and-tested-by: Mario Limonciello <mario.limonciello@amd.com> Closes: https://gitlab.freedesktop.org/drm/amd/-/issues/3124 Link: https://lore.kernel.org/all/20240123021155.2775-1-mario.limonciello@amd.com/ Reported-and-tested-by: Vlastimil Babka <vbabka@suse.cz> Closes: https://lore.kernel.org/dri-devel/05ddb2da-b182-4791-8ef7-82179fd159a8@amd.com/T/#m0c31d4d1b9ae9995bb880974c4f1dbaddc33a48a Signed-off-by: Matthew Brost <matthew.brost@intel.com> Signed-off-by: Dave Airlie <airlied@redhat.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240124210811.1639040-1-matthew.brost@intel.com drm/drm_exec: Work around a WW mutex lockdep oddity If *any* object of a certain WW mutex class is locked, lockdep will consider *all* mutexes of that class as locked. Also the lock allocation tracking code will apparently register only the address of the first mutex of a given class locked in a sequence. This has the odd consequence that if that first mutex is unlocked while other mutexes of the same class remain locked and then its memory then freed, the lock alloc tracking code will incorrectly assume that memory is freed with a held lock in there. For now, work around that for drm_exec by releasing the first grabbed object lock last. v2: - Fix a typo (Danilo Krummrich) - Reword the commit message a bit. - Add a Fixes: tag Related lock alloc tracking warning: [ 322.660067] ========================= [ 322.660070] WARNING: held lock freed! [ 322.660074] 6.5.0-rc7+ #155 Tainted: G U N [ 322.660078] ------------------------- [ 322.660081] kunit_try_catch/4981 is freeing memory ffff888112adc000-ffff888112adc3ff, with a lock still held there! [ 322.660089] ffff888112adc1a0 (reservation_ww_class_mutex){+.+.}-{3:3}, at: drm_exec_lock_obj+0x11a/0x600 [drm_exec] [ 322.660104] 2 locks held by kunit_try_catch/4981: [ 322.660108] #0: ffffc9000343fe18 (reservation_ww_class_acquire){+.+.}-{0:0}, at: test_early_put+0x22f/0x490 [drm_exec_test] [ 322.660123] #1: ffff888112adc1a0 (reservation_ww_class_mutex){+.+.}-{3:3}, at: drm_exec_lock_obj+0x11a/0x600 [drm_exec] [ 322.660135] stack backtrace: [ 322.660139] CPU: 7 PID: 4981 Comm: kunit_try_catch Tainted: G U N 6.5.0-rc7+ #155 [ 322.660146] Hardware name: ASUS System Product Name/PRIME B560M-A AC, BIOS 0403 01/26/2021 [ 322.660152] Call Trace: [ 322.660155] <TASK> [ 322.660158] dump_stack_lvl+0x57/0x90 [ 322.660164] debug_check_no_locks_freed+0x20b/0x2b0 [ 322.660172] slab_free_freelist_hook+0xa1/0x160 [ 322.660179] ? drm_exec_unlock_all+0x168/0x2a0 [drm_exec] [ 322.660186] __kmem_cache_free+0xb2/0x290 [ 322.660192] drm_exec_unlock_all+0x168/0x2a0 [drm_exec] [ 322.660200] drm_exec_fini+0xf/0x1c0 [drm_exec] [ 322.660206] test_early_put+0x289/0x490 [drm_exec_test] [ 322.660215] ? __pfx_test_early_put+0x10/0x10 [drm_exec_test] [ 322.660222] ? __kasan_check_byte+0xf/0x40 [ 322.660227] ? __ksize+0x63/0x140 [ 322.660233] ? drmm_add_final_kfree+0x3e/0xa0 [drm] [ 322.660289] ? _raw_spin_unlock_irqrestore+0x30/0x60 [ 322.660294] ? lockdep_hardirqs_on+0x7d/0x100 [ 322.660301] ? __pfx_kunit_try_run_case+0x10/0x10 [kunit] [ 322.660310] ? __pfx_kunit_generic_run_threadfn_adapter+0x10/0x10 [kunit] [ 322.660319] kunit_generic_run_threadfn_adapter+0x4a/0x90 [kunit] [ 322.660328] kthread+0x2e7/0x3c0 [ 322.660334] ? __pfx_kthread+0x10/0x10 [ 322.660339] ret_from_fork+0x2d/0x70 [ 322.660345] ? __pfx_kthread+0x10/0x10 [ 322.660349] ret_from_fork_asm+0x1b/0x30 [ 322.660358] </TASK> [ 322.660818] ok 8 test_early_put Cc: Christian König <christian.koenig@amd.com> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Danilo Krummrich <dakr@redhat.com> Cc: dri-devel@lists.freedesktop.org Fixes: 09593216bff1 ("drm: execution context for GEM buffers v7") Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Danilo Krummrich <dakr@redhat.com> Reviewed-by: Christian König <christian.koenig@amd.com> Link: https://patchwork.freedesktop.org/patch/msgid/20230906095039.3320-4-thomas.hellstrom@linux.intel.com drm/panthor: Add uAPI Panthor follows the lead of other recently submitted drivers with ioctls allowing us to support modern Vulkan features, like sparse memory binding: - Pretty standard GEM management ioctls (BO_CREATE and BO_MMAP_OFFSET), with the 'exclusive-VM' bit to speed-up BO reservation on job submission - VM management ioctls (VM_CREATE, VM_DESTROY and VM_BIND). The VM_BIND ioctl is loosely based on the Xe model, and can handle both asynchronous and synchronous requests - GPU execution context creation/destruction, tiler heap context creation and job submission. Those ioctls reflect how the hardware/scheduler works and are thus driver specific. We also have a way to expose IO regions, such that the usermode driver can directly access specific/well-isolate registers, like the LATEST_FLUSH register used to implement cache-flush reduction. This uAPI intentionally keeps usermode queues out of the scope, which explains why doorbell registers and command stream ring-buffers are not directly exposed to userspace. v6: - Add Maxime's and Heiko's acks v5: - Fix typo - Add Liviu's R-b v4: - Add a VM_GET_STATE ioctl - Fix doc - Expose the CORE_FEATURES register so we can deal with variants in the UMD - Add Steve's R-b v3: - Add the concept of sync-only VM operation - Fix support for 32-bit userspace - Rework drm_panthor_vm_create to pass the user VA size instead of the kernel VA size (suggested by Robin Murphy) - Typo fixes - Explicitly cast enums with top bit set to avoid compiler warnings in -pedantic mode. - Drop property core_group_count as it can be easily calculated by the number of bits set in l2_present. Co-developed-by: Steven Price <steven.price@arm.com> Signed-off-by: Steven Price <steven.price@arm.com> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Steven Price <steven.price@arm.com> Reviewed-by: Liviu Dudau <liviu.dudau@arm.com> Acked-by: Maxime Ripard <mripard@kernel.org> Acked-by: Heiko Stuebner <heiko@sntech.de> Link: https://patchwork.freedesktop.org/patch/msgid/20240229162230.2634044-2-boris.brezillon@collabora.com drm/panthor: Add GPU register definitions Those are the registers directly accessible through the MMIO range. FW registers are exposed in panthor_fw.h. v6: - Add Maxime's and Heiko's acks v4: - Add the CORE_FEATURES register (needed for GPU variants) - Add Steve's R-b v3: - Add macros to extract GPU ID info - Formatting changes - Remove AS_TRANSCFG_ADRMODE_LEGACY - it doesn't exist post-CSF - Remove CSF_GPU_LATEST_FLUSH_ID_DEFAULT - Add GPU_L2_FEATURES_LINE_SIZE for extracting the GPU cache line size Co-developed-by: Steven Price <steven.price@arm.com> Signed-off-by: Steven Price <steven.price@arm.com> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Acked-by: Steven Price <steven.price@arm.com> # MIT+GPL2 relicensing,Arm Acked-by: Grant Likely <grant.likely@linaro.org> # MIT+GPL2 relicensing,Linaro Acked-by: Boris Brezillon <boris.brezillon@collabora.com> # MIT+GPL2 relicensing,Collabora Reviewed-by: Steven Price <steven.price@arm.com> Acked-by: Maxime Ripard <mripard@kernel.org> Acked-by: Heiko Stuebner <heiko@sntech.de> Link: https://patchwork.freedesktop.org/patch/msgid/20240229162230.2634044-3-boris.brezillon@collabora.com drm/panthor: Add the device logical block The panthor driver is designed in a modular way, where each logical block is dealing with a specific HW-block or software feature. In order for those blocks to communicate with each other, we need a central panthor_device collecting all the blocks, and exposing some common features, like interrupt handling, power management, reset, ... This what this panthor_device logical block is about. v6: - Add Maxime's and Heiko's acks - Keep header inclusion alphabetically ordered v5: - Suspend the MMU/GPU blocks if panthor_fw_resume() fails in panthor_device_resume() - Move the pm_runtime_use_autosuspend() call before drm_dev_register() - Add Liviu's R-b v4: - Check drmm_mutex_init() return code - Fix panthor_device_reset_work() out path - Fix the race in the unplug logic - Fix typos - Unplug blocks when something fails in panthor_device_init() - Add Steve's R-b v3: - Add acks for the MIT+GPL2 relicensing - Fix 32-bit support - Shorten the sections protected by panthor_device::pm::mmio_lock to fix lock ordering issues. - Rename panthor_device::pm::lock into panthor_device::pm::mmio_lock to better reflect what this lock is protecting - Use dev_err_probe() - Make sure we call drm_dev_exit() when something fails half-way in panthor_device_reset_work() - Replace CSF_GPU_LATEST_FLUSH_ID_DEFAULT with a constant '1' and a comment to explain. Also remove setting the dummy flush ID on suspend. - Remove drm_WARN_ON() in panthor_exception_name() - Check pirq->suspended in panthor_xxx_irq_raw_handler() Co-developed-by: Steven Price <steven.price@arm.com> Signed-off-by: Steven Price <steven.price@arm.com> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Acked-by: Steven Price <steven.price@arm.com> # MIT+GPL2 relicensing,Arm Acked-by: Grant Likely <grant.likely@linaro.org> # MIT+GPL2 relicensing,Linaro Acked-by: Boris Brezillon <boris.brezillon@collabora.com> # MIT+GPL2 relicensing,Collabora Reviewed-by: Steven Price <steven.price@arm.com> Reviewed-by: Liviu Dudau <liviu.dudau@arm.com> Acked-by: Maxime Ripard <mripard@kernel.org> Acked-by: Heiko Stuebner <heiko@sntech.de> Link: https://patchwork.freedesktop.org/patch/msgid/20240229162230.2634044-4-boris.brezillon@collabora.com drm/panthor: Add the GPU logical block Handles everything that's not related to the FW, the MMU or the scheduler. This is the block dealing with the GPU property retrieval, the GPU block power on/off logic, and some global operations, like global cache flushing. v6: - Add Maxime's and Heiko's acks v5: - Fix GPU_MODEL() kernel doc - Fix test in panthor_gpu_block_power_off() - Add Steve's R-b v4: - Expose CORE_FEATURES through DEV_QUERY v3: - Add acks for the MIT/GPL2 relicensing - Use macros to extract GPU ID info - Make sure we reset clear pending_reqs bits when wait_event_timeout() times out but the corresponding bit is cleared in GPU_INT_RAWSTAT (can happen if the IRQ is masked or HW takes to long to call the IRQ handler) - GPU_MODEL now takes separate arch and product majors to be more readable. - Drop GPU_IRQ_MCU_STATUS_CHANGED from interrupt mask. - Handle GPU_IRQ_PROTM_FAULT correctly (don't output registers that are not updated for protected interrupts). - Minor code tidy ups Cc: Alexey Sheplyakov <asheplyakov@basealt.ru> # MIT+GPL2 relicensing Co-developed-by: Steven Price <steven.price@arm.com> Signed-off-by: Steven Price <steven.price@arm.com> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Acked-by: Steven Price <steven.price@arm.com> # MIT+GPL2 relicensing,Arm Acked-by: Grant Likely <grant.likely@linaro.org> # MIT+GPL2 relicensing,Linaro Acked-by: Boris Brezillon <boris.brezillon@collabora.com> # MIT+GPL2 relicensing,Collabora Reviewed-by: Steven Price <steven.price@arm.com> Acked-by: Maxime Ripard <mripard@kernel.org> Acked-by: Heiko Stuebner <heiko@sntech.de> Link: https://patchwork.freedesktop.org/patch/msgid/20240229162230.2634044-5-boris.brezillon@collabora.com drm/panthor: Add GEM logical block Anything relating to GEM object management is placed here. Nothing particularly interesting here, given the implementation is based on drm_gem_shmem_object, which is doing most of the work. v6: - Add Maxime's and Heiko's acks - Return a page-aligned BO size to userspace when creating a BO - Keep header inclusion alphabetically ordered v5: - Add Liviu's and Steve's R-b v4: - Force kernel BOs to be GPU mapped - Make panthor_kernel_bo_destroy() robust against ERR/NULL BO pointers to simplify the call sites v3: - Add acks for the MIT/GPL2 relicensing - Provide a panthor_kernel_bo abstraction for buffer objects managed by the kernel (will replace panthor_fw_mem and be used everywhere we were using panthor_gem_create_and_map() before) - Adjust things to match drm_gpuvm changes - Change return of panthor_gem_create_with_handle() to int Co-developed-by: Steven Price <steven.price@arm.com> Signed-off-by: Steven Price <steven.price@arm.com> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Acked-by: Steven Price <steven.price@arm.com> # MIT+GPL2 relicensing,Arm Acked-by: Grant Likely <grant.likely@linaro.org> # MIT+GPL2 relicensing,Linaro Acked-by: Boris Brezillon <boris.brezillon@collabora.com> # MIT+GPL2 relicensing,Collabora Reviewed-by: Liviu Dudau <liviu.dudau@arm.com> Reviewed-by: Steven Price <steven.price@arm.com> Acked-by: Maxime Ripard <mripard@kernel.org> Acked-by: Heiko Stuebner <heiko@sntech.de> Link: https://patchwork.freedesktop.org/patch/msgid/20240229162230.2634044-6-boris.brezillon@collabora.com drm/panthor: Add the devfreq logical block Every thing related to devfreq in placed in panthor_devfreq.c, and helpers that can be called by other logical blocks are exposed through panthor_devfreq.h. This implementation is loosely based on the panfrost implementation, the only difference being that we don't count device users, because the idle/active state will be managed by the scheduler logic. v6: - Add Maxime's and Heiko's acks - Keep header inclusion alphabetically ordered v4: - Add Clément's A-b for the relicensing v3: - Add acks for the MIT/GPL2 relicensing v2: - Added in v2 Cc: Clément Péron <peron.clem@gmail.com> # MIT+GPL2 relicensing Reviewed-by: Steven Price <steven.price@arm.com> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Acked-by: Steven Price <steven.price@arm.com> # MIT+GPL2 relicensing,Arm Acked-by: Grant Likely <grant.likely@linaro.org> # MIT+GPL2 relicensing,Linaro Acked-by: Boris Brezillon <boris.brezillon@collabora.com> # MIT+GPL2 relicensing,Collabora Acked-by: Clément Péron <peron.clem@gmail.com> # MIT+GPL2 relicensing Acked-by: Maxime Ripard <mripard@kernel.org> Acked-by: Heiko Stuebner <heiko@sntech.de> Link: https://patchwork.freedesktop.org/patch/msgid/20240229162230.2634044-7-boris.brezillon@collabora.com drm/panthor: Add the MMU/VM logical block MMU and VM management is related and placed in the same source file. Page table updates are delegated to the io-pgtable-arm driver that's in the iommu subsystem. The VM management logic is based on drm_gpuva_mgr, and is assuming the VA space is mostly managed by the usermode driver, except for a reserved portion of this VA-space that's used for kernel objects (like the heap contexts/chunks). Both asynchronous and synchronous VM operations are supported, and internal helpers are exposed to allow other logical blocks to map their buffers in the GPU VA space. There's one VM_BIND queue per-VM (meaning the Vulkan driver can only expose one sparse-binding queue), and this bind queue is managed with a 1:1 drm_sched_entity:drm_gpu_scheduler, such that each VM gets its own independent execution queue, avoiding VM operation serialization at the device level (things are still serialized at the VM level). The rest is just implementation details that are hopefully well explained in the documentation. v6: - Add Maxime's and Heiko's acks - Add Steve's R-b - Adjust the TRANSCFG value to account for SW VA space limitation on 32-bit systems - Keep header inclusion alphabetically ordered v5: - Fix a double panthor_vm_cleanup_op_ctx() call - Fix a race between panthor_vm_prepare_map_op_ctx() and panthor_vm_bo_put() - Fix panthor_vm_pool_destroy_vm() kernel doc - Fix paddr adjustment in panthor_vm_map_pages() - Fix bo_offset calculation in panthor_vm_get_bo_for_va() v4: - Add an helper to return the VM state - Check drmm_mutex_init() return code - Remove the VM from the AS reclaim list when panthor_vm_active() is called - Count the number of active VM users instead of considering there's at most one user (several scheduling groups can point to the same vM) - Pre-allocate a VMA object for unmap operations (unmaps can trigger a sm_step_remap() call) - Check vm->root_page_table instead of vm->pgtbl_ops to detect if the io-pgtable is trying to allocate the root page table - Don't memset() the va_node in panthor_vm_alloc_va(), make it a caller requirement - Fix the kernel doc in a few places - Drop the panthor_vm::base offset constraint and modify panthor_vm_put() to explicitly check for a NULL value - Fix unbalanced vm_bo refcount in panthor_gpuva_sm_step_remap() - Drop stale comments about the shared_bos list - Patch mmu_features::va_bits on 32-bit builds to reflect the io_pgtable limitation and let the UMD know about it v3: - Add acks for the MIT/GPL2 relicensing - Propagate MMU faults to the scheduler - Move pages pinning/unpinning out of the dma_signalling path - Fix 32-bit support - Rework the user/kernel VA range calculation - Make the auto-VA range explicit (auto-VA range doesn't cover the full kernel-VA range on the MCU VM) - Let callers of panthor_vm_alloc_va() allocate the drm_mm_node (embedded in panthor_kernel_bo now) - Adjust things to match the latest drm_gpuvm changes (extobj tracking, resv prep and more) - Drop the per-AS lock and use slots_lock (fixes a race on vm->as.id) - Set as.id to -1 when reusing an address space from the LRU list - Drop misleading comment about page faults - Remove check for irq being assigned in panthor_mmu_unplug() Co-developed-by: Steven Price <steven.price@arm.com> Signed-off-by: Steven Price <steven.price@arm.com> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Acked-by: Steven Price <steven.price@arm.com> # MIT+GPL2 relicensing,Arm Acked-by: Grant Likely <grant.likely@linaro.org> # MIT+GPL2 relicensing,Linaro Acked-by: Boris Brezillon <boris.brezillon@collabora.com> # MIT+GPL2 relicensing,Collabora Reviewed-by: Steven Price <steven.price@arm.com> Acked-by: Maxime Ripard <mripard@kernel.org> Acked-by: Heiko Stuebner <heiko@sntech.de> Link: https://patchwork.freedesktop.org/patch/msgid/20240229162230.2634044-8-boris.brezillon@collabora.com drm/panthor: Add the FW logical block Contains everything that's FW related, that includes the code dealing with the microcontroller unit (MCU) that's running the FW, and anything related to allocating memory shared between the FW and the CPU. A few global FW events are processed in the IRQ handler, the rest is forwarded to the scheduler, since scheduling is the primary reason for the FW existence, and also the main source of FW <-> kernel interactions. v6: - Add Maxime's and Heiko's acks - Keep header inclusion alphabetically ordered v5: - Fix typo in GLB_PERFCNT_SAMPLE definition - Fix unbalanced panthor_vm_idle/active() calls - Fallback to a slow reset when the fast reset fails - Add extra information when reporting a FW boot failure v4: - Add a MODULE_FIRMWARE() entry for gen 10.8 - Fix a wrong return ERR_PTR() in panthor_fw_load_section_entry() - Fix typos - Add Steve's R-b v3: - Make the FW path more future-proof (Liviu) - Use one waitqueue for all FW events - Simplify propagation of FW events to the scheduler logic - Drop the panthor_fw_mem abstraction and use panthor_kernel_bo instead - Account for the panthor_vm changes - Replace magic number with 0x7fffffff with ~0 to better signify that it's the maximum permitted value. - More accurate rounding when computing the firmware timeout. - Add a 'sub iterator' helper function. This also adds a check that a firmware entry doesn't overflow the firmware image. - Drop __packed from FW structures, natural alignment is good enough. - Other minor code improvements. Co-developed-by: Steven Price <steven.price@arm.com> Signed-off-by: Steven Price <steven.price@arm.com> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Steven Price <steven.price@arm.com> Acked-by: Maxime Ripard <mripard@kernel.org> Acked-by: Heiko Stuebner <heiko@sntech.de> Link: https://patchwork.freedesktop.org/patch/msgid/20240229162230.2634044-9-boris.brezillon@collabora.com drm/panthor: Add the heap logical block Tiler heap growing requires some kernel driver involvement: when the tiler runs out of heap memory, it will raise an exception which is either directly handled by the firmware if some free heap chunks are available in the heap context, or passed back to the kernel otherwise. The heap helpers will be used by the scheduler logic to allocate more heap chunks to a heap context, when such a situation happens. Heap context creation is explicitly requested by userspace (using the TILER_HEAP_CREATE ioctl), and the returned context is attached to a queue through some command stream instruction. All the kernel does is keep the list of heap chunks allocated to a context, so they can be freed when TILER_HEAP_DESTROY is called, or extended when the FW requests a new chunk. v6: - Add Maxime's and Heiko's acks v5: - Fix FIXME comment - Add Steve's R-b v4: - Rework locking to allow concurrent calls to panthor_heap_grow() - Add a helper to return a heap chunk if we couldn't pass it to the FW because the group was scheduled out v3: - Add a FIXME for the heap OOM deadlock - Use the panthor_kernel_bo abstraction for the heap context and heap chunks - Drop the panthor_heap_gpu_ctx struct as it is opaque to the driver - Ensure that the heap context is aligned to the GPU cache line size - Minor code tidy ups Co-developed-by: Steven Price <steven.price@arm.com> Signed-off-by: Steven Price <steven.price@arm.com> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Steven Price <steven.price@arm.com> Acked-by: Maxime Ripard <mripard@kernel.org> Acked-by: Heiko Stuebner <heiko@sntech.de> Link: https://patchwork.freedesktop.org/patch/msgid/20240229162230.2634044-10-boris.brezillon@collabora.com drm/panthor: Add the scheduler logical block This is the piece of software interacting with the FW scheduler, and taking care of some scheduling aspects when the FW comes short of slots scheduling slots. Indeed, the FW only expose a few slots, and the kernel has to give all submission contexts, a chance to execute their jobs. The kernel-side scheduler is timeslice-based, with a round-robin queue per priority level. Job submission is handled with a 1:1 drm_sched_entity:drm_gpu_scheduler, allowing us to delegate the dependency tracking to the core. All the gory details should be documented inline. v6: - Add Maxime's and Heiko's acks - Make sure the scheduler is initialized before queueing the tick work in the MMU fault handler - Keep header inclusion alphabetically ordered v5: - Fix typos - Call panthor_kernel_bo_destroy(group->syncobjs) unconditionally - Don't move the group to the waiting list tail when it was already waiting for a different syncobj - Fix fatal_queues flagging in the tiler OOM path - Don't warn when more than one job timesout on a group - Add a warning message when we fail to allocate a heap chunk - Add Steve's R-b v4: - Check drmm_mutex_init() return code - s/drm_gem_vmap_unlocked/drm_gem_vunmap_unlocked/ in panthor_queue_put_syncwait_obj() - Drop unneeded WARN_ON() in cs_slot_sync_queue_state_locked() - Use atomic_xchg() instead of atomic_fetch_and(0) - Fix typos - Let panthor_kernel_bo_destroy() check for IS_ERR_OR_NULL() BOs - Defer TILER_OOM event handling to a separate workqueue to prevent deadlocks when the heap chunk allocation is blocked on mem-reclaim. This is just a temporary solution, until we add support for non-blocking/failable allocations - Pass the scheduler workqueue to drm_sched instead of instantiating a separate one (no longer needed now that heap chunk allocation happens on a dedicated wq) - Set WQ_MEM_RECLAIM on the scheduler workqueue, so we can handle job timeouts when the system is under mem pressure, and hopefully free up some memory retained by these jobs v3: - Rework the FW event handling logic to avoid races - Make sure MMU faults kill the group immediately - Use the panthor_kernel_bo abstraction for group/queue buffers - Make in_progress an atomic_t, so we can check it without the reset lock held - Don't limit the number of groups per context to the FW scheduler capacity. Fix the limit to 128 for now. - Add a panthor_job_vm() helper - Account for panthor_vm changes - Add our job fence as DMA_RESV_USAGE_WRITE to all external objects (was previously DMA_RESV_USAGE_BOOKKEEP). I don't get why, given we're supposed to be fully-explicit, but other drivers do that, so there must be a good reason - Account for drm_sched changes - Provide a panthor_queue_put_syncwait_obj() - Unconditionally return groups to their idle list in panthor_sched_suspend() - Condition of sched_queue_{,delayed_}work fixed to be only when a reset isn't pending or in progress. - Several typos in comments fixed. Co-developed-by: Steven Price <steven.price@arm.com> Signed-off-by: Steven Price <steven.price@arm.com> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Steven Price <steven.price@arm.com> Acked-by: Maxime Ripard <mripard@kernel.org> Acked-by: Heiko Stuebner <heiko@sntech.de> Link: https://patchwork.freedesktop.org/patch/msgid/20240229162230.2634044-11-boris.brezillon@collabora.com drm/panthor: Add the driver frontend block This is the last piece missing to expose the driver to the outside world. This is basically a wrapper between the ioctls and the other logical blocks. v6: - Add Maxime's and Heiko's acks - Return a page-aligned BO size to userspace - Keep header inclusion alphabetically ordered v5: - Account for the drm_exec_init() prototype change - Include platform_device.h v4: - Add an ioctl to let the UMD query the VM state - Fix kernel doc - Let panthor_device_init() call panthor_device_init() - Fix cleanup ordering in the panthor_init() error path - Add Steve's and Liviu's R-b v3: - Add acks for the MIT/GPL2 relicensing - Fix 32-bit support - Account for panthor_vm and panthor_sched changes - Simplify the resv preparation/update logic - Use a linked list rather than xarray for list of signals. - Simplify panthor_get_uobj_array by returning the newly allocated array. - Drop the "DOC" for job submission helpers and move the relevant comments to panthor_ioctl_group_submit(). - Add helpers sync_op_is_signal()/sync_op_is_wait(). - Simplify return type of panthor_submit_ctx_add_sync_signal() and panthor_submit_ctx_get_sync_signal(). - Drop WARN_ON from panthor_submit_ctx_add_job(). - Fix typos in comments. Co-developed-by: Steven Price <steven.price@arm.com> Signed-off-by: Steven Price <steven.price@arm.com> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Acked-by: Steven Price <steven.price@arm.com> # MIT+GPL2 relicensing,Arm Acked-by: Grant Likely <grant.likely@linaro.org> # MIT+GPL2 relicensing,Linaro Acked-by: Boris Brezillon <boris.brezillon@collabora.com> # MIT+GPL2 relicensing,Collabora Reviewed-by: Steven Price <steven.price@arm.com> Reviewed-by: Liviu Dudau <liviu.dudau@arm.com> Acked-by: Maxime Ripard <mripard@kernel.org> Acked-by: Heiko Stuebner <heiko@sntech.de> Link: https://patchwork.freedesktop.org/patch/msgid/20240229162230.2634044-12-boris.brezillon@collabora.com drm/panthor: Allow driver compilation Now that all blocks are available, we can add/update Kconfig/Makefile files to allow compilation. v6: - Add Maxime's and Heiko's acks - Keep source files alphabetically ordered in the Makefile v4: - Add Steve's R-b v3: - Add a dep on DRM_GPUVM - Fix dependencies in Kconfig - Expand help text to (hopefully) describe which GPUs are to be supported by this driver and which are for panfrost. Co-developed-by: Steven Price <steven.price@arm.com> Signed-off-by: Steven Price <steven.price@arm.com> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Acked-by: Steven Price <steven.price@arm.com> # MIT+GPL2 relicensing,Arm Acked-by: Grant Likely <grant.likely@linaro.org> # MIT+GPL2 relicensing,Linaro Acked-by: Boris Brezillon <boris.brezillon@collabora.com> # MIT+GPL2 relicensing,Collabora Reviewed-by: Steven Price <steven.price@arm.com> Acked-by: Maxime Ripard <mripard@kernel.org> Acked-by: Heiko Stuebner <heiko@sntech.de> Link: https://patchwork.freedesktop.org/patch/msgid/20240229162230.2634044-13-boris.brezillon@collabora.com dt-bindings: gpu: mali-valhall-csf: Add support for Arm Mali CSF GPUs Arm has introduced a new v10 GPU architecture that replaces the Job Manager interface with a new Command Stream Frontend. It adds firmware driven command stream queues that can be used by kernel and user space to submit jobs to the GPU. Add the initial schema for the device tree that is based on support for RK3588 SoC. The minimum number of clocks is one for the IP, but on Rockchip platforms they will tend to expose the semi-independent clocks for better power management. v6: - Add Maxime's and Heiko's acks v5: - Move the opp-table node under the gpu node v4: - Fix formatting issue v3: - Cleanup commit message to remove redundant text - Added opp-table property and re-ordered entries - Clarified power-domains and power-domain-names requirements for RK3588. - Cleaned up example Note: power-domains and power-domain-names requirements for other platforms are still work in progress, hence the bindings are left incomplete here. v2: - New commit Signed-off-by: Liviu Dudau <liviu.dudau@arm.com> Cc: Krzysztof Kozlowski <krzysztof.kozlowski+dt@linaro.org> Cc: Rob Herring <robh+dt@kernel.org> Cc: Conor Dooley <conor+dt@kernel.org> Cc: devicetree@vger.kernel.org Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Rob Herring <robh@kernel.org> Acked-by: Maxime Ripard <mripard@kernel.org> Acked-by: Heiko Stuebner <heiko@sntech.de> Link: https://patchwork.freedesktop.org/patch/msgid/20240229162230.2634044-14-boris.brezillon@collabora.com drm/panthor: Add an entry to MAINTAINERS Add an entry for the Panthor driver to the MAINTAINERS file. v6: - Add Maxime's and Heiko's acks v4: - Add Steve's R-b v3: - Add bindings document as an 'F:' line. - Add Steven and Liviu as co-maintainers. Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Steven Price <steven.price@arm.com> Acked-by: Maxime Ripard <mripard@kernel.org> Acked-by: Heiko Stuebner <heiko@sntech.de> Link: https://patchwork.freedesktop.org/patch/msgid/20240229162230.2634044-15-boris.brezillon@collabora.com drm/panthor: remove debugfs dma-buf/dma-fence: Add deadline awareness Add a way to hint to the fence signaler of an upcoming deadline, such as vblank, which the fence waiter would prefer not to miss. This is to aid the fence signaler in making power management decisions, like boosting frequency as the deadline approaches and awareness of missing deadlines so that can be factored in to the frequency scaling. v2: Drop dma_fence::deadline and related logic to filter duplicate deadlines, to avoid increasing dma_fence size. The fence-context implementation will need similar logic to track deadlines of all the fences on the same timeline. [ckoenig] v3: Clarify locking wrt. set_deadline callback v4: Clarify in docs comment that this is a hint v5: Drop DMA_FENCE_FLAG_HAS_DEADLINE_BIT. v6: More docs v7: Fix typo, clarify past deadlines Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Christian König <christian.koenig@amd.com> Acked-by: Pekka Paalanen <pekka.paalanen@collabora.com> Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com> drm/gem: Take reservation lock for vmap/vunmap operations The new common dma-buf locking convention will require buffer importers to hold the reservation lock around mapping operations. Make DRM GEM core to take the lock around the vmapping operations and update DRM drivers to use the locked functions for the case where DRM core now holds the lock. This patch prepares DRM core and drivers to the common dynamic dma-buf locking convention. Acked-by: Christian König <christian.koenig@amd.com> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Link: https://patchwork.freedesktop.org/patch/msgid/20221017172229.42269-4-dmitry.osipenko@collabora.com dma-buf: Add unlocked variant of vmapping functions Add unlocked variant of dma_buf_vmap/vunmap() that will be utilized by drivers that don't take the reservation lock explicitly. Acked-by: Sumit Semwal <sumit.semwal@linaro.org> Acked-by: Christian König <christian.koenig@amd.com> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Link: https://patchwork.freedesktop.org/patch/msgid/20221017172229.42269-2-dmitry.osipenko@collabora.com dma-buf: Add unlocked variant of attachment-mapping functions Add unlocked variant of dma_buf_map/unmap_attachment() that will be used by drivers that don't take the reservation lock explicitly. Acked-by: Sumit Semwal <sumit.semwal@linaro.org> Acked-by: Christian König <christian.koenig@amd.com> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Link: https://patchwork.freedesktop.org/patch/msgid/20221017172229.42269-3-dmitry.osipenko@collabora.com dma-buf: Move dma_buf_vmap() to dynamic locking specification Move dma_buf_vmap/vunmap() functions to the dynamic locking specification by asserting that the reservation lock is held. Acked-by: Sumit Semwal <sumit.semwal@linaro.org> Acked-by: Christian König <christian.koenig@amd.com> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Link: https://patchwork.freedesktop.org/patch/msgid/20221017172229.42269-16-dmitry.osipenko@collabora.com drm/gpuvm: Helper to get range of unmap from a remap op. Determining the start and range of the unmap stage of a remap op is a common piece of code currently implemented by multiple drivers. Add a helper for this. Changes since v7: - Renamed helper to drm_gpuva_op_remap_to_unmap_range() - Improved documentation Changes since v6: - Remove use of __always_inline Signed-off-by: Donald Robson <donald.robson@imgtec.com> Signed-off-by: Sarah Walker <sarah.walker@imgtec.com> Reviewed-by: Danilo Krummrich <dakr@redhat.com> Link: https://lore.kernel.org/r/8a0a5b5eeec459d3c60fcdaa5a638ad14a18a59e.1700668843.git.donald.robson@imgtec.com Signed-off-by: Maxime Ripard <mripard@kernel.org> drm: Enable PRIME import/export for all drivers Call drm_gem_prime_handle_to_fd() and drm_gem_prime_fd_to_handle() by default if no PRIME import/export helpers have been set. Both functions are the default for almost all drivers. DRM drivers implement struct drm_driver.gem_prime_import_sg_table to import dma-buf objects from other drivers. Having the function drm_gem_prime_fd_to_handle() functions set by default allows each driver to import dma-buf objects to itself, even without support for other drivers. For drm_gem_prime_handle_to_fd() it is similar: using it by default allows each driver to export to itself, even without support for other drivers. This functionality enables userspace to share per-driver buffers across process boundaries via PRIME (e.g., wlroots requires this functionality). The patch generalizes a pattern that has previously been implemented by GEM VRAM helpers [1] to work with any driver. For example, gma500 can now run the wlroots-based sway compositor. v2: * clean up docs and TODO comments (Simon, Zack) * clean up style in drm_getcap() Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de> Link: https://lore.kernel.org/dri-devel/20230302143502.500661-1-contact@emersion.fr/ # 1 Reviewed-by: Simon Ser <contact@emersion.fr> Acked-by: Alex Deucher <alexander.deucher@amd.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Link: https://patchwork.freedesktop.org/patch/msgid/20230620080252.16368-2-tzimmermann@suse.de drm: Clear fd/handle callbacks in struct drm_driver Clear all assignments of struct drm_driver's fd/handle callbacks to drm_gem_prime_fd_to_handle() and drm_gem_prime_handle_to_fd(). These functions are called by default. Add a TODO item to convert vmwgfx to the defaults as well. v2: * remove TODO item (Zack) * also update amdgpu's amdgpu_partition_driver Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de> Reviewed-by: Simon Ser <contact@emersion.fr> Acked-by: Alex Deucher <alexander.deucher@amd.com> Acked-by: Jeffrey Hugo <quic_jhugo@quicinc.com> # qaic Link: https://patchwork.freedesktop.org/patch/msgid/20230620080252.16368-3-tzimmermann@suse.de drm/panthor: Fix IO-page mmap() for 32-bit userspace on 64-bit kernel When mapping an IO region, the pseudo-file offset is dependent on the userspace architecture. panthor_device_mmio_offset() abstracts that away for us by turning a userspace MMIO offset into its kernel equivalent, but we were not updating vm_area_struct::vm_pgoff accordingly, leading us to attach the MMIO region to the wrong file offset. This has implications when we start mixing 64 bit and 32 bit apps, but that's only really a problem when we start having more that 2^43 bytes of memory allocated, which is very unlikely to happen. What's more problematic is the fact this turns our unmap_mapping_range(DRM_PANTHOR_USER_MMIO_OFFSET) calls, which are supposed to kill the MMIO mapping when entering suspend, into NOPs. Which means we either keep the dummy flush_id mapping active at all times, or we risk a BUS_FAULT if the MMIO region was mapped, and the GPU is suspended after that. Solve that by patching vm_pgoff early in panthor_mmap(). With this in place, we no longer need the panthor_device_mmio_offset() helper. v3: - No changes v2: - Kill panthor_device_mmio_offset() Fixes: 5fe909cae118 ("drm/panthor: Add the device logical block") Reported-by: Adrián Larumbe <adrian.larumbe@collabora.com> Reported-by: Lukas F. Hartmann <lukas@mntmn.com> Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/10835 Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Steven Price <steven.price@arm.com> Reviewed-by: Liviu Dudau <liviu.dudau@arm.com> drm/panthor: Fix ordering in _irq_suspend() Make sure we set suspended=true last to avoid generating an irq storm in the unlikely case where an IRQ happens between the suspended=true assignment and the _INT_MASK update. We also move the mask=0 assignment before writing to the _INT_MASK register to prevent the thread handler from unmasking the interrupt behind our back. This means we might lose events if there were some pending when we get to suspend the IRQ, but that's fine. The synchronize_irq() we have in the _irq_suspend() path was not there to make sure all IRQs are processed, just to make sure we don't have registers accesses coming from the irq handlers after _irq_suspend() has been called. If there's a need to have all pending IRQs processed, it should happen before _irq_suspend() is called. v3: - Add Steve's R-b v2: - New patch Fixes: 5fe909cae118 ("drm/panthor: Add the device logical block") Reported-by: Steven Price <steven.price@arm.com> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Steven Price <steven.price@arm.com> Acked-by: Liviu Dudau <liviu.dudau@arm.com> drm/panthor: Drop the dev_enter/exit() sections in _irq_suspend/resume() There's no reason for _irq_suspend/resume() to be called after the device has been unplugged, and keeping this dev_enter/exit() section in _irq_suspend() is turns _irq_suspend() into a NOP when called from the _unplug() functions, which we don't want. v3: - New patch Fixes: 5fe909cae118 ("drm/panthor: Add the device logical block") Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com> Reviewed-by: Liviu Dudau <liviu.dudau@arm.com> Reviewed-by: Steven Price <steven.price@arm.com>
This commit is contained in:
Liviu Dudau
Mecid Urganci
parent
ce45bd9c03
commit
c4f16b4db2
100 changed files with 19998 additions and 893 deletions
147
Documentation/devicetree/bindings/gpu/arm,mali-valhall-csf.yaml
Normal file
147
Documentation/devicetree/bindings/gpu/arm,mali-valhall-csf.yaml
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause
|
||||
%YAML 1.2
|
||||
---
|
||||
$id: http://devicetree.org/schemas/gpu/arm,mali-valhall-csf.yaml#
|
||||
$schema: http://devicetree.org/meta-schemas/core.yaml#
|
||||
|
||||
title: ARM Mali Valhall GPU
|
||||
|
||||
maintainers:
|
||||
- Liviu Dudau <liviu.dudau@arm.com>
|
||||
- Boris Brezillon <boris.brezillon@collabora.com>
|
||||
|
||||
properties:
|
||||
$nodename:
|
||||
pattern: '^gpu@[a-f0-9]+$'
|
||||
|
||||
compatible:
|
||||
oneOf:
|
||||
- items:
|
||||
- enum:
|
||||
- rockchip,rk3588-mali
|
||||
- const: arm,mali-valhall-csf # Mali Valhall GPU model/revision is fully discoverable
|
||||
|
||||
reg:
|
||||
maxItems: 1
|
||||
|
||||
interrupts:
|
||||
items:
|
||||
- description: Job interrupt
|
||||
- description: MMU interrupt
|
||||
- description: GPU interrupt
|
||||
|
||||
interrupt-names:
|
||||
items:
|
||||
- const: job
|
||||
- const: mmu
|
||||
- const: gpu
|
||||
|
||||
clocks:
|
||||
minItems: 1
|
||||
maxItems: 3
|
||||
|
||||
clock-names:
|
||||
minItems: 1
|
||||
items:
|
||||
- const: core
|
||||
- const: coregroup
|
||||
- const: stacks
|
||||
|
||||
mali-supply: true
|
||||
|
||||
operating-points-v2: true
|
||||
opp-table:
|
||||
type: object
|
||||
|
||||
power-domains:
|
||||
minItems: 1
|
||||
maxItems: 5
|
||||
|
||||
power-domain-names:
|
||||
minItems: 1
|
||||
maxItems: 5
|
||||
|
||||
sram-supply: true
|
||||
|
||||
"#cooling-cells":
|
||||
const: 2
|
||||
|
||||
dynamic-power-coefficient:
|
||||
$ref: /schemas/types.yaml#/definitions/uint32
|
||||
description:
|
||||
A u32 value that represents the running time dynamic
|
||||
power coefficient in units of uW/MHz/V^2. The
|
||||
coefficient can either be calculated from power
|
||||
measurements or derived by analysis.
|
||||
|
||||
The dynamic power consumption of the GPU is
|
||||
proportional to the square of the Voltage (V) and
|
||||
the clock frequency (f). The coefficient is used to
|
||||
calculate the dynamic power as below -
|
||||
|
||||
Pdyn = dynamic-power-coefficient * V^2 * f
|
||||
|
||||
where voltage is in V, frequency is in MHz.
|
||||
|
||||
dma-coherent: true
|
||||
|
||||
required:
|
||||
- compatible
|
||||
- reg
|
||||
- interrupts
|
||||
- interrupt-names
|
||||
- clocks
|
||||
- mali-supply
|
||||
|
||||
additionalProperties: false
|
||||
|
||||
allOf:
|
||||
- if:
|
||||
properties:
|
||||
compatible:
|
||||
contains:
|
||||
const: rockchip,rk3588-mali
|
||||
then:
|
||||
properties:
|
||||
clocks:
|
||||
minItems: 3
|
||||
power-domains:
|
||||
maxItems: 1
|
||||
power-domain-names: false
|
||||
|
||||
examples:
|
||||
- |
|
||||
#include <dt-bindings/clock/rockchip,rk3588-cru.h>
|
||||
#include <dt-bindings/interrupt-controller/irq.h>
|
||||
#include <dt-bindings/interrupt-controller/arm-gic.h>
|
||||
#include <dt-bindings/power/rk3588-power.h>
|
||||
|
||||
gpu: gpu@fb000000 {
|
||||
compatible = "rockchip,rk3588-mali", "arm,mali-valhall-csf";
|
||||
reg = <0xfb000000 0x200000>;
|
||||
interrupts = <GIC_SPI 92 IRQ_TYPE_LEVEL_HIGH 0>,
|
||||
<GIC_SPI 93 IRQ_TYPE_LEVEL_HIGH 0>,
|
||||
<GIC_SPI 94 IRQ_TYPE_LEVEL_HIGH 0>;
|
||||
interrupt-names = "job", "mmu", "gpu";
|
||||
clock-names = "core", "coregroup", "stacks";
|
||||
clocks = <&cru CLK_GPU>, <&cru CLK_GPU_COREGROUP>,
|
||||
<&cru CLK_GPU_STACKS>;
|
||||
power-domains = <&power RK3588_PD_GPU>;
|
||||
operating-points-v2 = <&gpu_opp_table>;
|
||||
mali-supply = <&vdd_gpu_s0>;
|
||||
sram-supply = <&vdd_gpu_mem_s0>;
|
||||
|
||||
gpu_opp_table: opp-table {
|
||||
compatible = "operating-points-v2";
|
||||
opp-300000000 {
|
||||
opp-hz = /bits/ 64 <300000000>;
|
||||
opp-microvolt = <675000 675000 850000>;
|
||||
};
|
||||
opp-400000000 {
|
||||
opp-hz = /bits/ 64 <400000000>;
|
||||
opp-microvolt = <675000 675000 850000>;
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
...
|
||||
|
|
@ -158,6 +158,12 @@ DMA Fence Signalling Annotations
|
|||
.. kernel-doc:: drivers/dma-buf/dma-fence.c
|
||||
:doc: fence signalling annotation
|
||||
|
||||
DMA Fence Deadline Hints
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. kernel-doc:: drivers/dma-buf/dma-fence.c
|
||||
:doc: deadline hints
|
||||
|
||||
DMA Fences Functions Reference
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
|
|
|
|||
|
|
@ -6,3 +6,8 @@ drm/i915 uAPI
|
|||
=============
|
||||
|
||||
.. kernel-doc:: include/uapi/drm/i915_drm.h
|
||||
|
||||
drm/panthor uAPI
|
||||
================
|
||||
|
||||
.. kernel-doc:: i
|
||||
|
|
@ -466,6 +466,42 @@ DRM MM Range Allocator Function References
|
|||
.. kernel-doc:: drivers/gpu/drm/drm_mm.c
|
||||
:export:
|
||||
|
||||
DRM GPU VA Manager
|
||||
==================
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
.. kernel-doc:: drivers/gpu/drm/drm_gpuva_mgr.c
|
||||
:doc: Overview
|
||||
|
||||
Split and Merge
|
||||
---------------
|
||||
|
||||
.. kernel-doc:: drivers/gpu/drm/drm_gpuva_mgr.c
|
||||
:doc: Split and Merge
|
||||
|
||||
Locking
|
||||
-------
|
||||
|
||||
.. kernel-doc:: drivers/gpu/drm/drm_gpuva_mgr.c
|
||||
:doc: Locking
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
.. kernel-doc:: drivers/gpu/drm/drm_gpuva_mgr.c
|
||||
:doc: Examples
|
||||
|
||||
DRM GPU VA Manager Function References
|
||||
--------------------------------------
|
||||
|
||||
.. kernel-doc:: include/drm/drm_gpuva_mgr.h
|
||||
:internal:
|
||||
|
||||
.. kernel-doc:: drivers/gpu/drm/drm_gpuva_mgr.c
|
||||
:export:
|
||||
|
||||
DRM Buddy Allocator
|
||||
===================
|
||||
|
||||
|
|
@ -493,6 +529,18 @@ DRM Sync Objects
|
|||
.. kernel-doc:: drivers/gpu/drm/drm_syncobj.c
|
||||
:export:
|
||||
|
||||
DRM Execution context
|
||||
=====================
|
||||
|
||||
.. kernel-doc:: drivers/gpu/drm/drm_exec.c
|
||||
:doc: Overview
|
||||
|
||||
.. kernel-doc:: include/drm/drm_exec.h
|
||||
:internal:
|
||||
|
||||
.. kernel-doc:: drivers/gpu/drm/drm_exec.c
|
||||
:export:
|
||||
|
||||
GPU Scheduler
|
||||
=============
|
||||
|
||||
|
|
@ -502,6 +550,12 @@ Overview
|
|||
.. kernel-doc:: drivers/gpu/drm/scheduler/sched_main.c
|
||||
:doc: Overview
|
||||
|
||||
Flow Control
|
||||
------------
|
||||
|
||||
.. kernel-doc:: drivers/gpu/drm/scheduler/sched_main.c
|
||||
:doc: Flow Control
|
||||
|
||||
Scheduler Function References
|
||||
-----------------------------
|
||||
|
||||
|
|
|
|||
11
MAINTAINERS
11
MAINTAINERS
|
|
@ -1622,6 +1622,17 @@ T: git git://anongit.freedesktop.org/drm/drm-misc
|
|||
F: drivers/gpu/drm/panfrost/
|
||||
F: include/uapi/drm/panfrost_drm.h
|
||||
|
||||
ARM MALI PANTHOR DRM DRIVER
|
||||
M: Boris Brezillon <boris.brezillon@collabora.com>
|
||||
M: Steven Price <steven.price@arm.com>
|
||||
M: Liviu Dudau <liviu.dudau@arm.com>
|
||||
L: dri-devel@lists.freedesktop.org
|
||||
S: Supported
|
||||
T: git git://anongit.freedesktop.org/drm/drm-misc
|
||||
F: Documentation/devicetree/bindings/gpu/arm,mali-valhall-csf.yaml
|
||||
F: drivers/gpu/drm/panthor/
|
||||
F: include/uapi/drm/panthor_drm.h
|
||||
|
||||
ARM MALI-DP DRM DRIVER
|
||||
M: Liviu Dudau <liviu.dudau@arm.com>
|
||||
M: Brian Starkey <brian.starkey@arm.com>
|
||||
|
|
|
|||
|
|
@ -1667,6 +1667,8 @@ int dma_buf_vmap(struct dma_buf *dmabuf, struct iosys_map *map)
|
|||
if (WARN_ON(!dmabuf))
|
||||
return -EINVAL;
|
||||
|
||||
dma_resv_assert_held(dmabuf->resv);
|
||||
|
||||
if (!dmabuf->ops->vmap)
|
||||
return -EINVAL;
|
||||
|
||||
|
|
@ -1695,6 +1697,33 @@ out_unlock:
|
|||
}
|
||||
EXPORT_SYMBOL_NS_GPL(dma_buf_vmap, DMA_BUF);
|
||||
|
||||
/**
|
||||
* dma_buf_vmap_unlocked - Create virtual mapping for the buffer object into kernel
|
||||
* address space. Same restrictions as for vmap and friends apply.
|
||||
* @dmabuf: [in] buffer to vmap
|
||||
* @map: [out] returns the vmap pointer
|
||||
*
|
||||
* Unlocked version of dma_buf_vmap()
|
||||
*
|
||||
* Returns 0 on success, or a negative errno code otherwise.
|
||||
*/
|
||||
int dma_buf_vmap_unlocked(struct dma_buf *dmabuf, struct iosys_map *map)
|
||||
{
|
||||
int ret;
|
||||
|
||||
iosys_map_clear(map);
|
||||
|
||||
if (WARN_ON(!dmabuf))
|
||||
return -EINVAL;
|
||||
|
||||
dma_resv_lock(dmabuf->resv, NULL);
|
||||
ret = dma_buf_vmap(dmabuf, map);
|
||||
dma_resv_unlock(dmabuf->resv);
|
||||
|
||||
return ret;
|
||||
}
|
||||
EXPORT_SYMBOL_NS_GPL(dma_buf_vmap_unlocked, DMA_BUF);
|
||||
|
||||
/**
|
||||
* dma_buf_vunmap - Unmap a vmap obtained by dma_buf_vmap.
|
||||
* @dmabuf: [in] buffer to vunmap
|
||||
|
|
@ -1705,6 +1734,8 @@ void dma_buf_vunmap(struct dma_buf *dmabuf, struct iosys_map *map)
|
|||
if (WARN_ON(!dmabuf))
|
||||
return;
|
||||
|
||||
dma_resv_assert_held(dmabuf->resv);
|
||||
|
||||
BUG_ON(iosys_map_is_null(&dmabuf->vmap_ptr));
|
||||
BUG_ON(dmabuf->vmapping_counter == 0);
|
||||
BUG_ON(!iosys_map_is_equal(&dmabuf->vmap_ptr, map));
|
||||
|
|
@ -1719,6 +1750,22 @@ void dma_buf_vunmap(struct dma_buf *dmabuf, struct iosys_map *map)
|
|||
}
|
||||
EXPORT_SYMBOL_NS_GPL(dma_buf_vunmap, DMA_BUF);
|
||||
|
||||
/**
|
||||
* dma_buf_vunmap_unlocked - Unmap a vmap obtained by dma_buf_vmap.
|
||||
* @dmabuf: [in] buffer to vunmap
|
||||
* @map: [in] vmap pointer to vunmap
|
||||
*/
|
||||
void dma_buf_vunmap_unlocked(struct dma_buf *dmabuf, struct iosys_map *map)
|
||||
{
|
||||
if (WARN_ON(!dmabuf))
|
||||
return;
|
||||
|
||||
dma_resv_lock(dmabuf->resv, NULL);
|
||||
dma_buf_vunmap(dmabuf, map);
|
||||
dma_resv_unlock(dmabuf->resv);
|
||||
}
|
||||
EXPORT_SYMBOL_NS_GPL(dma_buf_vunmap_unlocked, DMA_BUF);
|
||||
|
||||
#ifdef CONFIG_DEBUG_FS
|
||||
static int dma_buf_debug_show(struct seq_file *s, void *unused)
|
||||
{
|
||||
|
|
|
|||
|
|
@ -913,6 +913,65 @@ err_free_cb:
|
|||
}
|
||||
EXPORT_SYMBOL(dma_fence_wait_any_timeout);
|
||||
|
||||
/**
|
||||
* DOC: deadline hints
|
||||
*
|
||||
* In an ideal world, it would be possible to pipeline a workload sufficiently
|
||||
* that a utilization based device frequency governor could arrive at a minimum
|
||||
* frequency that meets the requirements of the use-case, in order to minimize
|
||||
* power consumption. But in the real world there are many workloads which
|
||||
* defy this ideal. For example, but not limited to:
|
||||
*
|
||||
* * Workloads that ping-pong between device and CPU, with alternating periods
|
||||
* of CPU waiting for device, and device waiting on CPU. This can result in
|
||||
* devfreq and cpufreq seeing idle time in their respective domains and in
|
||||
* result reduce frequency.
|
||||
*
|
||||
* * Workloads that interact with a periodic time based deadline, such as double
|
||||
* buffered GPU rendering vs vblank sync'd page flipping. In this scenario,
|
||||
* missing a vblank deadline results in an *increase* in idle time on the GPU
|
||||
* (since it has to wait an additional vblank period), sending a signal to
|
||||
* the GPU's devfreq to reduce frequency, when in fact the opposite is what is
|
||||
* needed.
|
||||
*
|
||||
* To this end, deadline hint(s) can be set on a &dma_fence via &dma_fence_set_deadline.
|
||||
* The deadline hint provides a way for the waiting driver, or userspace, to
|
||||
* convey an appropriate sense of urgency to the signaling driver.
|
||||
*
|
||||
* A deadline hint is given in absolute ktime (CLOCK_MONOTONIC for userspace
|
||||
* facing APIs). The time could either be some point in the future (such as
|
||||
* the vblank based deadline for page-flipping, or the start of a compositor's
|
||||
* composition cycle), or the current time to indicate an immediate deadline
|
||||
* hint (Ie. forward progress cannot be made until this fence is signaled).
|
||||
*
|
||||
* Multiple deadlines may be set on a given fence, even in parallel. See the
|
||||
* documentation for &dma_fence_ops.set_deadline.
|
||||
*
|
||||
* The deadline hint is just that, a hint. The driver that created the fence
|
||||
* may react by increasing frequency, making different scheduling choices, etc.
|
||||
* Or doing nothing at all.
|
||||
*/
|
||||
|
||||
/**
|
||||
* dma_fence_set_deadline - set desired fence-wait deadline hint
|
||||
* @fence: the fence that is to be waited on
|
||||
* @deadline: the time by which the waiter hopes for the fence to be
|
||||
* signaled
|
||||
*
|
||||
* Give the fence signaler a hint about an upcoming deadline, such as
|
||||
* vblank, by which point the waiter would prefer the fence to be
|
||||
* signaled by. This is intended to give feedback to the fence signaler
|
||||
* to aid in power management decisions, such as boosting GPU frequency
|
||||
* if a periodic vblank deadline is approaching but the fence is not
|
||||
* yet signaled..
|
||||
*/
|
||||
void dma_fence_set_deadline(struct dma_fence *fence, ktime_t deadline)
|
||||
{
|
||||
if (fence->ops->set_deadline && !dma_fence_is_signaled(fence))
|
||||
fence->ops->set_deadline(fence, deadline);
|
||||
}
|
||||
EXPORT_SYMBOL(dma_fence_set_deadline);
|
||||
|
||||
/**
|
||||
* dma_fence_describe - Dump fence describtion into seq_file
|
||||
* @fence: the 6fence to describe
|
||||
|
|
|
|||
|
|
@ -207,6 +207,19 @@ config DRM_TTM
|
|||
GPU memory types. Will be enabled automatically if a device driver
|
||||
uses it.
|
||||
|
||||
config DRM_EXEC
|
||||
tristate
|
||||
depends on DRM
|
||||
help
|
||||
Execution context for command submissions
|
||||
|
||||
config DRM_GPUVM
|
||||
tristate
|
||||
depends on DRM
|
||||
help
|
||||
GPU-VM representation providing helpers to manage a GPUs virtual
|
||||
address space
|
||||
|
||||
config DRM_BUDDY
|
||||
tristate
|
||||
depends on DRM
|
||||
|
|
@ -415,6 +428,8 @@ source "drivers/gpu/drm/lima/Kconfig"
|
|||
|
||||
source "drivers/gpu/drm/panfrost/Kconfig"
|
||||
|
||||
source "drivers/gpu/drm/panthor/Kconfig"
|
||||
|
||||
source "drivers/gpu/drm/aspeed/Kconfig"
|
||||
|
||||
source "drivers/gpu/drm/mcde/Kconfig"
|
||||
|
|
|
|||
|
|
@ -40,6 +40,9 @@ obj-$(CONFIG_DRM_PANEL_ORIENTATION_QUIRKS) += drm_panel_orientation_quirks.o
|
|||
#
|
||||
# Memory-management helpers
|
||||
#
|
||||
#
|
||||
obj-$(CONFIG_DRM_EXEC) += drm_exec.o
|
||||
obj-$(CONFIG_DRM_GPUVM) += drm_gpuvm.o
|
||||
|
||||
obj-$(CONFIG_DRM_BUDDY) += drm_buddy.o
|
||||
|
||||
|
|
@ -141,6 +144,7 @@ obj-$(CONFIG_DRM_XEN) += xen/
|
|||
obj-$(CONFIG_DRM_VBOXVIDEO) += vboxvideo/
|
||||
obj-$(CONFIG_DRM_LIMA) += lima/
|
||||
obj-$(CONFIG_DRM_PANFROST) += panfrost/
|
||||
obj-$(CONFIG_DRM_PANTHOR) += panthor/
|
||||
obj-$(CONFIG_DRM_ASPEED_GFX) += aspeed/
|
||||
obj-$(CONFIG_DRM_MCDE) += mcde/
|
||||
obj-$(CONFIG_DRM_TIDSS) += tidss/
|
||||
|
|
|
|||
|
|
@ -68,10 +68,10 @@ amdgpu_ctx_to_drm_sched_prio(int32_t ctx_prio)
|
|||
return DRM_SCHED_PRIORITY_NORMAL;
|
||||
|
||||
case AMDGPU_CTX_PRIORITY_VERY_LOW:
|
||||
return DRM_SCHED_PRIORITY_MIN;
|
||||
return DRM_SCHED_PRIORITY_LOW;
|
||||
|
||||
case AMDGPU_CTX_PRIORITY_LOW:
|
||||
return DRM_SCHED_PRIORITY_MIN;
|
||||
return DRM_SCHED_PRIORITY_LOW;
|
||||
|
||||
case AMDGPU_CTX_PRIORITY_NORMAL:
|
||||
return DRM_SCHED_PRIORITY_NORMAL;
|
||||
|
|
|
|||
|
|
@ -1501,9 +1501,9 @@ static int amdgpu_debugfs_test_ib_show(struct seq_file *m, void *unused)
|
|||
for (i = 0; i < AMDGPU_MAX_RINGS; i++) {
|
||||
struct amdgpu_ring *ring = adev->rings[i];
|
||||
|
||||
if (!ring || !ring->sched.thread)
|
||||
if (!ring || !drm_sched_wqueue_ready(&ring->sched))
|
||||
continue;
|
||||
kthread_park(ring->sched.thread);
|
||||
drm_sched_wqueue_stop(&ring->sched);
|
||||
}
|
||||
|
||||
seq_printf(m, "run ib test:\n");
|
||||
|
|
@ -1517,9 +1517,9 @@ static int amdgpu_debugfs_test_ib_show(struct seq_file *m, void *unused)
|
|||
for (i = 0; i < AMDGPU_MAX_RINGS; i++) {
|
||||
struct amdgpu_ring *ring = adev->rings[i];
|
||||
|
||||
if (!ring || !ring->sched.thread)
|
||||
if (!ring || !drm_sched_wqueue_ready(&ring->sched))
|
||||
continue;
|
||||
kthread_unpark(ring->sched.thread);
|
||||
drm_sched_wqueue_start(&ring->sched);
|
||||
}
|
||||
|
||||
up_write(&adev->reset_domain->sem);
|
||||
|
|
@ -1739,7 +1739,8 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val)
|
|||
|
||||
ring = adev->rings[val];
|
||||
|
||||
if (!ring || !ring->funcs->preempt_ib || !ring->sched.thread)
|
||||
if (!ring || !ring->funcs->preempt_ib ||
|
||||
!drm_sched_wqueue_ready(&ring->sched))
|
||||
return -EINVAL;
|
||||
|
||||
/* the last preemption failed */
|
||||
|
|
@ -1757,7 +1758,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val)
|
|||
goto pro_end;
|
||||
|
||||
/* stop the scheduler */
|
||||
kthread_park(ring->sched.thread);
|
||||
drm_sched_wqueue_stop(&ring->sched);
|
||||
|
||||
resched = ttm_bo_lock_delayed_workqueue(&adev->mman.bdev);
|
||||
|
||||
|
|
@ -1793,7 +1794,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val)
|
|||
|
||||
failure:
|
||||
/* restart the scheduler */
|
||||
kthread_unpark(ring->sched.thread);
|
||||
drm_sched_wqueue_start(&ring->sched);
|
||||
|
||||
up_read(&adev->reset_domain->sem);
|
||||
|
||||
|
|
|
|||
|
|
@ -2355,7 +2355,8 @@ static int amdgpu_device_init_schedulers(struct amdgpu_device *adev)
|
|||
break;
|
||||
}
|
||||
|
||||
r = drm_sched_init(&ring->sched, &amdgpu_sched_ops,
|
||||
r = drm_sched_init(&ring->sched, &amdgpu_sched_ops, NULL,
|
||||
DRM_SCHED_PRIORITY_COUNT,
|
||||
ring->num_hw_submission, amdgpu_job_hang_limit,
|
||||
timeout, adev->reset_domain->wq,
|
||||
ring->sched_score, ring->name,
|
||||
|
|
@ -4691,7 +4692,7 @@ bool amdgpu_device_has_job_running(struct amdgpu_device *adev)
|
|||
for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
|
||||
struct amdgpu_ring *ring = adev->rings[i];
|
||||
|
||||
if (!ring || !ring->sched.thread)
|
||||
if (!ring || !drm_sched_wqueue_ready(&ring->sched))
|
||||
continue;
|
||||
|
||||
spin_lock(&ring->sched.job_list_lock);
|
||||
|
|
@ -4821,7 +4822,7 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
|
|||
for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
|
||||
struct amdgpu_ring *ring = adev->rings[i];
|
||||
|
||||
if (!ring || !ring->sched.thread)
|
||||
if (!ring || !drm_sched_wqueue_ready(&ring->sched))
|
||||
continue;
|
||||
|
||||
/*clear job fence from fence drv to avoid force_completion
|
||||
|
|
@ -5220,94 +5221,6 @@ static int amdgpu_device_suspend_display_audio(struct amdgpu_device *adev)
|
|||
return 0;
|
||||
}
|
||||
|
||||
static void amdgpu_device_recheck_guilty_jobs(
|
||||
struct amdgpu_device *adev, struct list_head *device_list_handle,
|
||||
struct amdgpu_reset_context *reset_context)
|
||||
{
|
||||
int i, r = 0;
|
||||
|
||||
for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
|
||||
struct amdgpu_ring *ring = adev->rings[i];
|
||||
int ret = 0;
|
||||
struct drm_sched_job *s_job;
|
||||
|
||||
if (!ring || !ring->sched.thread)
|
||||
continue;
|
||||
|
||||
s_job = list_first_entry_or_null(&ring->sched.pending_list,
|
||||
struct drm_sched_job, list);
|
||||
if (s_job == NULL)
|
||||
continue;
|
||||
|
||||
/* clear job's guilty and depend the folowing step to decide the real one */
|
||||
drm_sched_reset_karma(s_job);
|
||||
drm_sched_resubmit_jobs_ext(&ring->sched, 1);
|
||||
|
||||
if (!s_job->s_fence->parent) {
|
||||
DRM_WARN("Failed to get a HW fence for job!");
|
||||
continue;
|
||||
}
|
||||
|
||||
ret = dma_fence_wait_timeout(s_job->s_fence->parent, false, ring->sched.timeout);
|
||||
if (ret == 0) { /* timeout */
|
||||
DRM_ERROR("Found the real bad job! ring:%s, job_id:%llx\n",
|
||||
ring->sched.name, s_job->id);
|
||||
|
||||
|
||||
amdgpu_fence_driver_isr_toggle(adev, true);
|
||||
|
||||
/* Clear this failed job from fence array */
|
||||
amdgpu_fence_driver_clear_job_fences(ring);
|
||||
|
||||
amdgpu_fence_driver_isr_toggle(adev, false);
|
||||
|
||||
/* Since the job won't signal and we go for
|
||||
* another resubmit drop this parent pointer
|
||||
*/
|
||||
dma_fence_put(s_job->s_fence->parent);
|
||||
s_job->s_fence->parent = NULL;
|
||||
|
||||
/* set guilty */
|
||||
drm_sched_increase_karma(s_job);
|
||||
amdgpu_reset_prepare_hwcontext(adev, reset_context);
|
||||
retry:
|
||||
/* do hw reset */
|
||||
if (amdgpu_sriov_vf(adev)) {
|
||||
amdgpu_virt_fini_data_exchange(adev);
|
||||
r = amdgpu_device_reset_sriov(adev, false);
|
||||
if (r)
|
||||
adev->asic_reset_res = r;
|
||||
} else {
|
||||
clear_bit(AMDGPU_SKIP_HW_RESET,
|
||||
&reset_context->flags);
|
||||
r = amdgpu_do_asic_reset(device_list_handle,
|
||||
reset_context);
|
||||
if (r && r == -EAGAIN)
|
||||
goto retry;
|
||||
}
|
||||
|
||||
/*
|
||||
* add reset counter so that the following
|
||||
* resubmitted job could flush vmid
|
||||
*/
|
||||
atomic_inc(&adev->gpu_reset_counter);
|
||||
continue;
|
||||
}
|
||||
|
||||
/* got the hw fence, signal finished fence */
|
||||
atomic_dec(ring->sched.score);
|
||||
dma_fence_get(&s_job->s_fence->finished);
|
||||
dma_fence_signal(&s_job->s_fence->finished);
|
||||
dma_fence_put(&s_job->s_fence->finished);
|
||||
|
||||
/* remove node from list and free the job */
|
||||
spin_lock(&ring->sched.job_list_lock);
|
||||
list_del_init(&s_job->list);
|
||||
spin_unlock(&ring->sched.job_list_lock);
|
||||
ring->sched.ops->free_job(s_job);
|
||||
}
|
||||
}
|
||||
|
||||
static inline void amdgpu_device_stop_pending_resets(struct amdgpu_device *adev)
|
||||
{
|
||||
struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
|
||||
|
|
@ -5328,7 +5241,6 @@ static inline void amdgpu_device_stop_pending_resets(struct amdgpu_device *adev)
|
|||
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* amdgpu_device_gpu_recover - reset the asic and recover scheduler
|
||||
*
|
||||
|
|
@ -5351,7 +5263,6 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
|
|||
int i, r = 0;
|
||||
bool need_emergency_restart = false;
|
||||
bool audio_suspended = false;
|
||||
int tmp_vram_lost_counter;
|
||||
bool gpu_reset_for_dev_remove = false;
|
||||
|
||||
gpu_reset_for_dev_remove =
|
||||
|
|
@ -5451,7 +5362,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
|
|||
for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
|
||||
struct amdgpu_ring *ring = tmp_adev->rings[i];
|
||||
|
||||
if (!ring || !ring->sched.thread)
|
||||
if (!ring || !drm_sched_wqueue_ready(&ring->sched))
|
||||
continue;
|
||||
|
||||
drm_sched_stop(&ring->sched, job ? &job->base : NULL);
|
||||
|
|
@ -5498,7 +5409,6 @@ retry: /* Rest of adevs pre asic reset from XGMI hive. */
|
|||
amdgpu_device_stop_pending_resets(tmp_adev);
|
||||
}
|
||||
|
||||
tmp_vram_lost_counter = atomic_read(&((adev)->vram_lost_counter));
|
||||
/* Actual ASIC resets if needed.*/
|
||||
/* Host driver will handle XGMI hive reset for SRIOV */
|
||||
if (amdgpu_sriov_vf(adev)) {
|
||||
|
|
@ -5523,22 +5433,10 @@ skip_hw_reset:
|
|||
/* Post ASIC reset for all devs .*/
|
||||
list_for_each_entry(tmp_adev, device_list_handle, reset_list) {
|
||||
|
||||
/*
|
||||
* Sometimes a later bad compute job can block a good gfx job as gfx
|
||||
* and compute ring share internal GC HW mutually. We add an additional
|
||||
* guilty jobs recheck step to find the real guilty job, it synchronously
|
||||
* submits and pends for the first job being signaled. If it gets timeout,
|
||||
* we identify it as a real guilty job.
|
||||
*/
|
||||
if (amdgpu_gpu_recovery == 2 &&
|
||||
!(tmp_vram_lost_counter < atomic_read(&adev->vram_lost_counter)))
|
||||
amdgpu_device_recheck_guilty_jobs(
|
||||
tmp_adev, device_list_handle, reset_context);
|
||||
|
||||
for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
|
||||
struct amdgpu_ring *ring = tmp_adev->rings[i];
|
||||
|
||||
if (!ring || !ring->sched.thread)
|
||||
if (!ring || !drm_sched_wqueue_ready(&ring->sched))
|
||||
continue;
|
||||
|
||||
/* No point to resubmit jobs if we didn't HW reset*/
|
||||
|
|
@ -5867,7 +5765,7 @@ pci_ers_result_t amdgpu_pci_error_detected(struct pci_dev *pdev, pci_channel_sta
|
|||
for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
|
||||
struct amdgpu_ring *ring = adev->rings[i];
|
||||
|
||||
if (!ring || !ring->sched.thread)
|
||||
if (!ring || !drm_sched_wqueue_ready(&ring->sched))
|
||||
continue;
|
||||
|
||||
drm_sched_stop(&ring->sched, NULL);
|
||||
|
|
@ -5995,7 +5893,7 @@ void amdgpu_pci_resume(struct pci_dev *pdev)
|
|||
for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
|
||||
struct amdgpu_ring *ring = adev->rings[i];
|
||||
|
||||
if (!ring || !ring->sched.thread)
|
||||
if (!ring || !drm_sched_wqueue_ready(&ring->sched))
|
||||
continue;
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -534,7 +534,7 @@ module_param_named(compute_multipipe, amdgpu_compute_multipipe, int, 0444);
|
|||
* DOC: gpu_recovery (int)
|
||||
* Set to enable GPU recovery mechanism (1 = enable, 0 = disable). The default is -1 (auto, disabled except SRIOV).
|
||||
*/
|
||||
MODULE_PARM_DESC(gpu_recovery, "Enable GPU recovery mechanism, (2 = advanced tdr mode, 1 = enable, 0 = disable, -1 = auto)");
|
||||
MODULE_PARM_DESC(gpu_recovery, "Enable GPU recovery mechanism, (1 = enable, 0 = disable, -1 = auto)");
|
||||
module_param_named(gpu_recovery, amdgpu_gpu_recovery, int, 0444);
|
||||
|
||||
/**
|
||||
|
|
@ -2814,8 +2814,6 @@ static const struct drm_driver amdgpu_kms_driver = {
|
|||
.fops = &amdgpu_driver_kms_fops,
|
||||
.release = &amdgpu_driver_release_kms,
|
||||
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import = amdgpu_gem_prime_import,
|
||||
.gem_prime_mmap = drm_gem_prime_mmap,
|
||||
|
||||
|
|
|
|||
|
|
@ -111,6 +111,8 @@ int amdgpu_job_alloc(struct amdgpu_device *adev, unsigned num_ibs,
|
|||
(*job)->vm_pd_addr = AMDGPU_BO_INVALID_OFFSET;
|
||||
|
||||
return 0;
|
||||
|
||||
return drm_sched_job_init(&(*job)->base, entity, 1, owner);
|
||||
}
|
||||
|
||||
int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, unsigned size,
|
||||
|
|
@ -248,7 +250,8 @@ int amdgpu_job_submit_direct(struct amdgpu_job *job, struct amdgpu_ring *ring,
|
|||
return 0;
|
||||
}
|
||||
|
||||
static struct dma_fence *amdgpu_job_dependency(struct drm_sched_job *sched_job,
|
||||
static struct dma_fence *
|
||||
amdgpu_job_prepare_job(struct drm_sched_job *sched_job,
|
||||
struct drm_sched_entity *s_entity)
|
||||
{
|
||||
struct amdgpu_ring *ring = to_amdgpu_ring(s_entity->rq->sched);
|
||||
|
|
@ -329,8 +332,8 @@ void amdgpu_job_stop_all_jobs_on_sched(struct drm_gpu_scheduler *sched)
|
|||
int i;
|
||||
|
||||
/* Signal all jobs not yet scheduled */
|
||||
for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) {
|
||||
struct drm_sched_rq *rq = &sched->sched_rq[i];
|
||||
for (i = DRM_SCHED_PRIORITY_KERNEL; i < sched->num_rqs; i++) {
|
||||
struct drm_sched_rq *rq = sched->sched_rq[i];
|
||||
spin_lock(&rq->lock);
|
||||
list_for_each_entry(s_entity, &rq->entities, list) {
|
||||
while ((s_job = to_drm_sched_job(spsc_queue_pop(&s_entity->job_queue)))) {
|
||||
|
|
@ -354,7 +357,7 @@ void amdgpu_job_stop_all_jobs_on_sched(struct drm_gpu_scheduler *sched)
|
|||
}
|
||||
|
||||
const struct drm_sched_backend_ops amdgpu_sched_ops = {
|
||||
.dependency = amdgpu_job_dependency,
|
||||
.prepare_job = amdgpu_job_prepare_job,
|
||||
.run_job = amdgpu_job_run,
|
||||
.timedout_job = amdgpu_job_timedout,
|
||||
.free_job = amdgpu_job_free_cb
|
||||
|
|
|
|||
|
|
@ -38,8 +38,6 @@ DEFINE_DRM_GEM_FOPS(armada_drm_fops);
|
|||
|
||||
static const struct drm_driver armada_drm_driver = {
|
||||
.lastclose = drm_fb_helper_lastclose,
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import = armada_gem_prime_import,
|
||||
.dumb_create = armada_gem_dumb_create,
|
||||
.major = 1,
|
||||
|
|
|
|||
|
|
@ -344,7 +344,7 @@ drm_client_buffer_vmap(struct drm_client_buffer *buffer,
|
|||
* fd_install step out of the driver backend hooks, to make that
|
||||
* final step optional for internal users.
|
||||
*/
|
||||
ret = drm_gem_vmap(buffer->gem, map);
|
||||
ret = drm_gem_vmap_unlocked(buffer->gem, map);
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
|
|
@ -366,7 +366,7 @@ void drm_client_buffer_vunmap(struct drm_client_buffer *buffer)
|
|||
{
|
||||
struct iosys_map *map = &buffer->map;
|
||||
|
||||
drm_gem_vunmap(buffer->gem, map);
|
||||
drm_gem_vunmap_unlocked(buffer->gem, map);
|
||||
}
|
||||
EXPORT_SYMBOL(drm_client_buffer_vunmap);
|
||||
|
||||
|
|
|
|||
340
drivers/gpu/drm/drm_exec.c
Normal file
340
drivers/gpu/drm/drm_exec.c
Normal file
|
|
@ -0,0 +1,340 @@
|
|||
// SPDX-License-Identifier: GPL-2.0 OR MIT
|
||||
|
||||
#include <drm/drm_exec.h>
|
||||
#include <drm/drm_gem.h>
|
||||
#include <linux/dma-resv.h>
|
||||
|
||||
/**
|
||||
* DOC: Overview
|
||||
*
|
||||
* This component mainly abstracts the retry loop necessary for locking
|
||||
* multiple GEM objects while preparing hardware operations (e.g. command
|
||||
* submissions, page table updates etc..).
|
||||
*
|
||||
* If a contention is detected while locking a GEM object the cleanup procedure
|
||||
* unlocks all previously locked GEM objects and locks the contended one first
|
||||
* before locking any further objects.
|
||||
*
|
||||
* After an object is locked fences slots can optionally be reserved on the
|
||||
* dma_resv object inside the GEM object.
|
||||
*
|
||||
* A typical usage pattern should look like this::
|
||||
*
|
||||
* struct drm_gem_object *obj;
|
||||
* struct drm_exec exec;
|
||||
* unsigned long index;
|
||||
* int ret;
|
||||
*
|
||||
* drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT);
|
||||
* drm_exec_until_all_locked(&exec) {
|
||||
* ret = drm_exec_prepare_obj(&exec, boA, 1);
|
||||
* drm_exec_retry_on_contention(&exec);
|
||||
* if (ret)
|
||||
* goto error;
|
||||
*
|
||||
* ret = drm_exec_prepare_obj(&exec, boB, 1);
|
||||
* drm_exec_retry_on_contention(&exec);
|
||||
* if (ret)
|
||||
* goto error;
|
||||
* }
|
||||
*
|
||||
* drm_exec_for_each_locked_object(&exec, index, obj) {
|
||||
* dma_resv_add_fence(obj->resv, fence, DMA_RESV_USAGE_READ);
|
||||
* ...
|
||||
* }
|
||||
* drm_exec_fini(&exec);
|
||||
*
|
||||
* See struct dma_exec for more details.
|
||||
*/
|
||||
|
||||
/* Dummy value used to initially enter the retry loop */
|
||||
#define DRM_EXEC_DUMMY ((void *)~0)
|
||||
|
||||
/* Unlock all objects and drop references */
|
||||
static void drm_exec_unlock_all(struct drm_exec *exec)
|
||||
{
|
||||
struct drm_gem_object *obj;
|
||||
unsigned long index;
|
||||
|
||||
drm_exec_for_each_locked_object_reverse(exec, index, obj) {
|
||||
dma_resv_unlock(obj->resv);
|
||||
drm_gem_object_put(obj);
|
||||
}
|
||||
|
||||
drm_gem_object_put(exec->prelocked);
|
||||
exec->prelocked = NULL;
|
||||
}
|
||||
|
||||
/**
|
||||
* drm_exec_init - initialize a drm_exec object
|
||||
* @exec: the drm_exec object to initialize
|
||||
* @flags: controls locking behavior, see DRM_EXEC_* defines
|
||||
* @nr: the initial # of objects
|
||||
*
|
||||
* Initialize the object and make sure that we can track locked objects.
|
||||
*
|
||||
* If nr is non-zero then it is used as the initial objects table size.
|
||||
* In either case, the table will grow (be re-allocated) on demand.
|
||||
*/
|
||||
void drm_exec_init(struct drm_exec *exec, uint32_t flags, unsigned nr)
|
||||
{
|
||||
if (!nr)
|
||||
nr = PAGE_SIZE / sizeof(void *);
|
||||
|
||||
exec->flags = flags;
|
||||
exec->objects = kvmalloc_array(nr, sizeof(void *), GFP_KERNEL);
|
||||
|
||||
/* If allocation here fails, just delay that till the first use */
|
||||
exec->max_objects = exec->objects ? nr : 0;
|
||||
exec->num_objects = 0;
|
||||
exec->contended = DRM_EXEC_DUMMY;
|
||||
exec->prelocked = NULL;
|
||||
}
|
||||
EXPORT_SYMBOL(drm_exec_init);
|
||||
|
||||
/**
|
||||
* drm_exec_fini - finalize a drm_exec object
|
||||
* @exec: the drm_exec object to finalize
|
||||
*
|
||||
* Unlock all locked objects, drop the references to objects and free all memory
|
||||
* used for tracking the state.
|
||||
*/
|
||||
void drm_exec_fini(struct drm_exec *exec)
|
||||
{
|
||||
drm_exec_unlock_all(exec);
|
||||
kvfree(exec->objects);
|
||||
if (exec->contended != DRM_EXEC_DUMMY) {
|
||||
drm_gem_object_put(exec->contended);
|
||||
ww_acquire_fini(&exec->ticket);
|
||||
}
|
||||
}
|
||||
EXPORT_SYMBOL(drm_exec_fini);
|
||||
|
||||
/**
|
||||
* drm_exec_cleanup - cleanup when contention is detected
|
||||
* @exec: the drm_exec object to cleanup
|
||||
*
|
||||
* Cleanup the current state and return true if we should stay inside the retry
|
||||
* loop, false if there wasn't any contention detected and we can keep the
|
||||
* objects locked.
|
||||
*/
|
||||
bool drm_exec_cleanup(struct drm_exec *exec)
|
||||
{
|
||||
if (likely(!exec->contended)) {
|
||||
ww_acquire_done(&exec->ticket);
|
||||
return false;
|
||||
}
|
||||
|
||||
if (likely(exec->contended == DRM_EXEC_DUMMY)) {
|
||||
exec->contended = NULL;
|
||||
ww_acquire_init(&exec->ticket, &reservation_ww_class);
|
||||
return true;
|
||||
}
|
||||
|
||||
drm_exec_unlock_all(exec);
|
||||
exec->num_objects = 0;
|
||||
return true;
|
||||
}
|
||||
EXPORT_SYMBOL(drm_exec_cleanup);
|
||||
|
||||
/* Track the locked object in the array */
|
||||
static int drm_exec_obj_locked(struct drm_exec *exec,
|
||||
struct drm_gem_object *obj)
|
||||
{
|
||||
if (unlikely(exec->num_objects == exec->max_objects)) {
|
||||
size_t size = exec->max_objects * sizeof(void *);
|
||||
void *tmp;
|
||||
|
||||
tmp = kvrealloc(exec->objects, size, size + PAGE_SIZE,
|
||||
GFP_KERNEL);
|
||||
if (!tmp)
|
||||
return -ENOMEM;
|
||||
|
||||
exec->objects = tmp;
|
||||
exec->max_objects += PAGE_SIZE / sizeof(void *);
|
||||
}
|
||||
drm_gem_object_get(obj);
|
||||
exec->objects[exec->num_objects++] = obj;
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Make sure the contended object is locked first */
|
||||
static int drm_exec_lock_contended(struct drm_exec *exec)
|
||||
{
|
||||
struct drm_gem_object *obj = exec->contended;
|
||||
int ret;
|
||||
|
||||
if (likely(!obj))
|
||||
return 0;
|
||||
|
||||
/* Always cleanup the contention so that error handling can kick in */
|
||||
exec->contended = NULL;
|
||||
if (exec->flags & DRM_EXEC_INTERRUPTIBLE_WAIT) {
|
||||
ret = dma_resv_lock_slow_interruptible(obj->resv,
|
||||
&exec->ticket);
|
||||
if (unlikely(ret))
|
||||
goto error_dropref;
|
||||
} else {
|
||||
dma_resv_lock_slow(obj->resv, &exec->ticket);
|
||||
}
|
||||
|
||||
ret = drm_exec_obj_locked(exec, obj);
|
||||
if (unlikely(ret))
|
||||
goto error_unlock;
|
||||
|
||||
exec->prelocked = obj;
|
||||
return 0;
|
||||
|
||||
error_unlock:
|
||||
dma_resv_unlock(obj->resv);
|
||||
|
||||
error_dropref:
|
||||
drm_gem_object_put(obj);
|
||||
return ret;
|
||||
}
|
||||
|
||||
/**
|
||||
* drm_exec_lock_obj - lock a GEM object for use
|
||||
* @exec: the drm_exec object with the state
|
||||
* @obj: the GEM object to lock
|
||||
*
|
||||
* Lock a GEM object for use and grab a reference to it.
|
||||
*
|
||||
* Returns: -EDEADLK if a contention is detected, -EALREADY when object is
|
||||
* already locked (can be suppressed by setting the DRM_EXEC_IGNORE_DUPLICATES
|
||||
* flag), -ENOMEM when memory allocation failed and zero for success.
|
||||
*/
|
||||
int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj)
|
||||
{
|
||||
int ret;
|
||||
|
||||
ret = drm_exec_lock_contended(exec);
|
||||
if (unlikely(ret))
|
||||
return ret;
|
||||
|
||||
if (exec->prelocked == obj) {
|
||||
drm_gem_object_put(exec->prelocked);
|
||||
exec->prelocked = NULL;
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (exec->flags & DRM_EXEC_INTERRUPTIBLE_WAIT)
|
||||
ret = dma_resv_lock_interruptible(obj->resv, &exec->ticket);
|
||||
else
|
||||
ret = dma_resv_lock(obj->resv, &exec->ticket);
|
||||
|
||||
if (unlikely(ret == -EDEADLK)) {
|
||||
drm_gem_object_get(obj);
|
||||
exec->contended = obj;
|
||||
return -EDEADLK;
|
||||
}
|
||||
|
||||
if (unlikely(ret == -EALREADY) &&
|
||||
exec->flags & DRM_EXEC_IGNORE_DUPLICATES)
|
||||
return 0;
|
||||
|
||||
if (unlikely(ret))
|
||||
return ret;
|
||||
|
||||
ret = drm_exec_obj_locked(exec, obj);
|
||||
if (ret)
|
||||
goto error_unlock;
|
||||
|
||||
return 0;
|
||||
|
||||
error_unlock:
|
||||
dma_resv_unlock(obj->resv);
|
||||
return ret;
|
||||
}
|
||||
EXPORT_SYMBOL(drm_exec_lock_obj);
|
||||
|
||||
/**
|
||||
* drm_exec_unlock_obj - unlock a GEM object in this exec context
|
||||
* @exec: the drm_exec object with the state
|
||||
* @obj: the GEM object to unlock
|
||||
*
|
||||
* Unlock the GEM object and remove it from the collection of locked objects.
|
||||
* Should only be used to unlock the most recently locked objects. It's not time
|
||||
* efficient to unlock objects locked long ago.
|
||||
*/
|
||||
void drm_exec_unlock_obj(struct drm_exec *exec, struct drm_gem_object *obj)
|
||||
{
|
||||
unsigned int i;
|
||||
|
||||
for (i = exec->num_objects; i--;) {
|
||||
if (exec->objects[i] == obj) {
|
||||
dma_resv_unlock(obj->resv);
|
||||
for (++i; i < exec->num_objects; ++i)
|
||||
exec->objects[i - 1] = exec->objects[i];
|
||||
--exec->num_objects;
|
||||
drm_gem_object_put(obj);
|
||||
return;
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
EXPORT_SYMBOL(drm_exec_unlock_obj);
|
||||
|
||||
/**
|
||||
* drm_exec_prepare_obj - prepare a GEM object for use
|
||||
* @exec: the drm_exec object with the state
|
||||
* @obj: the GEM object to prepare
|
||||
* @num_fences: how many fences to reserve
|
||||
*
|
||||
* Prepare a GEM object for use by locking it and reserving fence slots.
|
||||
*
|
||||
* Returns: -EDEADLK if a contention is detected, -EALREADY when object is
|
||||
* already locked, -ENOMEM when memory allocation failed and zero for success.
|
||||
*/
|
||||
int drm_exec_prepare_obj(struct drm_exec *exec, struct drm_gem_object *obj,
|
||||
unsigned int num_fences)
|
||||
{
|
||||
int ret;
|
||||
|
||||
ret = drm_exec_lock_obj(exec, obj);
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
ret = dma_resv_reserve_fences(obj->resv, num_fences);
|
||||
if (ret) {
|
||||
drm_exec_unlock_obj(exec, obj);
|
||||
return ret;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
EXPORT_SYMBOL(drm_exec_prepare_obj);
|
||||
|
||||
/**
|
||||
* drm_exec_prepare_array - helper to prepare an array of objects
|
||||
* @exec: the drm_exec object with the state
|
||||
* @objects: array of GEM object to prepare
|
||||
* @num_objects: number of GEM objects in the array
|
||||
* @num_fences: number of fences to reserve on each GEM object
|
||||
*
|
||||
* Prepares all GEM objects in an array, aborts on first error.
|
||||
* Reserves @num_fences on each GEM object after locking it.
|
||||
*
|
||||
* Returns: -EDEADLOCK on contention, -EALREADY when object is already locked,
|
||||
* -ENOMEM when memory allocation failed and zero for success.
|
||||
*/
|
||||
int drm_exec_prepare_array(struct drm_exec *exec,
|
||||
struct drm_gem_object **objects,
|
||||
unsigned int num_objects,
|
||||
unsigned int num_fences)
|
||||
{
|
||||
int ret;
|
||||
|
||||
for (unsigned int i = 0; i < num_objects; ++i) {
|
||||
ret = drm_exec_prepare_obj(exec, objects[i], num_fences);
|
||||
if (unlikely(ret))
|
||||
return ret;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
EXPORT_SYMBOL(drm_exec_prepare_array);
|
||||
|
||||
MODULE_DESCRIPTION("DRM execution context");
|
||||
MODULE_LICENSE("Dual MIT/GPL");
|
||||
|
|
@ -164,11 +164,28 @@ void drm_gem_private_object_init(struct drm_device *dev,
|
|||
if (!obj->resv)
|
||||
obj->resv = &obj->_resv;
|
||||
|
||||
if (drm_core_check_feature(dev, DRIVER_GEM_GPUVA))
|
||||
drm_gem_gpuva_init(obj);
|
||||
|
||||
drm_vma_node_reset(&obj->vma_node);
|
||||
INIT_LIST_HEAD(&obj->lru_node);
|
||||
}
|
||||
EXPORT_SYMBOL(drm_gem_private_object_init);
|
||||
|
||||
/**
|
||||
* drm_gem_private_object_fini - Finalize a failed drm_gem_object
|
||||
* @obj: drm_gem_object
|
||||
*
|
||||
* Uninitialize an already allocated GEM object when it initialized failed
|
||||
*/
|
||||
void drm_gem_private_object_fini(struct drm_gem_object *obj)
|
||||
{
|
||||
WARN_ON(obj->dma_buf);
|
||||
|
||||
dma_resv_fini(&obj->_resv);
|
||||
}
|
||||
EXPORT_SYMBOL(drm_gem_private_object_fini);
|
||||
|
||||
/**
|
||||
* drm_gem_object_handle_free - release resources bound to userspace handles
|
||||
* @obj: GEM object to clean up.
|
||||
|
|
@ -930,12 +947,11 @@ drm_gem_release(struct drm_device *dev, struct drm_file *file_private)
|
|||
void
|
||||
drm_gem_object_release(struct drm_gem_object *obj)
|
||||
{
|
||||
WARN_ON(obj->dma_buf);
|
||||
|
||||
if (obj->filp)
|
||||
fput(obj->filp);
|
||||
|
||||
dma_resv_fini(&obj->_resv);
|
||||
drm_gem_private_object_fini(obj);
|
||||
|
||||
drm_gem_free_mmap_offset(obj);
|
||||
drm_gem_lru_remove(obj);
|
||||
}
|
||||
|
|
@ -1158,6 +1174,8 @@ int drm_gem_vmap(struct drm_gem_object *obj, struct iosys_map *map)
|
|||
{
|
||||
int ret;
|
||||
|
||||
dma_resv_assert_held(obj->resv);
|
||||
|
||||
if (!obj->funcs->vmap)
|
||||
return -EOPNOTSUPP;
|
||||
|
||||
|
|
@ -1173,6 +1191,8 @@ EXPORT_SYMBOL(drm_gem_vmap);
|
|||
|
||||
void drm_gem_vunmap(struct drm_gem_object *obj, struct iosys_map *map)
|
||||
{
|
||||
dma_resv_assert_held(obj->resv);
|
||||
|
||||
if (iosys_map_is_null(map))
|
||||
return;
|
||||
|
||||
|
|
@ -1184,6 +1204,26 @@ void drm_gem_vunmap(struct drm_gem_object *obj, struct iosys_map *map)
|
|||
}
|
||||
EXPORT_SYMBOL(drm_gem_vunmap);
|
||||
|
||||
int drm_gem_vmap_unlocked(struct drm_gem_object *obj, struct iosys_map *map)
|
||||
{
|
||||
int ret;
|
||||
|
||||
dma_resv_lock(obj->resv, NULL);
|
||||
ret = drm_gem_vmap(obj, map);
|
||||
dma_resv_unlock(obj->resv);
|
||||
|
||||
return ret;
|
||||
}
|
||||
EXPORT_SYMBOL(drm_gem_vmap_unlocked);
|
||||
|
||||
void drm_gem_vunmap_unlocked(struct drm_gem_object *obj, struct iosys_map *map)
|
||||
{
|
||||
dma_resv_lock(obj->resv, NULL);
|
||||
drm_gem_vunmap(obj, map);
|
||||
dma_resv_unlock(obj->resv);
|
||||
}
|
||||
EXPORT_SYMBOL(drm_gem_vunmap_unlocked);
|
||||
|
||||
/**
|
||||
* drm_gem_lock_reservations - Sets up the ww context and acquires
|
||||
* the lock on an array of GEM objects.
|
||||
|
|
|
|||
|
|
@ -230,7 +230,7 @@ void drm_gem_dma_free(struct drm_gem_dma_object *dma_obj)
|
|||
|
||||
if (gem_obj->import_attach) {
|
||||
if (dma_obj->vaddr)
|
||||
dma_buf_vunmap(gem_obj->import_attach->dmabuf, &map);
|
||||
dma_buf_vunmap_unlocked(gem_obj->import_attach->dmabuf, &map);
|
||||
drm_prime_gem_destroy(gem_obj, dma_obj->sgt);
|
||||
} else if (dma_obj->vaddr) {
|
||||
if (dma_obj->map_noncoherent)
|
||||
|
|
@ -477,8 +477,8 @@ drm_gem_dma_prime_import_sg_table(struct drm_device *dev,
|
|||
dma_obj->dma_addr = sg_dma_address(sgt->sgl);
|
||||
dma_obj->sgt = sgt;
|
||||
|
||||
DRM_DEBUG_PRIME("dma_addr = %pad, size = %zu\n", &dma_obj->dma_addr,
|
||||
attach->dmabuf->size);
|
||||
drm_dbg_prime(dev, "dma_addr = %pad, size = %zu\n", &dma_obj->dma_addr,
|
||||
attach->dmabuf->size);
|
||||
|
||||
return &dma_obj->base;
|
||||
}
|
||||
|
|
@ -581,7 +581,7 @@ drm_gem_dma_prime_import_sg_table_vmap(struct drm_device *dev,
|
|||
struct iosys_map map;
|
||||
int ret;
|
||||
|
||||
ret = dma_buf_vmap(attach->dmabuf, &map);
|
||||
ret = dma_buf_vmap_unlocked(attach->dmabuf, &map);
|
||||
if (ret) {
|
||||
DRM_ERROR("Failed to vmap PRIME buffer\n");
|
||||
return ERR_PTR(ret);
|
||||
|
|
@ -589,7 +589,7 @@ drm_gem_dma_prime_import_sg_table_vmap(struct drm_device *dev,
|
|||
|
||||
obj = drm_gem_dma_prime_import_sg_table(dev, attach, sgt);
|
||||
if (IS_ERR(obj)) {
|
||||
dma_buf_vunmap(attach->dmabuf, &map);
|
||||
dma_buf_vunmap_unlocked(attach->dmabuf, &map);
|
||||
return obj;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -354,7 +354,7 @@ int drm_gem_fb_vmap(struct drm_framebuffer *fb, struct iosys_map *map,
|
|||
ret = -EINVAL;
|
||||
goto err_drm_gem_vunmap;
|
||||
}
|
||||
ret = drm_gem_vmap(obj, &map[i]);
|
||||
ret = drm_gem_vmap_unlocked(obj, &map[i]);
|
||||
if (ret)
|
||||
goto err_drm_gem_vunmap;
|
||||
}
|
||||
|
|
@ -376,7 +376,7 @@ err_drm_gem_vunmap:
|
|||
obj = drm_gem_fb_get_obj(fb, i);
|
||||
if (!obj)
|
||||
continue;
|
||||
drm_gem_vunmap(obj, &map[i]);
|
||||
drm_gem_vunmap_unlocked(obj, &map[i]);
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
|
@ -403,7 +403,7 @@ void drm_gem_fb_vunmap(struct drm_framebuffer *fb, struct iosys_map *map)
|
|||
continue;
|
||||
if (iosys_map_is_null(&map[i]))
|
||||
continue;
|
||||
drm_gem_vunmap(obj, &map[i]);
|
||||
drm_gem_vunmap_unlocked(obj, &map[i]);
|
||||
}
|
||||
}
|
||||
EXPORT_SYMBOL(drm_gem_fb_vunmap);
|
||||
|
|
|
|||
|
|
@ -79,15 +79,15 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size, bool private)
|
|||
} else {
|
||||
ret = drm_gem_object_init(dev, obj, size);
|
||||
}
|
||||
if (ret)
|
||||
if (ret) {
|
||||
drm_gem_private_object_fini(obj);
|
||||
goto err_free;
|
||||
}
|
||||
|
||||
ret = drm_gem_create_mmap_offset(obj);
|
||||
if (ret)
|
||||
goto err_release;
|
||||
|
||||
mutex_init(&shmem->pages_lock);
|
||||
mutex_init(&shmem->vmap_lock);
|
||||
INIT_LIST_HEAD(&shmem->madv_list);
|
||||
|
||||
if (!private) {
|
||||
|
|
@ -139,11 +139,13 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
|
|||
{
|
||||
struct drm_gem_object *obj = &shmem->base;
|
||||
|
||||
WARN_ON(shmem->vmap_use_count);
|
||||
|
||||
if (obj->import_attach) {
|
||||
drm_prime_gem_destroy(obj, shmem->sgt);
|
||||
} else {
|
||||
dma_resv_lock(shmem->base.resv, NULL);
|
||||
|
||||
drm_WARN_ON(obj->dev, shmem->vmap_use_count);
|
||||
|
||||
if (shmem->sgt) {
|
||||
dma_unmap_sgtable(obj->dev->dev, shmem->sgt,
|
||||
DMA_BIDIRECTIONAL, 0);
|
||||
|
|
@ -152,28 +154,31 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
|
|||
}
|
||||
if (shmem->pages)
|
||||
drm_gem_shmem_put_pages(shmem);
|
||||
|
||||
drm_WARN_ON(obj->dev, shmem->pages_use_count);
|
||||
|
||||
dma_resv_unlock(shmem->base.resv);
|
||||
}
|
||||
|
||||
WARN_ON(shmem->pages_use_count);
|
||||
|
||||
drm_gem_object_release(obj);
|
||||
mutex_destroy(&shmem->pages_lock);
|
||||
mutex_destroy(&shmem->vmap_lock);
|
||||
kfree(shmem);
|
||||
}
|
||||
EXPORT_SYMBOL_GPL(drm_gem_shmem_free);
|
||||
|
||||
static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
|
||||
static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
|
||||
{
|
||||
struct drm_gem_object *obj = &shmem->base;
|
||||
struct page **pages;
|
||||
|
||||
dma_resv_assert_held(shmem->base.resv);
|
||||
|
||||
if (shmem->pages_use_count++ > 0)
|
||||
return 0;
|
||||
|
||||
pages = drm_gem_get_pages(obj);
|
||||
if (IS_ERR(pages)) {
|
||||
DRM_DEBUG_KMS("Failed to get pages (%ld)\n", PTR_ERR(pages));
|
||||
drm_dbg_kms(obj->dev, "Failed to get pages (%ld)\n",
|
||||
PTR_ERR(pages));
|
||||
shmem->pages_use_count = 0;
|
||||
return PTR_ERR(pages);
|
||||
}
|
||||
|
|
@ -194,36 +199,18 @@ static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
|
|||
}
|
||||
|
||||
/*
|
||||
* drm_gem_shmem_get_pages - Allocate backing pages for a shmem GEM object
|
||||
* drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object
|
||||
* @shmem: shmem GEM object
|
||||
*
|
||||
* This function makes sure that backing pages exists for the shmem GEM object
|
||||
* and increases the use count.
|
||||
*
|
||||
* Returns:
|
||||
* 0 on success or a negative error code on failure.
|
||||
* This function decreases the use count and puts the backing pages when use drops to zero.
|
||||
*/
|
||||
int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
|
||||
{
|
||||
int ret;
|
||||
|
||||
WARN_ON(shmem->base.import_attach);
|
||||
|
||||
ret = mutex_lock_interruptible(&shmem->pages_lock);
|
||||
if (ret)
|
||||
return ret;
|
||||
ret = drm_gem_shmem_get_pages_locked(shmem);
|
||||
mutex_unlock(&shmem->pages_lock);
|
||||
|
||||
return ret;
|
||||
}
|
||||
EXPORT_SYMBOL(drm_gem_shmem_get_pages);
|
||||
|
||||
static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
|
||||
void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
|
||||
{
|
||||
struct drm_gem_object *obj = &shmem->base;
|
||||
|
||||
if (WARN_ON_ONCE(!shmem->pages_use_count))
|
||||
dma_resv_assert_held(shmem->base.resv);
|
||||
|
||||
if (drm_WARN_ON_ONCE(obj->dev, !shmem->pages_use_count))
|
||||
return;
|
||||
|
||||
if (--shmem->pages_use_count > 0)
|
||||
|
|
@ -239,21 +226,26 @@ static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
|
|||
shmem->pages_mark_accessed_on_put);
|
||||
shmem->pages = NULL;
|
||||
}
|
||||
|
||||
/*
|
||||
* drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object
|
||||
* @shmem: shmem GEM object
|
||||
*
|
||||
* This function decreases the use count and puts the backing pages when use drops to zero.
|
||||
*/
|
||||
void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
|
||||
{
|
||||
mutex_lock(&shmem->pages_lock);
|
||||
drm_gem_shmem_put_pages_locked(shmem);
|
||||
mutex_unlock(&shmem->pages_lock);
|
||||
}
|
||||
EXPORT_SYMBOL(drm_gem_shmem_put_pages);
|
||||
|
||||
static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)
|
||||
{
|
||||
int ret;
|
||||
|
||||
dma_resv_assert_held(shmem->base.resv);
|
||||
|
||||
ret = drm_gem_shmem_get_pages(shmem);
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
static void drm_gem_shmem_unpin_locked(struct drm_gem_shmem_object *shmem)
|
||||
{
|
||||
dma_resv_assert_held(shmem->base.resv);
|
||||
|
||||
drm_gem_shmem_put_pages(shmem);
|
||||
}
|
||||
|
||||
/**
|
||||
* drm_gem_shmem_pin - Pin backing pages for a shmem GEM object
|
||||
* @shmem: shmem GEM object
|
||||
|
|
@ -266,9 +258,18 @@ EXPORT_SYMBOL(drm_gem_shmem_put_pages);
|
|||
*/
|
||||
int drm_gem_shmem_pin(struct drm_gem_shmem_object *shmem)
|
||||
{
|
||||
WARN_ON(shmem->base.import_attach);
|
||||
struct drm_gem_object *obj = &shmem->base;
|
||||
int ret;
|
||||
|
||||
return drm_gem_shmem_get_pages(shmem);
|
||||
drm_WARN_ON(obj->dev, obj->import_attach);
|
||||
|
||||
ret = dma_resv_lock_interruptible(shmem->base.resv, NULL);
|
||||
if (ret)
|
||||
return ret;
|
||||
ret = drm_gem_shmem_pin_locked(shmem);
|
||||
dma_resv_unlock(shmem->base.resv);
|
||||
|
||||
return ret;
|
||||
}
|
||||
EXPORT_SYMBOL(drm_gem_shmem_pin);
|
||||
|
||||
|
|
@ -281,66 +282,16 @@ EXPORT_SYMBOL(drm_gem_shmem_pin);
|
|||
*/
|
||||
void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)
|
||||
{
|
||||
WARN_ON(shmem->base.import_attach);
|
||||
struct drm_gem_object *obj = &shmem->base;
|
||||
|
||||
drm_gem_shmem_put_pages(shmem);
|
||||
drm_WARN_ON(obj->dev, obj->import_attach);
|
||||
|
||||
dma_resv_lock(shmem->base.resv, NULL);
|
||||
drm_gem_shmem_unpin_locked(shmem);
|
||||
dma_resv_unlock(shmem->base.resv);
|
||||
}
|
||||
EXPORT_SYMBOL(drm_gem_shmem_unpin);
|
||||
|
||||
static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem,
|
||||
struct iosys_map *map)
|
||||
{
|
||||
struct drm_gem_object *obj = &shmem->base;
|
||||
int ret = 0;
|
||||
|
||||
if (shmem->vmap_use_count++ > 0) {
|
||||
iosys_map_set_vaddr(map, shmem->vaddr);
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (obj->import_attach) {
|
||||
ret = dma_buf_vmap(obj->import_attach->dmabuf, map);
|
||||
if (!ret) {
|
||||
if (WARN_ON(map->is_iomem)) {
|
||||
dma_buf_vunmap(obj->import_attach->dmabuf, map);
|
||||
ret = -EIO;
|
||||
goto err_put_pages;
|
||||
}
|
||||
shmem->vaddr = map->vaddr;
|
||||
}
|
||||
} else {
|
||||
pgprot_t prot = PAGE_KERNEL;
|
||||
|
||||
ret = drm_gem_shmem_get_pages(shmem);
|
||||
if (ret)
|
||||
goto err_zero_use;
|
||||
|
||||
if (shmem->map_wc)
|
||||
prot = pgprot_writecombine(prot);
|
||||
shmem->vaddr = vmap(shmem->pages, obj->size >> PAGE_SHIFT,
|
||||
VM_MAP, prot);
|
||||
if (!shmem->vaddr)
|
||||
ret = -ENOMEM;
|
||||
else
|
||||
iosys_map_set_vaddr(map, shmem->vaddr);
|
||||
}
|
||||
|
||||
if (ret) {
|
||||
DRM_DEBUG_KMS("Failed to vmap pages, error %d\n", ret);
|
||||
goto err_put_pages;
|
||||
}
|
||||
|
||||
return 0;
|
||||
|
||||
err_put_pages:
|
||||
if (!obj->import_attach)
|
||||
drm_gem_shmem_put_pages(shmem);
|
||||
err_zero_use:
|
||||
shmem->vmap_use_count = 0;
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
/*
|
||||
* drm_gem_shmem_vmap - Create a virtual mapping for a shmem GEM object
|
||||
* @shmem: shmem GEM object
|
||||
|
|
@ -359,39 +310,58 @@ err_zero_use:
|
|||
int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,
|
||||
struct iosys_map *map)
|
||||
{
|
||||
int ret;
|
||||
struct drm_gem_object *obj = &shmem->base;
|
||||
int ret = 0;
|
||||
|
||||
ret = mutex_lock_interruptible(&shmem->vmap_lock);
|
||||
if (ret)
|
||||
return ret;
|
||||
ret = drm_gem_shmem_vmap_locked(shmem, map);
|
||||
mutex_unlock(&shmem->vmap_lock);
|
||||
if (obj->import_attach) {
|
||||
ret = dma_buf_vmap(obj->import_attach->dmabuf, map);
|
||||
if (!ret) {
|
||||
if (drm_WARN_ON(obj->dev, map->is_iomem)) {
|
||||
dma_buf_vunmap(obj->import_attach->dmabuf, map);
|
||||
return -EIO;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
pgprot_t prot = PAGE_KERNEL;
|
||||
|
||||
dma_resv_assert_held(shmem->base.resv);
|
||||
|
||||
if (shmem->vmap_use_count++ > 0) {
|
||||
iosys_map_set_vaddr(map, shmem->vaddr);
|
||||
return 0;
|
||||
}
|
||||
|
||||
ret = drm_gem_shmem_get_pages(shmem);
|
||||
if (ret)
|
||||
goto err_zero_use;
|
||||
|
||||
if (shmem->map_wc)
|
||||
prot = pgprot_writecombine(prot);
|
||||
shmem->vaddr = vmap(shmem->pages, obj->size >> PAGE_SHIFT,
|
||||
VM_MAP, prot);
|
||||
if (!shmem->vaddr)
|
||||
ret = -ENOMEM;
|
||||
else
|
||||
iosys_map_set_vaddr(map, shmem->vaddr);
|
||||
}
|
||||
|
||||
if (ret) {
|
||||
drm_dbg_kms(obj->dev, "Failed to vmap pages, error %d\n", ret);
|
||||
goto err_put_pages;
|
||||
}
|
||||
|
||||
return 0;
|
||||
|
||||
err_put_pages:
|
||||
if (!obj->import_attach)
|
||||
drm_gem_shmem_put_pages(shmem);
|
||||
err_zero_use:
|
||||
shmem->vmap_use_count = 0;
|
||||
|
||||
return ret;
|
||||
}
|
||||
EXPORT_SYMBOL(drm_gem_shmem_vmap);
|
||||
|
||||
static void drm_gem_shmem_vunmap_locked(struct drm_gem_shmem_object *shmem,
|
||||
struct iosys_map *map)
|
||||
{
|
||||
struct drm_gem_object *obj = &shmem->base;
|
||||
|
||||
if (WARN_ON_ONCE(!shmem->vmap_use_count))
|
||||
return;
|
||||
|
||||
if (--shmem->vmap_use_count > 0)
|
||||
return;
|
||||
|
||||
if (obj->import_attach) {
|
||||
dma_buf_vunmap(obj->import_attach->dmabuf, map);
|
||||
} else {
|
||||
vunmap(shmem->vaddr);
|
||||
drm_gem_shmem_put_pages(shmem);
|
||||
}
|
||||
|
||||
shmem->vaddr = NULL;
|
||||
}
|
||||
|
||||
/*
|
||||
* drm_gem_shmem_vunmap - Unmap a virtual mapping for a shmem GEM object
|
||||
* @shmem: shmem GEM object
|
||||
|
|
@ -407,13 +377,28 @@ static void drm_gem_shmem_vunmap_locked(struct drm_gem_shmem_object *shmem,
|
|||
void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,
|
||||
struct iosys_map *map)
|
||||
{
|
||||
mutex_lock(&shmem->vmap_lock);
|
||||
drm_gem_shmem_vunmap_locked(shmem, map);
|
||||
mutex_unlock(&shmem->vmap_lock);
|
||||
struct drm_gem_object *obj = &shmem->base;
|
||||
|
||||
if (obj->import_attach) {
|
||||
dma_buf_vunmap(obj->import_attach->dmabuf, map);
|
||||
} else {
|
||||
dma_resv_assert_held(shmem->base.resv);
|
||||
|
||||
if (drm_WARN_ON_ONCE(obj->dev, !shmem->vmap_use_count))
|
||||
return;
|
||||
|
||||
if (--shmem->vmap_use_count > 0)
|
||||
return;
|
||||
|
||||
vunmap(shmem->vaddr);
|
||||
drm_gem_shmem_put_pages(shmem);
|
||||
}
|
||||
|
||||
shmem->vaddr = NULL;
|
||||
}
|
||||
EXPORT_SYMBOL(drm_gem_shmem_vunmap);
|
||||
|
||||
static struct drm_gem_shmem_object *
|
||||
static int
|
||||
drm_gem_shmem_create_with_handle(struct drm_file *file_priv,
|
||||
struct drm_device *dev, size_t size,
|
||||
uint32_t *handle)
|
||||
|
|
@ -423,7 +408,7 @@ drm_gem_shmem_create_with_handle(struct drm_file *file_priv,
|
|||
|
||||
shmem = drm_gem_shmem_create(dev, size);
|
||||
if (IS_ERR(shmem))
|
||||
return shmem;
|
||||
return PTR_ERR(shmem);
|
||||
|
||||
/*
|
||||
* Allocate an id of idr table where the obj is registered
|
||||
|
|
@ -432,10 +417,8 @@ drm_gem_shmem_create_with_handle(struct drm_file *file_priv,
|
|||
ret = drm_gem_handle_create(file_priv, &shmem->base, handle);
|
||||
/* drop reference from allocate - handle holds it now. */
|
||||
drm_gem_object_put(&shmem->base);
|
||||
if (ret)
|
||||
return ERR_PTR(ret);
|
||||
|
||||
return shmem;
|
||||
return ret;
|
||||
}
|
||||
|
||||
/* Update madvise status, returns true if not purged, else
|
||||
|
|
@ -443,32 +426,32 @@ drm_gem_shmem_create_with_handle(struct drm_file *file_priv,
|
|||
*/
|
||||
int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv)
|
||||
{
|
||||
mutex_lock(&shmem->pages_lock);
|
||||
dma_resv_assert_held(shmem->base.resv);
|
||||
|
||||
if (shmem->madv >= 0)
|
||||
shmem->madv = madv;
|
||||
|
||||
madv = shmem->madv;
|
||||
|
||||
mutex_unlock(&shmem->pages_lock);
|
||||
|
||||
return (madv >= 0);
|
||||
}
|
||||
EXPORT_SYMBOL(drm_gem_shmem_madvise);
|
||||
|
||||
void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)
|
||||
void drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem)
|
||||
{
|
||||
struct drm_gem_object *obj = &shmem->base;
|
||||
struct drm_device *dev = obj->dev;
|
||||
|
||||
WARN_ON(!drm_gem_shmem_is_purgeable(shmem));
|
||||
dma_resv_assert_held(shmem->base.resv);
|
||||
|
||||
drm_WARN_ON(obj->dev, !drm_gem_shmem_is_purgeable(shmem));
|
||||
|
||||
dma_unmap_sgtable(dev->dev, shmem->sgt, DMA_BIDIRECTIONAL, 0);
|
||||
sg_free_table(shmem->sgt);
|
||||
kfree(shmem->sgt);
|
||||
shmem->sgt = NULL;
|
||||
|
||||
drm_gem_shmem_put_pages_locked(shmem);
|
||||
drm_gem_shmem_put_pages(shmem);
|
||||
|
||||
shmem->madv = -1;
|
||||
|
||||
|
|
@ -484,17 +467,6 @@ void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)
|
|||
|
||||
invalidate_mapping_pages(file_inode(obj->filp)->i_mapping, 0, (loff_t)-1);
|
||||
}
|
||||
EXPORT_SYMBOL(drm_gem_shmem_purge_locked);
|
||||
|
||||
bool drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem)
|
||||
{
|
||||
if (!mutex_trylock(&shmem->pages_lock))
|
||||
return false;
|
||||
drm_gem_shmem_purge_locked(shmem);
|
||||
mutex_unlock(&shmem->pages_lock);
|
||||
|
||||
return true;
|
||||
}
|
||||
EXPORT_SYMBOL(drm_gem_shmem_purge);
|
||||
|
||||
/**
|
||||
|
|
@ -518,7 +490,6 @@ int drm_gem_shmem_dumb_create(struct drm_file *file, struct drm_device *dev,
|
|||
struct drm_mode_create_dumb *args)
|
||||
{
|
||||
u32 min_pitch = DIV_ROUND_UP(args->width * args->bpp, 8);
|
||||
struct drm_gem_shmem_object *shmem;
|
||||
|
||||
if (!args->pitch || !args->size) {
|
||||
args->pitch = min_pitch;
|
||||
|
|
@ -531,9 +502,7 @@ int drm_gem_shmem_dumb_create(struct drm_file *file, struct drm_device *dev,
|
|||
args->size = PAGE_ALIGN(args->pitch * args->height);
|
||||
}
|
||||
|
||||
shmem = drm_gem_shmem_create_with_handle(file, dev, args->size, &args->handle);
|
||||
|
||||
return PTR_ERR_OR_ZERO(shmem);
|
||||
return drm_gem_shmem_create_with_handle(file, dev, args->size, &args->handle);
|
||||
}
|
||||
EXPORT_SYMBOL_GPL(drm_gem_shmem_dumb_create);
|
||||
|
||||
|
|
@ -550,10 +519,10 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)
|
|||
/* We don't use vmf->pgoff since that has the fake offset */
|
||||
page_offset = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
|
||||
|
||||
mutex_lock(&shmem->pages_lock);
|
||||
dma_resv_lock(shmem->base.resv, NULL);
|
||||
|
||||
if (page_offset >= num_pages ||
|
||||
WARN_ON_ONCE(!shmem->pages) ||
|
||||
drm_WARN_ON_ONCE(obj->dev, !shmem->pages) ||
|
||||
shmem->madv < 0) {
|
||||
ret = VM_FAULT_SIGBUS;
|
||||
} else {
|
||||
|
|
@ -562,7 +531,7 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)
|
|||
ret = vmf_insert_pfn(vma, vmf->address, page_to_pfn(page));
|
||||
}
|
||||
|
||||
mutex_unlock(&shmem->pages_lock);
|
||||
dma_resv_unlock(shmem->base.resv);
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
|
@ -572,19 +541,19 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma)
|
|||
struct drm_gem_object *obj = vma->vm_private_data;
|
||||
struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
|
||||
|
||||
WARN_ON(shmem->base.import_attach);
|
||||
drm_WARN_ON(obj->dev, obj->import_attach);
|
||||
|
||||
mutex_lock(&shmem->pages_lock);
|
||||
dma_resv_lock(shmem->base.resv, NULL);
|
||||
|
||||
/*
|
||||
* We should have already pinned the pages when the buffer was first
|
||||
* mmap'd, vm_open() just grabs an additional reference for the new
|
||||
* mm the vma is getting copied into (ie. on fork()).
|
||||
*/
|
||||
if (!WARN_ON_ONCE(!shmem->pages_use_count))
|
||||
if (!drm_WARN_ON_ONCE(obj->dev, !shmem->pages_use_count))
|
||||
shmem->pages_use_count++;
|
||||
|
||||
mutex_unlock(&shmem->pages_lock);
|
||||
dma_resv_unlock(shmem->base.resv);
|
||||
|
||||
drm_gem_vm_open(vma);
|
||||
}
|
||||
|
|
@ -594,7 +563,10 @@ static void drm_gem_shmem_vm_close(struct vm_area_struct *vma)
|
|||
struct drm_gem_object *obj = vma->vm_private_data;
|
||||
struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
|
||||
|
||||
dma_resv_lock(shmem->base.resv, NULL);
|
||||
drm_gem_shmem_put_pages(shmem);
|
||||
dma_resv_unlock(shmem->base.resv);
|
||||
|
||||
drm_gem_vm_close(vma);
|
||||
}
|
||||
|
||||
|
|
@ -642,6 +614,8 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_struct
|
|||
return -EINVAL;
|
||||
|
||||
ret = drm_gem_shmem_get_pages(shmem);
|
||||
dma_resv_unlock(shmem->base.resv);
|
||||
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
|
|
@ -663,6 +637,9 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_mmap);
|
|||
void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,
|
||||
struct drm_printer *p, unsigned int indent)
|
||||
{
|
||||
if (shmem->base.import_attach)
|
||||
return;
|
||||
|
||||
drm_printf_indent(p, indent, "pages_use_count=%u\n", shmem->pages_use_count);
|
||||
drm_printf_indent(p, indent, "vmap_use_count=%u\n", shmem->vmap_use_count);
|
||||
drm_printf_indent(p, indent, "vaddr=%p\n", shmem->vaddr);
|
||||
|
|
@ -687,7 +664,7 @@ struct sg_table *drm_gem_shmem_get_sg_table(struct drm_gem_shmem_object *shmem)
|
|||
{
|
||||
struct drm_gem_object *obj = &shmem->base;
|
||||
|
||||
WARN_ON(shmem->base.import_attach);
|
||||
drm_WARN_ON(obj->dev, obj->import_attach);
|
||||
|
||||
return drm_prime_pages_to_sg(obj->dev, shmem->pages, obj->size >> PAGE_SHIFT);
|
||||
}
|
||||
|
|
@ -702,9 +679,9 @@ static struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct drm_gem_shmem_
|
|||
if (shmem->sgt)
|
||||
return shmem->sgt;
|
||||
|
||||
WARN_ON(obj->import_attach);
|
||||
drm_WARN_ON(obj->dev, obj->import_attach);
|
||||
|
||||
ret = drm_gem_shmem_get_pages_locked(shmem);
|
||||
ret = drm_gem_shmem_get_pages(shmem);
|
||||
if (ret)
|
||||
return ERR_PTR(ret);
|
||||
|
||||
|
|
@ -726,7 +703,7 @@ err_free_sgt:
|
|||
sg_free_table(sgt);
|
||||
kfree(sgt);
|
||||
err_put_pages:
|
||||
drm_gem_shmem_put_pages_locked(shmem);
|
||||
drm_gem_shmem_put_pages(shmem);
|
||||
return ERR_PTR(ret);
|
||||
}
|
||||
|
||||
|
|
@ -751,11 +728,11 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem)
|
|||
int ret;
|
||||
struct sg_table *sgt;
|
||||
|
||||
ret = mutex_lock_interruptible(&shmem->pages_lock);
|
||||
ret = dma_resv_lock_interruptible(shmem->base.resv, NULL);
|
||||
if (ret)
|
||||
return ERR_PTR(ret);
|
||||
sgt = drm_gem_shmem_get_pages_sgt_locked(shmem);
|
||||
mutex_unlock(&shmem->pages_lock);
|
||||
dma_resv_unlock(shmem->base.resv);
|
||||
|
||||
return sgt;
|
||||
}
|
||||
|
|
@ -790,7 +767,7 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev,
|
|||
|
||||
shmem->sgt = sgt;
|
||||
|
||||
DRM_DEBUG_PRIME("size = %zu\n", size);
|
||||
drm_dbg_prime(dev, "size = %zu\n", size);
|
||||
|
||||
return &shmem->base;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -64,13 +64,8 @@ int drm_gem_ttm_vmap(struct drm_gem_object *gem,
|
|||
struct iosys_map *map)
|
||||
{
|
||||
struct ttm_buffer_object *bo = drm_gem_ttm_of_gem(gem);
|
||||
int ret;
|
||||
|
||||
dma_resv_lock(gem->resv, NULL);
|
||||
ret = ttm_bo_vmap(bo, map);
|
||||
dma_resv_unlock(gem->resv);
|
||||
|
||||
return ret;
|
||||
return ttm_bo_vmap(bo, map);
|
||||
}
|
||||
EXPORT_SYMBOL(drm_gem_ttm_vmap);
|
||||
|
||||
|
|
@ -87,9 +82,7 @@ void drm_gem_ttm_vunmap(struct drm_gem_object *gem,
|
|||
{
|
||||
struct ttm_buffer_object *bo = drm_gem_ttm_of_gem(gem);
|
||||
|
||||
dma_resv_lock(gem->resv, NULL);
|
||||
ttm_bo_vunmap(bo, map);
|
||||
dma_resv_unlock(gem->resv);
|
||||
}
|
||||
EXPORT_SYMBOL(drm_gem_ttm_vunmap);
|
||||
|
||||
|
|
|
|||
2752
drivers/gpu/drm/drm_gpuvm.c
Normal file
2752
drivers/gpu/drm/drm_gpuvm.c
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -245,8 +245,7 @@ static int drm_getcap(struct drm_device *dev, void *data, struct drm_file *file_
|
|||
req->value = 1;
|
||||
return 0;
|
||||
case DRM_CAP_PRIME:
|
||||
req->value |= dev->driver->prime_fd_to_handle ? DRM_PRIME_CAP_IMPORT : 0;
|
||||
req->value |= dev->driver->prime_handle_to_fd ? DRM_PRIME_CAP_EXPORT : 0;
|
||||
req->value = DRM_PRIME_CAP_IMPORT | DRM_PRIME_CAP_EXPORT;
|
||||
return 0;
|
||||
case DRM_CAP_SYNCOBJ:
|
||||
req->value = drm_core_check_feature(dev, DRIVER_SYNCOBJ);
|
||||
|
|
|
|||
|
|
@ -51,15 +51,10 @@ MODULE_IMPORT_NS(DMA_BUF);
|
|||
* between applications, they can't be guessed like the globally unique GEM
|
||||
* names.
|
||||
*
|
||||
* Drivers that support the PRIME API implement the
|
||||
* &drm_driver.prime_handle_to_fd and &drm_driver.prime_fd_to_handle operations.
|
||||
* GEM based drivers must use drm_gem_prime_handle_to_fd() and
|
||||
* drm_gem_prime_fd_to_handle() to implement these. For GEM based drivers the
|
||||
* actual driver interfaces is provided through the &drm_gem_object_funcs.export
|
||||
* and &drm_driver.gem_prime_import hooks.
|
||||
*
|
||||
* &dma_buf_ops implementations for GEM drivers are all individually exported
|
||||
* for drivers which need to overwrite or reimplement some of them.
|
||||
* Drivers that support the PRIME API implement the drm_gem_object_funcs.export
|
||||
* and &drm_driver.gem_prime_import hooks. &dma_buf_ops implementations for
|
||||
* drivers are all individually exported for drivers which need to overwrite
|
||||
* or reimplement some of them.
|
||||
*
|
||||
* Reference Counting for GEM Drivers
|
||||
* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
@ -372,11 +367,12 @@ int drm_prime_fd_to_handle_ioctl(struct drm_device *dev, void *data,
|
|||
{
|
||||
struct drm_prime_handle *args = data;
|
||||
|
||||
if (!dev->driver->prime_fd_to_handle)
|
||||
return -ENOSYS;
|
||||
if (dev->driver->prime_fd_to_handle) {
|
||||
return dev->driver->prime_fd_to_handle(dev, file_priv, args->fd,
|
||||
&args->handle);
|
||||
}
|
||||
|
||||
return dev->driver->prime_fd_to_handle(dev, file_priv,
|
||||
args->fd, &args->handle);
|
||||
return drm_gem_prime_fd_to_handle(dev, file_priv, args->fd, &args->handle);
|
||||
}
|
||||
|
||||
static struct dma_buf *export_and_register_object(struct drm_device *dev,
|
||||
|
|
@ -518,15 +514,17 @@ int drm_prime_handle_to_fd_ioctl(struct drm_device *dev, void *data,
|
|||
{
|
||||
struct drm_prime_handle *args = data;
|
||||
|
||||
if (!dev->driver->prime_handle_to_fd)
|
||||
return -ENOSYS;
|
||||
|
||||
/* check flags are valid */
|
||||
if (args->flags & ~(DRM_CLOEXEC | DRM_RDWR))
|
||||
return -EINVAL;
|
||||
|
||||
return dev->driver->prime_handle_to_fd(dev, file_priv,
|
||||
args->handle, args->flags, &args->fd);
|
||||
if (dev->driver->prime_handle_to_fd) {
|
||||
return dev->driver->prime_handle_to_fd(dev, file_priv,
|
||||
args->handle, args->flags,
|
||||
&args->fd);
|
||||
}
|
||||
return drm_gem_prime_handle_to_fd(dev, file_priv, args->handle,
|
||||
args->flags, &args->fd);
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -474,8 +474,6 @@ static const struct drm_driver etnaviv_drm_driver = {
|
|||
.driver_features = DRIVER_GEM | DRIVER_RENDER,
|
||||
.open = etnaviv_open,
|
||||
.postclose = etnaviv_postclose,
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import_sg_table = etnaviv_gem_prime_import_sg_table,
|
||||
.gem_prime_mmap = drm_gem_prime_mmap,
|
||||
#ifdef CONFIG_DEBUG_FS
|
||||
|
|
|
|||
|
|
@ -531,7 +531,7 @@ int etnaviv_ioctl_gem_submit(struct drm_device *dev, void *data,
|
|||
|
||||
ret = drm_sched_job_init(&submit->sched_job,
|
||||
&ctx->sched_entity[args->pipe],
|
||||
submit->ctx);
|
||||
1, submit->ctx);
|
||||
if (ret)
|
||||
goto err_submit_put;
|
||||
|
||||
|
|
|
|||
|
|
@ -1870,7 +1870,7 @@ static int etnaviv_gpu_rpm_suspend(struct device *dev)
|
|||
u32 idle, mask;
|
||||
|
||||
/* If there are any jobs in the HW queue, we're not idle */
|
||||
if (atomic_read(&gpu->sched.hw_rq_count))
|
||||
if (atomic_read(&gpu->sched.credit_count))
|
||||
return -EBUSY;
|
||||
|
||||
/* Check whether the hardware (except FE and MC) is idle */
|
||||
|
|
|
|||
|
|
@ -132,7 +132,8 @@ int etnaviv_sched_init(struct etnaviv_gpu *gpu)
|
|||
{
|
||||
int ret;
|
||||
|
||||
ret = drm_sched_init(&gpu->sched, &etnaviv_sched_ops,
|
||||
ret = drm_sched_init(&gpu->sched, &etnaviv_sched_ops, NULL,
|
||||
DRM_SCHED_PRIORITY_COUNT,
|
||||
etnaviv_hw_jobs_limit, etnaviv_job_hang_limit,
|
||||
msecs_to_jiffies(500), NULL, NULL,
|
||||
dev_name(gpu->dev), gpu->dev);
|
||||
|
|
|
|||
|
|
@ -111,8 +111,6 @@ static const struct drm_driver exynos_drm_driver = {
|
|||
.lastclose = drm_fb_helper_lastclose,
|
||||
.postclose = exynos_drm_postclose,
|
||||
.dumb_create = exynos_drm_gem_dumb_create,
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import = exynos_drm_gem_prime_import,
|
||||
.gem_prime_import_sg_table = exynos_drm_gem_prime_import_sg_table,
|
||||
.gem_prime_mmap = drm_gem_prime_mmap,
|
||||
|
|
|
|||
|
|
@ -1921,8 +1921,6 @@ static const struct drm_driver i915_drm_driver = {
|
|||
.lastclose = i915_driver_lastclose,
|
||||
.postclose = i915_driver_postclose,
|
||||
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import = i915_gem_prime_import,
|
||||
|
||||
.dumb_create = i915_gem_dumb_create,
|
||||
|
|
|
|||
|
|
@ -514,7 +514,7 @@ int lima_device_suspend(struct device *dev)
|
|||
|
||||
/* check any task running */
|
||||
for (i = 0; i < lima_pipe_num; i++) {
|
||||
if (atomic_read(&ldev->pipe[i].base.hw_rq_count))
|
||||
if (atomic_read(&ldev->pipe[i].base.credit_count))
|
||||
return -EBUSY;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -276,9 +276,7 @@ static const struct drm_driver lima_drm_driver = {
|
|||
.patchlevel = 0,
|
||||
|
||||
.gem_create_object = lima_gem_create_object,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import_sg_table = drm_gem_shmem_prime_import_sg_table,
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.gem_prime_mmap = drm_gem_prime_mmap,
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm)
|
|||
|
||||
new_size = min(new_size, bo->base.base.size);
|
||||
|
||||
mutex_lock(&bo->base.pages_lock);
|
||||
dma_resv_lock(bo->base.base.resv, NULL);
|
||||
|
||||
if (bo->base.pages) {
|
||||
pages = bo->base.pages;
|
||||
|
|
@ -42,7 +42,7 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm)
|
|||
pages = kvmalloc_array(bo->base.base.size >> PAGE_SHIFT,
|
||||
sizeof(*pages), GFP_KERNEL | __GFP_ZERO);
|
||||
if (!pages) {
|
||||
mutex_unlock(&bo->base.pages_lock);
|
||||
dma_resv_unlock(bo->base.base.resv);
|
||||
return -ENOMEM;
|
||||
}
|
||||
|
||||
|
|
@ -56,13 +56,13 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm)
|
|||
struct page *page = shmem_read_mapping_page(mapping, i);
|
||||
|
||||
if (IS_ERR(page)) {
|
||||
mutex_unlock(&bo->base.pages_lock);
|
||||
dma_resv_unlock(bo->base.base.resv);
|
||||
return PTR_ERR(page);
|
||||
}
|
||||
pages[i] = page;
|
||||
}
|
||||
|
||||
mutex_unlock(&bo->base.pages_lock);
|
||||
dma_resv_unlock(bo->base.base.resv);
|
||||
|
||||
ret = sg_alloc_table_from_pages(&sgt, pages, i, 0,
|
||||
new_size, GFP_KERNEL);
|
||||
|
|
|
|||
|
|
@ -123,7 +123,7 @@ int lima_sched_task_init(struct lima_sched_task *task,
|
|||
for (i = 0; i < num_bos; i++)
|
||||
drm_gem_object_get(&bos[i]->base.base);
|
||||
|
||||
err = drm_sched_job_init(&task->base, &context->base, vm);
|
||||
err = drm_sched_job_init(&task->base, &context->base, 1, vm);
|
||||
if (err) {
|
||||
kfree(task->bos);
|
||||
return err;
|
||||
|
|
@ -371,7 +371,7 @@ static void lima_sched_build_error_task_list(struct lima_sched_task *task)
|
|||
} else {
|
||||
buffer_chunk->size = lima_bo_size(bo);
|
||||
|
||||
ret = drm_gem_shmem_vmap(&bo->base, &map);
|
||||
ret = drm_gem_vmap_unlocked(&bo->base.base, &map);
|
||||
if (ret) {
|
||||
kvfree(et);
|
||||
goto out;
|
||||
|
|
@ -379,7 +379,7 @@ static void lima_sched_build_error_task_list(struct lima_sched_task *task)
|
|||
|
||||
memcpy(buffer_chunk + 1, map.vaddr, buffer_chunk->size);
|
||||
|
||||
drm_gem_shmem_vunmap(&bo->base, &map);
|
||||
drm_gem_vunmap_unlocked(&bo->base.base, &map);
|
||||
}
|
||||
|
||||
buffer_chunk = (void *)(buffer_chunk + 1) + buffer_chunk->size;
|
||||
|
|
@ -495,7 +495,9 @@ int lima_sched_pipe_init(struct lima_sched_pipe *pipe, const char *name)
|
|||
|
||||
INIT_WORK(&pipe->recover_work, lima_sched_recover_work);
|
||||
|
||||
return drm_sched_init(&pipe->base, &lima_sched_ops, 1,
|
||||
return drm_sched_init(&pipe->base, &lima_sched_ops, NULL,
|
||||
DRM_SCHED_PRIORITY_COUNT,
|
||||
1,
|
||||
lima_job_hang_limit,
|
||||
msecs_to_jiffies(timeout), NULL,
|
||||
NULL, name, pipe->ldev->dev);
|
||||
|
|
|
|||
|
|
@ -472,8 +472,6 @@ static const struct drm_driver mtk_drm_driver = {
|
|||
|
||||
.dumb_create = mtk_drm_gem_dumb_create,
|
||||
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import = mtk_drm_gem_prime_import,
|
||||
.gem_prime_import_sg_table = mtk_gem_prime_import_sg_table,
|
||||
.gem_prime_mmap = drm_gem_prime_mmap,
|
||||
|
|
|
|||
|
|
@ -667,7 +667,8 @@ static void suspend_scheduler(struct msm_gpu *gpu)
|
|||
*/
|
||||
for (i = 0; i < gpu->nr_rings; i++) {
|
||||
struct drm_gpu_scheduler *sched = &gpu->rb[i]->sched;
|
||||
kthread_park(sched->thread);
|
||||
|
||||
drm_sched_wqueue_stop(sched);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -677,7 +678,8 @@ static void resume_scheduler(struct msm_gpu *gpu)
|
|||
|
||||
for (i = 0; i < gpu->nr_rings; i++) {
|
||||
struct drm_gpu_scheduler *sched = &gpu->rb[i]->sched;
|
||||
kthread_unpark(sched->thread);
|
||||
|
||||
drm_sched_wqueue_start(sched);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1083,8 +1083,6 @@ static const struct drm_driver msm_driver = {
|
|||
.lastclose = drm_fb_helper_lastclose,
|
||||
.dumb_create = msm_gem_dumb_create,
|
||||
.dumb_map_offset = msm_gem_dumb_map_offset,
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import_sg_table = msm_gem_prime_import_sg_table,
|
||||
.gem_prime_mmap = msm_gem_prime_mmap,
|
||||
#ifdef CONFIG_DEBUG_FS
|
||||
|
|
|
|||
|
|
@ -41,7 +41,7 @@ static struct msm_gem_submit *submit_create(struct drm_device *dev,
|
|||
if (!submit)
|
||||
return ERR_PTR(-ENOMEM);
|
||||
|
||||
ret = drm_sched_job_init(&submit->base, queue->entity, queue);
|
||||
ret = drm_sched_job_init(&submit->base, queue->entity, 1, queue);
|
||||
if (ret) {
|
||||
kfree(submit);
|
||||
return ERR_PTR(ret);
|
||||
|
|
|
|||
|
|
@ -337,7 +337,7 @@ struct msm_gpu_perfcntr {
|
|||
* DRM_SCHED_PRIORITY_KERNEL priority level is treated specially in some
|
||||
* cases, so we don't use it (no need for kernel generated jobs).
|
||||
*/
|
||||
#define NR_SCHED_PRIORITIES (1 + DRM_SCHED_PRIORITY_HIGH - DRM_SCHED_PRIORITY_MIN)
|
||||
#define NR_SCHED_PRIORITIES (1 + DRM_SCHED_PRIORITY_LOW - DRM_SCHED_PRIORITY_HIGH)
|
||||
|
||||
/**
|
||||
* struct msm_file_private - per-drm_file context
|
||||
|
|
|
|||
|
|
@ -95,9 +95,10 @@ struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id,
|
|||
/* currently managing hangcheck ourselves: */
|
||||
sched_timeout = MAX_SCHEDULE_TIMEOUT;
|
||||
|
||||
ret = drm_sched_init(&ring->sched, &msm_sched_ops,
|
||||
num_hw_submissions, 0, sched_timeout,
|
||||
NULL, NULL, to_msm_bo(ring->bo)->name, gpu->dev->dev);
|
||||
ret = drm_sched_init(&ring->sched, &msm_sched_ops, NULL,
|
||||
DRM_SCHED_PRIORITY_COUNT,
|
||||
num_hw_submissions, 0, sched_timeout,
|
||||
NULL, NULL, to_msm_bo(ring->bo)->name, gpu->dev->dev);
|
||||
if (ret) {
|
||||
goto fail;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1260,8 +1260,6 @@ driver_stub = {
|
|||
.num_ioctls = ARRAY_SIZE(nouveau_ioctls),
|
||||
.fops = &nouveau_driver_fops,
|
||||
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import_sg_table = nouveau_gem_prime_import_sg_table,
|
||||
.gem_prime_mmap = drm_gem_prime_mmap,
|
||||
|
||||
|
|
|
|||
|
|
@ -693,8 +693,6 @@ static const struct drm_driver omap_drm_driver = {
|
|||
#ifdef CONFIG_DEBUG_FS
|
||||
.debugfs_init = omap_debugfs_init,
|
||||
#endif
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import = omap_gem_prime_import,
|
||||
.dumb_create = omap_gem_dumb_create,
|
||||
.dumb_map_offset = omap_gem_dumb_map_offset,
|
||||
|
|
|
|||
|
|
@ -278,7 +278,7 @@ static int panfrost_ioctl_submit(struct drm_device *dev, void *data,
|
|||
|
||||
ret = drm_sched_job_init(&job->base,
|
||||
&file_priv->sched_entity[slot],
|
||||
NULL);
|
||||
1, NULL);
|
||||
if (ret)
|
||||
goto out_put_job;
|
||||
|
||||
|
|
@ -414,6 +414,10 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data,
|
|||
|
||||
bo = to_panfrost_bo(gem_obj);
|
||||
|
||||
ret = dma_resv_lock_interruptible(bo->base.base.resv, NULL);
|
||||
if (ret)
|
||||
goto out_put_object;
|
||||
|
||||
mutex_lock(&pfdev->shrinker_lock);
|
||||
mutex_lock(&bo->mappings.lock);
|
||||
if (args->madv == PANFROST_MADV_DONTNEED) {
|
||||
|
|
@ -451,7 +455,8 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data,
|
|||
out_unlock_mappings:
|
||||
mutex_unlock(&bo->mappings.lock);
|
||||
mutex_unlock(&pfdev->shrinker_lock);
|
||||
|
||||
dma_resv_unlock(bo->base.base.resv);
|
||||
out_put_object:
|
||||
drm_gem_object_put(gem_obj);
|
||||
return ret;
|
||||
}
|
||||
|
|
@ -546,8 +551,6 @@ static const struct drm_driver panfrost_drm_driver = {
|
|||
.minor = 2,
|
||||
|
||||
.gem_create_object = panfrost_gem_create_object,
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import_sg_table = panfrost_gem_prime_import_sg_table,
|
||||
.gem_prime_mmap = drm_gem_prime_mmap,
|
||||
};
|
||||
|
|
|
|||
|
|
@ -209,7 +209,7 @@ void panfrost_core_dump(struct panfrost_job *job)
|
|||
goto dump_header;
|
||||
}
|
||||
|
||||
ret = drm_gem_shmem_vmap(&bo->base, &map);
|
||||
ret = drm_gem_vmap_unlocked(&bo->base.base, &map);
|
||||
if (ret) {
|
||||
dev_err(pfdev->dev, "Panfrost Dump: couldn't map Buffer Object\n");
|
||||
iter.hdr->bomap.valid = 0;
|
||||
|
|
@ -236,7 +236,7 @@ void panfrost_core_dump(struct panfrost_job *job)
|
|||
vaddr = map.vaddr;
|
||||
memcpy(iter.data, vaddr, bo->base.base.size);
|
||||
|
||||
drm_gem_shmem_vunmap(&bo->base, &map);
|
||||
drm_gem_vunmap_unlocked(&bo->base.base, &map);
|
||||
|
||||
iter.hdr->bomap.valid = 1;
|
||||
|
||||
|
|
|
|||
|
|
@ -48,14 +48,14 @@ static bool panfrost_gem_purge(struct drm_gem_object *obj)
|
|||
if (!mutex_trylock(&bo->mappings.lock))
|
||||
return false;
|
||||
|
||||
if (!mutex_trylock(&shmem->pages_lock))
|
||||
if (!dma_resv_trylock(shmem->base.resv))
|
||||
goto unlock_mappings;
|
||||
|
||||
panfrost_gem_teardown_mappings_locked(bo);
|
||||
drm_gem_shmem_purge_locked(&bo->base);
|
||||
drm_gem_shmem_purge(&bo->base);
|
||||
ret = true;
|
||||
|
||||
mutex_unlock(&shmem->pages_lock);
|
||||
dma_resv_unlock(shmem->base.resv);
|
||||
|
||||
unlock_mappings:
|
||||
mutex_unlock(&bo->mappings.lock);
|
||||
|
|
|
|||
|
|
@ -815,7 +815,8 @@ int panfrost_job_init(struct panfrost_device *pfdev)
|
|||
js->queue[j].fence_context = dma_fence_context_alloc(1);
|
||||
|
||||
ret = drm_sched_init(&js->queue[j].sched,
|
||||
&panfrost_sched_ops,
|
||||
&panfrost_sched_ops, NULL,
|
||||
DRM_SCHED_PRIORITY_COUNT,
|
||||
nentries, 0,
|
||||
msecs_to_jiffies(JOB_TIMEOUT_MS),
|
||||
pfdev->reset.wq,
|
||||
|
|
@ -922,7 +923,7 @@ int panfrost_job_is_idle(struct panfrost_device *pfdev)
|
|||
|
||||
for (i = 0; i < NUM_JOB_SLOTS; i++) {
|
||||
/* If there are any jobs in the HW queue, we're not idle */
|
||||
if (atomic_read(&js->queue[i].sched.hw_rq_count))
|
||||
if (atomic_read(&js->queue[i].sched.credit_count))
|
||||
return false;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -443,6 +443,7 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
|
|||
struct panfrost_gem_mapping *bomapping;
|
||||
struct panfrost_gem_object *bo;
|
||||
struct address_space *mapping;
|
||||
struct drm_gem_object *obj;
|
||||
pgoff_t page_offset;
|
||||
struct sg_table *sgt;
|
||||
struct page **pages;
|
||||
|
|
@ -465,15 +466,16 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
|
|||
page_offset = addr >> PAGE_SHIFT;
|
||||
page_offset -= bomapping->mmnode.start;
|
||||
|
||||
mutex_lock(&bo->base.pages_lock);
|
||||
obj = &bo->base.base;
|
||||
|
||||
dma_resv_lock(obj->resv, NULL);
|
||||
|
||||
if (!bo->base.pages) {
|
||||
bo->sgts = kvmalloc_array(bo->base.base.size / SZ_2M,
|
||||
sizeof(struct sg_table), GFP_KERNEL | __GFP_ZERO);
|
||||
if (!bo->sgts) {
|
||||
mutex_unlock(&bo->base.pages_lock);
|
||||
ret = -ENOMEM;
|
||||
goto err_bo;
|
||||
goto err_unlock;
|
||||
}
|
||||
|
||||
pages = kvmalloc_array(bo->base.base.size >> PAGE_SHIFT,
|
||||
|
|
@ -481,9 +483,8 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
|
|||
if (!pages) {
|
||||
kvfree(bo->sgts);
|
||||
bo->sgts = NULL;
|
||||
mutex_unlock(&bo->base.pages_lock);
|
||||
ret = -ENOMEM;
|
||||
goto err_bo;
|
||||
goto err_unlock;
|
||||
}
|
||||
bo->base.pages = pages;
|
||||
bo->base.pages_use_count = 1;
|
||||
|
|
@ -491,7 +492,6 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
|
|||
pages = bo->base.pages;
|
||||
if (pages[page_offset]) {
|
||||
/* Pages are already mapped, bail out. */
|
||||
mutex_unlock(&bo->base.pages_lock);
|
||||
goto out;
|
||||
}
|
||||
}
|
||||
|
|
@ -502,15 +502,12 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
|
|||
for (i = page_offset; i < page_offset + NUM_FAULT_PAGES; i++) {
|
||||
pages[i] = shmem_read_mapping_page(mapping, i);
|
||||
if (IS_ERR(pages[i])) {
|
||||
mutex_unlock(&bo->base.pages_lock);
|
||||
ret = PTR_ERR(pages[i]);
|
||||
pages[i] = NULL;
|
||||
goto err_pages;
|
||||
}
|
||||
}
|
||||
|
||||
mutex_unlock(&bo->base.pages_lock);
|
||||
|
||||
sgt = &bo->sgts[page_offset / (SZ_2M / PAGE_SIZE)];
|
||||
ret = sg_alloc_table_from_pages(sgt, pages + page_offset,
|
||||
NUM_FAULT_PAGES, 0, SZ_2M, GFP_KERNEL);
|
||||
|
|
@ -529,6 +526,8 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
|
|||
dev_dbg(pfdev->dev, "mapped page fault @ AS%d %llx", as, addr);
|
||||
|
||||
out:
|
||||
dma_resv_unlock(obj->resv);
|
||||
|
||||
panfrost_gem_mapping_put(bomapping);
|
||||
|
||||
return 0;
|
||||
|
|
@ -537,6 +536,8 @@ err_map:
|
|||
sg_free_table(sgt);
|
||||
err_pages:
|
||||
drm_gem_shmem_put_pages(&bo->base);
|
||||
err_unlock:
|
||||
dma_resv_unlock(obj->resv);
|
||||
err_bo:
|
||||
panfrost_gem_mapping_put(bomapping);
|
||||
return ret;
|
||||
|
|
|
|||
|
|
@ -106,7 +106,7 @@ static int panfrost_perfcnt_enable_locked(struct panfrost_device *pfdev,
|
|||
goto err_close_bo;
|
||||
}
|
||||
|
||||
ret = drm_gem_shmem_vmap(bo, &map);
|
||||
ret = drm_gem_vmap_unlocked(&bo->base, &map);
|
||||
if (ret)
|
||||
goto err_put_mapping;
|
||||
perfcnt->buf = map.vaddr;
|
||||
|
|
@ -165,7 +165,7 @@ static int panfrost_perfcnt_enable_locked(struct panfrost_device *pfdev,
|
|||
return 0;
|
||||
|
||||
err_vunmap:
|
||||
drm_gem_shmem_vunmap(bo, &map);
|
||||
drm_gem_vunmap_unlocked(&bo->base, &map);
|
||||
err_put_mapping:
|
||||
panfrost_gem_mapping_put(perfcnt->mapping);
|
||||
err_close_bo:
|
||||
|
|
@ -195,7 +195,7 @@ static int panfrost_perfcnt_disable_locked(struct panfrost_device *pfdev,
|
|||
GPU_PERFCNT_CFG_MODE(GPU_PERFCNT_CFG_MODE_OFF));
|
||||
|
||||
perfcnt->user = NULL;
|
||||
drm_gem_shmem_vunmap(&perfcnt->mapping->obj->base, &map);
|
||||
drm_gem_vunmap_unlocked(&perfcnt->mapping->obj->base.base, &map);
|
||||
perfcnt->buf = NULL;
|
||||
panfrost_gem_close(&perfcnt->mapping->obj->base.base, file_priv);
|
||||
panfrost_mmu_as_put(pfdev, perfcnt->mapping->mmu);
|
||||
|
|
|
|||
23
drivers/gpu/drm/panthor/Kconfig
Normal file
23
drivers/gpu/drm/panthor/Kconfig
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
# SPDX-License-Identifier: GPL-2.0 or MIT
|
||||
|
||||
config DRM_PANTHOR
|
||||
tristate "Panthor (DRM support for ARM Mali CSF-based GPUs)"
|
||||
depends on DRM
|
||||
depends on ARM || ARM64 || COMPILE_TEST
|
||||
depends on !GENERIC_ATOMIC64 # for IOMMU_IO_PGTABLE_LPAE
|
||||
depends on MMU
|
||||
select DEVFREQ_GOV_SIMPLE_ONDEMAND
|
||||
select DRM_EXEC
|
||||
select DRM_GEM_SHMEM_HELPER
|
||||
select DRM_GPUVM
|
||||
select DRM_SCHED
|
||||
select IOMMU_IO_PGTABLE_LPAE
|
||||
select IOMMU_SUPPORT
|
||||
select PM_DEVFREQ
|
||||
help
|
||||
DRM driver for ARM Mali CSF-based GPUs.
|
||||
|
||||
This driver is for Mali (or Immortalis) Valhall Gxxx GPUs.
|
||||
|
||||
Note that the Mali-G68 and Mali-G78, while Valhall architecture, will
|
||||
be supported with the panfrost driver as they are not CSF GPUs.
|
||||
14
drivers/gpu/drm/panthor/Makefile
Normal file
14
drivers/gpu/drm/panthor/Makefile
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
# SPDX-License-Identifier: GPL-2.0 or MIT
|
||||
|
||||
panthor-y := \
|
||||
panthor_devfreq.o \
|
||||
panthor_device.o \
|
||||
panthor_drv.o \
|
||||
panthor_fw.o \
|
||||
panthor_gem.o \
|
||||
panthor_gpu.o \
|
||||
panthor_heap.o \
|
||||
panthor_mmu.o \
|
||||
panthor_sched.o
|
||||
|
||||
obj-$(CONFIG_DRM_PANTHOR) += panthor.o
|
||||
283
drivers/gpu/drm/panthor/panthor_devfreq.c
Normal file
283
drivers/gpu/drm/panthor/panthor_devfreq.c
Normal file
|
|
@ -0,0 +1,283 @@
|
|||
// SPDX-License-Identifier: GPL-2.0 or MIT
|
||||
/* Copyright 2019 Collabora ltd. */
|
||||
|
||||
#include <linux/clk.h>
|
||||
#include <linux/devfreq.h>
|
||||
#include <linux/devfreq_cooling.h>
|
||||
#include <linux/platform_device.h>
|
||||
#include <linux/pm_opp.h>
|
||||
|
||||
#include <drm/drm_managed.h>
|
||||
|
||||
#include "panthor_devfreq.h"
|
||||
#include "panthor_device.h"
|
||||
|
||||
/**
|
||||
* struct panthor_devfreq - Device frequency management
|
||||
*/
|
||||
struct panthor_devfreq {
|
||||
/** @devfreq: devfreq device. */
|
||||
struct devfreq *devfreq;
|
||||
|
||||
/** @gov_data: Governor data. */
|
||||
struct devfreq_simple_ondemand_data gov_data;
|
||||
|
||||
/** @busy_time: Busy time. */
|
||||
ktime_t busy_time;
|
||||
|
||||
/** @idle_time: Idle time. */
|
||||
ktime_t idle_time;
|
||||
|
||||
/** @time_last_update: Last update time. */
|
||||
ktime_t time_last_update;
|
||||
|
||||
/** @last_busy_state: True if the GPU was busy last time we updated the state. */
|
||||
bool last_busy_state;
|
||||
|
||||
/*
|
||||
* @lock: Lock used to protect busy_time, idle_time, time_last_update and
|
||||
* last_busy_state.
|
||||
*
|
||||
* These fields can be accessed concurrently by panthor_devfreq_get_dev_status()
|
||||
* and panthor_devfreq_record_{busy,idle}().
|
||||
*/
|
||||
spinlock_t lock;
|
||||
};
|
||||
|
||||
static void panthor_devfreq_update_utilization(struct panthor_devfreq *pdevfreq)
|
||||
{
|
||||
ktime_t now, last;
|
||||
|
||||
now = ktime_get();
|
||||
last = pdevfreq->time_last_update;
|
||||
|
||||
if (pdevfreq->last_busy_state)
|
||||
pdevfreq->busy_time += ktime_sub(now, last);
|
||||
else
|
||||
pdevfreq->idle_time += ktime_sub(now, last);
|
||||
|
||||
pdevfreq->time_last_update = now;
|
||||
}
|
||||
|
||||
static int panthor_devfreq_target(struct device *dev, unsigned long *freq,
|
||||
u32 flags)
|
||||
{
|
||||
struct dev_pm_opp *opp;
|
||||
|
||||
opp = devfreq_recommended_opp(dev, freq, flags);
|
||||
if (IS_ERR(opp))
|
||||
return PTR_ERR(opp);
|
||||
dev_pm_opp_put(opp);
|
||||
|
||||
return dev_pm_opp_set_rate(dev, *freq);
|
||||
}
|
||||
|
||||
static void panthor_devfreq_reset(struct panthor_devfreq *pdevfreq)
|
||||
{
|
||||
pdevfreq->busy_time = 0;
|
||||
pdevfreq->idle_time = 0;
|
||||
pdevfreq->time_last_update = ktime_get();
|
||||
}
|
||||
|
||||
static int panthor_devfreq_get_dev_status(struct device *dev,
|
||||
struct devfreq_dev_status *status)
|
||||
{
|
||||
struct panthor_device *ptdev = dev_get_drvdata(dev);
|
||||
struct panthor_devfreq *pdevfreq = ptdev->devfreq;
|
||||
unsigned long irqflags;
|
||||
|
||||
status->current_frequency = clk_get_rate(ptdev->clks.core);
|
||||
|
||||
spin_lock_irqsave(&pdevfreq->lock, irqflags);
|
||||
|
||||
panthor_devfreq_update_utilization(pdevfreq);
|
||||
|
||||
status->total_time = ktime_to_ns(ktime_add(pdevfreq->busy_time,
|
||||
pdevfreq->idle_time));
|
||||
|
||||
status->busy_time = ktime_to_ns(pdevfreq->busy_time);
|
||||
|
||||
panthor_devfreq_reset(pdevfreq);
|
||||
|
||||
spin_unlock_irqrestore(&pdevfreq->lock, irqflags);
|
||||
|
||||
drm_dbg(&ptdev->base, "busy %lu total %lu %lu %% freq %lu MHz\n",
|
||||
status->busy_time, status->total_time,
|
||||
status->busy_time / (status->total_time / 100),
|
||||
status->current_frequency / 1000 / 1000);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
static struct devfreq_dev_profile panthor_devfreq_profile = {
|
||||
.timer = DEVFREQ_TIMER_DELAYED,
|
||||
.polling_ms = 50, /* ~3 frames */
|
||||
.target = panthor_devfreq_target,
|
||||
.get_dev_status = panthor_devfreq_get_dev_status,
|
||||
};
|
||||
|
||||
int panthor_devfreq_init(struct panthor_device *ptdev)
|
||||
{
|
||||
/* There's actually 2 regulators (mali and sram), but the OPP core only
|
||||
* supports one.
|
||||
*
|
||||
* We assume the sram regulator is coupled with the mali one and let
|
||||
* the coupling logic deal with voltage updates.
|
||||
*/
|
||||
static const char * const reg_names[] = { "mali", NULL };
|
||||
struct thermal_cooling_device *cooling;
|
||||
struct device *dev = ptdev->base.dev;
|
||||
struct panthor_devfreq *pdevfreq;
|
||||
struct dev_pm_opp *opp;
|
||||
unsigned long cur_freq;
|
||||
int ret;
|
||||
|
||||
pdevfreq = drmm_kzalloc(&ptdev->base, sizeof(*ptdev->devfreq), GFP_KERNEL);
|
||||
if (!pdevfreq)
|
||||
return -ENOMEM;
|
||||
|
||||
ptdev->devfreq = pdevfreq;
|
||||
|
||||
ret = devm_pm_opp_set_regulators(dev, reg_names);
|
||||
if (ret) {
|
||||
if (ret != -EPROBE_DEFER)
|
||||
DRM_DEV_ERROR(dev, "Couldn't set OPP regulators\n");
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
ret = devm_pm_opp_of_add_table(dev);
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
spin_lock_init(&pdevfreq->lock);
|
||||
|
||||
panthor_devfreq_reset(pdevfreq);
|
||||
|
||||
cur_freq = clk_get_rate(ptdev->clks.core);
|
||||
|
||||
opp = devfreq_recommended_opp(dev, &cur_freq, 0);
|
||||
if (IS_ERR(opp))
|
||||
return PTR_ERR(opp);
|
||||
|
||||
panthor_devfreq_profile.initial_freq = cur_freq;
|
||||
|
||||
/* Regulator coupling only takes care of synchronizing/balancing voltage
|
||||
* updates, but the coupled regulator needs to be enabled manually.
|
||||
*
|
||||
* We use devm_regulator_get_enable_optional() and keep the sram supply
|
||||
* enabled until the device is removed, just like we do for the mali
|
||||
* supply, which is enabled when dev_pm_opp_set_opp(dev, opp) is called,
|
||||
* and disabled when the opp_table is torn down, using the devm action.
|
||||
*
|
||||
* If we really care about disabling regulators on suspend, we should:
|
||||
* - use devm_regulator_get_optional() here
|
||||
* - call dev_pm_opp_set_opp(dev, NULL) before leaving this function
|
||||
* (this disables the regulator passed to the OPP layer)
|
||||
* - call dev_pm_opp_set_opp(dev, NULL) and
|
||||
* regulator_disable(ptdev->regulators.sram) in
|
||||
* panthor_devfreq_suspend()
|
||||
* - call dev_pm_opp_set_opp(dev, default_opp) and
|
||||
* regulator_enable(ptdev->regulators.sram) in
|
||||
* panthor_devfreq_resume()
|
||||
*
|
||||
* But without knowing if it's beneficial or not (in term of power
|
||||
* consumption), or how much it slows down the suspend/resume steps,
|
||||
* let's just keep regulators enabled for the device lifetime.
|
||||
*/
|
||||
ret = devm_regulator_get_enable_optional(dev, "sram");
|
||||
if (ret && ret != -ENODEV) {
|
||||
if (ret != -EPROBE_DEFER)
|
||||
DRM_DEV_ERROR(dev, "Couldn't retrieve/enable sram supply\n");
|
||||
return ret;
|
||||
}
|
||||
|
||||
/*
|
||||
* Set the recommend OPP this will enable and configure the regulator
|
||||
* if any and will avoid a switch off by regulator_late_cleanup()
|
||||
*/
|
||||
ret = dev_pm_opp_set_opp(dev, opp);
|
||||
if (ret) {
|
||||
DRM_DEV_ERROR(dev, "Couldn't set recommended OPP\n");
|
||||
return ret;
|
||||
}
|
||||
|
||||
dev_pm_opp_put(opp);
|
||||
|
||||
/*
|
||||
* Setup default thresholds for the simple_ondemand governor.
|
||||
* The values are chosen based on experiments.
|
||||
*/
|
||||
pdevfreq->gov_data.upthreshold = 45;
|
||||
pdevfreq->gov_data.downdifferential = 5;
|
||||
|
||||
pdevfreq->devfreq = devm_devfreq_add_device(dev, &panthor_devfreq_profile,
|
||||
DEVFREQ_GOV_SIMPLE_ONDEMAND,
|
||||
&pdevfreq->gov_data);
|
||||
if (IS_ERR(pdevfreq->devfreq)) {
|
||||
DRM_DEV_ERROR(dev, "Couldn't initialize GPU devfreq\n");
|
||||
ret = PTR_ERR(pdevfreq->devfreq);
|
||||
pdevfreq->devfreq = NULL;
|
||||
return ret;
|
||||
}
|
||||
|
||||
cooling = devfreq_cooling_em_register(pdevfreq->devfreq, NULL);
|
||||
if (IS_ERR(cooling))
|
||||
DRM_DEV_INFO(dev, "Failed to register cooling device\n");
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
int panthor_devfreq_resume(struct panthor_device *ptdev)
|
||||
{
|
||||
struct panthor_devfreq *pdevfreq = ptdev->devfreq;
|
||||
|
||||
if (!pdevfreq->devfreq)
|
||||
return 0;
|
||||
|
||||
panthor_devfreq_reset(pdevfreq);
|
||||
|
||||
return devfreq_resume_device(pdevfreq->devfreq);
|
||||
}
|
||||
|
||||
int panthor_devfreq_suspend(struct panthor_device *ptdev)
|
||||
{
|
||||
struct panthor_devfreq *pdevfreq = ptdev->devfreq;
|
||||
|
||||
if (!pdevfreq->devfreq)
|
||||
return 0;
|
||||
|
||||
return devfreq_suspend_device(pdevfreq->devfreq);
|
||||
}
|
||||
|
||||
void panthor_devfreq_record_busy(struct panthor_device *ptdev)
|
||||
{
|
||||
struct panthor_devfreq *pdevfreq = ptdev->devfreq;
|
||||
unsigned long irqflags;
|
||||
|
||||
if (!pdevfreq->devfreq)
|
||||
return;
|
||||
|
||||
spin_lock_irqsave(&pdevfreq->lock, irqflags);
|
||||
|
||||
panthor_devfreq_update_utilization(pdevfreq);
|
||||
pdevfreq->last_busy_state = true;
|
||||
|
||||
spin_unlock_irqrestore(&pdevfreq->lock, irqflags);
|
||||
}
|
||||
|
||||
void panthor_devfreq_record_idle(struct panthor_device *ptdev)
|
||||
{
|
||||
struct panthor_devfreq *pdevfreq = ptdev->devfreq;
|
||||
unsigned long irqflags;
|
||||
|
||||
if (!pdevfreq->devfreq)
|
||||
return;
|
||||
|
||||
spin_lock_irqsave(&pdevfreq->lock, irqflags);
|
||||
|
||||
panthor_devfreq_update_utilization(pdevfreq);
|
||||
pdevfreq->last_busy_state = false;
|
||||
|
||||
spin_unlock_irqrestore(&pdevfreq->lock, irqflags);
|
||||
}
|
||||
21
drivers/gpu/drm/panthor/panthor_devfreq.h
Normal file
21
drivers/gpu/drm/panthor/panthor_devfreq.h
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0 or MIT */
|
||||
/* Copyright 2019 Collabora ltd. */
|
||||
|
||||
#ifndef __PANTHOR_DEVFREQ_H__
|
||||
#define __PANTHOR_DEVFREQ_H__
|
||||
|
||||
struct devfreq;
|
||||
struct thermal_cooling_device;
|
||||
|
||||
struct panthor_device;
|
||||
struct panthor_devfreq;
|
||||
|
||||
int panthor_devfreq_init(struct panthor_device *ptdev);
|
||||
|
||||
int panthor_devfreq_resume(struct panthor_device *ptdev);
|
||||
int panthor_devfreq_suspend(struct panthor_device *ptdev);
|
||||
|
||||
void panthor_devfreq_record_busy(struct panthor_device *ptdev);
|
||||
void panthor_devfreq_record_idle(struct panthor_device *ptdev);
|
||||
|
||||
#endif /* __PANTHOR_DEVFREQ_H__ */
|
||||
548
drivers/gpu/drm/panthor/panthor_device.c
Normal file
548
drivers/gpu/drm/panthor/panthor_device.c
Normal file
|
|
@ -0,0 +1,548 @@
|
|||
// SPDX-License-Identifier: GPL-2.0 or MIT
|
||||
/* Copyright 2018 Marty E. Plummer <hanetzer@startmail.com> */
|
||||
/* Copyright 2019 Linaro, Ltd, Rob Herring <robh@kernel.org> */
|
||||
/* Copyright 2023 Collabora ltd. */
|
||||
|
||||
#include <linux/clk.h>
|
||||
#include <linux/platform_device.h>
|
||||
#include <linux/pm_domain.h>
|
||||
#include <linux/pm_runtime.h>
|
||||
#include <linux/regulator/consumer.h>
|
||||
#include <linux/reset.h>
|
||||
|
||||
#include <drm/drm_drv.h>
|
||||
#include <drm/drm_managed.h>
|
||||
|
||||
#include "panthor_devfreq.h"
|
||||
#include "panthor_device.h"
|
||||
#include "panthor_fw.h"
|
||||
#include "panthor_gpu.h"
|
||||
#include "panthor_mmu.h"
|
||||
#include "panthor_regs.h"
|
||||
#include "panthor_sched.h"
|
||||
|
||||
static int panthor_clk_init(struct panthor_device *ptdev)
|
||||
{
|
||||
ptdev->clks.core = devm_clk_get(ptdev->base.dev, NULL);
|
||||
if (IS_ERR(ptdev->clks.core))
|
||||
return dev_err_probe(ptdev->base.dev,
|
||||
PTR_ERR(ptdev->clks.core),
|
||||
"get 'core' clock failed");
|
||||
|
||||
ptdev->clks.stacks = devm_clk_get_optional(ptdev->base.dev, "stacks");
|
||||
if (IS_ERR(ptdev->clks.stacks))
|
||||
return dev_err_probe(ptdev->base.dev,
|
||||
PTR_ERR(ptdev->clks.stacks),
|
||||
"get 'stacks' clock failed");
|
||||
|
||||
ptdev->clks.coregroup = devm_clk_get_optional(ptdev->base.dev, "coregroup");
|
||||
if (IS_ERR(ptdev->clks.coregroup))
|
||||
return dev_err_probe(ptdev->base.dev,
|
||||
PTR_ERR(ptdev->clks.coregroup),
|
||||
"get 'coregroup' clock failed");
|
||||
|
||||
drm_info(&ptdev->base, "clock rate = %lu\n", clk_get_rate(ptdev->clks.core));
|
||||
return 0;
|
||||
}
|
||||
|
||||
void panthor_device_unplug(struct panthor_device *ptdev)
|
||||
{
|
||||
/* This function can be called from two different path: the reset work
|
||||
* and the platform device remove callback. drm_dev_unplug() doesn't
|
||||
* deal with concurrent callers, so we have to protect drm_dev_unplug()
|
||||
* calls with our own lock, and bail out if the device is already
|
||||
* unplugged.
|
||||
*/
|
||||
mutex_lock(&ptdev->unplug.lock);
|
||||
if (drm_dev_is_unplugged(&ptdev->base)) {
|
||||
/* Someone beat us, release the lock and wait for the unplug
|
||||
* operation to be reported as done.
|
||||
**/
|
||||
mutex_unlock(&ptdev->unplug.lock);
|
||||
wait_for_completion(&ptdev->unplug.done);
|
||||
return;
|
||||
}
|
||||
|
||||
/* Call drm_dev_unplug() so any access to HW blocks happening after
|
||||
* that point get rejected.
|
||||
*/
|
||||
drm_dev_unplug(&ptdev->base);
|
||||
|
||||
/* We do the rest of the unplug with the unplug lock released,
|
||||
* future callers will wait on ptdev->unplug.done anyway.
|
||||
*/
|
||||
mutex_unlock(&ptdev->unplug.lock);
|
||||
|
||||
drm_WARN_ON(&ptdev->base, pm_runtime_get_sync(ptdev->base.dev) < 0);
|
||||
|
||||
/* Now, try to cleanly shutdown the GPU before the device resources
|
||||
* get reclaimed.
|
||||
*/
|
||||
panthor_sched_unplug(ptdev);
|
||||
panthor_fw_unplug(ptdev);
|
||||
panthor_mmu_unplug(ptdev);
|
||||
panthor_gpu_unplug(ptdev);
|
||||
|
||||
pm_runtime_dont_use_autosuspend(ptdev->base.dev);
|
||||
pm_runtime_put_sync_suspend(ptdev->base.dev);
|
||||
|
||||
/* Report the unplug operation as done to unblock concurrent
|
||||
* panthor_device_unplug() callers.
|
||||
*/
|
||||
complete_all(&ptdev->unplug.done);
|
||||
}
|
||||
|
||||
static void panthor_device_reset_cleanup(struct drm_device *ddev, void *data)
|
||||
{
|
||||
struct panthor_device *ptdev = container_of(ddev, struct panthor_device, base);
|
||||
|
||||
cancel_work_sync(&ptdev->reset.work);
|
||||
destroy_workqueue(ptdev->reset.wq);
|
||||
}
|
||||
|
||||
static void panthor_device_reset_work(struct work_struct *work)
|
||||
{
|
||||
struct panthor_device *ptdev = container_of(work, struct panthor_device, reset.work);
|
||||
int ret = 0, cookie;
|
||||
|
||||
if (atomic_read(&ptdev->pm.state) != PANTHOR_DEVICE_PM_STATE_ACTIVE) {
|
||||
/*
|
||||
* No need for a reset as the device has been (or will be)
|
||||
* powered down
|
||||
*/
|
||||
atomic_set(&ptdev->reset.pending, 0);
|
||||
return;
|
||||
}
|
||||
|
||||
if (!drm_dev_enter(&ptdev->base, &cookie))
|
||||
return;
|
||||
|
||||
panthor_sched_pre_reset(ptdev);
|
||||
panthor_fw_pre_reset(ptdev, true);
|
||||
panthor_mmu_pre_reset(ptdev);
|
||||
panthor_gpu_soft_reset(ptdev);
|
||||
panthor_gpu_l2_power_on(ptdev);
|
||||
panthor_mmu_post_reset(ptdev);
|
||||
ret = panthor_fw_post_reset(ptdev);
|
||||
if (ret)
|
||||
goto out_dev_exit;
|
||||
|
||||
atomic_set(&ptdev->reset.pending, 0);
|
||||
panthor_sched_post_reset(ptdev);
|
||||
|
||||
out_dev_exit:
|
||||
drm_dev_exit(cookie);
|
||||
|
||||
if (ret) {
|
||||
panthor_device_unplug(ptdev);
|
||||
drm_err(&ptdev->base, "Failed to boot MCU after reset, making device unusable.");
|
||||
}
|
||||
}
|
||||
|
||||
static bool panthor_device_is_initialized(struct panthor_device *ptdev)
|
||||
{
|
||||
return !!ptdev->scheduler;
|
||||
}
|
||||
|
||||
static void panthor_device_free_page(struct drm_device *ddev, void *data)
|
||||
{
|
||||
free_page((unsigned long)data);
|
||||
}
|
||||
|
||||
int panthor_device_init(struct panthor_device *ptdev)
|
||||
{
|
||||
struct resource *res;
|
||||
struct page *p;
|
||||
int ret;
|
||||
|
||||
ptdev->coherent = device_get_dma_attr(ptdev->base.dev) == DEV_DMA_COHERENT;
|
||||
|
||||
init_completion(&ptdev->unplug.done);
|
||||
ret = drmm_mutex_init(&ptdev->base, &ptdev->unplug.lock);
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
ret = drmm_mutex_init(&ptdev->base, &ptdev->pm.mmio_lock);
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
atomic_set(&ptdev->pm.state, PANTHOR_DEVICE_PM_STATE_SUSPENDED);
|
||||
p = alloc_page(GFP_KERNEL | __GFP_ZERO);
|
||||
if (!p)
|
||||
return -ENOMEM;
|
||||
|
||||
ptdev->pm.dummy_latest_flush = page_address(p);
|
||||
ret = drmm_add_action_or_reset(&ptdev->base, panthor_device_free_page,
|
||||
ptdev->pm.dummy_latest_flush);
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
/*
|
||||
* Set the dummy page holding the latest flush to 1. This will cause the
|
||||
* flush to avoided as we know it isn't necessary if the submission
|
||||
* happens while the dummy page is mapped. Zero cannot be used because
|
||||
* that means 'always flush'.
|
||||
*/
|
||||
*ptdev->pm.dummy_latest_flush = 1;
|
||||
|
||||
INIT_WORK(&ptdev->reset.work, panthor_device_reset_work);
|
||||
ptdev->reset.wq = alloc_ordered_workqueue("panthor-reset-wq", 0);
|
||||
if (!ptdev->reset.wq)
|
||||
return -ENOMEM;
|
||||
|
||||
ret = drmm_add_action_or_reset(&ptdev->base, panthor_device_reset_cleanup, NULL);
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
ret = panthor_clk_init(ptdev);
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
ret = panthor_devfreq_init(ptdev);
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
ptdev->iomem = devm_platform_get_and_ioremap_resource(to_platform_device(ptdev->base.dev),
|
||||
0, &res);
|
||||
if (IS_ERR(ptdev->iomem))
|
||||
return PTR_ERR(ptdev->iomem);
|
||||
|
||||
ptdev->phys_addr = res->start;
|
||||
|
||||
ret = devm_pm_runtime_enable(ptdev->base.dev);
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
ret = pm_runtime_resume_and_get(ptdev->base.dev);
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
ret = panthor_gpu_init(ptdev);
|
||||
if (ret)
|
||||
goto err_rpm_put;
|
||||
|
||||
ret = panthor_mmu_init(ptdev);
|
||||
if (ret)
|
||||
goto err_unplug_gpu;
|
||||
|
||||
ret = panthor_fw_init(ptdev);
|
||||
if (ret)
|
||||
goto err_unplug_mmu;
|
||||
|
||||
ret = panthor_sched_init(ptdev);
|
||||
if (ret)
|
||||
goto err_unplug_fw;
|
||||
|
||||
/* ~3 frames */
|
||||
pm_runtime_set_autosuspend_delay(ptdev->base.dev, 50);
|
||||
pm_runtime_use_autosuspend(ptdev->base.dev);
|
||||
|
||||
ret = drm_dev_register(&ptdev->base, 0);
|
||||
if (ret)
|
||||
goto err_disable_autosuspend;
|
||||
|
||||
pm_runtime_put_autosuspend(ptdev->base.dev);
|
||||
return 0;
|
||||
|
||||
err_disable_autosuspend:
|
||||
pm_runtime_dont_use_autosuspend(ptdev->base.dev);
|
||||
panthor_sched_unplug(ptdev);
|
||||
|
||||
err_unplug_fw:
|
||||
panthor_fw_unplug(ptdev);
|
||||
|
||||
err_unplug_mmu:
|
||||
panthor_mmu_unplug(ptdev);
|
||||
|
||||
err_unplug_gpu:
|
||||
panthor_gpu_unplug(ptdev);
|
||||
|
||||
err_rpm_put:
|
||||
pm_runtime_put_sync_suspend(ptdev->base.dev);
|
||||
return ret;
|
||||
}
|
||||
|
||||
#define PANTHOR_EXCEPTION(id) \
|
||||
[DRM_PANTHOR_EXCEPTION_ ## id] = { \
|
||||
.name = #id, \
|
||||
}
|
||||
|
||||
struct panthor_exception_info {
|
||||
const char *name;
|
||||
};
|
||||
|
||||
static const struct panthor_exception_info panthor_exception_infos[] = {
|
||||
PANTHOR_EXCEPTION(OK),
|
||||
PANTHOR_EXCEPTION(TERMINATED),
|
||||
PANTHOR_EXCEPTION(KABOOM),
|
||||
PANTHOR_EXCEPTION(EUREKA),
|
||||
PANTHOR_EXCEPTION(ACTIVE),
|
||||
PANTHOR_EXCEPTION(CS_RES_TERM),
|
||||
PANTHOR_EXCEPTION(CS_CONFIG_FAULT),
|
||||
PANTHOR_EXCEPTION(CS_ENDPOINT_FAULT),
|
||||
PANTHOR_EXCEPTION(CS_BUS_FAULT),
|
||||
PANTHOR_EXCEPTION(CS_INSTR_INVALID),
|
||||
PANTHOR_EXCEPTION(CS_CALL_STACK_OVERFLOW),
|
||||
PANTHOR_EXCEPTION(CS_INHERIT_FAULT),
|
||||
PANTHOR_EXCEPTION(INSTR_INVALID_PC),
|
||||
PANTHOR_EXCEPTION(INSTR_INVALID_ENC),
|
||||
PANTHOR_EXCEPTION(INSTR_BARRIER_FAULT),
|
||||
PANTHOR_EXCEPTION(DATA_INVALID_FAULT),
|
||||
PANTHOR_EXCEPTION(TILE_RANGE_FAULT),
|
||||
PANTHOR_EXCEPTION(ADDR_RANGE_FAULT),
|
||||
PANTHOR_EXCEPTION(IMPRECISE_FAULT),
|
||||
PANTHOR_EXCEPTION(OOM),
|
||||
PANTHOR_EXCEPTION(CSF_FW_INTERNAL_ERROR),
|
||||
PANTHOR_EXCEPTION(CSF_RES_EVICTION_TIMEOUT),
|
||||
PANTHOR_EXCEPTION(GPU_BUS_FAULT),
|
||||
PANTHOR_EXCEPTION(GPU_SHAREABILITY_FAULT),
|
||||
PANTHOR_EXCEPTION(SYS_SHAREABILITY_FAULT),
|
||||
PANTHOR_EXCEPTION(GPU_CACHEABILITY_FAULT),
|
||||
PANTHOR_EXCEPTION(TRANSLATION_FAULT_0),
|
||||
PANTHOR_EXCEPTION(TRANSLATION_FAULT_1),
|
||||
PANTHOR_EXCEPTION(TRANSLATION_FAULT_2),
|
||||
PANTHOR_EXCEPTION(TRANSLATION_FAULT_3),
|
||||
PANTHOR_EXCEPTION(TRANSLATION_FAULT_4),
|
||||
PANTHOR_EXCEPTION(PERM_FAULT_0),
|
||||
PANTHOR_EXCEPTION(PERM_FAULT_1),
|
||||
PANTHOR_EXCEPTION(PERM_FAULT_2),
|
||||
PANTHOR_EXCEPTION(PERM_FAULT_3),
|
||||
PANTHOR_EXCEPTION(ACCESS_FLAG_1),
|
||||
PANTHOR_EXCEPTION(ACCESS_FLAG_2),
|
||||
PANTHOR_EXCEPTION(ACCESS_FLAG_3),
|
||||
PANTHOR_EXCEPTION(ADDR_SIZE_FAULT_IN),
|
||||
PANTHOR_EXCEPTION(ADDR_SIZE_FAULT_OUT0),
|
||||
PANTHOR_EXCEPTION(ADDR_SIZE_FAULT_OUT1),
|
||||
PANTHOR_EXCEPTION(ADDR_SIZE_FAULT_OUT2),
|
||||
PANTHOR_EXCEPTION(ADDR_SIZE_FAULT_OUT3),
|
||||
PANTHOR_EXCEPTION(MEM_ATTR_FAULT_0),
|
||||
PANTHOR_EXCEPTION(MEM_ATTR_FAULT_1),
|
||||
PANTHOR_EXCEPTION(MEM_ATTR_FAULT_2),
|
||||
PANTHOR_EXCEPTION(MEM_ATTR_FAULT_3),
|
||||
};
|
||||
|
||||
const char *panthor_exception_name(struct panthor_device *ptdev, u32 exception_code)
|
||||
{
|
||||
if (exception_code >= ARRAY_SIZE(panthor_exception_infos) ||
|
||||
!panthor_exception_infos[exception_code].name)
|
||||
return "Unknown exception type";
|
||||
|
||||
return panthor_exception_infos[exception_code].name;
|
||||
}
|
||||
|
||||
static vm_fault_t panthor_mmio_vm_fault(struct vm_fault *vmf)
|
||||
{
|
||||
struct vm_area_struct *vma = vmf->vma;
|
||||
struct panthor_device *ptdev = vma->vm_private_data;
|
||||
u64 offset = (u64)vma->vm_pgoff << PAGE_SHIFT;
|
||||
unsigned long pfn;
|
||||
pgprot_t pgprot;
|
||||
vm_fault_t ret;
|
||||
bool active;
|
||||
int cookie;
|
||||
|
||||
if (!drm_dev_enter(&ptdev->base, &cookie))
|
||||
return VM_FAULT_SIGBUS;
|
||||
|
||||
mutex_lock(&ptdev->pm.mmio_lock);
|
||||
active = atomic_read(&ptdev->pm.state) == PANTHOR_DEVICE_PM_STATE_ACTIVE;
|
||||
|
||||
switch (offset) {
|
||||
case DRM_PANTHOR_USER_FLUSH_ID_MMIO_OFFSET:
|
||||
if (active)
|
||||
pfn = __phys_to_pfn(ptdev->phys_addr + CSF_GPU_LATEST_FLUSH_ID);
|
||||
else
|
||||
pfn = virt_to_pfn(ptdev->pm.dummy_latest_flush);
|
||||
break;
|
||||
|
||||
default:
|
||||
ret = VM_FAULT_SIGBUS;
|
||||
goto out_unlock;
|
||||
}
|
||||
|
||||
pgprot = vma->vm_page_prot;
|
||||
if (active)
|
||||
pgprot = pgprot_noncached(pgprot);
|
||||
|
||||
ret = vmf_insert_pfn_prot(vma, vmf->address, pfn, pgprot);
|
||||
|
||||
out_unlock:
|
||||
mutex_unlock(&ptdev->pm.mmio_lock);
|
||||
drm_dev_exit(cookie);
|
||||
return ret;
|
||||
}
|
||||
|
||||
static const struct vm_operations_struct panthor_mmio_vm_ops = {
|
||||
.fault = panthor_mmio_vm_fault,
|
||||
};
|
||||
|
||||
int panthor_device_mmap_io(struct panthor_device *ptdev, struct vm_area_struct *vma)
|
||||
{
|
||||
u64 offset = (u64)vma->vm_pgoff << PAGE_SHIFT;
|
||||
|
||||
switch (offset) {
|
||||
case DRM_PANTHOR_USER_FLUSH_ID_MMIO_OFFSET:
|
||||
if (vma->vm_end - vma->vm_start != PAGE_SIZE ||
|
||||
(vma->vm_flags & (VM_WRITE | VM_EXEC)))
|
||||
return -EINVAL;
|
||||
|
||||
break;
|
||||
|
||||
default:
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
/* Defer actual mapping to the fault handler. */
|
||||
vma->vm_private_data = ptdev;
|
||||
vma->vm_ops = &panthor_mmio_vm_ops;
|
||||
vma->vm_flags |= VM_IO | VM_DONTCOPY | VM_DONTEXPAND |
|
||||
VM_NORESERVE | VM_DONTDUMP | VM_PFNMAP;
|
||||
return 0;
|
||||
}
|
||||
|
||||
#ifdef CONFIG_PM
|
||||
int panthor_device_resume(struct device *dev)
|
||||
{
|
||||
struct panthor_device *ptdev = dev_get_drvdata(dev);
|
||||
int ret, cookie;
|
||||
|
||||
if (atomic_read(&ptdev->pm.state) != PANTHOR_DEVICE_PM_STATE_SUSPENDED)
|
||||
return -EINVAL;
|
||||
|
||||
atomic_set(&ptdev->pm.state, PANTHOR_DEVICE_PM_STATE_RESUMING);
|
||||
|
||||
ret = clk_prepare_enable(ptdev->clks.core);
|
||||
if (ret)
|
||||
goto err_set_suspended;
|
||||
|
||||
ret = clk_prepare_enable(ptdev->clks.stacks);
|
||||
if (ret)
|
||||
goto err_disable_core_clk;
|
||||
|
||||
ret = clk_prepare_enable(ptdev->clks.coregroup);
|
||||
if (ret)
|
||||
goto err_disable_stacks_clk;
|
||||
|
||||
ret = panthor_devfreq_resume(ptdev);
|
||||
if (ret)
|
||||
goto err_disable_coregroup_clk;
|
||||
|
||||
if (panthor_device_is_initialized(ptdev) &&
|
||||
drm_dev_enter(&ptdev->base, &cookie)) {
|
||||
panthor_gpu_resume(ptdev);
|
||||
panthor_mmu_resume(ptdev);
|
||||
ret = drm_WARN_ON(&ptdev->base, panthor_fw_resume(ptdev));
|
||||
if (!ret) {
|
||||
panthor_sched_resume(ptdev);
|
||||
} else {
|
||||
panthor_mmu_suspend(ptdev);
|
||||
panthor_gpu_suspend(ptdev);
|
||||
}
|
||||
|
||||
drm_dev_exit(cookie);
|
||||
|
||||
if (ret)
|
||||
goto err_suspend_devfreq;
|
||||
}
|
||||
|
||||
if (atomic_read(&ptdev->reset.pending))
|
||||
queue_work(ptdev->reset.wq, &ptdev->reset.work);
|
||||
|
||||
/* Clear all IOMEM mappings pointing to this device after we've
|
||||
* resumed. This way the fake mappings pointing to the dummy pages
|
||||
* are removed and the real iomem mapping will be restored on next
|
||||
* access.
|
||||
*/
|
||||
mutex_lock(&ptdev->pm.mmio_lock);
|
||||
unmap_mapping_range(ptdev->base.anon_inode->i_mapping,
|
||||
DRM_PANTHOR_USER_MMIO_OFFSET, 0, 1);
|
||||
atomic_set(&ptdev->pm.state, PANTHOR_DEVICE_PM_STATE_ACTIVE);
|
||||
mutex_unlock(&ptdev->pm.mmio_lock);
|
||||
return 0;
|
||||
|
||||
err_suspend_devfreq:
|
||||
panthor_devfreq_suspend(ptdev);
|
||||
|
||||
err_disable_coregroup_clk:
|
||||
clk_disable_unprepare(ptdev->clks.coregroup);
|
||||
|
||||
err_disable_stacks_clk:
|
||||
clk_disable_unprepare(ptdev->clks.stacks);
|
||||
|
||||
err_disable_core_clk:
|
||||
clk_disable_unprepare(ptdev->clks.core);
|
||||
|
||||
err_set_suspended:
|
||||
atomic_set(&ptdev->pm.state, PANTHOR_DEVICE_PM_STATE_SUSPENDED);
|
||||
return ret;
|
||||
}
|
||||
|
||||
int panthor_device_suspend(struct device *dev)
|
||||
{
|
||||
struct panthor_device *ptdev = dev_get_drvdata(dev);
|
||||
int ret, cookie;
|
||||
|
||||
if (atomic_read(&ptdev->pm.state) != PANTHOR_DEVICE_PM_STATE_ACTIVE)
|
||||
return -EINVAL;
|
||||
|
||||
/* Clear all IOMEM mappings pointing to this device before we
|
||||
* shutdown the power-domain and clocks. Failing to do that results
|
||||
* in external aborts when the process accesses the iomem region.
|
||||
* We change the state and call unmap_mapping_range() with the
|
||||
* mmio_lock held to make sure the vm_fault handler won't set up
|
||||
* invalid mappings.
|
||||
*/
|
||||
mutex_lock(&ptdev->pm.mmio_lock);
|
||||
atomic_set(&ptdev->pm.state, PANTHOR_DEVICE_PM_STATE_SUSPENDING);
|
||||
unmap_mapping_range(ptdev->base.anon_inode->i_mapping,
|
||||
DRM_PANTHOR_USER_MMIO_OFFSET, 0, 1);
|
||||
mutex_unlock(&ptdev->pm.mmio_lock);
|
||||
|
||||
if (panthor_device_is_initialized(ptdev) &&
|
||||
drm_dev_enter(&ptdev->base, &cookie)) {
|
||||
cancel_work_sync(&ptdev->reset.work);
|
||||
|
||||
/* We prepare everything as if we were resetting the GPU.
|
||||
* The end of the reset will happen in the resume path though.
|
||||
*/
|
||||
panthor_sched_suspend(ptdev);
|
||||
panthor_fw_suspend(ptdev);
|
||||
panthor_mmu_suspend(ptdev);
|
||||
panthor_gpu_suspend(ptdev);
|
||||
drm_dev_exit(cookie);
|
||||
}
|
||||
|
||||
ret = panthor_devfreq_suspend(ptdev);
|
||||
if (ret) {
|
||||
if (panthor_device_is_initialized(ptdev) &&
|
||||
drm_dev_enter(&ptdev->base, &cookie)) {
|
||||
panthor_gpu_resume(ptdev);
|
||||
panthor_mmu_resume(ptdev);
|
||||
drm_WARN_ON(&ptdev->base, panthor_fw_resume(ptdev));
|
||||
panthor_sched_resume(ptdev);
|
||||
drm_dev_exit(cookie);
|
||||
}
|
||||
|
||||
goto err_set_active;
|
||||
}
|
||||
|
||||
clk_disable_unprepare(ptdev->clks.coregroup);
|
||||
clk_disable_unprepare(ptdev->clks.stacks);
|
||||
clk_disable_unprepare(ptdev->clks.core);
|
||||
atomic_set(&ptdev->pm.state, PANTHOR_DEVICE_PM_STATE_SUSPENDED);
|
||||
return 0;
|
||||
|
||||
err_set_active:
|
||||
/* If something failed and we have to revert back to an
|
||||
* active state, we also need to clear the MMIO userspace
|
||||
* mappings, so any dumb pages that were mapped while we
|
||||
* were trying to suspend gets invalidated.
|
||||
*/
|
||||
mutex_lock(&ptdev->pm.mmio_lock);
|
||||
atomic_set(&ptdev->pm.state, PANTHOR_DEVICE_PM_STATE_ACTIVE);
|
||||
unmap_mapping_range(ptdev->base.anon_inode->i_mapping,
|
||||
DRM_PANTHOR_USER_MMIO_OFFSET, 0, 1);
|
||||
mutex_unlock(&ptdev->pm.mmio_lock);
|
||||
return ret;
|
||||
}
|
||||
#endif
|
||||
361
drivers/gpu/drm/panthor/panthor_device.h
Normal file
361
drivers/gpu/drm/panthor/panthor_device.h
Normal file
|
|
@ -0,0 +1,361 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0 or MIT */
|
||||
/* Copyright 2018 Marty E. Plummer <hanetzer@startmail.com> */
|
||||
/* Copyright 2019 Linaro, Ltd, Rob Herring <robh@kernel.org> */
|
||||
/* Copyright 2023 Collabora ltd. */
|
||||
|
||||
#ifndef __PANTHOR_DEVICE_H__
|
||||
#define __PANTHOR_DEVICE_H__
|
||||
|
||||
#include <linux/atomic.h>
|
||||
#include <linux/io-pgtable.h>
|
||||
#include <linux/regulator/consumer.h>
|
||||
#include <linux/sched.h>
|
||||
#include <linux/spinlock.h>
|
||||
|
||||
#include <drm/drm_device.h>
|
||||
#include <drm/drm_mm.h>
|
||||
#include <drm/gpu_scheduler.h>
|
||||
#include <drm/panthor_drm.h>
|
||||
|
||||
struct panthor_csf;
|
||||
struct panthor_csf_ctx;
|
||||
struct panthor_device;
|
||||
struct panthor_gpu;
|
||||
struct panthor_group_pool;
|
||||
struct panthor_heap_pool;
|
||||
struct panthor_job;
|
||||
struct panthor_mmu;
|
||||
struct panthor_fw;
|
||||
struct panthor_perfcnt;
|
||||
struct panthor_vm;
|
||||
struct panthor_vm_pool;
|
||||
|
||||
/**
|
||||
* enum panthor_device_pm_state - PM state
|
||||
*/
|
||||
enum panthor_device_pm_state {
|
||||
/** @PANTHOR_DEVICE_PM_STATE_SUSPENDED: Device is suspended. */
|
||||
PANTHOR_DEVICE_PM_STATE_SUSPENDED = 0,
|
||||
|
||||
/** @PANTHOR_DEVICE_PM_STATE_RESUMING: Device is being resumed. */
|
||||
PANTHOR_DEVICE_PM_STATE_RESUMING,
|
||||
|
||||
/** @PANTHOR_DEVICE_PM_STATE_ACTIVE: Device is active. */
|
||||
PANTHOR_DEVICE_PM_STATE_ACTIVE,
|
||||
|
||||
/** @PANTHOR_DEVICE_PM_STATE_SUSPENDING: Device is being suspended. */
|
||||
PANTHOR_DEVICE_PM_STATE_SUSPENDING,
|
||||
};
|
||||
|
||||
/**
|
||||
* struct panthor_irq - IRQ data
|
||||
*
|
||||
* Used to automate IRQ handling for the 3 different IRQs we have in this driver.
|
||||
*/
|
||||
struct panthor_irq {
|
||||
/** @ptdev: Panthor device */
|
||||
struct panthor_device *ptdev;
|
||||
|
||||
/** @irq: IRQ number. */
|
||||
int irq;
|
||||
|
||||
/** @mask: Current mask being applied to xxx_INT_MASK. */
|
||||
u32 mask;
|
||||
|
||||
/** @suspended: Set to true when the IRQ is suspended. */
|
||||
atomic_t suspended;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct panthor_device - Panthor device
|
||||
*/
|
||||
struct panthor_device {
|
||||
/** @base: Base drm_device. */
|
||||
struct drm_device base;
|
||||
|
||||
/** @phys_addr: Physical address of the iomem region. */
|
||||
phys_addr_t phys_addr;
|
||||
|
||||
/** @iomem: CPU mapping of the IOMEM region. */
|
||||
void __iomem *iomem;
|
||||
|
||||
/** @clks: GPU clocks. */
|
||||
struct {
|
||||
/** @core: Core clock. */
|
||||
struct clk *core;
|
||||
|
||||
/** @stacks: Stacks clock. This clock is optional. */
|
||||
struct clk *stacks;
|
||||
|
||||
/** @coregroup: Core group clock. This clock is optional. */
|
||||
struct clk *coregroup;
|
||||
} clks;
|
||||
|
||||
/** @coherent: True if the CPU/GPU are memory coherent. */
|
||||
bool coherent;
|
||||
|
||||
/** @gpu_info: GPU information. */
|
||||
struct drm_panthor_gpu_info gpu_info;
|
||||
|
||||
/** @csif_info: Command stream interface information. */
|
||||
struct drm_panthor_csif_info csif_info;
|
||||
|
||||
/** @gpu: GPU management data. */
|
||||
struct panthor_gpu *gpu;
|
||||
|
||||
/** @fw: FW management data. */
|
||||
struct panthor_fw *fw;
|
||||
|
||||
/** @mmu: MMU management data. */
|
||||
struct panthor_mmu *mmu;
|
||||
|
||||
/** @scheduler: Scheduler management data. */
|
||||
struct panthor_scheduler *scheduler;
|
||||
|
||||
/** @devfreq: Device frequency scaling management data. */
|
||||
struct panthor_devfreq *devfreq;
|
||||
|
||||
/** @unplug: Device unplug related fields. */
|
||||
struct {
|
||||
/** @lock: Lock used to serialize unplug operations. */
|
||||
struct mutex lock;
|
||||
|
||||
/**
|
||||
* @done: Completion object signaled when the unplug
|
||||
* operation is done.
|
||||
*/
|
||||
struct completion done;
|
||||
} unplug;
|
||||
|
||||
/** @reset: Reset related fields. */
|
||||
struct {
|
||||
/** @wq: Ordered worqueud used to schedule reset operations. */
|
||||
struct workqueue_struct *wq;
|
||||
|
||||
/** @work: Reset work. */
|
||||
struct work_struct work;
|
||||
|
||||
/** @pending: Set to true if a reset is pending. */
|
||||
atomic_t pending;
|
||||
} reset;
|
||||
|
||||
/** @pm: Power management related data. */
|
||||
struct {
|
||||
/** @state: Power state. */
|
||||
atomic_t state;
|
||||
|
||||
/**
|
||||
* @mmio_lock: Lock protecting MMIO userspace CPU mappings.
|
||||
*
|
||||
* This is needed to ensure we map the dummy IO pages when
|
||||
* the device is being suspended, and the real IO pages when
|
||||
* the device is being resumed. We can't just do with the
|
||||
* state atomicity to deal with this race.
|
||||
*/
|
||||
struct mutex mmio_lock;
|
||||
|
||||
/**
|
||||
* @dummy_latest_flush: Dummy LATEST_FLUSH page.
|
||||
*
|
||||
* Used to replace the real LATEST_FLUSH page when the GPU
|
||||
* is suspended.
|
||||
*/
|
||||
u32 *dummy_latest_flush;
|
||||
} pm;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct panthor_file - Panthor file
|
||||
*/
|
||||
struct panthor_file {
|
||||
/** @ptdev: Device attached to this file. */
|
||||
struct panthor_device *ptdev;
|
||||
|
||||
/** @vms: VM pool attached to this file. */
|
||||
struct panthor_vm_pool *vms;
|
||||
|
||||
/** @groups: Scheduling group pool attached to this file. */
|
||||
struct panthor_group_pool *groups;
|
||||
};
|
||||
|
||||
int panthor_device_init(struct panthor_device *ptdev);
|
||||
void panthor_device_unplug(struct panthor_device *ptdev);
|
||||
|
||||
/**
|
||||
* panthor_device_schedule_reset() - Schedules a reset operation
|
||||
*/
|
||||
static inline void panthor_device_schedule_reset(struct panthor_device *ptdev)
|
||||
{
|
||||
if (!atomic_cmpxchg(&ptdev->reset.pending, 0, 1) &&
|
||||
atomic_read(&ptdev->pm.state) == PANTHOR_DEVICE_PM_STATE_ACTIVE)
|
||||
queue_work(ptdev->reset.wq, &ptdev->reset.work);
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_device_reset_is_pending() - Checks if a reset is pending.
|
||||
*
|
||||
* Return: true if a reset is pending, false otherwise.
|
||||
*/
|
||||
static inline bool panthor_device_reset_is_pending(struct panthor_device *ptdev)
|
||||
{
|
||||
return atomic_read(&ptdev->reset.pending) != 0;
|
||||
}
|
||||
|
||||
int panthor_device_mmap_io(struct panthor_device *ptdev,
|
||||
struct vm_area_struct *vma);
|
||||
|
||||
int panthor_device_resume(struct device *dev);
|
||||
int panthor_device_suspend(struct device *dev);
|
||||
|
||||
enum drm_panthor_exception_type {
|
||||
DRM_PANTHOR_EXCEPTION_OK = 0x00,
|
||||
DRM_PANTHOR_EXCEPTION_TERMINATED = 0x04,
|
||||
DRM_PANTHOR_EXCEPTION_KABOOM = 0x05,
|
||||
DRM_PANTHOR_EXCEPTION_EUREKA = 0x06,
|
||||
DRM_PANTHOR_EXCEPTION_ACTIVE = 0x08,
|
||||
DRM_PANTHOR_EXCEPTION_CS_RES_TERM = 0x0f,
|
||||
DRM_PANTHOR_EXCEPTION_MAX_NON_FAULT = 0x3f,
|
||||
DRM_PANTHOR_EXCEPTION_CS_CONFIG_FAULT = 0x40,
|
||||
DRM_PANTHOR_EXCEPTION_CS_ENDPOINT_FAULT = 0x44,
|
||||
DRM_PANTHOR_EXCEPTION_CS_BUS_FAULT = 0x48,
|
||||
DRM_PANTHOR_EXCEPTION_CS_INSTR_INVALID = 0x49,
|
||||
DRM_PANTHOR_EXCEPTION_CS_CALL_STACK_OVERFLOW = 0x4a,
|
||||
DRM_PANTHOR_EXCEPTION_CS_INHERIT_FAULT = 0x4b,
|
||||
DRM_PANTHOR_EXCEPTION_INSTR_INVALID_PC = 0x50,
|
||||
DRM_PANTHOR_EXCEPTION_INSTR_INVALID_ENC = 0x51,
|
||||
DRM_PANTHOR_EXCEPTION_INSTR_BARRIER_FAULT = 0x55,
|
||||
DRM_PANTHOR_EXCEPTION_DATA_INVALID_FAULT = 0x58,
|
||||
DRM_PANTHOR_EXCEPTION_TILE_RANGE_FAULT = 0x59,
|
||||
DRM_PANTHOR_EXCEPTION_ADDR_RANGE_FAULT = 0x5a,
|
||||
DRM_PANTHOR_EXCEPTION_IMPRECISE_FAULT = 0x5b,
|
||||
DRM_PANTHOR_EXCEPTION_OOM = 0x60,
|
||||
DRM_PANTHOR_EXCEPTION_CSF_FW_INTERNAL_ERROR = 0x68,
|
||||
DRM_PANTHOR_EXCEPTION_CSF_RES_EVICTION_TIMEOUT = 0x69,
|
||||
DRM_PANTHOR_EXCEPTION_GPU_BUS_FAULT = 0x80,
|
||||
DRM_PANTHOR_EXCEPTION_GPU_SHAREABILITY_FAULT = 0x88,
|
||||
DRM_PANTHOR_EXCEPTION_SYS_SHAREABILITY_FAULT = 0x89,
|
||||
DRM_PANTHOR_EXCEPTION_GPU_CACHEABILITY_FAULT = 0x8a,
|
||||
DRM_PANTHOR_EXCEPTION_TRANSLATION_FAULT_0 = 0xc0,
|
||||
DRM_PANTHOR_EXCEPTION_TRANSLATION_FAULT_1 = 0xc1,
|
||||
DRM_PANTHOR_EXCEPTION_TRANSLATION_FAULT_2 = 0xc2,
|
||||
DRM_PANTHOR_EXCEPTION_TRANSLATION_FAULT_3 = 0xc3,
|
||||
DRM_PANTHOR_EXCEPTION_TRANSLATION_FAULT_4 = 0xc4,
|
||||
DRM_PANTHOR_EXCEPTION_PERM_FAULT_0 = 0xc8,
|
||||
DRM_PANTHOR_EXCEPTION_PERM_FAULT_1 = 0xc9,
|
||||
DRM_PANTHOR_EXCEPTION_PERM_FAULT_2 = 0xca,
|
||||
DRM_PANTHOR_EXCEPTION_PERM_FAULT_3 = 0xcb,
|
||||
DRM_PANTHOR_EXCEPTION_ACCESS_FLAG_1 = 0xd9,
|
||||
DRM_PANTHOR_EXCEPTION_ACCESS_FLAG_2 = 0xda,
|
||||
DRM_PANTHOR_EXCEPTION_ACCESS_FLAG_3 = 0xdb,
|
||||
DRM_PANTHOR_EXCEPTION_ADDR_SIZE_FAULT_IN = 0xe0,
|
||||
DRM_PANTHOR_EXCEPTION_ADDR_SIZE_FAULT_OUT0 = 0xe4,
|
||||
DRM_PANTHOR_EXCEPTION_ADDR_SIZE_FAULT_OUT1 = 0xe5,
|
||||
DRM_PANTHOR_EXCEPTION_ADDR_SIZE_FAULT_OUT2 = 0xe6,
|
||||
DRM_PANTHOR_EXCEPTION_ADDR_SIZE_FAULT_OUT3 = 0xe7,
|
||||
DRM_PANTHOR_EXCEPTION_MEM_ATTR_FAULT_0 = 0xe8,
|
||||
DRM_PANTHOR_EXCEPTION_MEM_ATTR_FAULT_1 = 0xe9,
|
||||
DRM_PANTHOR_EXCEPTION_MEM_ATTR_FAULT_2 = 0xea,
|
||||
DRM_PANTHOR_EXCEPTION_MEM_ATTR_FAULT_3 = 0xeb,
|
||||
};
|
||||
|
||||
/**
|
||||
* panthor_exception_is_fault() - Checks if an exception is a fault.
|
||||
*
|
||||
* Return: true if the exception is a fault, false otherwise.
|
||||
*/
|
||||
static inline bool
|
||||
panthor_exception_is_fault(u32 exception_code)
|
||||
{
|
||||
return exception_code > DRM_PANTHOR_EXCEPTION_MAX_NON_FAULT;
|
||||
}
|
||||
|
||||
const char *panthor_exception_name(struct panthor_device *ptdev,
|
||||
u32 exception_code);
|
||||
|
||||
/**
|
||||
* PANTHOR_IRQ_HANDLER() - Define interrupt handlers and the interrupt
|
||||
* registration function.
|
||||
*
|
||||
* The boiler-plate to gracefully deal with shared interrupts is
|
||||
* auto-generated. All you have to do is call PANTHOR_IRQ_HANDLER()
|
||||
* just after the actual handler. The handler prototype is:
|
||||
*
|
||||
* void (*handler)(struct panthor_device *, u32 status);
|
||||
*/
|
||||
#define PANTHOR_IRQ_HANDLER(__name, __reg_prefix, __handler) \
|
||||
static irqreturn_t panthor_ ## __name ## _irq_raw_handler(int irq, void *data) \
|
||||
{ \
|
||||
struct panthor_irq *pirq = data; \
|
||||
struct panthor_device *ptdev = pirq->ptdev; \
|
||||
\
|
||||
if (atomic_read(&pirq->suspended)) \
|
||||
return IRQ_NONE; \
|
||||
if (!gpu_read(ptdev, __reg_prefix ## _INT_STAT)) \
|
||||
return IRQ_NONE; \
|
||||
\
|
||||
gpu_write(ptdev, __reg_prefix ## _INT_MASK, 0); \
|
||||
return IRQ_WAKE_THREAD; \
|
||||
} \
|
||||
\
|
||||
static irqreturn_t panthor_ ## __name ## _irq_threaded_handler(int irq, void *data) \
|
||||
{ \
|
||||
struct panthor_irq *pirq = data; \
|
||||
struct panthor_device *ptdev = pirq->ptdev; \
|
||||
irqreturn_t ret = IRQ_NONE; \
|
||||
\
|
||||
while (true) { \
|
||||
u32 status = gpu_read(ptdev, __reg_prefix ## _INT_RAWSTAT) & pirq->mask; \
|
||||
\
|
||||
if (!status) \
|
||||
break; \
|
||||
\
|
||||
gpu_write(ptdev, __reg_prefix ## _INT_CLEAR, status); \
|
||||
\
|
||||
__handler(ptdev, status); \
|
||||
ret = IRQ_HANDLED; \
|
||||
} \
|
||||
\
|
||||
if (!atomic_read(&pirq->suspended)) \
|
||||
gpu_write(ptdev, __reg_prefix ## _INT_MASK, pirq->mask); \
|
||||
\
|
||||
return ret; \
|
||||
} \
|
||||
\
|
||||
static inline void panthor_ ## __name ## _irq_suspend(struct panthor_irq *pirq) \
|
||||
{ \
|
||||
int cookie; \
|
||||
\
|
||||
pirq->mask = 0; \
|
||||
gpu_write(pirq->ptdev, __reg_prefix ## _INT_MASK, 0); \
|
||||
synchronize_irq(pirq->irq); \
|
||||
atomic_set(&pirq->suspended, true); \
|
||||
} \
|
||||
\
|
||||
static inline void panthor_ ## __name ## _irq_resume(struct panthor_irq *pirq, u32 mask) \
|
||||
{ \
|
||||
int cookie; \
|
||||
\
|
||||
atomic_set(&pirq->suspended, false); \
|
||||
pirq->mask = mask; \
|
||||
gpu_write(pirq->ptdev, __reg_prefix ## _INT_CLEAR, mask); \
|
||||
gpu_write(pirq->ptdev, __reg_prefix ## _INT_MASK, mask); \
|
||||
} \
|
||||
\
|
||||
static int panthor_request_ ## __name ## _irq(struct panthor_device *ptdev, \
|
||||
struct panthor_irq *pirq, \
|
||||
int irq, u32 mask) \
|
||||
{ \
|
||||
pirq->ptdev = ptdev; \
|
||||
pirq->irq = irq; \
|
||||
panthor_ ## __name ## _irq_resume(pirq, mask); \
|
||||
\
|
||||
return devm_request_threaded_irq(ptdev->base.dev, irq, \
|
||||
panthor_ ## __name ## _irq_raw_handler, \
|
||||
panthor_ ## __name ## _irq_threaded_handler, \
|
||||
IRQF_SHARED, KBUILD_MODNAME "-" # __name, \
|
||||
pirq); \
|
||||
}
|
||||
|
||||
extern struct workqueue_struct *panthor_cleanup_wq;
|
||||
|
||||
#endif
|
||||
1477
drivers/gpu/drm/panthor/panthor_drv.c
Normal file
1477
drivers/gpu/drm/panthor/panthor_drv.c
Normal file
File diff suppressed because it is too large
Load diff
1362
drivers/gpu/drm/panthor/panthor_fw.c
Normal file
1362
drivers/gpu/drm/panthor/panthor_fw.c
Normal file
File diff suppressed because it is too large
Load diff
503
drivers/gpu/drm/panthor/panthor_fw.h
Normal file
503
drivers/gpu/drm/panthor/panthor_fw.h
Normal file
|
|
@ -0,0 +1,503 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0 or MIT */
|
||||
/* Copyright 2023 Collabora ltd. */
|
||||
|
||||
#ifndef __PANTHOR_MCU_H__
|
||||
#define __PANTHOR_MCU_H__
|
||||
|
||||
#include <linux/types.h>
|
||||
|
||||
struct panthor_device;
|
||||
struct panthor_kernel_bo;
|
||||
|
||||
#define MAX_CSGS 31
|
||||
#define MAX_CS_PER_CSG 32
|
||||
|
||||
struct panthor_fw_ringbuf_input_iface {
|
||||
u64 insert;
|
||||
u64 extract;
|
||||
};
|
||||
|
||||
struct panthor_fw_ringbuf_output_iface {
|
||||
u64 extract;
|
||||
u32 active;
|
||||
};
|
||||
|
||||
struct panthor_fw_cs_control_iface {
|
||||
#define CS_FEATURES_WORK_REGS(x) (((x) & GENMASK(7, 0)) + 1)
|
||||
#define CS_FEATURES_SCOREBOARDS(x) (((x) & GENMASK(15, 8)) >> 8)
|
||||
#define CS_FEATURES_COMPUTE BIT(16)
|
||||
#define CS_FEATURES_FRAGMENT BIT(17)
|
||||
#define CS_FEATURES_TILER BIT(18)
|
||||
u32 features;
|
||||
u32 input_va;
|
||||
u32 output_va;
|
||||
};
|
||||
|
||||
struct panthor_fw_cs_input_iface {
|
||||
#define CS_STATE_MASK GENMASK(2, 0)
|
||||
#define CS_STATE_STOP 0
|
||||
#define CS_STATE_START 1
|
||||
#define CS_EXTRACT_EVENT BIT(4)
|
||||
#define CS_IDLE_SYNC_WAIT BIT(8)
|
||||
#define CS_IDLE_PROTM_PENDING BIT(9)
|
||||
#define CS_IDLE_EMPTY BIT(10)
|
||||
#define CS_IDLE_RESOURCE_REQ BIT(11)
|
||||
#define CS_TILER_OOM BIT(26)
|
||||
#define CS_PROTM_PENDING BIT(27)
|
||||
#define CS_FATAL BIT(30)
|
||||
#define CS_FAULT BIT(31)
|
||||
#define CS_REQ_MASK (CS_STATE_MASK | \
|
||||
CS_EXTRACT_EVENT | \
|
||||
CS_IDLE_SYNC_WAIT | \
|
||||
CS_IDLE_PROTM_PENDING | \
|
||||
CS_IDLE_EMPTY | \
|
||||
CS_IDLE_RESOURCE_REQ)
|
||||
#define CS_EVT_MASK (CS_TILER_OOM | \
|
||||
CS_PROTM_PENDING | \
|
||||
CS_FATAL | \
|
||||
CS_FAULT)
|
||||
u32 req;
|
||||
|
||||
#define CS_CONFIG_PRIORITY(x) ((x) & GENMASK(3, 0))
|
||||
#define CS_CONFIG_DOORBELL(x) (((x) << 8) & GENMASK(15, 8))
|
||||
u32 config;
|
||||
u32 reserved1;
|
||||
u32 ack_irq_mask;
|
||||
u64 ringbuf_base;
|
||||
u32 ringbuf_size;
|
||||
u32 reserved2;
|
||||
u64 heap_start;
|
||||
u64 heap_end;
|
||||
u64 ringbuf_input;
|
||||
u64 ringbuf_output;
|
||||
u32 instr_config;
|
||||
u32 instrbuf_size;
|
||||
u64 instrbuf_base;
|
||||
u64 instrbuf_offset_ptr;
|
||||
};
|
||||
|
||||
struct panthor_fw_cs_output_iface {
|
||||
u32 ack;
|
||||
u32 reserved1[15];
|
||||
u64 status_cmd_ptr;
|
||||
|
||||
#define CS_STATUS_WAIT_SB_MASK GENMASK(15, 0)
|
||||
#define CS_STATUS_WAIT_SB_SRC_MASK GENMASK(19, 16)
|
||||
#define CS_STATUS_WAIT_SB_SRC_NONE (0 << 16)
|
||||
#define CS_STATUS_WAIT_SB_SRC_WAIT (8 << 16)
|
||||
#define CS_STATUS_WAIT_SYNC_COND_LE (0 << 24)
|
||||
#define CS_STATUS_WAIT_SYNC_COND_GT (1 << 24)
|
||||
#define CS_STATUS_WAIT_SYNC_COND_MASK GENMASK(27, 24)
|
||||
#define CS_STATUS_WAIT_PROGRESS BIT(28)
|
||||
#define CS_STATUS_WAIT_PROTM BIT(29)
|
||||
#define CS_STATUS_WAIT_SYNC_64B BIT(30)
|
||||
#define CS_STATUS_WAIT_SYNC BIT(31)
|
||||
u32 status_wait;
|
||||
u32 status_req_resource;
|
||||
u64 status_wait_sync_ptr;
|
||||
u32 status_wait_sync_value;
|
||||
u32 status_scoreboards;
|
||||
|
||||
#define CS_STATUS_BLOCKED_REASON_UNBLOCKED 0
|
||||
#define CS_STATUS_BLOCKED_REASON_SB_WAIT 1
|
||||
#define CS_STATUS_BLOCKED_REASON_PROGRESS_WAIT 2
|
||||
#define CS_STATUS_BLOCKED_REASON_SYNC_WAIT 3
|
||||
#define CS_STATUS_BLOCKED_REASON_DEFERRED 5
|
||||
#define CS_STATUS_BLOCKED_REASON_RES 6
|
||||
#define CS_STATUS_BLOCKED_REASON_FLUSH 7
|
||||
#define CS_STATUS_BLOCKED_REASON_MASK GENMASK(3, 0)
|
||||
u32 status_blocked_reason;
|
||||
u32 status_wait_sync_value_hi;
|
||||
u32 reserved2[6];
|
||||
|
||||
#define CS_EXCEPTION_TYPE(x) ((x) & GENMASK(7, 0))
|
||||
#define CS_EXCEPTION_DATA(x) (((x) >> 8) & GENMASK(23, 0))
|
||||
u32 fault;
|
||||
u32 fatal;
|
||||
u64 fault_info;
|
||||
u64 fatal_info;
|
||||
u32 reserved3[10];
|
||||
u32 heap_vt_start;
|
||||
u32 heap_vt_end;
|
||||
u32 reserved4;
|
||||
u32 heap_frag_end;
|
||||
u64 heap_address;
|
||||
};
|
||||
|
||||
struct panthor_fw_csg_control_iface {
|
||||
u32 features;
|
||||
u32 input_va;
|
||||
u32 output_va;
|
||||
u32 suspend_size;
|
||||
u32 protm_suspend_size;
|
||||
u32 stream_num;
|
||||
u32 stream_stride;
|
||||
};
|
||||
|
||||
struct panthor_fw_csg_input_iface {
|
||||
#define CSG_STATE_MASK GENMASK(2, 0)
|
||||
#define CSG_STATE_TERMINATE 0
|
||||
#define CSG_STATE_START 1
|
||||
#define CSG_STATE_SUSPEND 2
|
||||
#define CSG_STATE_RESUME 3
|
||||
#define CSG_ENDPOINT_CONFIG BIT(4)
|
||||
#define CSG_STATUS_UPDATE BIT(5)
|
||||
#define CSG_SYNC_UPDATE BIT(28)
|
||||
#define CSG_IDLE BIT(29)
|
||||
#define CSG_DOORBELL BIT(30)
|
||||
#define CSG_PROGRESS_TIMER_EVENT BIT(31)
|
||||
#define CSG_REQ_MASK (CSG_STATE_MASK | \
|
||||
CSG_ENDPOINT_CONFIG | \
|
||||
CSG_STATUS_UPDATE)
|
||||
#define CSG_EVT_MASK (CSG_SYNC_UPDATE | \
|
||||
CSG_IDLE | \
|
||||
CSG_PROGRESS_TIMER_EVENT)
|
||||
u32 req;
|
||||
u32 ack_irq_mask;
|
||||
|
||||
u32 doorbell_req;
|
||||
u32 cs_irq_ack;
|
||||
u32 reserved1[4];
|
||||
u64 allow_compute;
|
||||
u64 allow_fragment;
|
||||
u32 allow_other;
|
||||
|
||||
#define CSG_EP_REQ_COMPUTE(x) ((x) & GENMASK(7, 0))
|
||||
#define CSG_EP_REQ_FRAGMENT(x) (((x) << 8) & GENMASK(15, 8))
|
||||
#define CSG_EP_REQ_TILER(x) (((x) << 16) & GENMASK(19, 16))
|
||||
#define CSG_EP_REQ_EXCL_COMPUTE BIT(20)
|
||||
#define CSG_EP_REQ_EXCL_FRAGMENT BIT(21)
|
||||
#define CSG_EP_REQ_PRIORITY(x) (((x) << 28) & GENMASK(31, 28))
|
||||
#define CSG_EP_REQ_PRIORITY_MASK GENMASK(31, 28)
|
||||
u32 endpoint_req;
|
||||
u32 reserved2[2];
|
||||
u64 suspend_buf;
|
||||
u64 protm_suspend_buf;
|
||||
u32 config;
|
||||
u32 iter_trace_config;
|
||||
};
|
||||
|
||||
struct panthor_fw_csg_output_iface {
|
||||
u32 ack;
|
||||
u32 reserved1;
|
||||
u32 doorbell_ack;
|
||||
u32 cs_irq_req;
|
||||
u32 status_endpoint_current;
|
||||
u32 status_endpoint_req;
|
||||
|
||||
#define CSG_STATUS_STATE_IS_IDLE BIT(0)
|
||||
u32 status_state;
|
||||
u32 resource_dep;
|
||||
};
|
||||
|
||||
struct panthor_fw_global_control_iface {
|
||||
u32 version;
|
||||
u32 features;
|
||||
u32 input_va;
|
||||
u32 output_va;
|
||||
u32 group_num;
|
||||
u32 group_stride;
|
||||
u32 perfcnt_size;
|
||||
u32 instr_features;
|
||||
};
|
||||
|
||||
struct panthor_fw_global_input_iface {
|
||||
#define GLB_HALT BIT(0)
|
||||
#define GLB_CFG_PROGRESS_TIMER BIT(1)
|
||||
#define GLB_CFG_ALLOC_EN BIT(2)
|
||||
#define GLB_CFG_POWEROFF_TIMER BIT(3)
|
||||
#define GLB_PROTM_ENTER BIT(4)
|
||||
#define GLB_PERFCNT_EN BIT(5)
|
||||
#define GLB_PERFCNT_SAMPLE BIT(6)
|
||||
#define GLB_COUNTER_EN BIT(7)
|
||||
#define GLB_PING BIT(8)
|
||||
#define GLB_FWCFG_UPDATE BIT(9)
|
||||
#define GLB_IDLE_EN BIT(10)
|
||||
#define GLB_SLEEP BIT(12)
|
||||
#define GLB_INACTIVE_COMPUTE BIT(20)
|
||||
#define GLB_INACTIVE_FRAGMENT BIT(21)
|
||||
#define GLB_INACTIVE_TILER BIT(22)
|
||||
#define GLB_PROTM_EXIT BIT(23)
|
||||
#define GLB_PERFCNT_THRESHOLD BIT(24)
|
||||
#define GLB_PERFCNT_OVERFLOW BIT(25)
|
||||
#define GLB_IDLE BIT(26)
|
||||
#define GLB_DBG_CSF BIT(30)
|
||||
#define GLB_DBG_HOST BIT(31)
|
||||
#define GLB_REQ_MASK GENMASK(10, 0)
|
||||
#define GLB_EVT_MASK GENMASK(26, 20)
|
||||
u32 req;
|
||||
u32 ack_irq_mask;
|
||||
u32 doorbell_req;
|
||||
u32 reserved1;
|
||||
u32 progress_timer;
|
||||
|
||||
#define GLB_TIMER_VAL(x) ((x) & GENMASK(30, 0))
|
||||
#define GLB_TIMER_SOURCE_GPU_COUNTER BIT(31)
|
||||
u32 poweroff_timer;
|
||||
u64 core_en_mask;
|
||||
u32 reserved2;
|
||||
u32 perfcnt_as;
|
||||
u64 perfcnt_base;
|
||||
u32 perfcnt_extract;
|
||||
u32 reserved3[3];
|
||||
u32 perfcnt_config;
|
||||
u32 perfcnt_csg_select;
|
||||
u32 perfcnt_fw_enable;
|
||||
u32 perfcnt_csg_enable;
|
||||
u32 perfcnt_csf_enable;
|
||||
u32 perfcnt_shader_enable;
|
||||
u32 perfcnt_tiler_enable;
|
||||
u32 perfcnt_mmu_l2_enable;
|
||||
u32 reserved4[8];
|
||||
u32 idle_timer;
|
||||
};
|
||||
|
||||
enum panthor_fw_halt_status {
|
||||
PANTHOR_FW_HALT_OK = 0,
|
||||
PANTHOR_FW_HALT_ON_PANIC = 0x4e,
|
||||
PANTHOR_FW_HALT_ON_WATCHDOG_EXPIRATION = 0x4f,
|
||||
};
|
||||
|
||||
struct panthor_fw_global_output_iface {
|
||||
u32 ack;
|
||||
u32 reserved1;
|
||||
u32 doorbell_ack;
|
||||
u32 reserved2;
|
||||
u32 halt_status;
|
||||
u32 perfcnt_status;
|
||||
u32 perfcnt_insert;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct panthor_fw_cs_iface - Firmware command stream slot interface
|
||||
*/
|
||||
struct panthor_fw_cs_iface {
|
||||
/**
|
||||
* @lock: Lock protecting access to the panthor_fw_cs_input_iface::req
|
||||
* field.
|
||||
*
|
||||
* Needed so we can update the req field concurrently from the interrupt
|
||||
* handler and the scheduler logic.
|
||||
*
|
||||
* TODO: Ideally we'd want to use a cmpxchg() to update the req, but FW
|
||||
* interface sections are mapped uncached/write-combined right now, and
|
||||
* using cmpxchg() on such mappings leads to SError faults. Revisit when
|
||||
* we have 'SHARED' GPU mappings hooked up.
|
||||
*/
|
||||
spinlock_t lock;
|
||||
|
||||
/**
|
||||
* @control: Command stream slot control interface.
|
||||
*
|
||||
* Used to expose command stream slot properties.
|
||||
*
|
||||
* This interface is read-only.
|
||||
*/
|
||||
struct panthor_fw_cs_control_iface *control;
|
||||
|
||||
/**
|
||||
* @input: Command stream slot input interface.
|
||||
*
|
||||
* Used for host updates/events.
|
||||
*/
|
||||
struct panthor_fw_cs_input_iface *input;
|
||||
|
||||
/**
|
||||
* @output: Command stream slot output interface.
|
||||
*
|
||||
* Used for FW updates/events.
|
||||
*
|
||||
* This interface is read-only.
|
||||
*/
|
||||
const struct panthor_fw_cs_output_iface *output;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct panthor_fw_csg_iface - Firmware command stream group slot interface
|
||||
*/
|
||||
struct panthor_fw_csg_iface {
|
||||
/**
|
||||
* @lock: Lock protecting access to the panthor_fw_csg_input_iface::req
|
||||
* field.
|
||||
*
|
||||
* Needed so we can update the req field concurrently from the interrupt
|
||||
* handler and the scheduler logic.
|
||||
*
|
||||
* TODO: Ideally we'd want to use a cmpxchg() to update the req, but FW
|
||||
* interface sections are mapped uncached/write-combined right now, and
|
||||
* using cmpxchg() on such mappings leads to SError faults. Revisit when
|
||||
* we have 'SHARED' GPU mappings hooked up.
|
||||
*/
|
||||
spinlock_t lock;
|
||||
|
||||
/**
|
||||
* @control: Command stream group slot control interface.
|
||||
*
|
||||
* Used to expose command stream group slot properties.
|
||||
*
|
||||
* This interface is read-only.
|
||||
*/
|
||||
const struct panthor_fw_csg_control_iface *control;
|
||||
|
||||
/**
|
||||
* @input: Command stream slot input interface.
|
||||
*
|
||||
* Used for host updates/events.
|
||||
*/
|
||||
struct panthor_fw_csg_input_iface *input;
|
||||
|
||||
/**
|
||||
* @output: Command stream group slot output interface.
|
||||
*
|
||||
* Used for FW updates/events.
|
||||
*
|
||||
* This interface is read-only.
|
||||
*/
|
||||
const struct panthor_fw_csg_output_iface *output;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct panthor_fw_global_iface - Firmware global interface
|
||||
*/
|
||||
struct panthor_fw_global_iface {
|
||||
/**
|
||||
* @lock: Lock protecting access to the panthor_fw_global_input_iface::req
|
||||
* field.
|
||||
*
|
||||
* Needed so we can update the req field concurrently from the interrupt
|
||||
* handler and the scheduler/FW management logic.
|
||||
*
|
||||
* TODO: Ideally we'd want to use a cmpxchg() to update the req, but FW
|
||||
* interface sections are mapped uncached/write-combined right now, and
|
||||
* using cmpxchg() on such mappings leads to SError faults. Revisit when
|
||||
* we have 'SHARED' GPU mappings hooked up.
|
||||
*/
|
||||
spinlock_t lock;
|
||||
|
||||
/**
|
||||
* @control: Command stream group slot control interface.
|
||||
*
|
||||
* Used to expose global FW properties.
|
||||
*
|
||||
* This interface is read-only.
|
||||
*/
|
||||
const struct panthor_fw_global_control_iface *control;
|
||||
|
||||
/**
|
||||
* @input: Global input interface.
|
||||
*
|
||||
* Used for host updates/events.
|
||||
*/
|
||||
struct panthor_fw_global_input_iface *input;
|
||||
|
||||
/**
|
||||
* @output: Global output interface.
|
||||
*
|
||||
* Used for FW updates/events.
|
||||
*
|
||||
* This interface is read-only.
|
||||
*/
|
||||
const struct panthor_fw_global_output_iface *output;
|
||||
};
|
||||
|
||||
/**
|
||||
* panthor_fw_toggle_reqs() - Toggle acknowledge bits to send an event to the FW
|
||||
* @__iface: The interface to operate on.
|
||||
* @__in_reg: Name of the register to update in the input section of the interface.
|
||||
* @__out_reg: Name of the register to take as a reference in the output section of the
|
||||
* interface.
|
||||
* @__mask: Mask to apply to the update.
|
||||
*
|
||||
* The Host -> FW event/message passing was designed to be lockless, with each side of
|
||||
* the channel having its writeable section. Events are signaled as a difference between
|
||||
* the host and FW side in the req/ack registers (when a bit differs, there's an event
|
||||
* pending, when they are the same, nothing needs attention).
|
||||
*
|
||||
* This helper allows one to update the req register based on the current value of the
|
||||
* ack register managed by the FW. Toggling a specific bit will flag an event. In order
|
||||
* for events to be re-evaluated, the interface doorbell needs to be rung.
|
||||
*
|
||||
* Concurrent accesses to the same req register is covered.
|
||||
*
|
||||
* Anything requiring atomic updates to multiple registers requires a dedicated lock.
|
||||
*/
|
||||
#define panthor_fw_toggle_reqs(__iface, __in_reg, __out_reg, __mask) \
|
||||
do { \
|
||||
u32 __cur_val, __new_val, __out_val; \
|
||||
spin_lock(&(__iface)->lock); \
|
||||
__cur_val = READ_ONCE((__iface)->input->__in_reg); \
|
||||
__out_val = READ_ONCE((__iface)->output->__out_reg); \
|
||||
__new_val = ((__out_val ^ (__mask)) & (__mask)) | (__cur_val & ~(__mask)); \
|
||||
WRITE_ONCE((__iface)->input->__in_reg, __new_val); \
|
||||
spin_unlock(&(__iface)->lock); \
|
||||
} while (0)
|
||||
|
||||
/**
|
||||
* panthor_fw_update_reqs() - Update bits to reflect a configuration change
|
||||
* @__iface: The interface to operate on.
|
||||
* @__in_reg: Name of the register to update in the input section of the interface.
|
||||
* @__val: Value to set.
|
||||
* @__mask: Mask to apply to the update.
|
||||
*
|
||||
* Some configuration get passed through req registers that are also used to
|
||||
* send events to the FW. Those req registers being updated from the interrupt
|
||||
* handler, they require special helpers to update the configuration part as well.
|
||||
*
|
||||
* Concurrent accesses to the same req register is covered.
|
||||
*
|
||||
* Anything requiring atomic updates to multiple registers requires a dedicated lock.
|
||||
*/
|
||||
#define panthor_fw_update_reqs(__iface, __in_reg, __val, __mask) \
|
||||
do { \
|
||||
u32 __cur_val, __new_val; \
|
||||
spin_lock(&(__iface)->lock); \
|
||||
__cur_val = READ_ONCE((__iface)->input->__in_reg); \
|
||||
__new_val = (__cur_val & ~(__mask)) | ((__val) & (__mask)); \
|
||||
WRITE_ONCE((__iface)->input->__in_reg, __new_val); \
|
||||
spin_unlock(&(__iface)->lock); \
|
||||
} while (0)
|
||||
|
||||
struct panthor_fw_global_iface *
|
||||
panthor_fw_get_glb_iface(struct panthor_device *ptdev);
|
||||
|
||||
struct panthor_fw_csg_iface *
|
||||
panthor_fw_get_csg_iface(struct panthor_device *ptdev, u32 csg_slot);
|
||||
|
||||
struct panthor_fw_cs_iface *
|
||||
panthor_fw_get_cs_iface(struct panthor_device *ptdev, u32 csg_slot, u32 cs_slot);
|
||||
|
||||
int panthor_fw_csg_wait_acks(struct panthor_device *ptdev, u32 csg_id, u32 req_mask,
|
||||
u32 *acked, u32 timeout_ms);
|
||||
|
||||
int panthor_fw_glb_wait_acks(struct panthor_device *ptdev, u32 req_mask, u32 *acked,
|
||||
u32 timeout_ms);
|
||||
|
||||
void panthor_fw_ring_csg_doorbells(struct panthor_device *ptdev, u32 csg_slot);
|
||||
|
||||
struct panthor_kernel_bo *
|
||||
panthor_fw_alloc_queue_iface_mem(struct panthor_device *ptdev,
|
||||
struct panthor_fw_ringbuf_input_iface **input,
|
||||
const struct panthor_fw_ringbuf_output_iface **output,
|
||||
u32 *input_fw_va, u32 *output_fw_va);
|
||||
struct panthor_kernel_bo *
|
||||
panthor_fw_alloc_suspend_buf_mem(struct panthor_device *ptdev, size_t size);
|
||||
|
||||
struct panthor_vm *panthor_fw_vm(struct panthor_device *ptdev);
|
||||
|
||||
void panthor_fw_pre_reset(struct panthor_device *ptdev, bool on_hang);
|
||||
int panthor_fw_post_reset(struct panthor_device *ptdev);
|
||||
|
||||
static inline void panthor_fw_suspend(struct panthor_device *ptdev)
|
||||
{
|
||||
panthor_fw_pre_reset(ptdev, false);
|
||||
}
|
||||
|
||||
static inline int panthor_fw_resume(struct panthor_device *ptdev)
|
||||
{
|
||||
return panthor_fw_post_reset(ptdev);
|
||||
}
|
||||
|
||||
int panthor_fw_init(struct panthor_device *ptdev);
|
||||
void panthor_fw_unplug(struct panthor_device *ptdev);
|
||||
|
||||
#endif
|
||||
230
drivers/gpu/drm/panthor/panthor_gem.c
Normal file
230
drivers/gpu/drm/panthor/panthor_gem.c
Normal file
|
|
@ -0,0 +1,230 @@
|
|||
// SPDX-License-Identifier: GPL-2.0 or MIT
|
||||
/* Copyright 2019 Linaro, Ltd, Rob Herring <robh@kernel.org> */
|
||||
/* Copyright 2023 Collabora ltd. */
|
||||
|
||||
#include <linux/dma-buf.h>
|
||||
#include <linux/dma-mapping.h>
|
||||
#include <linux/err.h>
|
||||
#include <linux/slab.h>
|
||||
|
||||
#include <drm/panthor_drm.h>
|
||||
|
||||
#include "panthor_device.h"
|
||||
#include "panthor_gem.h"
|
||||
#include "panthor_mmu.h"
|
||||
|
||||
static void panthor_gem_free_object(struct drm_gem_object *obj)
|
||||
{
|
||||
struct panthor_gem_object *bo = to_panthor_bo(obj);
|
||||
struct drm_gem_object *vm_root_gem = bo->exclusive_vm_root_gem;
|
||||
|
||||
drm_gem_free_mmap_offset(&bo->base.base);
|
||||
mutex_destroy(&bo->gpuva_list_lock);
|
||||
drm_gem_shmem_free(&bo->base);
|
||||
drm_gem_object_put(vm_root_gem);
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_kernel_bo_destroy() - Destroy a kernel buffer object
|
||||
* @vm: The VM this BO was mapped to.
|
||||
* @bo: Kernel buffer object to destroy. If NULL or an ERR_PTR(), the destruction
|
||||
* is skipped.
|
||||
*/
|
||||
void panthor_kernel_bo_destroy(struct panthor_vm *vm,
|
||||
struct panthor_kernel_bo *bo)
|
||||
{
|
||||
int ret;
|
||||
|
||||
if (IS_ERR_OR_NULL(bo))
|
||||
return;
|
||||
|
||||
panthor_kernel_bo_vunmap(bo);
|
||||
|
||||
if (drm_WARN_ON(bo->obj->dev,
|
||||
to_panthor_bo(bo->obj)->exclusive_vm_root_gem != panthor_vm_root_gem(vm)))
|
||||
goto out_free_bo;
|
||||
|
||||
ret = panthor_vm_unmap_range(vm, bo->va_node.start,
|
||||
panthor_kernel_bo_size(bo));
|
||||
if (ret)
|
||||
goto out_free_bo;
|
||||
|
||||
panthor_vm_free_va(vm, &bo->va_node);
|
||||
drm_gem_object_put(bo->obj);
|
||||
|
||||
out_free_bo:
|
||||
kfree(bo);
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_kernel_bo_create() - Create and map a GEM object to a VM
|
||||
* @ptdev: Device.
|
||||
* @vm: VM to map the GEM to. If NULL, the kernel object is not GPU mapped.
|
||||
* @size: Size of the buffer object.
|
||||
* @bo_flags: Combination of drm_panthor_bo_flags flags.
|
||||
* @vm_map_flags: Combination of drm_panthor_vm_bind_op_flags (only those
|
||||
* that are related to map operations).
|
||||
* @gpu_va: GPU address assigned when mapping to the VM.
|
||||
* If gpu_va == PANTHOR_VM_KERNEL_AUTO_VA, the virtual address will be
|
||||
* automatically allocated.
|
||||
*
|
||||
* Return: A valid pointer in case of success, an ERR_PTR() otherwise.
|
||||
*/
|
||||
struct panthor_kernel_bo *
|
||||
panthor_kernel_bo_create(struct panthor_device *ptdev, struct panthor_vm *vm,
|
||||
size_t size, u32 bo_flags, u32 vm_map_flags,
|
||||
u64 gpu_va)
|
||||
{
|
||||
struct drm_gem_shmem_object *obj;
|
||||
struct panthor_kernel_bo *kbo;
|
||||
struct panthor_gem_object *bo;
|
||||
int ret;
|
||||
|
||||
if (drm_WARN_ON(&ptdev->base, !vm))
|
||||
return ERR_PTR(-EINVAL);
|
||||
|
||||
kbo = kzalloc(sizeof(*kbo), GFP_KERNEL);
|
||||
if (!kbo)
|
||||
return ERR_PTR(-ENOMEM);
|
||||
|
||||
obj = drm_gem_shmem_create(&ptdev->base, size);
|
||||
if (IS_ERR(obj)) {
|
||||
ret = PTR_ERR(obj);
|
||||
goto err_free_bo;
|
||||
}
|
||||
|
||||
bo = to_panthor_bo(&obj->base);
|
||||
size = obj->base.size;
|
||||
kbo->obj = &obj->base;
|
||||
bo->flags = bo_flags;
|
||||
|
||||
ret = panthor_vm_alloc_va(vm, gpu_va, size, &kbo->va_node);
|
||||
if (ret)
|
||||
goto err_put_obj;
|
||||
|
||||
ret = panthor_vm_map_bo_range(vm, bo, 0, size, kbo->va_node.start, vm_map_flags);
|
||||
if (ret)
|
||||
goto err_free_va;
|
||||
|
||||
bo->exclusive_vm_root_gem = panthor_vm_root_gem(vm);
|
||||
drm_gem_object_get(bo->exclusive_vm_root_gem);
|
||||
bo->base.base.resv = bo->exclusive_vm_root_gem->resv;
|
||||
return kbo;
|
||||
|
||||
err_free_va:
|
||||
panthor_vm_free_va(vm, &kbo->va_node);
|
||||
|
||||
err_put_obj:
|
||||
drm_gem_object_put(&obj->base);
|
||||
|
||||
err_free_bo:
|
||||
kfree(kbo);
|
||||
return ERR_PTR(ret);
|
||||
}
|
||||
|
||||
static int panthor_gem_mmap(struct drm_gem_object *obj, struct vm_area_struct *vma)
|
||||
{
|
||||
struct panthor_gem_object *bo = to_panthor_bo(obj);
|
||||
|
||||
/* Don't allow mmap on objects that have the NO_MMAP flag set. */
|
||||
if (bo->flags & DRM_PANTHOR_BO_NO_MMAP)
|
||||
return -EINVAL;
|
||||
|
||||
return drm_gem_shmem_object_mmap(obj, vma);
|
||||
}
|
||||
|
||||
static struct dma_buf *
|
||||
panthor_gem_prime_export(struct drm_gem_object *obj, int flags)
|
||||
{
|
||||
/* We can't export GEMs that have an exclusive VM. */
|
||||
if (to_panthor_bo(obj)->exclusive_vm_root_gem)
|
||||
return ERR_PTR(-EINVAL);
|
||||
|
||||
return drm_gem_prime_export(obj, flags);
|
||||
}
|
||||
|
||||
static const struct drm_gem_object_funcs panthor_gem_funcs = {
|
||||
.free = panthor_gem_free_object,
|
||||
.print_info = drm_gem_shmem_object_print_info,
|
||||
.pin = drm_gem_shmem_object_pin,
|
||||
.unpin = drm_gem_shmem_object_unpin,
|
||||
.get_sg_table = drm_gem_shmem_object_get_sg_table,
|
||||
.vmap = drm_gem_shmem_object_vmap,
|
||||
.vunmap = drm_gem_shmem_object_vunmap,
|
||||
.mmap = panthor_gem_mmap,
|
||||
.export = panthor_gem_prime_export,
|
||||
.vm_ops = &drm_gem_shmem_vm_ops,
|
||||
};
|
||||
|
||||
/**
|
||||
* panthor_gem_create_object - Implementation of driver->gem_create_object.
|
||||
* @ddev: DRM device
|
||||
* @size: Size in bytes of the memory the object will reference
|
||||
*
|
||||
* This lets the GEM helpers allocate object structs for us, and keep
|
||||
* our BO stats correct.
|
||||
*/
|
||||
struct drm_gem_object *panthor_gem_create_object(struct drm_device *ddev, size_t size)
|
||||
{
|
||||
struct panthor_device *ptdev = container_of(ddev, struct panthor_device, base);
|
||||
struct panthor_gem_object *obj;
|
||||
|
||||
obj = kzalloc(sizeof(*obj), GFP_KERNEL);
|
||||
if (!obj)
|
||||
return ERR_PTR(-ENOMEM);
|
||||
|
||||
obj->base.base.funcs = &panthor_gem_funcs;
|
||||
obj->base.map_wc = !ptdev->coherent;
|
||||
mutex_init(&obj->gpuva_list_lock);
|
||||
drm_gem_gpuva_set_lock(&obj->base.base, &obj->gpuva_list_lock);
|
||||
|
||||
return &obj->base.base;
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_gem_create_with_handle() - Create a GEM object and attach it to a handle.
|
||||
* @file: DRM file.
|
||||
* @ddev: DRM device.
|
||||
* @exclusive_vm: Exclusive VM. Not NULL if the GEM object can't be shared.
|
||||
* @size: Size of the GEM object to allocate.
|
||||
* @flags: Combination of drm_panthor_bo_flags flags.
|
||||
* @handle: Pointer holding the handle pointing to the new GEM object.
|
||||
*
|
||||
* Return: Zero on success
|
||||
*/
|
||||
int
|
||||
panthor_gem_create_with_handle(struct drm_file *file,
|
||||
struct drm_device *ddev,
|
||||
struct panthor_vm *exclusive_vm,
|
||||
u64 *size, u32 flags, u32 *handle)
|
||||
{
|
||||
int ret;
|
||||
struct drm_gem_shmem_object *shmem;
|
||||
struct panthor_gem_object *bo;
|
||||
|
||||
shmem = drm_gem_shmem_create(ddev, *size);
|
||||
if (IS_ERR(shmem))
|
||||
return PTR_ERR(shmem);
|
||||
|
||||
bo = to_panthor_bo(&shmem->base);
|
||||
bo->flags = flags;
|
||||
|
||||
if (exclusive_vm) {
|
||||
bo->exclusive_vm_root_gem = panthor_vm_root_gem(exclusive_vm);
|
||||
drm_gem_object_get(bo->exclusive_vm_root_gem);
|
||||
bo->base.base.resv = bo->exclusive_vm_root_gem->resv;
|
||||
}
|
||||
|
||||
/*
|
||||
* Allocate an id of idr table where the obj is registered
|
||||
* and handle has the id what user can see.
|
||||
*/
|
||||
ret = drm_gem_handle_create(file, &shmem->base, handle);
|
||||
if (!ret)
|
||||
*size = bo->base.base.size;
|
||||
|
||||
/* drop reference from allocate - handle holds it now. */
|
||||
drm_gem_object_put(&shmem->base);
|
||||
|
||||
return ret;
|
||||
}
|
||||
142
drivers/gpu/drm/panthor/panthor_gem.h
Normal file
142
drivers/gpu/drm/panthor/panthor_gem.h
Normal file
|
|
@ -0,0 +1,142 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0 or MIT */
|
||||
/* Copyright 2019 Linaro, Ltd, Rob Herring <robh@kernel.org> */
|
||||
/* Copyright 2023 Collabora ltd. */
|
||||
|
||||
#ifndef __PANTHOR_GEM_H__
|
||||
#define __PANTHOR_GEM_H__
|
||||
|
||||
#include <drm/drm_gem_shmem_helper.h>
|
||||
#include <drm/drm_mm.h>
|
||||
|
||||
#include <linux/iosys-map.h>
|
||||
#include <linux/rwsem.h>
|
||||
|
||||
struct panthor_vm;
|
||||
|
||||
/**
|
||||
* struct panthor_gem_object - Driver specific GEM object.
|
||||
*/
|
||||
struct panthor_gem_object {
|
||||
/** @base: Inherit from drm_gem_shmem_object. */
|
||||
struct drm_gem_shmem_object base;
|
||||
|
||||
/**
|
||||
* @exclusive_vm_root_gem: Root GEM of the exclusive VM this GEM object
|
||||
* is attached to.
|
||||
*
|
||||
* If @exclusive_vm_root_gem != NULL, any attempt to bind the GEM to a
|
||||
* different VM will fail.
|
||||
*
|
||||
* All FW memory objects have this field set to the root GEM of the MCU
|
||||
* VM.
|
||||
*/
|
||||
struct drm_gem_object *exclusive_vm_root_gem;
|
||||
|
||||
/**
|
||||
* @gpuva_list_lock: Custom GPUVA lock.
|
||||
*
|
||||
* Used to protect insertion of drm_gpuva elements to the
|
||||
* drm_gem_object.gpuva.list list.
|
||||
*
|
||||
* We can't use the GEM resv for that, because drm_gpuva_link() is
|
||||
* called in a dma-signaling path, where we're not allowed to take
|
||||
* resv locks.
|
||||
*/
|
||||
struct mutex gpuva_list_lock;
|
||||
|
||||
/** @flags: Combination of drm_panthor_bo_flags flags. */
|
||||
u32 flags;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct panthor_kernel_bo - Kernel buffer object.
|
||||
*
|
||||
* These objects are only manipulated by the kernel driver and not
|
||||
* directly exposed to the userspace. The GPU address of a kernel
|
||||
* BO might be passed to userspace though.
|
||||
*/
|
||||
struct panthor_kernel_bo {
|
||||
/**
|
||||
* @obj: The GEM object backing this kernel buffer object.
|
||||
*/
|
||||
struct drm_gem_object *obj;
|
||||
|
||||
/**
|
||||
* @va_node: VA space allocated to this GEM.
|
||||
*/
|
||||
struct drm_mm_node va_node;
|
||||
|
||||
/**
|
||||
* @kmap: Kernel CPU mapping of @gem.
|
||||
*/
|
||||
void *kmap;
|
||||
};
|
||||
|
||||
static inline
|
||||
struct panthor_gem_object *to_panthor_bo(struct drm_gem_object *obj)
|
||||
{
|
||||
return container_of(to_drm_gem_shmem_obj(obj), struct panthor_gem_object, base);
|
||||
}
|
||||
|
||||
struct drm_gem_object *panthor_gem_create_object(struct drm_device *ddev, size_t size);
|
||||
|
||||
struct drm_gem_object *
|
||||
panthor_gem_prime_import_sg_table(struct drm_device *ddev,
|
||||
struct dma_buf_attachment *attach,
|
||||
struct sg_table *sgt);
|
||||
|
||||
int
|
||||
panthor_gem_create_with_handle(struct drm_file *file,
|
||||
struct drm_device *ddev,
|
||||
struct panthor_vm *exclusive_vm,
|
||||
u64 *size, u32 flags, uint32_t *handle);
|
||||
|
||||
static inline u64
|
||||
panthor_kernel_bo_gpuva(struct panthor_kernel_bo *bo)
|
||||
{
|
||||
return bo->va_node.start;
|
||||
}
|
||||
|
||||
static inline size_t
|
||||
panthor_kernel_bo_size(struct panthor_kernel_bo *bo)
|
||||
{
|
||||
return bo->obj->size;
|
||||
}
|
||||
|
||||
static inline int
|
||||
panthor_kernel_bo_vmap(struct panthor_kernel_bo *bo)
|
||||
{
|
||||
struct iosys_map map;
|
||||
int ret;
|
||||
|
||||
if (bo->kmap)
|
||||
return 0;
|
||||
|
||||
ret = drm_gem_vmap_unlocked(bo->obj, &map);
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
bo->kmap = map.vaddr;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static inline void
|
||||
panthor_kernel_bo_vunmap(struct panthor_kernel_bo *bo)
|
||||
{
|
||||
if (bo->kmap) {
|
||||
struct iosys_map map = IOSYS_MAP_INIT_VADDR(bo->kmap);
|
||||
|
||||
drm_gem_vunmap_unlocked(bo->obj, &map);
|
||||
bo->kmap = NULL;
|
||||
}
|
||||
}
|
||||
|
||||
struct panthor_kernel_bo *
|
||||
panthor_kernel_bo_create(struct panthor_device *ptdev, struct panthor_vm *vm,
|
||||
size_t size, u32 bo_flags, u32 vm_map_flags,
|
||||
u64 gpu_va);
|
||||
|
||||
void panthor_kernel_bo_destroy(struct panthor_vm *vm,
|
||||
struct panthor_kernel_bo *bo);
|
||||
|
||||
#endif /* __PANTHOR_GEM_H__ */
|
||||
482
drivers/gpu/drm/panthor/panthor_gpu.c
Normal file
482
drivers/gpu/drm/panthor/panthor_gpu.c
Normal file
|
|
@ -0,0 +1,482 @@
|
|||
// SPDX-License-Identifier: GPL-2.0 or MIT
|
||||
/* Copyright 2018 Marty E. Plummer <hanetzer@startmail.com> */
|
||||
/* Copyright 2019 Linaro, Ltd., Rob Herring <robh@kernel.org> */
|
||||
/* Copyright 2019 Collabora ltd. */
|
||||
|
||||
#include <linux/bitfield.h>
|
||||
#include <linux/bitmap.h>
|
||||
#include <linux/delay.h>
|
||||
#include <linux/dma-mapping.h>
|
||||
#include <linux/interrupt.h>
|
||||
#include <linux/io.h>
|
||||
#include <linux/iopoll.h>
|
||||
#include <linux/platform_device.h>
|
||||
#include <linux/pm_runtime.h>
|
||||
|
||||
#include <drm/drm_drv.h>
|
||||
#include <drm/drm_managed.h>
|
||||
|
||||
#include "panthor_device.h"
|
||||
#include "panthor_gpu.h"
|
||||
#include "panthor_regs.h"
|
||||
|
||||
/**
|
||||
* struct panthor_gpu - GPU block management data.
|
||||
*/
|
||||
struct panthor_gpu {
|
||||
/** @irq: GPU irq. */
|
||||
struct panthor_irq irq;
|
||||
|
||||
/** @reqs_lock: Lock protecting access to pending_reqs. */
|
||||
spinlock_t reqs_lock;
|
||||
|
||||
/** @pending_reqs: Pending GPU requests. */
|
||||
u32 pending_reqs;
|
||||
|
||||
/** @reqs_acked: GPU request wait queue. */
|
||||
wait_queue_head_t reqs_acked;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct panthor_model - GPU model description
|
||||
*/
|
||||
struct panthor_model {
|
||||
/** @name: Model name. */
|
||||
const char *name;
|
||||
|
||||
/** @arch_major: Major version number of architecture. */
|
||||
u8 arch_major;
|
||||
|
||||
/** @product_major: Major version number of product. */
|
||||
u8 product_major;
|
||||
};
|
||||
|
||||
/**
|
||||
* GPU_MODEL() - Define a GPU model. A GPU product can be uniquely identified
|
||||
* by a combination of the major architecture version and the major product
|
||||
* version.
|
||||
* @_name: Name for the GPU model.
|
||||
* @_arch_major: Architecture major.
|
||||
* @_product_major: Product major.
|
||||
*/
|
||||
#define GPU_MODEL(_name, _arch_major, _product_major) \
|
||||
{\
|
||||
.name = __stringify(_name), \
|
||||
.arch_major = _arch_major, \
|
||||
.product_major = _product_major, \
|
||||
}
|
||||
|
||||
static const struct panthor_model gpu_models[] = {
|
||||
GPU_MODEL(g610, 10, 7),
|
||||
{},
|
||||
};
|
||||
|
||||
#define GPU_INTERRUPTS_MASK \
|
||||
(GPU_IRQ_FAULT | \
|
||||
GPU_IRQ_PROTM_FAULT | \
|
||||
GPU_IRQ_RESET_COMPLETED | \
|
||||
GPU_IRQ_CLEAN_CACHES_COMPLETED)
|
||||
|
||||
static void panthor_gpu_init_info(struct panthor_device *ptdev)
|
||||
{
|
||||
const struct panthor_model *model;
|
||||
u32 arch_major, product_major;
|
||||
u32 major, minor, status;
|
||||
unsigned int i;
|
||||
|
||||
ptdev->gpu_info.gpu_id = gpu_read(ptdev, GPU_ID);
|
||||
ptdev->gpu_info.csf_id = gpu_read(ptdev, GPU_CSF_ID);
|
||||
ptdev->gpu_info.gpu_rev = gpu_read(ptdev, GPU_REVID);
|
||||
ptdev->gpu_info.core_features = gpu_read(ptdev, GPU_CORE_FEATURES);
|
||||
ptdev->gpu_info.l2_features = gpu_read(ptdev, GPU_L2_FEATURES);
|
||||
ptdev->gpu_info.tiler_features = gpu_read(ptdev, GPU_TILER_FEATURES);
|
||||
ptdev->gpu_info.mem_features = gpu_read(ptdev, GPU_MEM_FEATURES);
|
||||
ptdev->gpu_info.mmu_features = gpu_read(ptdev, GPU_MMU_FEATURES);
|
||||
ptdev->gpu_info.thread_features = gpu_read(ptdev, GPU_THREAD_FEATURES);
|
||||
ptdev->gpu_info.max_threads = gpu_read(ptdev, GPU_THREAD_MAX_THREADS);
|
||||
ptdev->gpu_info.thread_max_workgroup_size = gpu_read(ptdev, GPU_THREAD_MAX_WORKGROUP_SIZE);
|
||||
ptdev->gpu_info.thread_max_barrier_size = gpu_read(ptdev, GPU_THREAD_MAX_BARRIER_SIZE);
|
||||
ptdev->gpu_info.coherency_features = gpu_read(ptdev, GPU_COHERENCY_FEATURES);
|
||||
for (i = 0; i < 4; i++)
|
||||
ptdev->gpu_info.texture_features[i] = gpu_read(ptdev, GPU_TEXTURE_FEATURES(i));
|
||||
|
||||
ptdev->gpu_info.as_present = gpu_read(ptdev, GPU_AS_PRESENT);
|
||||
|
||||
ptdev->gpu_info.shader_present = gpu_read(ptdev, GPU_SHADER_PRESENT_LO);
|
||||
ptdev->gpu_info.shader_present |= (u64)gpu_read(ptdev, GPU_SHADER_PRESENT_HI) << 32;
|
||||
|
||||
ptdev->gpu_info.tiler_present = gpu_read(ptdev, GPU_TILER_PRESENT_LO);
|
||||
ptdev->gpu_info.tiler_present |= (u64)gpu_read(ptdev, GPU_TILER_PRESENT_HI) << 32;
|
||||
|
||||
ptdev->gpu_info.l2_present = gpu_read(ptdev, GPU_L2_PRESENT_LO);
|
||||
ptdev->gpu_info.l2_present |= (u64)gpu_read(ptdev, GPU_L2_PRESENT_HI) << 32;
|
||||
|
||||
arch_major = GPU_ARCH_MAJOR(ptdev->gpu_info.gpu_id);
|
||||
product_major = GPU_PROD_MAJOR(ptdev->gpu_info.gpu_id);
|
||||
major = GPU_VER_MAJOR(ptdev->gpu_info.gpu_id);
|
||||
minor = GPU_VER_MINOR(ptdev->gpu_info.gpu_id);
|
||||
status = GPU_VER_STATUS(ptdev->gpu_info.gpu_id);
|
||||
|
||||
for (model = gpu_models; model->name; model++) {
|
||||
if (model->arch_major == arch_major &&
|
||||
model->product_major == product_major)
|
||||
break;
|
||||
}
|
||||
|
||||
drm_info(&ptdev->base,
|
||||
"mali-%s id 0x%x major 0x%x minor 0x%x status 0x%x",
|
||||
model->name ?: "unknown", ptdev->gpu_info.gpu_id >> 16,
|
||||
major, minor, status);
|
||||
|
||||
drm_info(&ptdev->base,
|
||||
"Features: L2:%#x Tiler:%#x Mem:%#x MMU:%#x AS:%#x",
|
||||
ptdev->gpu_info.l2_features,
|
||||
ptdev->gpu_info.tiler_features,
|
||||
ptdev->gpu_info.mem_features,
|
||||
ptdev->gpu_info.mmu_features,
|
||||
ptdev->gpu_info.as_present);
|
||||
|
||||
drm_info(&ptdev->base,
|
||||
"shader_present=0x%0llx l2_present=0x%0llx tiler_present=0x%0llx",
|
||||
ptdev->gpu_info.shader_present, ptdev->gpu_info.l2_present,
|
||||
ptdev->gpu_info.tiler_present);
|
||||
}
|
||||
|
||||
static void panthor_gpu_irq_handler(struct panthor_device *ptdev, u32 status)
|
||||
{
|
||||
if (status & GPU_IRQ_FAULT) {
|
||||
u32 fault_status = gpu_read(ptdev, GPU_FAULT_STATUS);
|
||||
u64 address = ((u64)gpu_read(ptdev, GPU_FAULT_ADDR_HI) << 32) |
|
||||
gpu_read(ptdev, GPU_FAULT_ADDR_LO);
|
||||
|
||||
drm_warn(&ptdev->base, "GPU Fault 0x%08x (%s) at 0x%016llx\n",
|
||||
fault_status, panthor_exception_name(ptdev, fault_status & 0xFF),
|
||||
address);
|
||||
}
|
||||
if (status & GPU_IRQ_PROTM_FAULT)
|
||||
drm_warn(&ptdev->base, "GPU Fault in protected mode\n");
|
||||
|
||||
spin_lock(&ptdev->gpu->reqs_lock);
|
||||
if (status & ptdev->gpu->pending_reqs) {
|
||||
ptdev->gpu->pending_reqs &= ~status;
|
||||
wake_up_all(&ptdev->gpu->reqs_acked);
|
||||
}
|
||||
spin_unlock(&ptdev->gpu->reqs_lock);
|
||||
}
|
||||
PANTHOR_IRQ_HANDLER(gpu, GPU, panthor_gpu_irq_handler);
|
||||
|
||||
/**
|
||||
* panthor_gpu_unplug() - Called when the GPU is unplugged.
|
||||
* @ptdev: Device to unplug.
|
||||
*/
|
||||
void panthor_gpu_unplug(struct panthor_device *ptdev)
|
||||
{
|
||||
unsigned long flags;
|
||||
|
||||
/* Make sure the IRQ handler is not running after that point. */
|
||||
panthor_gpu_irq_suspend(&ptdev->gpu->irq);
|
||||
|
||||
/* Wake-up all waiters. */
|
||||
spin_lock_irqsave(&ptdev->gpu->reqs_lock, flags);
|
||||
ptdev->gpu->pending_reqs = 0;
|
||||
wake_up_all(&ptdev->gpu->reqs_acked);
|
||||
spin_unlock_irqrestore(&ptdev->gpu->reqs_lock, flags);
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_gpu_init() - Initialize the GPU block
|
||||
* @ptdev: Device.
|
||||
*
|
||||
* Return: 0 on success, a negative error code otherwise.
|
||||
*/
|
||||
int panthor_gpu_init(struct panthor_device *ptdev)
|
||||
{
|
||||
struct panthor_gpu *gpu;
|
||||
u32 pa_bits;
|
||||
int ret, irq;
|
||||
|
||||
gpu = drmm_kzalloc(&ptdev->base, sizeof(*gpu), GFP_KERNEL);
|
||||
if (!gpu)
|
||||
return -ENOMEM;
|
||||
|
||||
spin_lock_init(&gpu->reqs_lock);
|
||||
init_waitqueue_head(&gpu->reqs_acked);
|
||||
ptdev->gpu = gpu;
|
||||
panthor_gpu_init_info(ptdev);
|
||||
|
||||
dma_set_max_seg_size(ptdev->base.dev, UINT_MAX);
|
||||
pa_bits = GPU_MMU_FEATURES_PA_BITS(ptdev->gpu_info.mmu_features);
|
||||
ret = dma_set_mask_and_coherent(ptdev->base.dev, DMA_BIT_MASK(pa_bits));
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
irq = platform_get_irq_byname(to_platform_device(ptdev->base.dev), "gpu");
|
||||
if (irq <= 0)
|
||||
return ret;
|
||||
|
||||
ret = panthor_request_gpu_irq(ptdev, &ptdev->gpu->irq, irq, GPU_INTERRUPTS_MASK);
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_gpu_block_power_off() - Power-off a specific block of the GPU
|
||||
* @ptdev: Device.
|
||||
* @blk_name: Block name.
|
||||
* @pwroff_reg: Power-off register for this block.
|
||||
* @pwrtrans_reg: Power transition register for this block.
|
||||
* @mask: Sub-elements to power-off.
|
||||
* @timeout_us: Timeout in microseconds.
|
||||
*
|
||||
* Return: 0 on success, a negative error code otherwise.
|
||||
*/
|
||||
int panthor_gpu_block_power_off(struct panthor_device *ptdev,
|
||||
const char *blk_name,
|
||||
u32 pwroff_reg, u32 pwrtrans_reg,
|
||||
u64 mask, u32 timeout_us)
|
||||
{
|
||||
u32 val, i;
|
||||
int ret;
|
||||
|
||||
for (i = 0; i < 2; i++) {
|
||||
u32 mask32 = mask >> (i * 32);
|
||||
|
||||
if (!mask32)
|
||||
continue;
|
||||
|
||||
ret = readl_relaxed_poll_timeout(ptdev->iomem + pwrtrans_reg + (i * 4),
|
||||
val, !(mask32 & val),
|
||||
100, timeout_us);
|
||||
if (ret) {
|
||||
drm_err(&ptdev->base, "timeout waiting on %s:%llx power transition",
|
||||
blk_name, mask);
|
||||
return ret;
|
||||
}
|
||||
}
|
||||
|
||||
if (mask & GENMASK(31, 0))
|
||||
gpu_write(ptdev, pwroff_reg, mask);
|
||||
|
||||
if (mask >> 32)
|
||||
gpu_write(ptdev, pwroff_reg + 4, mask >> 32);
|
||||
|
||||
for (i = 0; i < 2; i++) {
|
||||
u32 mask32 = mask >> (i * 32);
|
||||
|
||||
if (!mask32)
|
||||
continue;
|
||||
|
||||
ret = readl_relaxed_poll_timeout(ptdev->iomem + pwrtrans_reg + (i * 4),
|
||||
val, !(mask32 & val),
|
||||
100, timeout_us);
|
||||
if (ret) {
|
||||
drm_err(&ptdev->base, "timeout waiting on %s:%llx power transition",
|
||||
blk_name, mask);
|
||||
return ret;
|
||||
}
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_gpu_block_power_on() - Power-on a specific block of the GPU
|
||||
* @ptdev: Device.
|
||||
* @blk_name: Block name.
|
||||
* @pwron_reg: Power-on register for this block.
|
||||
* @pwrtrans_reg: Power transition register for this block.
|
||||
* @rdy_reg: Power transition ready register.
|
||||
* @mask: Sub-elements to power-on.
|
||||
* @timeout_us: Timeout in microseconds.
|
||||
*
|
||||
* Return: 0 on success, a negative error code otherwise.
|
||||
*/
|
||||
int panthor_gpu_block_power_on(struct panthor_device *ptdev,
|
||||
const char *blk_name,
|
||||
u32 pwron_reg, u32 pwrtrans_reg,
|
||||
u32 rdy_reg, u64 mask, u32 timeout_us)
|
||||
{
|
||||
u32 val, i;
|
||||
int ret;
|
||||
|
||||
for (i = 0; i < 2; i++) {
|
||||
u32 mask32 = mask >> (i * 32);
|
||||
|
||||
if (!mask32)
|
||||
continue;
|
||||
|
||||
ret = readl_relaxed_poll_timeout(ptdev->iomem + pwrtrans_reg + (i * 4),
|
||||
val, !(mask32 & val),
|
||||
100, timeout_us);
|
||||
if (ret) {
|
||||
drm_err(&ptdev->base, "timeout waiting on %s:%llx power transition",
|
||||
blk_name, mask);
|
||||
return ret;
|
||||
}
|
||||
}
|
||||
|
||||
if (mask & GENMASK(31, 0))
|
||||
gpu_write(ptdev, pwron_reg, mask);
|
||||
|
||||
if (mask >> 32)
|
||||
gpu_write(ptdev, pwron_reg + 4, mask >> 32);
|
||||
|
||||
for (i = 0; i < 2; i++) {
|
||||
u32 mask32 = mask >> (i * 32);
|
||||
|
||||
if (!mask32)
|
||||
continue;
|
||||
|
||||
ret = readl_relaxed_poll_timeout(ptdev->iomem + rdy_reg + (i * 4),
|
||||
val, (mask32 & val) == mask32,
|
||||
100, timeout_us);
|
||||
if (ret) {
|
||||
drm_err(&ptdev->base, "timeout waiting on %s:%llx readyness",
|
||||
blk_name, mask);
|
||||
return ret;
|
||||
}
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_gpu_l2_power_on() - Power-on the L2-cache
|
||||
* @ptdev: Device.
|
||||
*
|
||||
* Return: 0 on success, a negative error code otherwise.
|
||||
*/
|
||||
int panthor_gpu_l2_power_on(struct panthor_device *ptdev)
|
||||
{
|
||||
if (ptdev->gpu_info.l2_present != 1) {
|
||||
/*
|
||||
* Only support one core group now.
|
||||
* ~(l2_present - 1) unsets all bits in l2_present except
|
||||
* the bottom bit. (l2_present - 2) has all the bits in
|
||||
* the first core group set. AND them together to generate
|
||||
* a mask of cores in the first core group.
|
||||
*/
|
||||
u64 core_mask = ~(ptdev->gpu_info.l2_present - 1) &
|
||||
(ptdev->gpu_info.l2_present - 2);
|
||||
drm_info_once(&ptdev->base, "using only 1st core group (%lu cores from %lu)\n",
|
||||
hweight64(core_mask),
|
||||
hweight64(ptdev->gpu_info.shader_present));
|
||||
}
|
||||
|
||||
return panthor_gpu_power_on(ptdev, L2, 1, 20000);
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_gpu_flush_caches() - Flush caches
|
||||
* @ptdev: Device.
|
||||
* @l2: L2 flush type.
|
||||
* @lsc: LSC flush type.
|
||||
* @other: Other flush type.
|
||||
*
|
||||
* Return: 0 on success, a negative error code otherwise.
|
||||
*/
|
||||
int panthor_gpu_flush_caches(struct panthor_device *ptdev,
|
||||
u32 l2, u32 lsc, u32 other)
|
||||
{
|
||||
bool timedout = false;
|
||||
unsigned long flags;
|
||||
|
||||
spin_lock_irqsave(&ptdev->gpu->reqs_lock, flags);
|
||||
if (!drm_WARN_ON(&ptdev->base,
|
||||
ptdev->gpu->pending_reqs & GPU_IRQ_CLEAN_CACHES_COMPLETED)) {
|
||||
ptdev->gpu->pending_reqs |= GPU_IRQ_CLEAN_CACHES_COMPLETED;
|
||||
gpu_write(ptdev, GPU_CMD, GPU_FLUSH_CACHES(l2, lsc, other));
|
||||
}
|
||||
spin_unlock_irqrestore(&ptdev->gpu->reqs_lock, flags);
|
||||
|
||||
if (!wait_event_timeout(ptdev->gpu->reqs_acked,
|
||||
!(ptdev->gpu->pending_reqs & GPU_IRQ_CLEAN_CACHES_COMPLETED),
|
||||
msecs_to_jiffies(100))) {
|
||||
spin_lock_irqsave(&ptdev->gpu->reqs_lock, flags);
|
||||
if ((ptdev->gpu->pending_reqs & GPU_IRQ_CLEAN_CACHES_COMPLETED) != 0 &&
|
||||
!(gpu_read(ptdev, GPU_INT_RAWSTAT) & GPU_IRQ_CLEAN_CACHES_COMPLETED))
|
||||
timedout = true;
|
||||
else
|
||||
ptdev->gpu->pending_reqs &= ~GPU_IRQ_CLEAN_CACHES_COMPLETED;
|
||||
spin_unlock_irqrestore(&ptdev->gpu->reqs_lock, flags);
|
||||
}
|
||||
|
||||
if (timedout) {
|
||||
drm_err(&ptdev->base, "Flush caches timeout");
|
||||
return -ETIMEDOUT;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_gpu_soft_reset() - Issue a soft-reset
|
||||
* @ptdev: Device.
|
||||
*
|
||||
* Return: 0 on success, a negative error code otherwise.
|
||||
*/
|
||||
int panthor_gpu_soft_reset(struct panthor_device *ptdev)
|
||||
{
|
||||
bool timedout = false;
|
||||
unsigned long flags;
|
||||
|
||||
spin_lock_irqsave(&ptdev->gpu->reqs_lock, flags);
|
||||
if (!drm_WARN_ON(&ptdev->base,
|
||||
ptdev->gpu->pending_reqs & GPU_IRQ_RESET_COMPLETED)) {
|
||||
ptdev->gpu->pending_reqs |= GPU_IRQ_RESET_COMPLETED;
|
||||
gpu_write(ptdev, GPU_INT_CLEAR, GPU_IRQ_RESET_COMPLETED);
|
||||
gpu_write(ptdev, GPU_CMD, GPU_SOFT_RESET);
|
||||
}
|
||||
spin_unlock_irqrestore(&ptdev->gpu->reqs_lock, flags);
|
||||
|
||||
if (!wait_event_timeout(ptdev->gpu->reqs_acked,
|
||||
!(ptdev->gpu->pending_reqs & GPU_IRQ_RESET_COMPLETED),
|
||||
msecs_to_jiffies(100))) {
|
||||
spin_lock_irqsave(&ptdev->gpu->reqs_lock, flags);
|
||||
if ((ptdev->gpu->pending_reqs & GPU_IRQ_RESET_COMPLETED) != 0 &&
|
||||
!(gpu_read(ptdev, GPU_INT_RAWSTAT) & GPU_IRQ_RESET_COMPLETED))
|
||||
timedout = true;
|
||||
else
|
||||
ptdev->gpu->pending_reqs &= ~GPU_IRQ_RESET_COMPLETED;
|
||||
spin_unlock_irqrestore(&ptdev->gpu->reqs_lock, flags);
|
||||
}
|
||||
|
||||
if (timedout) {
|
||||
drm_err(&ptdev->base, "Soft reset timeout");
|
||||
return -ETIMEDOUT;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_gpu_suspend() - Suspend the GPU block.
|
||||
* @ptdev: Device.
|
||||
*
|
||||
* Suspend the GPU irq. This should be called last in the suspend procedure,
|
||||
* after all other blocks have been suspented.
|
||||
*/
|
||||
void panthor_gpu_suspend(struct panthor_device *ptdev)
|
||||
{
|
||||
/*
|
||||
* It may be preferable to simply power down the L2, but for now just
|
||||
* soft-reset which will leave the L2 powered down.
|
||||
*/
|
||||
panthor_gpu_soft_reset(ptdev);
|
||||
panthor_gpu_irq_suspend(&ptdev->gpu->irq);
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_gpu_resume() - Resume the GPU block.
|
||||
* @ptdev: Device.
|
||||
*
|
||||
* Resume the IRQ handler and power-on the L2-cache.
|
||||
* The FW takes care of powering the other blocks.
|
||||
*/
|
||||
void panthor_gpu_resume(struct panthor_device *ptdev)
|
||||
{
|
||||
panthor_gpu_irq_resume(&ptdev->gpu->irq, GPU_INTERRUPTS_MASK);
|
||||
panthor_gpu_l2_power_on(ptdev);
|
||||
}
|
||||
52
drivers/gpu/drm/panthor/panthor_gpu.h
Normal file
52
drivers/gpu/drm/panthor/panthor_gpu.h
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0 or MIT */
|
||||
/* Copyright 2018 Marty E. Plummer <hanetzer@startmail.com> */
|
||||
/* Copyright 2019 Collabora ltd. */
|
||||
|
||||
#ifndef __PANTHOR_GPU_H__
|
||||
#define __PANTHOR_GPU_H__
|
||||
|
||||
struct panthor_device;
|
||||
|
||||
int panthor_gpu_init(struct panthor_device *ptdev);
|
||||
void panthor_gpu_unplug(struct panthor_device *ptdev);
|
||||
void panthor_gpu_suspend(struct panthor_device *ptdev);
|
||||
void panthor_gpu_resume(struct panthor_device *ptdev);
|
||||
|
||||
int panthor_gpu_block_power_on(struct panthor_device *ptdev,
|
||||
const char *blk_name,
|
||||
u32 pwron_reg, u32 pwrtrans_reg,
|
||||
u32 rdy_reg, u64 mask, u32 timeout_us);
|
||||
int panthor_gpu_block_power_off(struct panthor_device *ptdev,
|
||||
const char *blk_name,
|
||||
u32 pwroff_reg, u32 pwrtrans_reg,
|
||||
u64 mask, u32 timeout_us);
|
||||
|
||||
/**
|
||||
* panthor_gpu_power_on() - Power on the GPU block.
|
||||
*
|
||||
* Return: 0 on success, a negative error code otherwise.
|
||||
*/
|
||||
#define panthor_gpu_power_on(ptdev, type, mask, timeout_us) \
|
||||
panthor_gpu_block_power_on(ptdev, #type, \
|
||||
type ## _PWRON_LO, \
|
||||
type ## _PWRTRANS_LO, \
|
||||
type ## _READY_LO, \
|
||||
mask, timeout_us)
|
||||
|
||||
/**
|
||||
* panthor_gpu_power_off() - Power off the GPU block.
|
||||
*
|
||||
* Return: 0 on success, a negative error code otherwise.
|
||||
*/
|
||||
#define panthor_gpu_power_off(ptdev, type, mask, timeout_us) \
|
||||
panthor_gpu_block_power_off(ptdev, #type, \
|
||||
type ## _PWROFF_LO, \
|
||||
type ## _PWRTRANS_LO, \
|
||||
mask, timeout_us)
|
||||
|
||||
int panthor_gpu_l2_power_on(struct panthor_device *ptdev);
|
||||
int panthor_gpu_flush_caches(struct panthor_device *ptdev,
|
||||
u32 l2, u32 lsc, u32 other);
|
||||
int panthor_gpu_soft_reset(struct panthor_device *ptdev);
|
||||
|
||||
#endif
|
||||
597
drivers/gpu/drm/panthor/panthor_heap.c
Normal file
597
drivers/gpu/drm/panthor/panthor_heap.c
Normal file
|
|
@ -0,0 +1,597 @@
|
|||
// SPDX-License-Identifier: GPL-2.0 or MIT
|
||||
/* Copyright 2023 Collabora ltd. */
|
||||
|
||||
#include <linux/iosys-map.h>
|
||||
#include <linux/rwsem.h>
|
||||
|
||||
#include <drm/panthor_drm.h>
|
||||
|
||||
#include "panthor_device.h"
|
||||
#include "panthor_gem.h"
|
||||
#include "panthor_heap.h"
|
||||
#include "panthor_mmu.h"
|
||||
#include "panthor_regs.h"
|
||||
|
||||
/*
|
||||
* The GPU heap context is an opaque structure used by the GPU to track the
|
||||
* heap allocations. The driver should only touch it to initialize it (zero all
|
||||
* fields). Because the CPU and GPU can both access this structure it is
|
||||
* required to be GPU cache line aligned.
|
||||
*/
|
||||
#define HEAP_CONTEXT_SIZE 32
|
||||
|
||||
/**
|
||||
* struct panthor_heap_chunk_header - Heap chunk header
|
||||
*/
|
||||
struct panthor_heap_chunk_header {
|
||||
/**
|
||||
* @next: Next heap chunk in the list.
|
||||
*
|
||||
* This is a GPU VA.
|
||||
*/
|
||||
u64 next;
|
||||
|
||||
/** @unknown: MBZ. */
|
||||
u32 unknown[14];
|
||||
};
|
||||
|
||||
/**
|
||||
* struct panthor_heap_chunk - Structure used to keep track of allocated heap chunks.
|
||||
*/
|
||||
struct panthor_heap_chunk {
|
||||
/** @node: Used to insert the heap chunk in panthor_heap::chunks. */
|
||||
struct list_head node;
|
||||
|
||||
/** @bo: Buffer object backing the heap chunk. */
|
||||
struct panthor_kernel_bo *bo;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct panthor_heap - Structure used to manage tiler heap contexts.
|
||||
*/
|
||||
struct panthor_heap {
|
||||
/** @chunks: List containing all heap chunks allocated so far. */
|
||||
struct list_head chunks;
|
||||
|
||||
/** @lock: Lock protecting insertion in the chunks list. */
|
||||
struct mutex lock;
|
||||
|
||||
/** @chunk_size: Size of each chunk. */
|
||||
u32 chunk_size;
|
||||
|
||||
/** @max_chunks: Maximum number of chunks. */
|
||||
u32 max_chunks;
|
||||
|
||||
/**
|
||||
* @target_in_flight: Number of in-flight render passes after which
|
||||
* we'd let the FW wait for fragment job to finish instead of allocating new chunks.
|
||||
*/
|
||||
u32 target_in_flight;
|
||||
|
||||
/** @chunk_count: Number of heap chunks currently allocated. */
|
||||
u32 chunk_count;
|
||||
};
|
||||
|
||||
#define MAX_HEAPS_PER_POOL 128
|
||||
|
||||
/**
|
||||
* struct panthor_heap_pool - Pool of heap contexts
|
||||
*
|
||||
* The pool is attached to a panthor_file and can't be shared across processes.
|
||||
*/
|
||||
struct panthor_heap_pool {
|
||||
/** @refcount: Reference count. */
|
||||
struct kref refcount;
|
||||
|
||||
/** @ptdev: Device. */
|
||||
struct panthor_device *ptdev;
|
||||
|
||||
/** @vm: VM this pool is bound to. */
|
||||
struct panthor_vm *vm;
|
||||
|
||||
/** @lock: Lock protecting access to @xa. */
|
||||
struct rw_semaphore lock;
|
||||
|
||||
/** @xa: Array storing panthor_heap objects. */
|
||||
struct xarray xa;
|
||||
|
||||
/** @gpu_contexts: Buffer object containing the GPU heap contexts. */
|
||||
struct panthor_kernel_bo *gpu_contexts;
|
||||
};
|
||||
|
||||
static int panthor_heap_ctx_stride(struct panthor_device *ptdev)
|
||||
{
|
||||
u32 l2_features = ptdev->gpu_info.l2_features;
|
||||
u32 gpu_cache_line_size = GPU_L2_FEATURES_LINE_SIZE(l2_features);
|
||||
|
||||
return ALIGN(HEAP_CONTEXT_SIZE, gpu_cache_line_size);
|
||||
}
|
||||
|
||||
static int panthor_get_heap_ctx_offset(struct panthor_heap_pool *pool, int id)
|
||||
{
|
||||
return panthor_heap_ctx_stride(pool->ptdev) * id;
|
||||
}
|
||||
|
||||
static void *panthor_get_heap_ctx(struct panthor_heap_pool *pool, int id)
|
||||
{
|
||||
return pool->gpu_contexts->kmap +
|
||||
panthor_get_heap_ctx_offset(pool, id);
|
||||
}
|
||||
|
||||
static void panthor_free_heap_chunk(struct panthor_vm *vm,
|
||||
struct panthor_heap *heap,
|
||||
struct panthor_heap_chunk *chunk)
|
||||
{
|
||||
mutex_lock(&heap->lock);
|
||||
list_del(&chunk->node);
|
||||
heap->chunk_count--;
|
||||
mutex_unlock(&heap->lock);
|
||||
|
||||
panthor_kernel_bo_destroy(vm, chunk->bo);
|
||||
kfree(chunk);
|
||||
}
|
||||
|
||||
static int panthor_alloc_heap_chunk(struct panthor_device *ptdev,
|
||||
struct panthor_vm *vm,
|
||||
struct panthor_heap *heap,
|
||||
bool initial_chunk)
|
||||
{
|
||||
struct panthor_heap_chunk *chunk;
|
||||
struct panthor_heap_chunk_header *hdr;
|
||||
int ret;
|
||||
|
||||
chunk = kmalloc(sizeof(*chunk), GFP_KERNEL);
|
||||
if (!chunk)
|
||||
return -ENOMEM;
|
||||
|
||||
chunk->bo = panthor_kernel_bo_create(ptdev, vm, heap->chunk_size,
|
||||
DRM_PANTHOR_BO_NO_MMAP,
|
||||
DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC,
|
||||
PANTHOR_VM_KERNEL_AUTO_VA);
|
||||
if (IS_ERR(chunk->bo)) {
|
||||
ret = PTR_ERR(chunk->bo);
|
||||
goto err_free_chunk;
|
||||
}
|
||||
|
||||
ret = panthor_kernel_bo_vmap(chunk->bo);
|
||||
if (ret)
|
||||
goto err_destroy_bo;
|
||||
|
||||
hdr = chunk->bo->kmap;
|
||||
memset(hdr, 0, sizeof(*hdr));
|
||||
|
||||
if (initial_chunk && !list_empty(&heap->chunks)) {
|
||||
struct panthor_heap_chunk *prev_chunk;
|
||||
u64 prev_gpuva;
|
||||
|
||||
prev_chunk = list_first_entry(&heap->chunks,
|
||||
struct panthor_heap_chunk,
|
||||
node);
|
||||
|
||||
prev_gpuva = panthor_kernel_bo_gpuva(prev_chunk->bo);
|
||||
hdr->next = (prev_gpuva & GENMASK_ULL(63, 12)) |
|
||||
(heap->chunk_size >> 12);
|
||||
}
|
||||
|
||||
panthor_kernel_bo_vunmap(chunk->bo);
|
||||
|
||||
mutex_lock(&heap->lock);
|
||||
list_add(&chunk->node, &heap->chunks);
|
||||
heap->chunk_count++;
|
||||
mutex_unlock(&heap->lock);
|
||||
|
||||
return 0;
|
||||
|
||||
err_destroy_bo:
|
||||
panthor_kernel_bo_destroy(vm, chunk->bo);
|
||||
|
||||
err_free_chunk:
|
||||
kfree(chunk);
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
static void panthor_free_heap_chunks(struct panthor_vm *vm,
|
||||
struct panthor_heap *heap)
|
||||
{
|
||||
struct panthor_heap_chunk *chunk, *tmp;
|
||||
|
||||
list_for_each_entry_safe(chunk, tmp, &heap->chunks, node)
|
||||
panthor_free_heap_chunk(vm, heap, chunk);
|
||||
}
|
||||
|
||||
static int panthor_alloc_heap_chunks(struct panthor_device *ptdev,
|
||||
struct panthor_vm *vm,
|
||||
struct panthor_heap *heap,
|
||||
u32 chunk_count)
|
||||
{
|
||||
int ret;
|
||||
u32 i;
|
||||
|
||||
for (i = 0; i < chunk_count; i++) {
|
||||
ret = panthor_alloc_heap_chunk(ptdev, vm, heap, true);
|
||||
if (ret)
|
||||
return ret;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int
|
||||
panthor_heap_destroy_locked(struct panthor_heap_pool *pool, u32 handle)
|
||||
{
|
||||
struct panthor_heap *heap;
|
||||
|
||||
heap = xa_erase(&pool->xa, handle);
|
||||
if (!heap)
|
||||
return -EINVAL;
|
||||
|
||||
panthor_free_heap_chunks(pool->vm, heap);
|
||||
mutex_destroy(&heap->lock);
|
||||
kfree(heap);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_heap_destroy() - Destroy a heap context
|
||||
* @pool: Pool this context belongs to.
|
||||
* @handle: Handle returned by panthor_heap_create().
|
||||
*/
|
||||
int panthor_heap_destroy(struct panthor_heap_pool *pool, u32 handle)
|
||||
{
|
||||
int ret;
|
||||
|
||||
down_write(&pool->lock);
|
||||
ret = panthor_heap_destroy_locked(pool, handle);
|
||||
up_write(&pool->lock);
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_heap_create() - Create a heap context
|
||||
* @pool: Pool to instantiate the heap context from.
|
||||
* @initial_chunk_count: Number of chunk allocated at initialization time.
|
||||
* Must be at least 1.
|
||||
* @chunk_size: The size of each chunk. Must be a power of two between 256k
|
||||
* and 2M.
|
||||
* @max_chunks: Maximum number of chunks that can be allocated.
|
||||
* @target_in_flight: Maximum number of in-flight render passes.
|
||||
* @heap_ctx_gpu_va: Pointer holding the GPU address of the allocated heap
|
||||
* context.
|
||||
* @first_chunk_gpu_va: Pointer holding the GPU address of the first chunk
|
||||
* assigned to the heap context.
|
||||
*
|
||||
* Return: a positive handle on success, a negative error otherwise.
|
||||
*/
|
||||
int panthor_heap_create(struct panthor_heap_pool *pool,
|
||||
u32 initial_chunk_count,
|
||||
u32 chunk_size,
|
||||
u32 max_chunks,
|
||||
u32 target_in_flight,
|
||||
u64 *heap_ctx_gpu_va,
|
||||
u64 *first_chunk_gpu_va)
|
||||
{
|
||||
struct panthor_heap *heap;
|
||||
struct panthor_heap_chunk *first_chunk;
|
||||
struct panthor_vm *vm;
|
||||
int ret = 0;
|
||||
u32 id;
|
||||
|
||||
if (initial_chunk_count == 0)
|
||||
return -EINVAL;
|
||||
|
||||
if (hweight32(chunk_size) != 1 ||
|
||||
chunk_size < SZ_256K || chunk_size > SZ_2M)
|
||||
return -EINVAL;
|
||||
|
||||
down_read(&pool->lock);
|
||||
vm = panthor_vm_get(pool->vm);
|
||||
up_read(&pool->lock);
|
||||
|
||||
/* The pool has been destroyed, we can't create a new heap. */
|
||||
if (!vm)
|
||||
return -EINVAL;
|
||||
|
||||
heap = kzalloc(sizeof(*heap), GFP_KERNEL);
|
||||
if (!heap) {
|
||||
ret = -ENOMEM;
|
||||
goto err_put_vm;
|
||||
}
|
||||
|
||||
mutex_init(&heap->lock);
|
||||
INIT_LIST_HEAD(&heap->chunks);
|
||||
heap->chunk_size = chunk_size;
|
||||
heap->max_chunks = max_chunks;
|
||||
heap->target_in_flight = target_in_flight;
|
||||
|
||||
ret = panthor_alloc_heap_chunks(pool->ptdev, vm, heap,
|
||||
initial_chunk_count);
|
||||
if (ret)
|
||||
goto err_free_heap;
|
||||
|
||||
first_chunk = list_first_entry(&heap->chunks,
|
||||
struct panthor_heap_chunk,
|
||||
node);
|
||||
*first_chunk_gpu_va = panthor_kernel_bo_gpuva(first_chunk->bo);
|
||||
|
||||
down_write(&pool->lock);
|
||||
/* The pool has been destroyed, we can't create a new heap. */
|
||||
if (!pool->vm) {
|
||||
ret = -EINVAL;
|
||||
} else {
|
||||
ret = xa_alloc(&pool->xa, &id, heap, XA_LIMIT(1, MAX_HEAPS_PER_POOL), GFP_KERNEL);
|
||||
if (!ret) {
|
||||
void *gpu_ctx = panthor_get_heap_ctx(pool, id);
|
||||
|
||||
memset(gpu_ctx, 0, panthor_heap_ctx_stride(pool->ptdev));
|
||||
*heap_ctx_gpu_va = panthor_kernel_bo_gpuva(pool->gpu_contexts) +
|
||||
panthor_get_heap_ctx_offset(pool, id);
|
||||
}
|
||||
}
|
||||
up_write(&pool->lock);
|
||||
|
||||
if (ret)
|
||||
goto err_free_heap;
|
||||
|
||||
panthor_vm_put(vm);
|
||||
return id;
|
||||
|
||||
err_free_heap:
|
||||
panthor_free_heap_chunks(pool->vm, heap);
|
||||
mutex_destroy(&heap->lock);
|
||||
kfree(heap);
|
||||
|
||||
err_put_vm:
|
||||
panthor_vm_put(vm);
|
||||
return ret;
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_heap_return_chunk() - Return an unused heap chunk
|
||||
* @pool: The pool this heap belongs to.
|
||||
* @heap_gpu_va: The GPU address of the heap context.
|
||||
* @chunk_gpu_va: The chunk VA to return.
|
||||
*
|
||||
* This function is used when a chunk allocated with panthor_heap_grow()
|
||||
* couldn't be linked to the heap context through the FW interface because
|
||||
* the group requesting the allocation was scheduled out in the meantime.
|
||||
*/
|
||||
int panthor_heap_return_chunk(struct panthor_heap_pool *pool,
|
||||
u64 heap_gpu_va,
|
||||
u64 chunk_gpu_va)
|
||||
{
|
||||
u64 offset = heap_gpu_va - panthor_kernel_bo_gpuva(pool->gpu_contexts);
|
||||
u32 heap_id = (u32)offset / panthor_heap_ctx_stride(pool->ptdev);
|
||||
struct panthor_heap_chunk *chunk, *tmp, *removed = NULL;
|
||||
struct panthor_heap *heap;
|
||||
int ret;
|
||||
|
||||
if (offset > U32_MAX || heap_id >= MAX_HEAPS_PER_POOL)
|
||||
return -EINVAL;
|
||||
|
||||
down_read(&pool->lock);
|
||||
heap = xa_load(&pool->xa, heap_id);
|
||||
if (!heap) {
|
||||
ret = -EINVAL;
|
||||
goto out_unlock;
|
||||
}
|
||||
|
||||
chunk_gpu_va &= GENMASK_ULL(63, 12);
|
||||
|
||||
mutex_lock(&heap->lock);
|
||||
list_for_each_entry_safe(chunk, tmp, &heap->chunks, node) {
|
||||
if (panthor_kernel_bo_gpuva(chunk->bo) == chunk_gpu_va) {
|
||||
removed = chunk;
|
||||
list_del(&chunk->node);
|
||||
heap->chunk_count--;
|
||||
break;
|
||||
}
|
||||
}
|
||||
mutex_unlock(&heap->lock);
|
||||
|
||||
if (removed) {
|
||||
panthor_kernel_bo_destroy(pool->vm, chunk->bo);
|
||||
kfree(chunk);
|
||||
ret = 0;
|
||||
} else {
|
||||
ret = -EINVAL;
|
||||
}
|
||||
|
||||
out_unlock:
|
||||
up_read(&pool->lock);
|
||||
return ret;
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_heap_grow() - Make a heap context grow.
|
||||
* @pool: The pool this heap belongs to.
|
||||
* @heap_gpu_va: The GPU address of the heap context.
|
||||
* @renderpasses_in_flight: Number of render passes currently in-flight.
|
||||
* @pending_frag_count: Number of fragment jobs waiting for execution/completion.
|
||||
* @new_chunk_gpu_va: Pointer used to return the chunk VA.
|
||||
*/
|
||||
int panthor_heap_grow(struct panthor_heap_pool *pool,
|
||||
u64 heap_gpu_va,
|
||||
u32 renderpasses_in_flight,
|
||||
u32 pending_frag_count,
|
||||
u64 *new_chunk_gpu_va)
|
||||
{
|
||||
u64 offset = heap_gpu_va - panthor_kernel_bo_gpuva(pool->gpu_contexts);
|
||||
u32 heap_id = (u32)offset / panthor_heap_ctx_stride(pool->ptdev);
|
||||
struct panthor_heap_chunk *chunk;
|
||||
struct panthor_heap *heap;
|
||||
int ret;
|
||||
|
||||
if (offset > U32_MAX || heap_id >= MAX_HEAPS_PER_POOL)
|
||||
return -EINVAL;
|
||||
|
||||
down_read(&pool->lock);
|
||||
heap = xa_load(&pool->xa, heap_id);
|
||||
if (!heap) {
|
||||
ret = -EINVAL;
|
||||
goto out_unlock;
|
||||
}
|
||||
|
||||
/* If we reached the target in-flight render passes, or if we
|
||||
* reached the maximum number of chunks, let the FW figure another way to
|
||||
* find some memory (wait for render passes to finish, or call the exception
|
||||
* handler provided by the userspace driver, if any).
|
||||
*/
|
||||
if (renderpasses_in_flight > heap->target_in_flight ||
|
||||
(pending_frag_count > 0 && heap->chunk_count >= heap->max_chunks)) {
|
||||
ret = -EBUSY;
|
||||
goto out_unlock;
|
||||
} else if (heap->chunk_count >= heap->max_chunks) {
|
||||
ret = -ENOMEM;
|
||||
goto out_unlock;
|
||||
}
|
||||
|
||||
/* FIXME: panthor_alloc_heap_chunk() triggers a kernel BO creation,
|
||||
* which goes through the blocking allocation path. Ultimately, we
|
||||
* want a non-blocking allocation, so we can immediately report to the
|
||||
* FW when the system is running out of memory. In that case, the FW
|
||||
* can call a user-provided exception handler, which might try to free
|
||||
* some tiler memory by issuing an intermediate fragment job. If the
|
||||
* exception handler can't do anything, it will flag the queue as
|
||||
* faulty so the job that triggered this tiler chunk allocation and all
|
||||
* further jobs in this queue fail immediately instead of having to
|
||||
* wait for the job timeout.
|
||||
*/
|
||||
ret = panthor_alloc_heap_chunk(pool->ptdev, pool->vm, heap, false);
|
||||
if (ret)
|
||||
goto out_unlock;
|
||||
|
||||
chunk = list_first_entry(&heap->chunks,
|
||||
struct panthor_heap_chunk,
|
||||
node);
|
||||
*new_chunk_gpu_va = (panthor_kernel_bo_gpuva(chunk->bo) & GENMASK_ULL(63, 12)) |
|
||||
(heap->chunk_size >> 12);
|
||||
ret = 0;
|
||||
|
||||
out_unlock:
|
||||
up_read(&pool->lock);
|
||||
return ret;
|
||||
}
|
||||
|
||||
static void panthor_heap_pool_release(struct kref *refcount)
|
||||
{
|
||||
struct panthor_heap_pool *pool =
|
||||
container_of(refcount, struct panthor_heap_pool, refcount);
|
||||
|
||||
xa_destroy(&pool->xa);
|
||||
kfree(pool);
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_heap_pool_put() - Release a heap pool reference
|
||||
* @pool: Pool to release the reference on. Can be NULL.
|
||||
*/
|
||||
void panthor_heap_pool_put(struct panthor_heap_pool *pool)
|
||||
{
|
||||
if (pool)
|
||||
kref_put(&pool->refcount, panthor_heap_pool_release);
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_heap_pool_get() - Get a heap pool reference
|
||||
* @pool: Pool to get the reference on. Can be NULL.
|
||||
*
|
||||
* Return: @pool.
|
||||
*/
|
||||
struct panthor_heap_pool *
|
||||
panthor_heap_pool_get(struct panthor_heap_pool *pool)
|
||||
{
|
||||
if (pool)
|
||||
kref_get(&pool->refcount);
|
||||
|
||||
return pool;
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_heap_pool_create() - Create a heap pool
|
||||
* @ptdev: Device.
|
||||
* @vm: The VM this heap pool will be attached to.
|
||||
*
|
||||
* Heap pools might contain up to 128 heap contexts, and are per-VM.
|
||||
*
|
||||
* Return: A valid pointer on success, a negative error code otherwise.
|
||||
*/
|
||||
struct panthor_heap_pool *
|
||||
panthor_heap_pool_create(struct panthor_device *ptdev, struct panthor_vm *vm)
|
||||
{
|
||||
size_t bosize = ALIGN(MAX_HEAPS_PER_POOL *
|
||||
panthor_heap_ctx_stride(ptdev),
|
||||
4096);
|
||||
struct panthor_heap_pool *pool;
|
||||
int ret = 0;
|
||||
|
||||
pool = kzalloc(sizeof(*pool), GFP_KERNEL);
|
||||
if (!pool)
|
||||
return ERR_PTR(-ENOMEM);
|
||||
|
||||
/* We want a weak ref here: the heap pool belongs to the VM, so we're
|
||||
* sure that, as long as the heap pool exists, the VM exists too.
|
||||
*/
|
||||
pool->vm = vm;
|
||||
pool->ptdev = ptdev;
|
||||
init_rwsem(&pool->lock);
|
||||
xa_init_flags(&pool->xa, XA_FLAGS_ALLOC1);
|
||||
kref_init(&pool->refcount);
|
||||
|
||||
pool->gpu_contexts = panthor_kernel_bo_create(ptdev, vm, bosize,
|
||||
DRM_PANTHOR_BO_NO_MMAP,
|
||||
DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC,
|
||||
PANTHOR_VM_KERNEL_AUTO_VA);
|
||||
if (IS_ERR(pool->gpu_contexts)) {
|
||||
ret = PTR_ERR(pool->gpu_contexts);
|
||||
goto err_destroy_pool;
|
||||
}
|
||||
|
||||
ret = panthor_kernel_bo_vmap(pool->gpu_contexts);
|
||||
if (ret)
|
||||
goto err_destroy_pool;
|
||||
|
||||
return pool;
|
||||
|
||||
err_destroy_pool:
|
||||
panthor_heap_pool_destroy(pool);
|
||||
return ERR_PTR(ret);
|
||||
}
|
||||
|
||||
/**
|
||||
* panthor_heap_pool_destroy() - Destroy a heap pool.
|
||||
* @pool: Pool to destroy.
|
||||
*
|
||||
* This function destroys all heap contexts and their resources. Thus
|
||||
* preventing any use of the heap context or the chunk attached to them
|
||||
* after that point.
|
||||
*
|
||||
* If the GPU still has access to some heap contexts, a fault should be
|
||||
* triggered, which should flag the command stream groups using these
|
||||
* context as faulty.
|
||||
*
|
||||
* The heap pool object is only released when all references to this pool
|
||||
* are released.
|
||||
*/
|
||||
void panthor_heap_pool_destroy(struct panthor_heap_pool *pool)
|
||||
{
|
||||
struct panthor_heap *heap;
|
||||
unsigned long i;
|
||||
|
||||
if (!pool)
|
||||
return;
|
||||
|
||||
down_write(&pool->lock);
|
||||
xa_for_each(&pool->xa, i, heap)
|
||||
drm_WARN_ON(&pool->ptdev->base, panthor_heap_destroy_locked(pool, i));
|
||||
|
||||
if (!IS_ERR_OR_NULL(pool->gpu_contexts))
|
||||
panthor_kernel_bo_destroy(pool->vm, pool->gpu_contexts);
|
||||
|
||||
/* Reflects the fact the pool has been destroyed. */
|
||||
pool->vm = NULL;
|
||||
up_write(&pool->lock);
|
||||
|
||||
panthor_heap_pool_put(pool);
|
||||
}
|
||||
39
drivers/gpu/drm/panthor/panthor_heap.h
Normal file
39
drivers/gpu/drm/panthor/panthor_heap.h
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0 or MIT */
|
||||
/* Copyright 2023 Collabora ltd. */
|
||||
|
||||
#ifndef __PANTHOR_HEAP_H__
|
||||
#define __PANTHOR_HEAP_H__
|
||||
|
||||
#include <linux/types.h>
|
||||
|
||||
struct panthor_device;
|
||||
struct panthor_heap_pool;
|
||||
struct panthor_vm;
|
||||
|
||||
int panthor_heap_create(struct panthor_heap_pool *pool,
|
||||
u32 initial_chunk_count,
|
||||
u32 chunk_size,
|
||||
u32 max_chunks,
|
||||
u32 target_in_flight,
|
||||
u64 *heap_ctx_gpu_va,
|
||||
u64 *first_chunk_gpu_va);
|
||||
int panthor_heap_destroy(struct panthor_heap_pool *pool, u32 handle);
|
||||
|
||||
struct panthor_heap_pool *
|
||||
panthor_heap_pool_create(struct panthor_device *ptdev, struct panthor_vm *vm);
|
||||
void panthor_heap_pool_destroy(struct panthor_heap_pool *pool);
|
||||
|
||||
struct panthor_heap_pool *
|
||||
panthor_heap_pool_get(struct panthor_heap_pool *pool);
|
||||
void panthor_heap_pool_put(struct panthor_heap_pool *pool);
|
||||
|
||||
int panthor_heap_grow(struct panthor_heap_pool *pool,
|
||||
u64 heap_gpu_va,
|
||||
u32 renderpasses_in_flight,
|
||||
u32 pending_frag_count,
|
||||
u64 *new_chunk_gpu_va);
|
||||
int panthor_heap_return_chunk(struct panthor_heap_pool *pool,
|
||||
u64 heap_gpu_va,
|
||||
u64 chunk_gpu_va);
|
||||
|
||||
#endif
|
||||
2717
drivers/gpu/drm/panthor/panthor_mmu.c
Normal file
2717
drivers/gpu/drm/panthor/panthor_mmu.c
Normal file
File diff suppressed because it is too large
Load diff
98
drivers/gpu/drm/panthor/panthor_mmu.h
Normal file
98
drivers/gpu/drm/panthor/panthor_mmu.h
Normal file
|
|
@ -0,0 +1,98 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0 or MIT */
|
||||
/* Copyright 2019 Linaro, Ltd, Rob Herring <robh@kernel.org> */
|
||||
/* Copyright 2023 Collabora ltd. */
|
||||
|
||||
#ifndef __PANTHOR_MMU_H__
|
||||
#define __PANTHOR_MMU_H__
|
||||
|
||||
#include <linux/dma-resv.h>
|
||||
|
||||
struct drm_exec;
|
||||
struct drm_sched_job;
|
||||
struct panthor_gem_object;
|
||||
struct panthor_heap_pool;
|
||||
struct panthor_vm;
|
||||
struct panthor_vma;
|
||||
struct panthor_mmu;
|
||||
|
||||
int panthor_mmu_init(struct panthor_device *ptdev);
|
||||
void panthor_mmu_unplug(struct panthor_device *ptdev);
|
||||
void panthor_mmu_pre_reset(struct panthor_device *ptdev);
|
||||
void panthor_mmu_post_reset(struct panthor_device *ptdev);
|
||||
void panthor_mmu_suspend(struct panthor_device *ptdev);
|
||||
void panthor_mmu_resume(struct panthor_device *ptdev);
|
||||
|
||||
int panthor_vm_map_bo_range(struct panthor_vm *vm, struct panthor_gem_object *bo,
|
||||
u64 offset, u64 size, u64 va, u32 flags);
|
||||
int panthor_vm_unmap_range(struct panthor_vm *vm, u64 va, u64 size);
|
||||
struct panthor_gem_object *
|
||||
panthor_vm_get_bo_for_va(struct panthor_vm *vm, u64 va, u64 *bo_offset);
|
||||
|
||||
int panthor_vm_active(struct panthor_vm *vm);
|
||||
void panthor_vm_idle(struct panthor_vm *vm);
|
||||
int panthor_vm_as(struct panthor_vm *vm);
|
||||
|
||||
struct panthor_heap_pool *
|
||||
panthor_vm_get_heap_pool(struct panthor_vm *vm, bool create);
|
||||
|
||||
struct panthor_vm *panthor_vm_get(struct panthor_vm *vm);
|
||||
void panthor_vm_put(struct panthor_vm *vm);
|
||||
struct panthor_vm *panthor_vm_create(struct panthor_device *ptdev, bool for_mcu,
|
||||
u64 kernel_va_start, u64 kernel_va_size,
|
||||
u64 kernel_auto_va_start,
|
||||
u64 kernel_auto_va_size);
|
||||
|
||||
int panthor_vm_prepare_mapped_bos_resvs(struct drm_exec *exec,
|
||||
struct panthor_vm *vm,
|
||||
u32 slot_count);
|
||||
int panthor_vm_add_bos_resvs_deps_to_job(struct panthor_vm *vm,
|
||||
struct drm_sched_job *job);
|
||||
void panthor_vm_add_job_fence_to_bos_resvs(struct panthor_vm *vm,
|
||||
struct drm_sched_job *job);
|
||||
|
||||
struct dma_resv *panthor_vm_resv(struct panthor_vm *vm);
|
||||
struct drm_gem_object *panthor_vm_root_gem(struct panthor_vm *vm);
|
||||
|
||||
void panthor_vm_pool_destroy(struct panthor_file *pfile);
|
||||
int panthor_vm_pool_create(struct panthor_file *pfile);
|
||||
int panthor_vm_pool_create_vm(struct panthor_device *ptdev,
|
||||
struct panthor_vm_pool *pool,
|
||||
struct drm_panthor_vm_create *args);
|
||||
int panthor_vm_pool_destroy_vm(struct panthor_vm_pool *pool, u32 handle);
|
||||
struct panthor_vm *panthor_vm_pool_get_vm(struct panthor_vm_pool *pool, u32 handle);
|
||||
|
||||
bool panthor_vm_has_unhandled_faults(struct panthor_vm *vm);
|
||||
bool panthor_vm_is_unusable(struct panthor_vm *vm);
|
||||
|
||||
/*
|
||||
* PANTHOR_VM_KERNEL_AUTO_VA: Use this magic address when you want the GEM
|
||||
* logic to auto-allocate the virtual address in the reserved kernel VA range.
|
||||
*/
|
||||
#define PANTHOR_VM_KERNEL_AUTO_VA ~0ull
|
||||
|
||||
int panthor_vm_alloc_va(struct panthor_vm *vm, u64 va, u64 size,
|
||||
struct drm_mm_node *va_node);
|
||||
void panthor_vm_free_va(struct panthor_vm *vm, struct drm_mm_node *va_node);
|
||||
|
||||
int panthor_vm_bind_exec_sync_op(struct drm_file *file,
|
||||
struct panthor_vm *vm,
|
||||
struct drm_panthor_vm_bind_op *op);
|
||||
|
||||
struct drm_sched_job *
|
||||
panthor_vm_bind_job_create(struct drm_file *file,
|
||||
struct panthor_vm *vm,
|
||||
const struct drm_panthor_vm_bind_op *op);
|
||||
void panthor_vm_bind_job_put(struct drm_sched_job *job);
|
||||
int panthor_vm_bind_job_prepare_resvs(struct drm_exec *exec,
|
||||
struct drm_sched_job *job);
|
||||
void panthor_vm_bind_job_update_resvs(struct drm_exec *exec, struct drm_sched_job *job);
|
||||
|
||||
void panthor_vm_update_resvs(struct panthor_vm *vm, struct drm_exec *exec,
|
||||
struct dma_fence *fence,
|
||||
enum dma_resv_usage private_usage,
|
||||
enum dma_resv_usage extobj_usage);
|
||||
|
||||
int panthor_mmu_pt_cache_init(void);
|
||||
void panthor_mmu_pt_cache_fini(void);
|
||||
|
||||
#endif
|
||||
239
drivers/gpu/drm/panthor/panthor_regs.h
Normal file
239
drivers/gpu/drm/panthor/panthor_regs.h
Normal file
|
|
@ -0,0 +1,239 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0 or MIT */
|
||||
/* Copyright 2018 Marty E. Plummer <hanetzer@startmail.com> */
|
||||
/* Copyright 2019 Linaro, Ltd, Rob Herring <robh@kernel.org> */
|
||||
/* Copyright 2023 Collabora ltd. */
|
||||
/*
|
||||
* Register definitions based on mali_kbase_gpu_regmap.h and
|
||||
* mali_kbase_gpu_regmap_csf.h
|
||||
* (C) COPYRIGHT 2010-2022 ARM Limited. All rights reserved.
|
||||
*/
|
||||
#ifndef __PANTHOR_REGS_H__
|
||||
#define __PANTHOR_REGS_H__
|
||||
|
||||
#define GPU_ID 0x0
|
||||
#define GPU_ARCH_MAJOR(x) ((x) >> 28)
|
||||
#define GPU_ARCH_MINOR(x) (((x) & GENMASK(27, 24)) >> 24)
|
||||
#define GPU_ARCH_REV(x) (((x) & GENMASK(23, 20)) >> 20)
|
||||
#define GPU_PROD_MAJOR(x) (((x) & GENMASK(19, 16)) >> 16)
|
||||
#define GPU_VER_MAJOR(x) (((x) & GENMASK(15, 12)) >> 12)
|
||||
#define GPU_VER_MINOR(x) (((x) & GENMASK(11, 4)) >> 4)
|
||||
#define GPU_VER_STATUS(x) ((x) & GENMASK(3, 0))
|
||||
|
||||
#define GPU_L2_FEATURES 0x4
|
||||
#define GPU_L2_FEATURES_LINE_SIZE(x) (1 << ((x) & GENMASK(7, 0)))
|
||||
|
||||
#define GPU_CORE_FEATURES 0x8
|
||||
|
||||
#define GPU_TILER_FEATURES 0xC
|
||||
#define GPU_MEM_FEATURES 0x10
|
||||
#define GROUPS_L2_COHERENT BIT(0)
|
||||
|
||||
#define GPU_MMU_FEATURES 0x14
|
||||
#define GPU_MMU_FEATURES_VA_BITS(x) ((x) & GENMASK(7, 0))
|
||||
#define GPU_MMU_FEATURES_PA_BITS(x) (((x) >> 8) & GENMASK(7, 0))
|
||||
#define GPU_AS_PRESENT 0x18
|
||||
#define GPU_CSF_ID 0x1C
|
||||
|
||||
#define GPU_INT_RAWSTAT 0x20
|
||||
#define GPU_INT_CLEAR 0x24
|
||||
#define GPU_INT_MASK 0x28
|
||||
#define GPU_INT_STAT 0x2c
|
||||
#define GPU_IRQ_FAULT BIT(0)
|
||||
#define GPU_IRQ_PROTM_FAULT BIT(1)
|
||||
#define GPU_IRQ_RESET_COMPLETED BIT(8)
|
||||
#define GPU_IRQ_POWER_CHANGED BIT(9)
|
||||
#define GPU_IRQ_POWER_CHANGED_ALL BIT(10)
|
||||
#define GPU_IRQ_CLEAN_CACHES_COMPLETED BIT(17)
|
||||
#define GPU_IRQ_DOORBELL_MIRROR BIT(18)
|
||||
#define GPU_IRQ_MCU_STATUS_CHANGED BIT(19)
|
||||
#define GPU_CMD 0x30
|
||||
#define GPU_CMD_DEF(type, payload) ((type) | ((payload) << 8))
|
||||
#define GPU_SOFT_RESET GPU_CMD_DEF(1, 1)
|
||||
#define GPU_HARD_RESET GPU_CMD_DEF(1, 2)
|
||||
#define CACHE_CLEAN BIT(0)
|
||||
#define CACHE_INV BIT(1)
|
||||
#define GPU_FLUSH_CACHES(l2, lsc, oth) \
|
||||
GPU_CMD_DEF(4, ((l2) << 0) | ((lsc) << 4) | ((oth) << 8))
|
||||
|
||||
#define GPU_STATUS 0x34
|
||||
#define GPU_STATUS_ACTIVE BIT(0)
|
||||
#define GPU_STATUS_PWR_ACTIVE BIT(1)
|
||||
#define GPU_STATUS_PAGE_FAULT BIT(4)
|
||||
#define GPU_STATUS_PROTM_ACTIVE BIT(7)
|
||||
#define GPU_STATUS_DBG_ENABLED BIT(8)
|
||||
|
||||
#define GPU_FAULT_STATUS 0x3C
|
||||
#define GPU_FAULT_ADDR_LO 0x40
|
||||
#define GPU_FAULT_ADDR_HI 0x44
|
||||
|
||||
#define GPU_PWR_KEY 0x50
|
||||
#define GPU_PWR_KEY_UNLOCK 0x2968A819
|
||||
#define GPU_PWR_OVERRIDE0 0x54
|
||||
#define GPU_PWR_OVERRIDE1 0x58
|
||||
|
||||
#define GPU_TIMESTAMP_OFFSET_LO 0x88
|
||||
#define GPU_TIMESTAMP_OFFSET_HI 0x8C
|
||||
#define GPU_CYCLE_COUNT_LO 0x90
|
||||
#define GPU_CYCLE_COUNT_HI 0x94
|
||||
#define GPU_TIMESTAMP_LO 0x98
|
||||
#define GPU_TIMESTAMP_HI 0x9C
|
||||
|
||||
#define GPU_THREAD_MAX_THREADS 0xA0
|
||||
#define GPU_THREAD_MAX_WORKGROUP_SIZE 0xA4
|
||||
#define GPU_THREAD_MAX_BARRIER_SIZE 0xA8
|
||||
#define GPU_THREAD_FEATURES 0xAC
|
||||
|
||||
#define GPU_TEXTURE_FEATURES(n) (0xB0 + ((n) * 4))
|
||||
|
||||
#define GPU_SHADER_PRESENT_LO 0x100
|
||||
#define GPU_SHADER_PRESENT_HI 0x104
|
||||
#define GPU_TILER_PRESENT_LO 0x110
|
||||
#define GPU_TILER_PRESENT_HI 0x114
|
||||
#define GPU_L2_PRESENT_LO 0x120
|
||||
#define GPU_L2_PRESENT_HI 0x124
|
||||
|
||||
#define SHADER_READY_LO 0x140
|
||||
#define SHADER_READY_HI 0x144
|
||||
#define TILER_READY_LO 0x150
|
||||
#define TILER_READY_HI 0x154
|
||||
#define L2_READY_LO 0x160
|
||||
#define L2_READY_HI 0x164
|
||||
|
||||
#define SHADER_PWRON_LO 0x180
|
||||
#define SHADER_PWRON_HI 0x184
|
||||
#define TILER_PWRON_LO 0x190
|
||||
#define TILER_PWRON_HI 0x194
|
||||
#define L2_PWRON_LO 0x1A0
|
||||
#define L2_PWRON_HI 0x1A4
|
||||
|
||||
#define SHADER_PWROFF_LO 0x1C0
|
||||
#define SHADER_PWROFF_HI 0x1C4
|
||||
#define TILER_PWROFF_LO 0x1D0
|
||||
#define TILER_PWROFF_HI 0x1D4
|
||||
#define L2_PWROFF_LO 0x1E0
|
||||
#define L2_PWROFF_HI 0x1E4
|
||||
|
||||
#define SHADER_PWRTRANS_LO 0x200
|
||||
#define SHADER_PWRTRANS_HI 0x204
|
||||
#define TILER_PWRTRANS_LO 0x210
|
||||
#define TILER_PWRTRANS_HI 0x214
|
||||
#define L2_PWRTRANS_LO 0x220
|
||||
#define L2_PWRTRANS_HI 0x224
|
||||
|
||||
#define SHADER_PWRACTIVE_LO 0x240
|
||||
#define SHADER_PWRACTIVE_HI 0x244
|
||||
#define TILER_PWRACTIVE_LO 0x250
|
||||
#define TILER_PWRACTIVE_HI 0x254
|
||||
#define L2_PWRACTIVE_LO 0x260
|
||||
#define L2_PWRACTIVE_HI 0x264
|
||||
|
||||
#define GPU_REVID 0x280
|
||||
|
||||
#define GPU_COHERENCY_FEATURES 0x300
|
||||
#define GPU_COHERENCY_PROT_BIT(name) BIT(GPU_COHERENCY_ ## name)
|
||||
|
||||
#define GPU_COHERENCY_PROTOCOL 0x304
|
||||
#define GPU_COHERENCY_ACE 0
|
||||
#define GPU_COHERENCY_ACE_LITE 1
|
||||
#define GPU_COHERENCY_NONE 31
|
||||
|
||||
#define MCU_CONTROL 0x700
|
||||
#define MCU_CONTROL_ENABLE 1
|
||||
#define MCU_CONTROL_AUTO 2
|
||||
#define MCU_CONTROL_DISABLE 0
|
||||
|
||||
#define MCU_STATUS 0x704
|
||||
#define MCU_STATUS_DISABLED 0
|
||||
#define MCU_STATUS_ENABLED 1
|
||||
#define MCU_STATUS_HALT 2
|
||||
#define MCU_STATUS_FATAL 3
|
||||
|
||||
/* Job Control regs */
|
||||
#define JOB_INT_RAWSTAT 0x1000
|
||||
#define JOB_INT_CLEAR 0x1004
|
||||
#define JOB_INT_MASK 0x1008
|
||||
#define JOB_INT_STAT 0x100c
|
||||
#define JOB_INT_GLOBAL_IF BIT(31)
|
||||
#define JOB_INT_CSG_IF(x) BIT(x)
|
||||
|
||||
/* MMU regs */
|
||||
#define MMU_INT_RAWSTAT 0x2000
|
||||
#define MMU_INT_CLEAR 0x2004
|
||||
#define MMU_INT_MASK 0x2008
|
||||
#define MMU_INT_STAT 0x200c
|
||||
|
||||
/* AS_COMMAND register commands */
|
||||
|
||||
#define MMU_BASE 0x2400
|
||||
#define MMU_AS_SHIFT 6
|
||||
#define MMU_AS(as) (MMU_BASE + ((as) << MMU_AS_SHIFT))
|
||||
|
||||
#define AS_TRANSTAB_LO(as) (MMU_AS(as) + 0x0)
|
||||
#define AS_TRANSTAB_HI(as) (MMU_AS(as) + 0x4)
|
||||
#define AS_MEMATTR_LO(as) (MMU_AS(as) + 0x8)
|
||||
#define AS_MEMATTR_HI(as) (MMU_AS(as) + 0xC)
|
||||
#define AS_MEMATTR_AARCH64_INNER_ALLOC_IMPL (2 << 2)
|
||||
#define AS_MEMATTR_AARCH64_INNER_ALLOC_EXPL(w, r) ((3 << 2) | \
|
||||
((w) ? BIT(0) : 0) | \
|
||||
((r) ? BIT(1) : 0))
|
||||
#define AS_MEMATTR_AARCH64_SH_MIDGARD_INNER (0 << 4)
|
||||
#define AS_MEMATTR_AARCH64_SH_CPU_INNER (1 << 4)
|
||||
#define AS_MEMATTR_AARCH64_SH_CPU_INNER_SHADER_COH (2 << 4)
|
||||
#define AS_MEMATTR_AARCH64_SHARED (0 << 6)
|
||||
#define AS_MEMATTR_AARCH64_INNER_OUTER_NC (1 << 6)
|
||||
#define AS_MEMATTR_AARCH64_INNER_OUTER_WB (2 << 6)
|
||||
#define AS_MEMATTR_AARCH64_FAULT (3 << 6)
|
||||
#define AS_LOCKADDR_LO(as) (MMU_AS(as) + 0x10)
|
||||
#define AS_LOCKADDR_HI(as) (MMU_AS(as) + 0x14)
|
||||
#define AS_COMMAND(as) (MMU_AS(as) + 0x18)
|
||||
#define AS_COMMAND_NOP 0
|
||||
#define AS_COMMAND_UPDATE 1
|
||||
#define AS_COMMAND_LOCK 2
|
||||
#define AS_COMMAND_UNLOCK 3
|
||||
#define AS_COMMAND_FLUSH_PT 4
|
||||
#define AS_COMMAND_FLUSH_MEM 5
|
||||
#define AS_LOCK_REGION_MIN_SIZE (1ULL << 15)
|
||||
#define AS_FAULTSTATUS(as) (MMU_AS(as) + 0x1C)
|
||||
#define AS_FAULTSTATUS_ACCESS_TYPE_MASK (0x3 << 8)
|
||||
#define AS_FAULTSTATUS_ACCESS_TYPE_ATOMIC (0x0 << 8)
|
||||
#define AS_FAULTSTATUS_ACCESS_TYPE_EX (0x1 << 8)
|
||||
#define AS_FAULTSTATUS_ACCESS_TYPE_READ (0x2 << 8)
|
||||
#define AS_FAULTSTATUS_ACCESS_TYPE_WRITE (0x3 << 8)
|
||||
#define AS_FAULTADDRESS_LO(as) (MMU_AS(as) + 0x20)
|
||||
#define AS_FAULTADDRESS_HI(as) (MMU_AS(as) + 0x24)
|
||||
#define AS_STATUS(as) (MMU_AS(as) + 0x28)
|
||||
#define AS_STATUS_AS_ACTIVE BIT(0)
|
||||
#define AS_TRANSCFG_LO(as) (MMU_AS(as) + 0x30)
|
||||
#define AS_TRANSCFG_HI(as) (MMU_AS(as) + 0x34)
|
||||
#define AS_TRANSCFG_ADRMODE_UNMAPPED (1 << 0)
|
||||
#define AS_TRANSCFG_ADRMODE_IDENTITY (2 << 0)
|
||||
#define AS_TRANSCFG_ADRMODE_AARCH64_4K (6 << 0)
|
||||
#define AS_TRANSCFG_ADRMODE_AARCH64_64K (8 << 0)
|
||||
#define AS_TRANSCFG_INA_BITS(x) ((x) << 6)
|
||||
#define AS_TRANSCFG_OUTA_BITS(x) ((x) << 14)
|
||||
#define AS_TRANSCFG_SL_CONCAT BIT(22)
|
||||
#define AS_TRANSCFG_PTW_MEMATTR_NC (1 << 24)
|
||||
#define AS_TRANSCFG_PTW_MEMATTR_WB (2 << 24)
|
||||
#define AS_TRANSCFG_PTW_SH_NS (0 << 28)
|
||||
#define AS_TRANSCFG_PTW_SH_OS (2 << 28)
|
||||
#define AS_TRANSCFG_PTW_SH_IS (3 << 28)
|
||||
#define AS_TRANSCFG_PTW_RA BIT(30)
|
||||
#define AS_TRANSCFG_DISABLE_HIER_AP BIT(33)
|
||||
#define AS_TRANSCFG_DISABLE_AF_FAULT BIT(34)
|
||||
#define AS_TRANSCFG_WXN BIT(35)
|
||||
#define AS_TRANSCFG_XREADABLE BIT(36)
|
||||
#define AS_FAULTEXTRA_LO(as) (MMU_AS(as) + 0x38)
|
||||
#define AS_FAULTEXTRA_HI(as) (MMU_AS(as) + 0x3C)
|
||||
|
||||
#define CSF_GPU_LATEST_FLUSH_ID 0x10000
|
||||
|
||||
#define CSF_DOORBELL(i) (0x80000 + ((i) * 0x10000))
|
||||
#define CSF_GLB_DOORBELL_ID 0
|
||||
|
||||
#define gpu_write(dev, reg, data) \
|
||||
writel(data, (dev)->iomem + (reg))
|
||||
|
||||
#define gpu_read(dev, reg) \
|
||||
readl((dev)->iomem + (reg))
|
||||
|
||||
#endif
|
||||
3502
drivers/gpu/drm/panthor/panthor_sched.c
Normal file
3502
drivers/gpu/drm/panthor/panthor_sched.c
Normal file
File diff suppressed because it is too large
Load diff
50
drivers/gpu/drm/panthor/panthor_sched.h
Normal file
50
drivers/gpu/drm/panthor/panthor_sched.h
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0 or MIT */
|
||||
/* Copyright 2023 Collabora ltd. */
|
||||
|
||||
#ifndef __PANTHOR_SCHED_H__
|
||||
#define __PANTHOR_SCHED_H__
|
||||
|
||||
struct drm_exec;
|
||||
struct dma_fence;
|
||||
struct drm_file;
|
||||
struct drm_gem_object;
|
||||
struct drm_sched_job;
|
||||
struct drm_panthor_group_create;
|
||||
struct drm_panthor_queue_create;
|
||||
struct drm_panthor_group_get_state;
|
||||
struct drm_panthor_queue_submit;
|
||||
struct panthor_device;
|
||||
struct panthor_file;
|
||||
struct panthor_group_pool;
|
||||
struct panthor_job;
|
||||
|
||||
int panthor_group_create(struct panthor_file *pfile,
|
||||
const struct drm_panthor_group_create *group_args,
|
||||
const struct drm_panthor_queue_create *queue_args);
|
||||
int panthor_group_destroy(struct panthor_file *pfile, u32 group_handle);
|
||||
int panthor_group_get_state(struct panthor_file *pfile,
|
||||
struct drm_panthor_group_get_state *get_state);
|
||||
|
||||
struct drm_sched_job *
|
||||
panthor_job_create(struct panthor_file *pfile,
|
||||
u16 group_handle,
|
||||
const struct drm_panthor_queue_submit *qsubmit);
|
||||
struct drm_sched_job *panthor_job_get(struct drm_sched_job *job);
|
||||
struct panthor_vm *panthor_job_vm(struct drm_sched_job *sched_job);
|
||||
void panthor_job_put(struct drm_sched_job *job);
|
||||
void panthor_job_update_resvs(struct drm_exec *exec, struct drm_sched_job *job);
|
||||
|
||||
int panthor_group_pool_create(struct panthor_file *pfile);
|
||||
void panthor_group_pool_destroy(struct panthor_file *pfile);
|
||||
|
||||
int panthor_sched_init(struct panthor_device *ptdev);
|
||||
void panthor_sched_unplug(struct panthor_device *ptdev);
|
||||
void panthor_sched_pre_reset(struct panthor_device *ptdev);
|
||||
void panthor_sched_post_reset(struct panthor_device *ptdev);
|
||||
void panthor_sched_suspend(struct panthor_device *ptdev);
|
||||
void panthor_sched_resume(struct panthor_device *ptdev);
|
||||
|
||||
void panthor_sched_report_mmu_fault(struct panthor_device *ptdev);
|
||||
void panthor_sched_report_fw_events(struct panthor_device *ptdev, u32 events);
|
||||
|
||||
#endif
|
||||
|
|
@ -224,8 +224,6 @@ static const struct drm_driver pl111_drm_driver = {
|
|||
.minor = 0,
|
||||
.patchlevel = 0,
|
||||
.dumb_create = drm_gem_dma_dumb_create,
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import_sg_table = pl111_gem_import_sg_table,
|
||||
.gem_prime_mmap = drm_gem_prime_mmap,
|
||||
|
||||
|
|
|
|||
|
|
@ -289,8 +289,6 @@ static struct drm_driver qxl_driver = {
|
|||
#if defined(CONFIG_DEBUG_FS)
|
||||
.debugfs_init = qxl_debugfs_init,
|
||||
#endif
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import_sg_table = qxl_gem_prime_import_sg_table,
|
||||
.fops = &qxl_fops,
|
||||
.ioctls = qxl_ioctls,
|
||||
|
|
|
|||
|
|
@ -168,9 +168,16 @@ int qxl_bo_vmap_locked(struct qxl_bo *bo, struct iosys_map *map)
|
|||
bo->map_count++;
|
||||
goto out;
|
||||
}
|
||||
r = ttm_bo_vmap(&bo->tbo, &bo->map);
|
||||
|
||||
r = __qxl_bo_pin(bo);
|
||||
if (r)
|
||||
return r;
|
||||
|
||||
r = ttm_bo_vmap(&bo->tbo, &bo->map);
|
||||
if (r) {
|
||||
__qxl_bo_unpin(bo);
|
||||
return r;
|
||||
}
|
||||
bo->map_count = 1;
|
||||
|
||||
/* TODO: Remove kptr in favor of map everywhere. */
|
||||
|
|
@ -192,12 +199,6 @@ int qxl_bo_vmap(struct qxl_bo *bo, struct iosys_map *map)
|
|||
if (r)
|
||||
return r;
|
||||
|
||||
r = __qxl_bo_pin(bo);
|
||||
if (r) {
|
||||
qxl_bo_unreserve(bo);
|
||||
return r;
|
||||
}
|
||||
|
||||
r = qxl_bo_vmap_locked(bo, map);
|
||||
qxl_bo_unreserve(bo);
|
||||
return r;
|
||||
|
|
@ -247,6 +248,7 @@ void qxl_bo_vunmap_locked(struct qxl_bo *bo)
|
|||
return;
|
||||
bo->kptr = NULL;
|
||||
ttm_bo_vunmap(&bo->tbo, &bo->map);
|
||||
__qxl_bo_unpin(bo);
|
||||
}
|
||||
|
||||
int qxl_bo_vunmap(struct qxl_bo *bo)
|
||||
|
|
@ -258,7 +260,6 @@ int qxl_bo_vunmap(struct qxl_bo *bo)
|
|||
return r;
|
||||
|
||||
qxl_bo_vunmap_locked(bo);
|
||||
__qxl_bo_unpin(bo);
|
||||
qxl_bo_unreserve(bo);
|
||||
return 0;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -59,7 +59,7 @@ int qxl_gem_prime_vmap(struct drm_gem_object *obj, struct iosys_map *map)
|
|||
struct qxl_bo *bo = gem_to_qxl_bo(obj);
|
||||
int ret;
|
||||
|
||||
ret = qxl_bo_vmap(bo, map);
|
||||
ret = qxl_bo_vmap_locked(bo, map);
|
||||
if (ret < 0)
|
||||
return ret;
|
||||
|
||||
|
|
@ -71,5 +71,5 @@ void qxl_gem_prime_vunmap(struct drm_gem_object *obj,
|
|||
{
|
||||
struct qxl_bo *bo = gem_to_qxl_bo(obj);
|
||||
|
||||
qxl_bo_vunmap(bo);
|
||||
qxl_bo_vunmap_locked(bo);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -605,8 +605,6 @@ static const struct drm_driver kms_driver = {
|
|||
.dumb_map_offset = radeon_mode_dumb_mmap,
|
||||
.fops = &radeon_driver_kms_fops,
|
||||
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import_sg_table = radeon_gem_prime_import_sg_table,
|
||||
.gem_prime_mmap = drm_gem_prime_mmap,
|
||||
|
||||
|
|
|
|||
|
|
@ -583,8 +583,6 @@ DEFINE_DRM_GEM_DMA_FOPS(rcar_du_fops);
|
|||
static const struct drm_driver rcar_du_driver = {
|
||||
.driver_features = DRIVER_GEM | DRIVER_MODESET | DRIVER_ATOMIC,
|
||||
.dumb_create = rcar_du_dumb_create,
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import_sg_table = rcar_du_gem_prime_import_sg_table,
|
||||
.gem_prime_mmap = drm_gem_prime_mmap,
|
||||
.fops = &rcar_du_fops,
|
||||
|
|
|
|||
|
|
@ -2229,8 +2229,6 @@ static struct drm_driver rockchip_drm_driver = {
|
|||
.lastclose = rockchip_drm_lastclose,
|
||||
.open = rockchip_drm_open,
|
||||
.dumb_create = rockchip_gem_dumb_create,
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import = rockchip_drm_gem_prime_import,
|
||||
.gem_prime_import_sg_table = rockchip_gem_prime_import_sg_table,
|
||||
.gem_prime_mmap = drm_gem_prime_mmap,
|
||||
|
|
|
|||
|
|
@ -51,7 +51,7 @@ DECLARE_EVENT_CLASS(drm_sched_job,
|
|||
__assign_str(name, sched_job->sched->name);
|
||||
__entry->job_count = spsc_queue_count(&entity->job_queue);
|
||||
__entry->hw_job_count = atomic_read(
|
||||
&sched_job->sched->hw_rq_count);
|
||||
&sched_job->sched->credit_count);
|
||||
),
|
||||
TP_printk("entity=%p, id=%llu, fence=%p, ring=%s, job count:%u, hw job count:%d",
|
||||
__entry->entity, __entry->id,
|
||||
|
|
|
|||
|
|
@ -72,15 +72,32 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,
|
|||
entity->num_sched_list = num_sched_list;
|
||||
entity->priority = priority;
|
||||
entity->sched_list = num_sched_list > 1 ? sched_list : NULL;
|
||||
entity->last_scheduled = NULL;
|
||||
RCU_INIT_POINTER(entity->last_scheduled, NULL);
|
||||
RB_CLEAR_NODE(&entity->rb_tree_node);
|
||||
|
||||
if(num_sched_list)
|
||||
entity->rq = &sched_list[0]->sched_rq[entity->priority];
|
||||
if (!sched_list[0]->sched_rq) {
|
||||
/* Warn drivers not to do this and to fix their DRM
|
||||
* calling order.
|
||||
*/
|
||||
pr_warn("%s: called with uninitialized scheduler\n", __func__);
|
||||
} else if (num_sched_list) {
|
||||
/* The "priority" of an entity cannot exceed the number of run-queues of a
|
||||
* scheduler. Protect against num_rqs being 0, by converting to signed. Choose
|
||||
* the lowest priority available.
|
||||
*/
|
||||
if (entity->priority >= sched_list[0]->num_rqs) {
|
||||
drm_err(sched_list[0], "entity with out-of-bounds priority:%u num_rqs:%u\n",
|
||||
entity->priority, sched_list[0]->num_rqs);
|
||||
entity->priority = max_t(s32, (s32) sched_list[0]->num_rqs - 1,
|
||||
(s32) DRM_SCHED_PRIORITY_KERNEL);
|
||||
}
|
||||
entity->rq = sched_list[0]->sched_rq[entity->priority];
|
||||
}
|
||||
|
||||
init_completion(&entity->entity_idle);
|
||||
|
||||
/* We start in an idle state. */
|
||||
complete(&entity->entity_idle);
|
||||
complete_all(&entity->entity_idle);
|
||||
|
||||
spin_lock_init(&entity->rq_lock);
|
||||
spsc_queue_init(&entity->job_queue);
|
||||
|
|
@ -141,6 +158,110 @@ bool drm_sched_entity_is_ready(struct drm_sched_entity *entity)
|
|||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* drm_sched_entity_error - return error of last scheduled job
|
||||
* @entity: scheduler entity to check
|
||||
*
|
||||
* Opportunistically return the error of the last scheduled job. Result can
|
||||
* change any time when new jobs are pushed to the hw.
|
||||
*/
|
||||
int drm_sched_entity_error(struct drm_sched_entity *entity)
|
||||
{
|
||||
struct dma_fence *fence;
|
||||
int r;
|
||||
|
||||
rcu_read_lock();
|
||||
fence = rcu_dereference(entity->last_scheduled);
|
||||
r = fence ? fence->error : 0;
|
||||
rcu_read_unlock();
|
||||
|
||||
return r;
|
||||
}
|
||||
EXPORT_SYMBOL(drm_sched_entity_error);
|
||||
|
||||
static void drm_sched_entity_kill_jobs_work(struct work_struct *wrk)
|
||||
{
|
||||
struct drm_sched_job *job = container_of(wrk, typeof(*job), work);
|
||||
|
||||
drm_sched_fence_finished(job->s_fence, -ESRCH);
|
||||
WARN_ON(job->s_fence->parent);
|
||||
job->sched->ops->free_job(job);
|
||||
}
|
||||
|
||||
/* Signal the scheduler finished fence when the entity in question is killed. */
|
||||
static void drm_sched_entity_kill_jobs_cb(struct dma_fence *f,
|
||||
struct dma_fence_cb *cb)
|
||||
{
|
||||
struct drm_sched_job *job = container_of(cb, struct drm_sched_job,
|
||||
finish_cb);
|
||||
unsigned long index;
|
||||
|
||||
dma_fence_put(f);
|
||||
|
||||
/* Wait for all dependencies to avoid data corruptions */
|
||||
xa_for_each(&job->dependencies, index, f) {
|
||||
struct drm_sched_fence *s_fence = to_drm_sched_fence(f);
|
||||
|
||||
if (s_fence && f == &s_fence->scheduled) {
|
||||
/* The dependencies array had a reference on the scheduled
|
||||
* fence, and the finished fence refcount might have
|
||||
* dropped to zero. Use dma_fence_get_rcu() so we get
|
||||
* a NULL fence in that case.
|
||||
*/
|
||||
f = dma_fence_get_rcu(&s_fence->finished);
|
||||
|
||||
/* Now that we have a reference on the finished fence,
|
||||
* we can release the reference the dependencies array
|
||||
* had on the scheduled fence.
|
||||
*/
|
||||
dma_fence_put(&s_fence->scheduled);
|
||||
}
|
||||
|
||||
xa_erase(&job->dependencies, index);
|
||||
if (f && !dma_fence_add_callback(f, &job->finish_cb,
|
||||
drm_sched_entity_kill_jobs_cb))
|
||||
return;
|
||||
|
||||
dma_fence_put(f);
|
||||
}
|
||||
|
||||
INIT_WORK(&job->work, drm_sched_entity_kill_jobs_work);
|
||||
schedule_work(&job->work);
|
||||
}
|
||||
|
||||
/* Remove the entity from the scheduler and kill all pending jobs */
|
||||
static void drm_sched_entity_kill(struct drm_sched_entity *entity)
|
||||
{
|
||||
struct drm_sched_job *job;
|
||||
struct dma_fence *prev;
|
||||
|
||||
if (!entity->rq)
|
||||
return;
|
||||
|
||||
spin_lock(&entity->rq_lock);
|
||||
entity->stopped = true;
|
||||
drm_sched_rq_remove_entity(entity->rq, entity);
|
||||
spin_unlock(&entity->rq_lock);
|
||||
|
||||
/* Make sure this entity is not used by the scheduler at the moment */
|
||||
wait_for_completion(&entity->entity_idle);
|
||||
|
||||
/* The entity is guaranteed to not be used by the scheduler */
|
||||
prev = rcu_dereference_check(entity->last_scheduled, true);
|
||||
dma_fence_get(prev);
|
||||
while ((job = to_drm_sched_job(spsc_queue_pop(&entity->job_queue)))) {
|
||||
struct drm_sched_fence *s_fence = job->s_fence;
|
||||
|
||||
dma_fence_get(&s_fence->finished);
|
||||
if (!prev || dma_fence_add_callback(prev, &job->finish_cb,
|
||||
drm_sched_entity_kill_jobs_cb))
|
||||
drm_sched_entity_kill_jobs_cb(NULL, &job->finish_cb);
|
||||
|
||||
prev = &s_fence->finished;
|
||||
}
|
||||
dma_fence_put(prev);
|
||||
}
|
||||
|
||||
/**
|
||||
* drm_sched_entity_flush - Flush a context entity
|
||||
*
|
||||
|
|
@ -181,91 +302,13 @@ long drm_sched_entity_flush(struct drm_sched_entity *entity, long timeout)
|
|||
/* For killed process disable any more IBs enqueue right now */
|
||||
last_user = cmpxchg(&entity->last_user, current->group_leader, NULL);
|
||||
if ((!last_user || last_user == current->group_leader) &&
|
||||
(current->flags & PF_EXITING) && (current->exit_code == SIGKILL)) {
|
||||
spin_lock(&entity->rq_lock);
|
||||
entity->stopped = true;
|
||||
drm_sched_rq_remove_entity(entity->rq, entity);
|
||||
spin_unlock(&entity->rq_lock);
|
||||
}
|
||||
(current->flags & PF_EXITING) && (current->exit_code == SIGKILL))
|
||||
drm_sched_entity_kill(entity);
|
||||
|
||||
return ret;
|
||||
}
|
||||
EXPORT_SYMBOL(drm_sched_entity_flush);
|
||||
|
||||
static void drm_sched_entity_kill_jobs_work(struct work_struct *wrk)
|
||||
{
|
||||
struct drm_sched_job *job = container_of(wrk, typeof(*job), work);
|
||||
|
||||
drm_sched_fence_finished(job->s_fence);
|
||||
WARN_ON(job->s_fence->parent);
|
||||
job->sched->ops->free_job(job);
|
||||
}
|
||||
|
||||
|
||||
/* Signal the scheduler finished fence when the entity in question is killed. */
|
||||
static void drm_sched_entity_kill_jobs_cb(struct dma_fence *f,
|
||||
struct dma_fence_cb *cb)
|
||||
{
|
||||
struct drm_sched_job *job = container_of(cb, struct drm_sched_job,
|
||||
finish_cb);
|
||||
|
||||
dma_fence_put(f);
|
||||
INIT_WORK(&job->work, drm_sched_entity_kill_jobs_work);
|
||||
schedule_work(&job->work);
|
||||
}
|
||||
|
||||
static struct dma_fence *
|
||||
drm_sched_job_dependency(struct drm_sched_job *job,
|
||||
struct drm_sched_entity *entity)
|
||||
{
|
||||
if (!xa_empty(&job->dependencies))
|
||||
return xa_erase(&job->dependencies, job->last_dependency++);
|
||||
|
||||
if (job->sched->ops->dependency)
|
||||
return job->sched->ops->dependency(job, entity);
|
||||
|
||||
return NULL;
|
||||
}
|
||||
|
||||
static void drm_sched_entity_kill_jobs(struct drm_sched_entity *entity)
|
||||
{
|
||||
struct drm_sched_job *job;
|
||||
struct dma_fence *f;
|
||||
int r;
|
||||
|
||||
while ((job = to_drm_sched_job(spsc_queue_pop(&entity->job_queue)))) {
|
||||
struct drm_sched_fence *s_fence = job->s_fence;
|
||||
|
||||
/* Wait for all dependencies to avoid data corruptions */
|
||||
while ((f = drm_sched_job_dependency(job, entity))) {
|
||||
dma_fence_wait(f, false);
|
||||
dma_fence_put(f);
|
||||
}
|
||||
|
||||
drm_sched_fence_scheduled(s_fence);
|
||||
dma_fence_set_error(&s_fence->finished, -ESRCH);
|
||||
|
||||
/*
|
||||
* When pipe is hanged by older entity, new entity might
|
||||
* not even have chance to submit it's first job to HW
|
||||
* and so entity->last_scheduled will remain NULL
|
||||
*/
|
||||
if (!entity->last_scheduled) {
|
||||
drm_sched_entity_kill_jobs_cb(NULL, &job->finish_cb);
|
||||
continue;
|
||||
}
|
||||
|
||||
dma_fence_get(entity->last_scheduled);
|
||||
r = dma_fence_add_callback(entity->last_scheduled,
|
||||
&job->finish_cb,
|
||||
drm_sched_entity_kill_jobs_cb);
|
||||
if (r == -ENOENT)
|
||||
drm_sched_entity_kill_jobs_cb(NULL, &job->finish_cb);
|
||||
else if (r)
|
||||
DRM_ERROR("fence add callback failed (%d)\n", r);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* drm_sched_entity_fini - Destroy a context entity
|
||||
*
|
||||
|
|
@ -279,37 +322,21 @@ static void drm_sched_entity_kill_jobs(struct drm_sched_entity *entity)
|
|||
*/
|
||||
void drm_sched_entity_fini(struct drm_sched_entity *entity)
|
||||
{
|
||||
struct drm_gpu_scheduler *sched = NULL;
|
||||
|
||||
if (entity->rq) {
|
||||
sched = entity->rq->sched;
|
||||
drm_sched_rq_remove_entity(entity->rq, entity);
|
||||
}
|
||||
|
||||
/* Consumption of existing IBs wasn't completed. Forcefully
|
||||
* remove them here.
|
||||
/*
|
||||
* If consumption of existing IBs wasn't completed. Forcefully remove
|
||||
* them here. Also makes sure that the scheduler won't touch this entity
|
||||
* any more.
|
||||
*/
|
||||
if (spsc_queue_count(&entity->job_queue)) {
|
||||
if (sched) {
|
||||
/*
|
||||
* Wait for thread to idle to make sure it isn't processing
|
||||
* this entity.
|
||||
*/
|
||||
wait_for_completion(&entity->entity_idle);
|
||||
drm_sched_entity_kill(entity);
|
||||
|
||||
}
|
||||
if (entity->dependency) {
|
||||
dma_fence_remove_callback(entity->dependency,
|
||||
&entity->cb);
|
||||
dma_fence_put(entity->dependency);
|
||||
entity->dependency = NULL;
|
||||
}
|
||||
|
||||
drm_sched_entity_kill_jobs(entity);
|
||||
if (entity->dependency) {
|
||||
dma_fence_remove_callback(entity->dependency, &entity->cb);
|
||||
dma_fence_put(entity->dependency);
|
||||
entity->dependency = NULL;
|
||||
}
|
||||
|
||||
dma_fence_put(entity->last_scheduled);
|
||||
entity->last_scheduled = NULL;
|
||||
dma_fence_put(rcu_dereference_check(entity->last_scheduled, true));
|
||||
RCU_INIT_POINTER(entity->last_scheduled, NULL);
|
||||
}
|
||||
EXPORT_SYMBOL(drm_sched_entity_fini);
|
||||
|
||||
|
|
@ -349,7 +376,7 @@ static void drm_sched_entity_wakeup(struct dma_fence *f,
|
|||
container_of(cb, struct drm_sched_entity, cb);
|
||||
|
||||
drm_sched_entity_clear_dep(f, cb);
|
||||
drm_sched_wakeup(entity->rq->sched);
|
||||
drm_sched_wakeup(entity->rq->sched, entity);
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -391,7 +418,7 @@ static bool drm_sched_entity_add_dependency_cb(struct drm_sched_entity *entity)
|
|||
}
|
||||
|
||||
s_fence = to_drm_sched_fence(fence);
|
||||
if (s_fence && s_fence->sched == sched &&
|
||||
if (!fence->error && s_fence && s_fence->sched == sched &&
|
||||
!test_bit(DRM_SCHED_FENCE_DONT_PIPELINE, &fence->flags)) {
|
||||
|
||||
/*
|
||||
|
|
@ -418,6 +445,28 @@ static bool drm_sched_entity_add_dependency_cb(struct drm_sched_entity *entity)
|
|||
return false;
|
||||
}
|
||||
|
||||
static struct dma_fence *
|
||||
drm_sched_job_dependency(struct drm_sched_job *job,
|
||||
struct drm_sched_entity *entity)
|
||||
{
|
||||
struct dma_fence *f;
|
||||
|
||||
/* We keep the fence around, so we can iterate over all dependencies
|
||||
* in drm_sched_entity_kill_jobs_cb() to ensure all deps are signaled
|
||||
* before killing the job.
|
||||
*/
|
||||
f = xa_load(&job->dependencies, job->last_dependency);
|
||||
if (f) {
|
||||
job->last_dependency++;
|
||||
return dma_fence_get(f);
|
||||
}
|
||||
|
||||
if (job->sched->ops->prepare_job)
|
||||
return job->sched->ops->prepare_job(job, entity);
|
||||
|
||||
return NULL;
|
||||
}
|
||||
|
||||
struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
|
||||
{
|
||||
struct drm_sched_job *sched_job;
|
||||
|
|
@ -438,9 +487,9 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
|
|||
if (entity->guilty && atomic_read(entity->guilty))
|
||||
dma_fence_set_error(&sched_job->s_fence->finished, -ECANCELED);
|
||||
|
||||
dma_fence_put(entity->last_scheduled);
|
||||
|
||||
entity->last_scheduled = dma_fence_get(&sched_job->s_fence->finished);
|
||||
dma_fence_put(rcu_dereference_check(entity->last_scheduled, true));
|
||||
rcu_assign_pointer(entity->last_scheduled,
|
||||
dma_fence_get(&sched_job->s_fence->finished));
|
||||
|
||||
/*
|
||||
* If the queue is empty we allow drm_sched_entity_select_rq() to
|
||||
|
|
@ -450,6 +499,25 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
|
|||
smp_wmb();
|
||||
|
||||
spsc_queue_pop(&entity->job_queue);
|
||||
|
||||
/*
|
||||
* Update the entity's location in the min heap according to
|
||||
* the timestamp of the next job, if any.
|
||||
*/
|
||||
if (drm_sched_policy == DRM_SCHED_POLICY_FIFO) {
|
||||
struct drm_sched_job *next;
|
||||
|
||||
next = to_drm_sched_job(spsc_queue_peek(&entity->job_queue));
|
||||
if (next)
|
||||
drm_sched_rq_update_fifo(entity, next->submit_ts);
|
||||
}
|
||||
|
||||
/* Jobs and entities might have different lifecycles. Since we're
|
||||
* removing the job from the entities queue, set the jobs entity pointer
|
||||
* to NULL to prevent any future access of the entity through this job.
|
||||
*/
|
||||
sched_job->entity = NULL;
|
||||
|
||||
return sched_job;
|
||||
}
|
||||
|
||||
|
|
@ -475,7 +543,7 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity)
|
|||
*/
|
||||
smp_rmb();
|
||||
|
||||
fence = entity->last_scheduled;
|
||||
fence = rcu_dereference_check(entity->last_scheduled, true);
|
||||
|
||||
/* stay on the same engine if the previous job hasn't finished */
|
||||
if (fence && !dma_fence_is_signaled(fence))
|
||||
|
|
@ -483,7 +551,7 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity)
|
|||
|
||||
spin_lock(&entity->rq_lock);
|
||||
sched = drm_sched_pick_best(entity->sched_list, entity->num_sched_list);
|
||||
rq = sched ? &sched->sched_rq[entity->priority] : NULL;
|
||||
rq = sched ? sched->sched_rq[entity->priority] : NULL;
|
||||
if (rq != entity->rq) {
|
||||
drm_sched_rq_remove_entity(entity->rq, entity);
|
||||
entity->rq = rq;
|
||||
|
|
@ -509,10 +577,18 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)
|
|||
{
|
||||
struct drm_sched_entity *entity = sched_job->entity;
|
||||
bool first;
|
||||
ktime_t submit_ts;
|
||||
|
||||
trace_drm_sched_job(sched_job, entity);
|
||||
atomic_inc(entity->rq->sched->score);
|
||||
WRITE_ONCE(entity->last_user, current->group_leader);
|
||||
|
||||
/*
|
||||
* After the sched_job is pushed into the entity queue, it may be
|
||||
* completed and freed up at any time. We can no longer access it.
|
||||
* Make sure to set the submit_ts first, to avoid a race.
|
||||
*/
|
||||
sched_job->submit_ts = submit_ts = ktime_get();
|
||||
first = spsc_queue_push(&entity->job_queue, &sched_job->queue_node);
|
||||
|
||||
/* first job wakes up scheduler */
|
||||
|
|
@ -525,9 +601,14 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job)
|
|||
DRM_ERROR("Trying to push to a killed entity\n");
|
||||
return;
|
||||
}
|
||||
|
||||
drm_sched_rq_add_entity(entity->rq, entity);
|
||||
spin_unlock(&entity->rq_lock);
|
||||
drm_sched_wakeup(entity->rq->sched);
|
||||
|
||||
if (drm_sched_policy == DRM_SCHED_POLICY_FIFO)
|
||||
drm_sched_rq_update_fifo(entity, submit_ts);
|
||||
|
||||
drm_sched_wakeup(entity->rq->sched, entity);
|
||||
}
|
||||
}
|
||||
EXPORT_SYMBOL(drm_sched_entity_push_job);
|
||||
|
|
|
|||
|
|
@ -48,13 +48,39 @@ static void __exit drm_sched_fence_slab_fini(void)
|
|||
kmem_cache_destroy(sched_fence_slab);
|
||||
}
|
||||
|
||||
void drm_sched_fence_scheduled(struct drm_sched_fence *fence)
|
||||
static void drm_sched_fence_set_parent(struct drm_sched_fence *s_fence,
|
||||
struct dma_fence *fence)
|
||||
{
|
||||
/*
|
||||
* smp_store_release() to ensure another thread racing us
|
||||
* in drm_sched_fence_set_deadline_finished() sees the
|
||||
* fence's parent set before test_bit()
|
||||
*/
|
||||
smp_store_release(&s_fence->parent, dma_fence_get(fence));
|
||||
if (test_bit(DRM_SCHED_FENCE_FLAG_HAS_DEADLINE_BIT,
|
||||
&s_fence->finished.flags))
|
||||
dma_fence_set_deadline(fence, s_fence->deadline);
|
||||
}
|
||||
|
||||
void drm_sched_fence_scheduled(struct drm_sched_fence *fence,
|
||||
struct dma_fence *parent)
|
||||
{
|
||||
/* Set the parent before signaling the scheduled fence, such that,
|
||||
* any waiter expecting the parent to be filled after the job has
|
||||
* been scheduled (which is the case for drivers delegating waits
|
||||
* to some firmware) doesn't have to busy wait for parent to show
|
||||
* up.
|
||||
*/
|
||||
if (!IS_ERR_OR_NULL(parent))
|
||||
drm_sched_fence_set_parent(fence, parent);
|
||||
|
||||
dma_fence_signal(&fence->scheduled);
|
||||
}
|
||||
|
||||
void drm_sched_fence_finished(struct drm_sched_fence *fence)
|
||||
void drm_sched_fence_finished(struct drm_sched_fence *fence, int result)
|
||||
{
|
||||
if (result)
|
||||
dma_fence_set_error(&fence->finished, result);
|
||||
dma_fence_signal(&fence->finished);
|
||||
}
|
||||
|
||||
|
|
@ -123,6 +149,37 @@ static void drm_sched_fence_release_finished(struct dma_fence *f)
|
|||
dma_fence_put(&fence->scheduled);
|
||||
}
|
||||
|
||||
static void drm_sched_fence_set_deadline_finished(struct dma_fence *f,
|
||||
ktime_t deadline)
|
||||
{
|
||||
struct drm_sched_fence *fence = to_drm_sched_fence(f);
|
||||
struct dma_fence *parent;
|
||||
unsigned long flags;
|
||||
|
||||
spin_lock_irqsave(&fence->lock, flags);
|
||||
|
||||
/* If we already have an earlier deadline, keep it: */
|
||||
if (test_bit(DRM_SCHED_FENCE_FLAG_HAS_DEADLINE_BIT, &f->flags) &&
|
||||
ktime_before(fence->deadline, deadline)) {
|
||||
spin_unlock_irqrestore(&fence->lock, flags);
|
||||
return;
|
||||
}
|
||||
|
||||
fence->deadline = deadline;
|
||||
set_bit(DRM_SCHED_FENCE_FLAG_HAS_DEADLINE_BIT, &f->flags);
|
||||
|
||||
spin_unlock_irqrestore(&fence->lock, flags);
|
||||
|
||||
/*
|
||||
* smp_load_aquire() to ensure that if we are racing another
|
||||
* thread calling drm_sched_fence_set_parent(), that we see
|
||||
* the parent set before it calls test_bit(HAS_DEADLINE_BIT)
|
||||
*/
|
||||
parent = smp_load_acquire(&fence->parent);
|
||||
if (parent)
|
||||
dma_fence_set_deadline(parent, deadline);
|
||||
}
|
||||
|
||||
static const struct dma_fence_ops drm_sched_fence_ops_scheduled = {
|
||||
.get_driver_name = drm_sched_fence_get_driver_name,
|
||||
.get_timeline_name = drm_sched_fence_get_timeline_name,
|
||||
|
|
@ -133,6 +190,7 @@ static const struct dma_fence_ops drm_sched_fence_ops_finished = {
|
|||
.get_driver_name = drm_sched_fence_get_driver_name,
|
||||
.get_timeline_name = drm_sched_fence_get_timeline_name,
|
||||
.release = drm_sched_fence_release_finished,
|
||||
.set_deadline = drm_sched_fence_set_deadline_finished,
|
||||
};
|
||||
|
||||
struct drm_sched_fence *to_drm_sched_fence(struct dma_fence *f)
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -891,8 +891,6 @@ static const struct drm_driver tegra_drm_driver = {
|
|||
.debugfs_init = tegra_debugfs_init,
|
||||
#endif
|
||||
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import = tegra_gem_prime_import,
|
||||
|
||||
.dumb_create = tegra_bo_dumb_create,
|
||||
|
|
|
|||
|
|
@ -172,8 +172,6 @@ static const struct drm_driver v3d_drm_driver = {
|
|||
#endif
|
||||
|
||||
.gem_create_object = v3d_create_object,
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import_sg_table = v3d_prime_import_sg_table,
|
||||
.gem_prime_mmap = drm_gem_prime_mmap,
|
||||
|
||||
|
|
|
|||
|
|
@ -473,7 +473,7 @@ v3d_job_init(struct v3d_dev *v3d, struct drm_file *file_priv,
|
|||
job->free = free;
|
||||
|
||||
ret = drm_sched_job_init(&job->base, &v3d_priv->sched_entity[queue],
|
||||
v3d_priv);
|
||||
1, v3d_priv);
|
||||
if (ret)
|
||||
goto fail;
|
||||
|
||||
|
|
|
|||
|
|
@ -388,7 +388,8 @@ v3d_sched_init(struct v3d_dev *v3d)
|
|||
int ret;
|
||||
|
||||
ret = drm_sched_init(&v3d->queue[V3D_BIN].sched,
|
||||
&v3d_bin_sched_ops,
|
||||
&v3d_bin_sched_ops, NULL,
|
||||
DRM_SCHED_PRIORITY_COUNT,
|
||||
hw_jobs_limit, job_hang_limit,
|
||||
msecs_to_jiffies(hang_limit_ms), NULL,
|
||||
NULL, "v3d_bin", v3d->drm.dev);
|
||||
|
|
@ -396,7 +397,8 @@ v3d_sched_init(struct v3d_dev *v3d)
|
|||
return ret;
|
||||
|
||||
ret = drm_sched_init(&v3d->queue[V3D_RENDER].sched,
|
||||
&v3d_render_sched_ops,
|
||||
&v3d_render_sched_ops, NULL,
|
||||
DRM_SCHED_PRIORITY_COUNT,
|
||||
hw_jobs_limit, job_hang_limit,
|
||||
msecs_to_jiffies(hang_limit_ms), NULL,
|
||||
NULL, "v3d_render", v3d->drm.dev);
|
||||
|
|
@ -404,7 +406,8 @@ v3d_sched_init(struct v3d_dev *v3d)
|
|||
goto fail;
|
||||
|
||||
ret = drm_sched_init(&v3d->queue[V3D_TFU].sched,
|
||||
&v3d_tfu_sched_ops,
|
||||
&v3d_tfu_sched_ops, NULL,
|
||||
DRM_SCHED_PRIORITY_COUNT,
|
||||
hw_jobs_limit, job_hang_limit,
|
||||
msecs_to_jiffies(hang_limit_ms), NULL,
|
||||
NULL, "v3d_tfu", v3d->drm.dev);
|
||||
|
|
@ -413,7 +416,8 @@ v3d_sched_init(struct v3d_dev *v3d)
|
|||
|
||||
if (v3d_has_csd(v3d)) {
|
||||
ret = drm_sched_init(&v3d->queue[V3D_CSD].sched,
|
||||
&v3d_csd_sched_ops,
|
||||
&v3d_csd_sched_ops, NULL,
|
||||
DRM_SCHED_PRIORITY_COUNT,
|
||||
hw_jobs_limit, job_hang_limit,
|
||||
msecs_to_jiffies(hang_limit_ms), NULL,
|
||||
NULL, "v3d_csd", v3d->drm.dev);
|
||||
|
|
@ -421,7 +425,8 @@ v3d_sched_init(struct v3d_dev *v3d)
|
|||
goto fail;
|
||||
|
||||
ret = drm_sched_init(&v3d->queue[V3D_CACHE_CLEAN].sched,
|
||||
&v3d_cache_clean_sched_ops,
|
||||
&v3d_cache_clean_sched_ops, NULL,
|
||||
DRM_SCHED_PRIORITY_COUNT,
|
||||
hw_jobs_limit, job_hang_limit,
|
||||
msecs_to_jiffies(hang_limit_ms), NULL,
|
||||
NULL, "v3d_cache_clean", v3d->drm.dev);
|
||||
|
|
|
|||
|
|
@ -182,8 +182,6 @@ static const struct drm_driver driver = {
|
|||
#if defined(CONFIG_DEBUG_FS)
|
||||
.debugfs_init = virtio_gpu_debugfs_init,
|
||||
#endif
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_mmap = drm_gem_prime_mmap,
|
||||
.gem_prime_import = virtgpu_gem_prime_import,
|
||||
.gem_prime_import_sg_table = virtgpu_gem_prime_import_sg_table,
|
||||
|
|
|
|||
|
|
@ -474,8 +474,6 @@ DEFINE_DRM_GEM_FOPS(xen_drm_dev_fops);
|
|||
static const struct drm_driver xen_drm_driver = {
|
||||
.driver_features = DRIVER_GEM | DRIVER_MODESET | DRIVER_ATOMIC,
|
||||
.release = xen_drm_drv_release,
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
|
||||
.gem_prime_import_sg_table = xen_drm_front_gem_import_sg_table,
|
||||
.gem_prime_mmap = drm_gem_prime_mmap,
|
||||
.dumb_create = xen_drm_drv_dumb_create,
|
||||
|
|
|
|||
|
|
@ -188,20 +188,28 @@ static dma_addr_t __arm_lpae_dma_addr(void *pages)
|
|||
}
|
||||
|
||||
static void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp,
|
||||
struct io_pgtable_cfg *cfg)
|
||||
struct io_pgtable_cfg *cfg,
|
||||
void *cookie)
|
||||
{
|
||||
struct device *dev = cfg->iommu_dev;
|
||||
int order = get_order(size);
|
||||
struct page *p;
|
||||
dma_addr_t dma;
|
||||
void *pages;
|
||||
|
||||
VM_BUG_ON((gfp & __GFP_HIGHMEM));
|
||||
p = alloc_pages_node(dev_to_node(dev), gfp | __GFP_ZERO, order);
|
||||
if (!p)
|
||||
|
||||
if (cfg->alloc) {
|
||||
pages = cfg->alloc(cookie, size, gfp);
|
||||
} else {
|
||||
struct page *p;
|
||||
|
||||
p = alloc_pages_node(dev_to_node(dev), gfp | __GFP_ZERO, order);
|
||||
pages = p ? page_address(p) : NULL;
|
||||
}
|
||||
|
||||
if (!pages)
|
||||
return NULL;
|
||||
|
||||
pages = page_address(p);
|
||||
if (!cfg->coherent_walk) {
|
||||
dma = dma_map_single(dev, pages, size, DMA_TO_DEVICE);
|
||||
if (dma_mapping_error(dev, dma))
|
||||
|
|
@ -220,18 +228,28 @@ static void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp,
|
|||
out_unmap:
|
||||
dev_err(dev, "Cannot accommodate DMA translation for IOMMU page tables\n");
|
||||
dma_unmap_single(dev, dma, size, DMA_TO_DEVICE);
|
||||
|
||||
out_free:
|
||||
__free_pages(p, order);
|
||||
if (cfg->free)
|
||||
cfg->free(cookie, pages, size);
|
||||
else
|
||||
free_pages((unsigned long)pages, order);
|
||||
|
||||
return NULL;
|
||||
}
|
||||
|
||||
static void __arm_lpae_free_pages(void *pages, size_t size,
|
||||
struct io_pgtable_cfg *cfg)
|
||||
struct io_pgtable_cfg *cfg,
|
||||
void *cookie)
|
||||
{
|
||||
if (!cfg->coherent_walk)
|
||||
dma_unmap_single(cfg->iommu_dev, __arm_lpae_dma_addr(pages),
|
||||
size, DMA_TO_DEVICE);
|
||||
free_pages((unsigned long)pages, get_order(size));
|
||||
|
||||
if (cfg->free)
|
||||
cfg->free(cookie, pages, size);
|
||||
else
|
||||
free_pages((unsigned long)pages, get_order(size));
|
||||
}
|
||||
|
||||
static void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries,
|
||||
|
|
@ -373,13 +391,13 @@ static int __arm_lpae_map(struct arm_lpae_io_pgtable *data, unsigned long iova,
|
|||
/* Grab a pointer to the next level */
|
||||
pte = READ_ONCE(*ptep);
|
||||
if (!pte) {
|
||||
cptep = __arm_lpae_alloc_pages(tblsz, gfp, cfg);
|
||||
cptep = __arm_lpae_alloc_pages(tblsz, gfp, cfg, data->iop.cookie);
|
||||
if (!cptep)
|
||||
return -ENOMEM;
|
||||
|
||||
pte = arm_lpae_install_table(cptep, ptep, 0, data);
|
||||
if (pte)
|
||||
__arm_lpae_free_pages(cptep, tblsz, cfg);
|
||||
__arm_lpae_free_pages(cptep, tblsz, cfg, data->iop.cookie);
|
||||
} else if (!cfg->coherent_walk && !(pte & ARM_LPAE_PTE_SW_SYNC)) {
|
||||
__arm_lpae_sync_pte(ptep, 1, cfg);
|
||||
}
|
||||
|
|
@ -530,7 +548,7 @@ static void __arm_lpae_free_pgtable(struct arm_lpae_io_pgtable *data, int lvl,
|
|||
__arm_lpae_free_pgtable(data, lvl + 1, iopte_deref(pte, data));
|
||||
}
|
||||
|
||||
__arm_lpae_free_pages(start, table_size, &data->iop.cfg);
|
||||
__arm_lpae_free_pages(start, table_size, &data->iop.cfg, data->iop.cookie);
|
||||
}
|
||||
|
||||
static void arm_lpae_free_pgtable(struct io_pgtable *iop)
|
||||
|
|
@ -558,7 +576,7 @@ static size_t arm_lpae_split_blk_unmap(struct arm_lpae_io_pgtable *data,
|
|||
if (WARN_ON(lvl == ARM_LPAE_MAX_LEVELS))
|
||||
return 0;
|
||||
|
||||
tablep = __arm_lpae_alloc_pages(tablesz, GFP_ATOMIC, cfg);
|
||||
tablep = __arm_lpae_alloc_pages(tablesz, GFP_ATOMIC, cfg, data->iop.cookie);
|
||||
if (!tablep)
|
||||
return 0; /* Bytes unmapped */
|
||||
|
||||
|
|
@ -581,7 +599,7 @@ static size_t arm_lpae_split_blk_unmap(struct arm_lpae_io_pgtable *data,
|
|||
|
||||
pte = arm_lpae_install_table(tablep, ptep, blk_pte, data);
|
||||
if (pte != blk_pte) {
|
||||
__arm_lpae_free_pages(tablep, tablesz, cfg);
|
||||
__arm_lpae_free_pages(tablep, tablesz, cfg, data->iop.cookie);
|
||||
/*
|
||||
* We may race against someone unmapping another part of this
|
||||
* block, but anything else is invalid. We can't misinterpret
|
||||
|
|
@ -896,7 +914,7 @@ arm_64_lpae_alloc_pgtable_s1(struct io_pgtable_cfg *cfg, void *cookie)
|
|||
|
||||
/* Looking good; allocate a pgd */
|
||||
data->pgd = __arm_lpae_alloc_pages(ARM_LPAE_PGD_SIZE(data),
|
||||
GFP_KERNEL, cfg);
|
||||
GFP_KERNEL, cfg, cookie);
|
||||
if (!data->pgd)
|
||||
goto out_free_data;
|
||||
|
||||
|
|
@ -998,7 +1016,7 @@ arm_64_lpae_alloc_pgtable_s2(struct io_pgtable_cfg *cfg, void *cookie)
|
|||
|
||||
/* Allocate pgd pages */
|
||||
data->pgd = __arm_lpae_alloc_pages(ARM_LPAE_PGD_SIZE(data),
|
||||
GFP_KERNEL, cfg);
|
||||
GFP_KERNEL, cfg, cookie);
|
||||
if (!data->pgd)
|
||||
goto out_free_data;
|
||||
|
||||
|
|
@ -1073,7 +1091,7 @@ arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie)
|
|||
<< ARM_LPAE_MAIR_ATTR_SHIFT(ARM_LPAE_MAIR_ATTR_IDX_DEV));
|
||||
|
||||
data->pgd = __arm_lpae_alloc_pages(ARM_LPAE_PGD_SIZE(data), GFP_KERNEL,
|
||||
cfg);
|
||||
cfg, cookie);
|
||||
if (!data->pgd)
|
||||
goto out_free_data;
|
||||
|
||||
|
|
@ -1094,26 +1112,31 @@ out_free_data:
|
|||
}
|
||||
|
||||
struct io_pgtable_init_fns io_pgtable_arm_64_lpae_s1_init_fns = {
|
||||
.caps = IO_PGTABLE_CAP_CUSTOM_ALLOCATOR,
|
||||
.alloc = arm_64_lpae_alloc_pgtable_s1,
|
||||
.free = arm_lpae_free_pgtable,
|
||||
};
|
||||
|
||||
struct io_pgtable_init_fns io_pgtable_arm_64_lpae_s2_init_fns = {
|
||||
.caps = IO_PGTABLE_CAP_CUSTOM_ALLOCATOR,
|
||||
.alloc = arm_64_lpae_alloc_pgtable_s2,
|
||||
.free = arm_lpae_free_pgtable,
|
||||
};
|
||||
|
||||
struct io_pgtable_init_fns io_pgtable_arm_32_lpae_s1_init_fns = {
|
||||
.caps = IO_PGTABLE_CAP_CUSTOM_ALLOCATOR,
|
||||
.alloc = arm_32_lpae_alloc_pgtable_s1,
|
||||
.free = arm_lpae_free_pgtable,
|
||||
};
|
||||
|
||||
struct io_pgtable_init_fns io_pgtable_arm_32_lpae_s2_init_fns = {
|
||||
.caps = IO_PGTABLE_CAP_CUSTOM_ALLOCATOR,
|
||||
.alloc = arm_32_lpae_alloc_pgtable_s2,
|
||||
.free = arm_lpae_free_pgtable,
|
||||
};
|
||||
|
||||
struct io_pgtable_init_fns io_pgtable_arm_mali_lpae_init_fns = {
|
||||
.caps = IO_PGTABLE_CAP_CUSTOM_ALLOCATOR,
|
||||
.alloc = arm_mali_lpae_alloc_pgtable,
|
||||
.free = arm_lpae_free_pgtable,
|
||||
};
|
||||
|
|
|
|||
|
|
@ -34,6 +34,26 @@ io_pgtable_init_table[IO_PGTABLE_NUM_FMTS] = {
|
|||
#endif
|
||||
};
|
||||
|
||||
static int check_custom_allocator(enum io_pgtable_fmt fmt,
|
||||
struct io_pgtable_cfg *cfg)
|
||||
{
|
||||
/* No custom allocator, no need to check the format. */
|
||||
if (!cfg->alloc && !cfg->free)
|
||||
return 0;
|
||||
|
||||
/* When passing a custom allocator, both the alloc and free
|
||||
* functions should be provided.
|
||||
*/
|
||||
if (!cfg->alloc || !cfg->free)
|
||||
return -EINVAL;
|
||||
|
||||
/* Make sure the format supports custom allocators. */
|
||||
if (io_pgtable_init_table[fmt]->caps & IO_PGTABLE_CAP_CUSTOM_ALLOCATOR)
|
||||
return 0;
|
||||
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
struct io_pgtable_ops *alloc_io_pgtable_ops(enum io_pgtable_fmt fmt,
|
||||
struct io_pgtable_cfg *cfg,
|
||||
void *cookie)
|
||||
|
|
@ -44,6 +64,9 @@ struct io_pgtable_ops *alloc_io_pgtable_ops(enum io_pgtable_fmt fmt,
|
|||
if (fmt >= IO_PGTABLE_NUM_FMTS)
|
||||
return NULL;
|
||||
|
||||
if (check_custom_allocator(fmt, cfg))
|
||||
return NULL;
|
||||
|
||||
fns = io_pgtable_init_table[fmt];
|
||||
if (!fns)
|
||||
return NULL;
|
||||
|
|
|
|||
|
|
@ -94,6 +94,12 @@ enum drm_driver_feature {
|
|||
* synchronization of command submission.
|
||||
*/
|
||||
DRIVER_SYNCOBJ_TIMELINE = BIT(6),
|
||||
/**
|
||||
* @DRIVER_GEM_GPUVA:
|
||||
*
|
||||
* Driver supports user defined GPU VA bindings for GEM objects.
|
||||
*/
|
||||
DRIVER_GEM_GPUVA = BIT(8),
|
||||
|
||||
/* IMPORTANT: Below are all the legacy flags, add new ones above. */
|
||||
|
||||
|
|
@ -301,22 +307,14 @@ struct drm_driver {
|
|||
/**
|
||||
* @prime_handle_to_fd:
|
||||
*
|
||||
* Main PRIME export function. Should be implemented with
|
||||
* drm_gem_prime_handle_to_fd() for GEM based drivers.
|
||||
*
|
||||
* For an in-depth discussion see :ref:`PRIME buffer sharing
|
||||
* documentation <prime_buffer_sharing>`.
|
||||
* PRIME export function. Only used by vmwgfx.
|
||||
*/
|
||||
int (*prime_handle_to_fd)(struct drm_device *dev, struct drm_file *file_priv,
|
||||
uint32_t handle, uint32_t flags, int *prime_fd);
|
||||
/**
|
||||
* @prime_fd_to_handle:
|
||||
*
|
||||
* Main PRIME import function. Should be implemented with
|
||||
* drm_gem_prime_fd_to_handle() for GEM based drivers.
|
||||
*
|
||||
* For an in-depth discussion see :ref:`PRIME buffer sharing
|
||||
* documentation <prime_buffer_sharing>`.
|
||||
* PRIME import function. Only used by vmwgfx.
|
||||
*/
|
||||
int (*prime_fd_to_handle)(struct drm_device *dev, struct drm_file *file_priv,
|
||||
int prime_fd, uint32_t *handle);
|
||||
|
|
|
|||
150
include/drm/drm_exec.h
Normal file
150
include/drm/drm_exec.h
Normal file
|
|
@ -0,0 +1,150 @@
|
|||
/* SPDX-License-Identifier: GPL-2.0 OR MIT */
|
||||
|
||||
#ifndef __DRM_EXEC_H__
|
||||
#define __DRM_EXEC_H__
|
||||
|
||||
#include <linux/ww_mutex.h>
|
||||
|
||||
#define DRM_EXEC_INTERRUPTIBLE_WAIT BIT(0)
|
||||
#define DRM_EXEC_IGNORE_DUPLICATES BIT(1)
|
||||
|
||||
struct drm_gem_object;
|
||||
|
||||
/**
|
||||
* struct drm_exec - Execution context
|
||||
*/
|
||||
struct drm_exec {
|
||||
/**
|
||||
* @flags: Flags to control locking behavior
|
||||
*/
|
||||
uint32_t flags;
|
||||
|
||||
/**
|
||||
* @ticket: WW ticket used for acquiring locks
|
||||
*/
|
||||
struct ww_acquire_ctx ticket;
|
||||
|
||||
/**
|
||||
* @num_objects: number of objects locked
|
||||
*/
|
||||
unsigned int num_objects;
|
||||
|
||||
/**
|
||||
* @max_objects: maximum objects in array
|
||||
*/
|
||||
unsigned int max_objects;
|
||||
|
||||
/**
|
||||
* @objects: array of the locked objects
|
||||
*/
|
||||
struct drm_gem_object **objects;
|
||||
|
||||
/**
|
||||
* @contended: contended GEM object we backed off for
|
||||
*/
|
||||
struct drm_gem_object *contended;
|
||||
|
||||
/**
|
||||
* @prelocked: already locked GEM object due to contention
|
||||
*/
|
||||
struct drm_gem_object *prelocked;
|
||||
};
|
||||
|
||||
/**
|
||||
* drm_exec_obj() - Return the object for a give drm_exec index
|
||||
* @exec: Pointer to the drm_exec context
|
||||
* @index: The index.
|
||||
*
|
||||
* Return: Pointer to the locked object corresponding to @index if
|
||||
* index is within the number of locked objects. NULL otherwise.
|
||||
*/
|
||||
static inline struct drm_gem_object *
|
||||
drm_exec_obj(struct drm_exec *exec, unsigned long index)
|
||||
{
|
||||
return index < exec->num_objects ? exec->objects[index] : NULL;
|
||||
}
|
||||
|
||||
/**
|
||||
* drm_exec_for_each_locked_object - iterate over all the locked objects
|
||||
* @exec: drm_exec object
|
||||
* @index: unsigned long index for the iteration
|
||||
* @obj: the current GEM object
|
||||
*
|
||||
* Iterate over all the locked GEM objects inside the drm_exec object.
|
||||
*/
|
||||
#define drm_exec_for_each_locked_object(exec, index, obj) \
|
||||
for ((index) = 0; ((obj) = drm_exec_obj(exec, index)); ++(index))
|
||||
|
||||
/**
|
||||
* drm_exec_for_each_locked_object_reverse - iterate over all the locked
|
||||
* objects in reverse locking order
|
||||
* @exec: drm_exec object
|
||||
* @index: unsigned long index for the iteration
|
||||
* @obj: the current GEM object
|
||||
*
|
||||
* Iterate over all the locked GEM objects inside the drm_exec object in
|
||||
* reverse locking order. Note that @index may go below zero and wrap,
|
||||
* but that will be caught by drm_exec_obj(), returning a NULL object.
|
||||
*/
|
||||
#define drm_exec_for_each_locked_object_reverse(exec, index, obj) \
|
||||
for ((index) = (exec)->num_objects - 1; \
|
||||
((obj) = drm_exec_obj(exec, index)); --(index))
|
||||
|
||||
/**
|
||||
* drm_exec_until_all_locked - loop until all GEM objects are locked
|
||||
* @exec: drm_exec object
|
||||
*
|
||||
* Core functionality of the drm_exec object. Loops until all GEM objects are
|
||||
* locked and no more contention exists. At the beginning of the loop it is
|
||||
* guaranteed that no GEM object is locked.
|
||||
*
|
||||
* Since labels can't be defined local to the loops body we use a jump pointer
|
||||
* to make sure that the retry is only used from within the loops body.
|
||||
*/
|
||||
#define drm_exec_until_all_locked(exec) \
|
||||
for (void *__drm_exec_retry_ptr; ({ \
|
||||
__label__ __drm_exec_retry; \
|
||||
__drm_exec_retry: \
|
||||
__drm_exec_retry_ptr = &&__drm_exec_retry; \
|
||||
(void)__drm_exec_retry_ptr; \
|
||||
drm_exec_cleanup(exec); \
|
||||
});)
|
||||
|
||||
/**
|
||||
* drm_exec_retry_on_contention - restart the loop to grap all locks
|
||||
* @exec: drm_exec object
|
||||
*
|
||||
* Control flow helper to continue when a contention was detected and we need to
|
||||
* clean up and re-start the loop to prepare all GEM objects.
|
||||
*/
|
||||
#define drm_exec_retry_on_contention(exec) \
|
||||
do { \
|
||||
if (unlikely(drm_exec_is_contended(exec))) \
|
||||
goto *__drm_exec_retry_ptr; \
|
||||
} while (0)
|
||||
|
||||
/**
|
||||
* drm_exec_is_contended - check for contention
|
||||
* @exec: drm_exec object
|
||||
*
|
||||
* Returns true if the drm_exec object has run into some contention while
|
||||
* locking a GEM object and needs to clean up.
|
||||
*/
|
||||
static inline bool drm_exec_is_contended(struct drm_exec *exec)
|
||||
{
|
||||
return !!exec->contended;
|
||||
}
|
||||
|
||||
void drm_exec_init(struct drm_exec *exec, uint32_t flags, unsigned nr);
|
||||
void drm_exec_fini(struct drm_exec *exec);
|
||||
bool drm_exec_cleanup(struct drm_exec *exec);
|
||||
int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj);
|
||||
void drm_exec_unlock_obj(struct drm_exec *exec, struct drm_gem_object *obj);
|
||||
int drm_exec_prepare_obj(struct drm_exec *exec, struct drm_gem_object *obj,
|
||||
unsigned int num_fences);
|
||||
int drm_exec_prepare_array(struct drm_exec *exec,
|
||||
struct drm_gem_object **objects,
|
||||
unsigned int num_objects,
|
||||
unsigned int num_fences);
|
||||
|
||||
#endif
|
||||
|
|
@ -36,6 +36,8 @@
|
|||
|
||||
#include <linux/kref.h>
|
||||
#include <linux/dma-resv.h>
|
||||
#include <linux/list.h>
|
||||
#include <linux/mutex.h>
|
||||
|
||||
#include <drm/drm_vma_manager.h>
|
||||
|
||||
|
|
@ -337,6 +339,22 @@ struct drm_gem_object {
|
|||
*/
|
||||
struct dma_resv _resv;
|
||||
|
||||
/**
|
||||
* @gpuva:
|
||||
*
|
||||
* Provides the list of GPU VAs attached to this GEM object.
|
||||
*
|
||||
* Drivers should lock list accesses with the GEMs &dma_resv lock
|
||||
* (&drm_gem_object.resv) or a custom lock if one is provided.
|
||||
*/
|
||||
struct {
|
||||
struct list_head list;
|
||||
|
||||
#ifdef CONFIG_LOCKDEP
|
||||
struct lockdep_map *lock_dep_map;
|
||||
#endif
|
||||
} gpuva;
|
||||
|
||||
/**
|
||||
* @funcs:
|
||||
*
|
||||
|
|
@ -405,6 +423,7 @@ int drm_gem_object_init(struct drm_device *dev,
|
|||
struct drm_gem_object *obj, size_t size);
|
||||
void drm_gem_private_object_init(struct drm_device *dev,
|
||||
struct drm_gem_object *obj, size_t size);
|
||||
void drm_gem_private_object_fini(struct drm_gem_object *obj);
|
||||
void drm_gem_vm_open(struct vm_area_struct *vma);
|
||||
void drm_gem_vm_close(struct vm_area_struct *vma);
|
||||
int drm_gem_mmap_obj(struct drm_gem_object *obj, unsigned long obj_size,
|
||||
|
|
@ -457,6 +476,9 @@ struct page **drm_gem_get_pages(struct drm_gem_object *obj);
|
|||
void drm_gem_put_pages(struct drm_gem_object *obj, struct page **pages,
|
||||
bool dirty, bool accessed);
|
||||
|
||||
int drm_gem_vmap_unlocked(struct drm_gem_object *obj, struct iosys_map *map);
|
||||
void drm_gem_vunmap_unlocked(struct drm_gem_object *obj, struct iosys_map *map);
|
||||
|
||||
int drm_gem_objects_lookup(struct drm_file *filp, void __user *bo_handles,
|
||||
int count, struct drm_gem_object ***objs_out);
|
||||
struct drm_gem_object *drm_gem_object_lookup(struct drm_file *filp, u32 handle);
|
||||
|
|
@ -477,4 +499,65 @@ unsigned long drm_gem_lru_scan(struct drm_gem_lru *lru,
|
|||
unsigned long *remaining,
|
||||
bool (*shrink)(struct drm_gem_object *obj));
|
||||
|
||||
#ifdef CONFIG_LOCKDEP
|
||||
/**
|
||||
* drm_gem_gpuva_set_lock() - Set the lock protecting accesses to the gpuva list.
|
||||
* @obj: the &drm_gem_object
|
||||
* @lock: the lock used to protect the gpuva list. The locking primitive
|
||||
* must contain a dep_map field.
|
||||
*
|
||||
* Call this if you're not proctecting access to the gpuva list
|
||||
* with the dma-resv lock, otherwise, drm_gem_gpuva_init() takes care
|
||||
* of initializing lock_dep_map for you.
|
||||
*/
|
||||
#define drm_gem_gpuva_set_lock(obj, lock) \
|
||||
if (!(obj)->gpuva.lock_dep_map) \
|
||||
(obj)->gpuva.lock_dep_map = &(lock)->dep_map
|
||||
#define drm_gem_gpuva_assert_lock_held(obj) \
|
||||
lockdep_assert(lock_is_held((obj)->gpuva.lock_dep_map))
|
||||
#else
|
||||
#define drm_gem_gpuva_set_lock(obj, lock) do {} while (0)
|
||||
#define drm_gem_gpuva_assert_lock_held(obj) do {} while (0)
|
||||
#endif
|
||||
|
||||
/**
|
||||
* drm_gem_gpuva_init() - initialize the gpuva list of a GEM object
|
||||
* @obj: the &drm_gem_object
|
||||
*
|
||||
* This initializes the &drm_gem_object's &drm_gpuvm_bo list.
|
||||
*
|
||||
* Calling this function is only necessary for drivers intending to support the
|
||||
* &drm_driver_feature DRIVER_GEM_GPUVA.
|
||||
*/
|
||||
static inline void drm_gem_gpuva_init(struct drm_gem_object *obj)
|
||||
{
|
||||
INIT_LIST_HEAD(&obj->gpuva.list);
|
||||
drm_gem_gpuva_set_lock(obj, &obj->resv->lock.base);
|
||||
}
|
||||
|
||||
/**
|
||||
* drm_gem_for_each_gpuvm_bo() - iterator to walk over a list of &drm_gpuvm_bo
|
||||
* @entry__: &drm_gpuvm_bo structure to assign to in each iteration step
|
||||
* @obj__: the &drm_gem_object the &drm_gpuvm_bo to walk are associated with
|
||||
*
|
||||
* This iterator walks over all &drm_gpuvm_bo structures associated with the
|
||||
* &drm_gem_object.
|
||||
*/
|
||||
#define drm_gem_for_each_gpuvm_bo(entry__, obj__) \
|
||||
list_for_each_entry(entry__, &(obj__)->gpuva.list, list.entry.gem)
|
||||
|
||||
/**
|
||||
* drm_gem_for_each_gpuvm_bo_safe() - iterator to safely walk over a list of
|
||||
* &drm_gpuvm_bo
|
||||
* @entry__: &drm_gpuvm_bostructure to assign to in each iteration step
|
||||
* @next__: &next &drm_gpuvm_bo to store the next step
|
||||
* @obj__: the &drm_gem_object the &drm_gpuvm_bo to walk are associated with
|
||||
*
|
||||
* This iterator walks over all &drm_gpuvm_bo structures associated with the
|
||||
* &drm_gem_object. It is implemented with list_for_each_entry_safe(), hence
|
||||
* it is save against removal of elements.
|
||||
*/
|
||||
#define drm_gem_for_each_gpuvm_bo_safe(entry__, next__, obj__) \
|
||||
list_for_each_entry_safe(entry__, next__, &(obj__)->gpuva.list, list.entry.gem)
|
||||
|
||||
#endif /* __DRM_GEM_H__ */
|
||||
|
|
|
|||
|
|
@ -167,8 +167,6 @@ drm_gem_dma_prime_import_sg_table(struct drm_device *dev,
|
|||
*/
|
||||
#define DRM_GEM_DMA_DRIVER_OPS_WITH_DUMB_CREATE(dumb_create_func) \
|
||||
.dumb_create = (dumb_create_func), \
|
||||
.prime_handle_to_fd = drm_gem_prime_handle_to_fd, \
|
||||
.prime_fd_to_handle = drm_gem_prime_fd_to_handle, \
|
||||
.gem_prime_import_sg_table = drm_gem_dma_prime_import_sg_table, \
|
||||
.gem_prime_mmap = drm_gem_prime_mmap
|
||||
|
||||
|
|
|
|||
|
|
@ -26,11 +26,6 @@ struct drm_gem_shmem_object {
|
|||
*/
|
||||
struct drm_gem_object base;
|
||||
|
||||
/**
|
||||
* @pages_lock: Protects the page table and use count
|
||||
*/
|
||||
struct mutex pages_lock;
|
||||
|
||||
/**
|
||||
* @pages: Page table
|
||||
*/
|
||||
|
|
@ -79,11 +74,6 @@ struct drm_gem_shmem_object {
|
|||
*/
|
||||
struct sg_table *sgt;
|
||||
|
||||
/**
|
||||
* @vmap_lock: Protects the vmap address and use count
|
||||
*/
|
||||
struct mutex vmap_lock;
|
||||
|
||||
/**
|
||||
* @vaddr: Kernel virtual address of the backing memory
|
||||
*/
|
||||
|
|
@ -109,7 +99,6 @@ struct drm_gem_shmem_object {
|
|||
struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t size);
|
||||
void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem);
|
||||
|
||||
int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem);
|
||||
void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem);
|
||||
int drm_gem_shmem_pin(struct drm_gem_shmem_object *shmem);
|
||||
void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem);
|
||||
|
|
@ -128,8 +117,7 @@ static inline bool drm_gem_shmem_is_purgeable(struct drm_gem_shmem_object *shmem
|
|||
!shmem->base.dma_buf && !shmem->base.import_attach;
|
||||
}
|
||||
|
||||
void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem);
|
||||
bool drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem);
|
||||
void drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem);
|
||||
|
||||
struct sg_table *drm_gem_shmem_get_sg_table(struct drm_gem_shmem_object *shmem);
|
||||
struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem);
|
||||
|
|
|
|||
|
|
@ -159,8 +159,7 @@ void drm_gem_vram_simple_display_pipe_cleanup_fb(
|
|||
#define DRM_GEM_VRAM_DRIVER \
|
||||
.debugfs_init = drm_vram_mm_debugfs_init, \
|
||||
.dumb_create = drm_gem_vram_driver_dumb_create, \
|
||||
.dumb_map_offset = drm_gem_ttm_dumb_map_offset, \
|
||||
.gem_prime_mmap = drm_gem_prime_mmap
|
||||
.dumb_map_offset = drm_gem_ttm_dumb_map_offset
|
||||
|
||||
/*
|
||||
* VRAM memory manager
|
||||
|
|
|
|||
1227
include/drm/drm_gpuvm.h
Normal file
1227
include/drm/drm_gpuvm.h
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -41,23 +41,42 @@
|
|||
*/
|
||||
#define DRM_SCHED_FENCE_DONT_PIPELINE DMA_FENCE_FLAG_USER_BITS
|
||||
|
||||
/**
|
||||
* DRM_SCHED_FENCE_FLAG_HAS_DEADLINE_BIT - A fence deadline hint has been set
|
||||
*
|
||||
* Because we could have a deadline hint can be set before the backing hw
|
||||
* fence is created, we need to keep track of whether a deadline has already
|
||||
* been set.
|
||||
*/
|
||||
#define DRM_SCHED_FENCE_FLAG_HAS_DEADLINE_BIT (DMA_FENCE_FLAG_USER_BITS + 1)
|
||||
|
||||
enum dma_resv_usage;
|
||||
struct dma_resv;
|
||||
struct drm_gem_object;
|
||||
|
||||
struct drm_gpu_scheduler;
|
||||
struct drm_sched_rq;
|
||||
|
||||
struct drm_file;
|
||||
|
||||
/* These are often used as an (initial) index
|
||||
* to an array, and as such should start at 0.
|
||||
*/
|
||||
enum drm_sched_priority {
|
||||
DRM_SCHED_PRIORITY_MIN,
|
||||
DRM_SCHED_PRIORITY_NORMAL,
|
||||
DRM_SCHED_PRIORITY_HIGH,
|
||||
DRM_SCHED_PRIORITY_KERNEL,
|
||||
DRM_SCHED_PRIORITY_HIGH,
|
||||
DRM_SCHED_PRIORITY_NORMAL,
|
||||
DRM_SCHED_PRIORITY_LOW,
|
||||
|
||||
DRM_SCHED_PRIORITY_COUNT
|
||||
};
|
||||
|
||||
/* Used to chose between FIFO and RR jobs scheduling */
|
||||
extern int drm_sched_policy;
|
||||
|
||||
#define DRM_SCHED_POLICY_RR 0
|
||||
#define DRM_SCHED_POLICY_FIFO 1
|
||||
|
||||
/**
|
||||
* struct drm_sched_entity - A wrapper around a job queue (typically
|
||||
* attached to the DRM file_priv).
|
||||
|
|
@ -181,7 +200,7 @@ struct drm_sched_entity {
|
|||
* by the scheduler thread, can be accessed locklessly from
|
||||
* drm_sched_job_arm() iff the queue is empty.
|
||||
*/
|
||||
struct dma_fence *last_scheduled;
|
||||
struct dma_fence __rcu *last_scheduled;
|
||||
|
||||
/**
|
||||
* @last_user: last group leader pushing a job into the entity.
|
||||
|
|
@ -204,6 +223,21 @@ struct drm_sched_entity {
|
|||
* drm_sched_entity_fini().
|
||||
*/
|
||||
struct completion entity_idle;
|
||||
|
||||
/**
|
||||
* @oldest_job_waiting:
|
||||
*
|
||||
* Marks earliest job waiting in SW queue
|
||||
*/
|
||||
ktime_t oldest_job_waiting;
|
||||
|
||||
/**
|
||||
* @rb_tree_node:
|
||||
*
|
||||
* The node used to insert this entity into time based priority queue
|
||||
*/
|
||||
struct rb_node rb_tree_node;
|
||||
|
||||
};
|
||||
|
||||
/**
|
||||
|
|
@ -213,6 +247,7 @@ struct drm_sched_entity {
|
|||
* @sched: the scheduler to which this rq belongs to.
|
||||
* @entities: list of the entities to be scheduled.
|
||||
* @current_entity: the entity which is to be scheduled.
|
||||
* @rb_tree_root: root of time based priory queue of entities for FIFO scheduling
|
||||
*
|
||||
* Run queue is a set of entities scheduling command submissions for
|
||||
* one specific ring. It implements the scheduling policy that selects
|
||||
|
|
@ -223,6 +258,7 @@ struct drm_sched_rq {
|
|||
struct drm_gpu_scheduler *sched;
|
||||
struct list_head entities;
|
||||
struct drm_sched_entity *current_entity;
|
||||
struct rb_root_cached rb_tree_root;
|
||||
};
|
||||
|
||||
/**
|
||||
|
|
@ -247,6 +283,12 @@ struct drm_sched_fence {
|
|||
*/
|
||||
struct dma_fence finished;
|
||||
|
||||
/**
|
||||
* @deadline: deadline set on &drm_sched_fence.finished which
|
||||
* potentially needs to be propagated to &drm_sched_fence.parent
|
||||
*/
|
||||
ktime_t deadline;
|
||||
|
||||
/**
|
||||
* @parent: the fence returned by &drm_sched_backend_ops.run_job
|
||||
* when scheduling the job on hardware. We signal the
|
||||
|
|
@ -278,6 +320,7 @@ struct drm_sched_fence *to_drm_sched_fence(struct dma_fence *f);
|
|||
* @sched: the scheduler instance on which this job is scheduled.
|
||||
* @s_fence: contains the fences for the scheduling of job.
|
||||
* @finish_cb: the callback for the finished fence.
|
||||
* @credits: the number of credits this job contributes to the scheduler
|
||||
* @work: Helper to reschdeule job kill to different context.
|
||||
* @id: a unique id assigned to each job scheduled on the scheduler.
|
||||
* @karma: increment on every hang caused by this job. If this exceeds the hang
|
||||
|
|
@ -297,13 +340,15 @@ struct drm_sched_job {
|
|||
struct drm_gpu_scheduler *sched;
|
||||
struct drm_sched_fence *s_fence;
|
||||
|
||||
u32 credits;
|
||||
|
||||
/*
|
||||
* work is used only after finish_cb has been used and will not be
|
||||
* accessed anymore.
|
||||
*/
|
||||
union {
|
||||
struct dma_fence_cb finish_cb;
|
||||
struct work_struct work;
|
||||
struct work_struct work;
|
||||
};
|
||||
|
||||
uint64_t id;
|
||||
|
|
@ -322,6 +367,13 @@ struct drm_sched_job {
|
|||
|
||||
/** @last_dependency: tracks @dependencies as they signal */
|
||||
unsigned long last_dependency;
|
||||
|
||||
/**
|
||||
* @submit_ts:
|
||||
*
|
||||
* When the job was pushed into the entity queue.
|
||||
*/
|
||||
ktime_t submit_ts;
|
||||
};
|
||||
|
||||
static inline bool drm_sched_invalidate_job(struct drm_sched_job *s_job,
|
||||
|
|
@ -344,18 +396,17 @@ enum drm_gpu_sched_stat {
|
|||
*/
|
||||
struct drm_sched_backend_ops {
|
||||
/**
|
||||
* @dependency:
|
||||
* @prepare_job:
|
||||
*
|
||||
* Called when the scheduler is considering scheduling this job next, to
|
||||
* get another struct dma_fence for this job to block on. Once it
|
||||
* returns NULL, run_job() may be called.
|
||||
*
|
||||
* If a driver exclusively uses drm_sched_job_add_dependency() and
|
||||
* drm_sched_job_add_implicit_dependencies() this can be ommitted and
|
||||
* left as NULL.
|
||||
* Can be NULL if no additional preparation to the dependencies are
|
||||
* necessary. Skipped when jobs are killed instead of run.
|
||||
*/
|
||||
struct dma_fence *(*dependency)(struct drm_sched_job *sched_job,
|
||||
struct drm_sched_entity *s_entity);
|
||||
struct dma_fence *(*prepare_job)(struct drm_sched_job *sched_job,
|
||||
struct drm_sched_entity *s_entity);
|
||||
|
||||
/**
|
||||
* @run_job: Called to execute the job once all of the dependencies
|
||||
|
|
@ -414,27 +465,42 @@ struct drm_sched_backend_ops {
|
|||
* and it's time to clean it up.
|
||||
*/
|
||||
void (*free_job)(struct drm_sched_job *sched_job);
|
||||
|
||||
/**
|
||||
* @update_job_credits: Called when the scheduler is considering this
|
||||
* job for execution.
|
||||
*
|
||||
* This callback returns the number of credits the job would take if
|
||||
* pushed to the hardware. Drivers may use this to dynamically update
|
||||
* the job's credit count. For instance, deduct the number of credits
|
||||
* for already signalled native fences.
|
||||
*
|
||||
* This callback is optional.
|
||||
*/
|
||||
u32 (*update_job_credits)(struct drm_sched_job *sched_job);
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_gpu_scheduler - scheduler instance-specific data
|
||||
*
|
||||
* @ops: backend operations provided by the driver.
|
||||
* @hw_submission_limit: the max size of the hardware queue.
|
||||
* @credit_limit: the credit limit of this scheduler
|
||||
* @credit_count: the current credit count of this scheduler
|
||||
* @timeout: the time after which a job is removed from the scheduler.
|
||||
* @name: name of the ring for which this scheduler is being used.
|
||||
* @sched_rq: priority wise array of run queues.
|
||||
* @wake_up_worker: the wait queue on which the scheduler sleeps until a job
|
||||
* is ready to be scheduled.
|
||||
* @num_rqs: Number of run-queues. This is at most DRM_SCHED_PRIORITY_COUNT,
|
||||
* as there's usually one run-queue per priority, but could be less.
|
||||
* @sched_rq: An allocated array of run-queues of size @num_rqs;
|
||||
* @job_scheduled: once @drm_sched_entity_do_release is called the scheduler
|
||||
* waits on this wait queue until all the scheduled jobs are
|
||||
* finished.
|
||||
* @hw_rq_count: the number of jobs currently in the hardware queue.
|
||||
* @job_id_count: used to assign unique id to the each job.
|
||||
* @submit_wq: workqueue used to queue @work_run_job and @work_free_job
|
||||
* @timeout_wq: workqueue used to queue @work_tdr
|
||||
* @work_run_job: work which calls run_job op of each scheduler.
|
||||
* @work_free_job: work which calls free_job op of each scheduler.
|
||||
* @work_tdr: schedules a delayed call to @drm_sched_job_timedout after the
|
||||
* timeout interval is over.
|
||||
* @thread: the kthread on which the scheduler which run.
|
||||
* @pending_list: the list of jobs which are currently in the job queue.
|
||||
* @job_list_lock: lock to protect the pending_list.
|
||||
* @hang_limit: once the hangs by a job crosses this limit then it is marked
|
||||
|
|
@ -443,23 +509,27 @@ struct drm_sched_backend_ops {
|
|||
* @_score: score used when the driver doesn't provide one
|
||||
* @ready: marks if the underlying HW is ready to work
|
||||
* @free_guilty: A hit to time out handler to free the guilty job.
|
||||
* @pause_submit: pause queuing of @work_run_job on @submit_wq
|
||||
* @own_submit_wq: scheduler owns allocation of @submit_wq
|
||||
* @dev: system &struct device
|
||||
*
|
||||
* One scheduler is implemented for each hardware ring.
|
||||
*/
|
||||
struct drm_gpu_scheduler {
|
||||
const struct drm_sched_backend_ops *ops;
|
||||
uint32_t hw_submission_limit;
|
||||
u32 credit_limit;
|
||||
atomic_t credit_count;
|
||||
long timeout;
|
||||
const char *name;
|
||||
struct drm_sched_rq sched_rq[DRM_SCHED_PRIORITY_COUNT];
|
||||
wait_queue_head_t wake_up_worker;
|
||||
u32 num_rqs;
|
||||
struct drm_sched_rq **sched_rq;
|
||||
wait_queue_head_t job_scheduled;
|
||||
atomic_t hw_rq_count;
|
||||
atomic64_t job_id_count;
|
||||
struct workqueue_struct *submit_wq;
|
||||
struct workqueue_struct *timeout_wq;
|
||||
struct work_struct work_run_job;
|
||||
struct work_struct work_free_job;
|
||||
struct delayed_work work_tdr;
|
||||
struct task_struct *thread;
|
||||
struct list_head pending_list;
|
||||
spinlock_t job_list_lock;
|
||||
int hang_limit;
|
||||
|
|
@ -467,22 +537,32 @@ struct drm_gpu_scheduler {
|
|||
atomic_t _score;
|
||||
bool ready;
|
||||
bool free_guilty;
|
||||
bool pause_submit;
|
||||
bool own_submit_wq;
|
||||
struct device *dev;
|
||||
};
|
||||
|
||||
int drm_sched_init(struct drm_gpu_scheduler *sched,
|
||||
const struct drm_sched_backend_ops *ops,
|
||||
uint32_t hw_submission, unsigned hang_limit,
|
||||
struct workqueue_struct *submit_wq,
|
||||
u32 num_rqs, u32 credit_limit, unsigned int hang_limit,
|
||||
long timeout, struct workqueue_struct *timeout_wq,
|
||||
atomic_t *score, const char *name, struct device *dev);
|
||||
|
||||
void drm_sched_fini(struct drm_gpu_scheduler *sched);
|
||||
int drm_sched_job_init(struct drm_sched_job *job,
|
||||
struct drm_sched_entity *entity,
|
||||
void *owner);
|
||||
u32 credits, void *owner);
|
||||
void drm_sched_job_arm(struct drm_sched_job *job);
|
||||
int drm_sched_job_add_dependency(struct drm_sched_job *job,
|
||||
struct dma_fence *fence);
|
||||
int drm_sched_job_add_syncobj_dependency(struct drm_sched_job *job,
|
||||
struct drm_file *file,
|
||||
u32 handle,
|
||||
u32 point);
|
||||
int drm_sched_job_add_resv_dependencies(struct drm_sched_job *job,
|
||||
struct dma_resv *resv,
|
||||
enum dma_resv_usage usage);
|
||||
int drm_sched_job_add_implicit_dependencies(struct drm_sched_job *job,
|
||||
struct drm_gem_object *obj,
|
||||
bool write);
|
||||
|
|
@ -492,17 +572,16 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity,
|
|||
struct drm_gpu_scheduler **sched_list,
|
||||
unsigned int num_sched_list);
|
||||
|
||||
void drm_sched_tdr_queue_imm(struct drm_gpu_scheduler *sched);
|
||||
void drm_sched_job_cleanup(struct drm_sched_job *job);
|
||||
void drm_sched_wakeup(struct drm_gpu_scheduler *sched);
|
||||
void drm_sched_wakeup(struct drm_gpu_scheduler *sched, struct drm_sched_entity *entity);
|
||||
bool drm_sched_wqueue_ready(struct drm_gpu_scheduler *sched);
|
||||
void drm_sched_wqueue_stop(struct drm_gpu_scheduler *sched);
|
||||
void drm_sched_wqueue_start(struct drm_gpu_scheduler *sched);
|
||||
void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad);
|
||||
void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery);
|
||||
void drm_sched_resubmit_jobs(struct drm_gpu_scheduler *sched);
|
||||
void drm_sched_resubmit_jobs_ext(struct drm_gpu_scheduler *sched, int max);
|
||||
void drm_sched_increase_karma(struct drm_sched_job *bad);
|
||||
void drm_sched_reset_karma(struct drm_sched_job *bad);
|
||||
void drm_sched_increase_karma_ext(struct drm_sched_job *bad, int type);
|
||||
bool drm_sched_dependency_optimized(struct dma_fence* fence,
|
||||
struct drm_sched_entity *entity);
|
||||
void drm_sched_fault(struct drm_gpu_scheduler *sched);
|
||||
void drm_sched_job_kickout(struct drm_sched_job *s_job);
|
||||
|
||||
|
|
@ -511,6 +590,8 @@ void drm_sched_rq_add_entity(struct drm_sched_rq *rq,
|
|||
void drm_sched_rq_remove_entity(struct drm_sched_rq *rq,
|
||||
struct drm_sched_entity *entity);
|
||||
|
||||
void drm_sched_rq_update_fifo(struct drm_sched_entity *entity, ktime_t ts);
|
||||
|
||||
int drm_sched_entity_init(struct drm_sched_entity *entity,
|
||||
enum drm_sched_priority priority,
|
||||
struct drm_gpu_scheduler **sched_list,
|
||||
|
|
@ -525,6 +606,7 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job);
|
|||
void drm_sched_entity_set_priority(struct drm_sched_entity *entity,
|
||||
enum drm_sched_priority priority);
|
||||
bool drm_sched_entity_is_ready(struct drm_sched_entity *entity);
|
||||
int drm_sched_entity_error(struct drm_sched_entity *entity);
|
||||
|
||||
struct drm_sched_fence *drm_sched_fence_alloc(
|
||||
struct drm_sched_entity *s_entity, void *owner);
|
||||
|
|
@ -532,8 +614,9 @@ void drm_sched_fence_init(struct drm_sched_fence *fence,
|
|||
struct drm_sched_entity *entity);
|
||||
void drm_sched_fence_free(struct drm_sched_fence *fence);
|
||||
|
||||
void drm_sched_fence_scheduled(struct drm_sched_fence *fence);
|
||||
void drm_sched_fence_finished(struct drm_sched_fence *fence);
|
||||
void drm_sched_fence_scheduled(struct drm_sched_fence *fence,
|
||||
struct dma_fence *parent);
|
||||
void drm_sched_fence_finished(struct drm_sched_fence *fence, int result);
|
||||
|
||||
unsigned long drm_sched_suspend_timeout(struct drm_gpu_scheduler *sched);
|
||||
void drm_sched_resume_timeout(struct drm_gpu_scheduler *sched,
|
||||
|
|
|
|||
|
|
@ -752,4 +752,6 @@ static inline size_t dma_buf_get_peak_size(void) { return 0; }
|
|||
static inline size_t dma_buf_get_total_size(void) { return 0; }
|
||||
#endif
|
||||
|
||||
int dma_buf_vmap_unlocked(struct dma_buf *dmabuf, struct iosys_map *map);
|
||||
void dma_buf_vunmap_unlocked(struct dma_buf *dmabuf, struct iosys_map *map);
|
||||
#endif /* __DMA_BUF_H__ */
|
||||
|
|
|
|||
|
|
@ -257,6 +257,26 @@ struct dma_fence_ops {
|
|||
*/
|
||||
void (*timeline_value_str)(struct dma_fence *fence,
|
||||
char *str, int size);
|
||||
|
||||
/**
|
||||
* @set_deadline:
|
||||
*
|
||||
* Callback to allow a fence waiter to inform the fence signaler of
|
||||
* an upcoming deadline, such as vblank, by which point the waiter
|
||||
* would prefer the fence to be signaled by. This is intended to
|
||||
* give feedback to the fence signaler to aid in power management
|
||||
* decisions, such as boosting GPU frequency.
|
||||
*
|
||||
* This is called without &dma_fence.lock held, it can be called
|
||||
* multiple times and from any context. Locking is up to the callee
|
||||
* if it has some state to manage. If multiple deadlines are set,
|
||||
* the expectation is to track the soonest one. If the deadline is
|
||||
* before the current time, it should be interpreted as an immediate
|
||||
* deadline.
|
||||
*
|
||||
* This callback is optional.
|
||||
*/
|
||||
void (*set_deadline)(struct dma_fence *fence, ktime_t deadline);
|
||||
};
|
||||
|
||||
void dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,
|
||||
|
|
@ -617,6 +637,8 @@ static inline signed long dma_fence_wait(struct dma_fence *fence, bool intr)
|
|||
return ret < 0 ? ret : 0;
|
||||
}
|
||||
|
||||
void dma_fence_set_deadline(struct dma_fence *fence, ktime_t deadline);
|
||||
|
||||
struct dma_fence *dma_fence_get_stub(void);
|
||||
struct dma_fence *dma_fence_allocate_private_stub(ktime_t timestamp);
|
||||
u64 dma_fence_context_alloc(unsigned num);
|
||||
|
|
|
|||
|
|
@ -100,6 +100,30 @@ struct io_pgtable_cfg {
|
|||
const struct iommu_flush_ops *tlb;
|
||||
struct device *iommu_dev;
|
||||
|
||||
/**
|
||||
* @alloc: Custom page allocator.
|
||||
*
|
||||
* Optional hook used to allocate page tables. If this function is NULL,
|
||||
* @free must be NULL too.
|
||||
*
|
||||
* Memory returned should be zeroed and suitable for dma_map_single() and
|
||||
* virt_to_phys().
|
||||
*
|
||||
* Not all formats support custom page allocators. Before considering
|
||||
* passing a non-NULL value, make sure the chosen page format supports
|
||||
* this feature.
|
||||
*/
|
||||
void *(*alloc)(void *cookie, size_t size, gfp_t gfp);
|
||||
|
||||
/**
|
||||
* @free: Custom page de-allocator.
|
||||
*
|
||||
* Optional hook used to free page tables allocated with the @alloc
|
||||
* hook. Must be non-NULL if @alloc is not NULL, must be NULL
|
||||
* otherwise.
|
||||
*/
|
||||
void (*free)(void *cookie, void *pages, size_t size);
|
||||
|
||||
/* Low-level data specific to the table format */
|
||||
union {
|
||||
struct {
|
||||
|
|
@ -243,16 +267,26 @@ io_pgtable_tlb_add_page(struct io_pgtable *iop,
|
|||
iop->cfg.tlb->tlb_add_page(gather, iova, granule, iop->cookie);
|
||||
}
|
||||
|
||||
/**
|
||||
* enum io_pgtable_caps - IO page table backend capabilities.
|
||||
*/
|
||||
enum io_pgtable_caps {
|
||||
/** @IO_PGTABLE_CAP_CUSTOM_ALLOCATOR: Backend accepts custom page table allocators. */
|
||||
IO_PGTABLE_CAP_CUSTOM_ALLOCATOR = BIT(0),
|
||||
};
|
||||
|
||||
/**
|
||||
* struct io_pgtable_init_fns - Alloc/free a set of page tables for a
|
||||
* particular format.
|
||||
*
|
||||
* @alloc: Allocate a set of page tables described by cfg.
|
||||
* @free: Free the page tables associated with iop.
|
||||
* @caps: Combination of @io_pgtable_caps flags encoding the backend capabilities.
|
||||
*/
|
||||
struct io_pgtable_init_fns {
|
||||
struct io_pgtable *(*alloc)(struct io_pgtable_cfg *cfg, void *cookie);
|
||||
void (*free)(struct io_pgtable *iop);
|
||||
u32 caps;
|
||||
};
|
||||
|
||||
extern struct io_pgtable_init_fns io_pgtable_arm_32_lpae_s1_init_fns;
|
||||
|
|
|
|||
945
include/uapi/drm/panthor_drm.h
Normal file
945
include/uapi/drm/panthor_drm.h
Normal file
|
|
@ -0,0 +1,945 @@
|
|||
/* SPDX-License-Identifier: MIT */
|
||||
/* Copyright (C) 2023 Collabora ltd. */
|
||||
#ifndef _PANTHOR_DRM_H_
|
||||
#define _PANTHOR_DRM_H_
|
||||
|
||||
#include "drm.h"
|
||||
|
||||
#if defined(__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
/**
|
||||
* DOC: Introduction
|
||||
*
|
||||
* This documentation describes the Panthor IOCTLs.
|
||||
*
|
||||
* Just a few generic rules about the data passed to the Panthor IOCTLs:
|
||||
*
|
||||
* - Structures must be aligned on 64-bit/8-byte. If the object is not
|
||||
* naturally aligned, a padding field must be added.
|
||||
* - Fields must be explicitly aligned to their natural type alignment with
|
||||
* pad[0..N] fields.
|
||||
* - All padding fields will be checked by the driver to make sure they are
|
||||
* zeroed.
|
||||
* - Flags can be added, but not removed/replaced.
|
||||
* - New fields can be added to the main structures (the structures
|
||||
* directly passed to the ioctl). Those fields can be added at the end of
|
||||
* the structure, or replace existing padding fields. Any new field being
|
||||
* added must preserve the behavior that existed before those fields were
|
||||
* added when a value of zero is passed.
|
||||
* - New fields can be added to indirect objects (objects pointed by the
|
||||
* main structure), iff those objects are passed a size to reflect the
|
||||
* size known by the userspace driver (see drm_panthor_obj_array::stride
|
||||
* or drm_panthor_dev_query::size).
|
||||
* - If the kernel driver is too old to know some fields, those will be
|
||||
* ignored if zero, and otherwise rejected (and so will be zero on output).
|
||||
* - If userspace is too old to know some fields, those will be zeroed
|
||||
* (input) before the structure is parsed by the kernel driver.
|
||||
* - Each new flag/field addition must come with a driver version update so
|
||||
* the userspace driver doesn't have to trial and error to know which
|
||||
* flags are supported.
|
||||
* - Structures should not contain unions, as this would defeat the
|
||||
* extensibility of such structures.
|
||||
* - IOCTLs can't be removed or replaced. New IOCTL IDs should be placed
|
||||
* at the end of the drm_panthor_ioctl_id enum.
|
||||
*/
|
||||
|
||||
/**
|
||||
* DOC: MMIO regions exposed to userspace.
|
||||
*
|
||||
* .. c:macro:: DRM_PANTHOR_USER_MMIO_OFFSET
|
||||
*
|
||||
* File offset for all MMIO regions being exposed to userspace. Don't use
|
||||
* this value directly, use DRM_PANTHOR_USER_<name>_OFFSET values instead.
|
||||
* pgoffset passed to mmap2() is an unsigned long, which forces us to use a
|
||||
* different offset on 32-bit and 64-bit systems.
|
||||
*
|
||||
* .. c:macro:: DRM_PANTHOR_USER_FLUSH_ID_MMIO_OFFSET
|
||||
*
|
||||
* File offset for the LATEST_FLUSH_ID register. The Userspace driver controls
|
||||
* GPU cache flushing through CS instructions, but the flush reduction
|
||||
* mechanism requires a flush_id. This flush_id could be queried with an
|
||||
* ioctl, but Arm provides a well-isolated register page containing only this
|
||||
* read-only register, so let's expose this page through a static mmap offset
|
||||
* and allow direct mapping of this MMIO region so we can avoid the
|
||||
* user <-> kernel round-trip.
|
||||
*/
|
||||
#define DRM_PANTHOR_USER_MMIO_OFFSET_32BIT (1ull << 43)
|
||||
#define DRM_PANTHOR_USER_MMIO_OFFSET_64BIT (1ull << 56)
|
||||
#define DRM_PANTHOR_USER_MMIO_OFFSET (sizeof(unsigned long) < 8 ? \
|
||||
DRM_PANTHOR_USER_MMIO_OFFSET_32BIT : \
|
||||
DRM_PANTHOR_USER_MMIO_OFFSET_64BIT)
|
||||
#define DRM_PANTHOR_USER_FLUSH_ID_MMIO_OFFSET (DRM_PANTHOR_USER_MMIO_OFFSET | 0)
|
||||
|
||||
/**
|
||||
* DOC: IOCTL IDs
|
||||
*
|
||||
* enum drm_panthor_ioctl_id - IOCTL IDs
|
||||
*
|
||||
* Place new ioctls at the end, don't re-order, don't replace or remove entries.
|
||||
*
|
||||
* These IDs are not meant to be used directly. Use the DRM_IOCTL_PANTHOR_xxx
|
||||
* definitions instead.
|
||||
*/
|
||||
enum drm_panthor_ioctl_id {
|
||||
/** @DRM_PANTHOR_DEV_QUERY: Query device information. */
|
||||
DRM_PANTHOR_DEV_QUERY = 0,
|
||||
|
||||
/** @DRM_PANTHOR_VM_CREATE: Create a VM. */
|
||||
DRM_PANTHOR_VM_CREATE,
|
||||
|
||||
/** @DRM_PANTHOR_VM_DESTROY: Destroy a VM. */
|
||||
DRM_PANTHOR_VM_DESTROY,
|
||||
|
||||
/** @DRM_PANTHOR_VM_BIND: Bind/unbind memory to a VM. */
|
||||
DRM_PANTHOR_VM_BIND,
|
||||
|
||||
/** @DRM_PANTHOR_VM_GET_STATE: Get VM state. */
|
||||
DRM_PANTHOR_VM_GET_STATE,
|
||||
|
||||
/** @DRM_PANTHOR_BO_CREATE: Create a buffer object. */
|
||||
DRM_PANTHOR_BO_CREATE,
|
||||
|
||||
/**
|
||||
* @DRM_PANTHOR_BO_MMAP_OFFSET: Get the file offset to pass to
|
||||
* mmap to map a GEM object.
|
||||
*/
|
||||
DRM_PANTHOR_BO_MMAP_OFFSET,
|
||||
|
||||
/** @DRM_PANTHOR_GROUP_CREATE: Create a scheduling group. */
|
||||
DRM_PANTHOR_GROUP_CREATE,
|
||||
|
||||
/** @DRM_PANTHOR_GROUP_DESTROY: Destroy a scheduling group. */
|
||||
DRM_PANTHOR_GROUP_DESTROY,
|
||||
|
||||
/**
|
||||
* @DRM_PANTHOR_GROUP_SUBMIT: Submit jobs to queues belonging
|
||||
* to a specific scheduling group.
|
||||
*/
|
||||
DRM_PANTHOR_GROUP_SUBMIT,
|
||||
|
||||
/** @DRM_PANTHOR_GROUP_GET_STATE: Get the state of a scheduling group. */
|
||||
DRM_PANTHOR_GROUP_GET_STATE,
|
||||
|
||||
/** @DRM_PANTHOR_TILER_HEAP_CREATE: Create a tiler heap. */
|
||||
DRM_PANTHOR_TILER_HEAP_CREATE,
|
||||
|
||||
/** @DRM_PANTHOR_TILER_HEAP_DESTROY: Destroy a tiler heap. */
|
||||
DRM_PANTHOR_TILER_HEAP_DESTROY,
|
||||
};
|
||||
|
||||
/**
|
||||
* DRM_IOCTL_PANTHOR() - Build a Panthor IOCTL number
|
||||
* @__access: Access type. Must be R, W or RW.
|
||||
* @__id: One of the DRM_PANTHOR_xxx id.
|
||||
* @__type: Suffix of the type being passed to the IOCTL.
|
||||
*
|
||||
* Don't use this macro directly, use the DRM_IOCTL_PANTHOR_xxx
|
||||
* values instead.
|
||||
*
|
||||
* Return: An IOCTL number to be passed to ioctl() from userspace.
|
||||
*/
|
||||
#define DRM_IOCTL_PANTHOR(__access, __id, __type) \
|
||||
DRM_IO ## __access(DRM_COMMAND_BASE + DRM_PANTHOR_ ## __id, \
|
||||
struct drm_panthor_ ## __type)
|
||||
|
||||
#define DRM_IOCTL_PANTHOR_DEV_QUERY \
|
||||
DRM_IOCTL_PANTHOR(WR, DEV_QUERY, dev_query)
|
||||
#define DRM_IOCTL_PANTHOR_VM_CREATE \
|
||||
DRM_IOCTL_PANTHOR(WR, VM_CREATE, vm_create)
|
||||
#define DRM_IOCTL_PANTHOR_VM_DESTROY \
|
||||
DRM_IOCTL_PANTHOR(WR, VM_DESTROY, vm_destroy)
|
||||
#define DRM_IOCTL_PANTHOR_VM_BIND \
|
||||
DRM_IOCTL_PANTHOR(WR, VM_BIND, vm_bind)
|
||||
#define DRM_IOCTL_PANTHOR_VM_GET_STATE \
|
||||
DRM_IOCTL_PANTHOR(WR, VM_GET_STATE, vm_get_state)
|
||||
#define DRM_IOCTL_PANTHOR_BO_CREATE \
|
||||
DRM_IOCTL_PANTHOR(WR, BO_CREATE, bo_create)
|
||||
#define DRM_IOCTL_PANTHOR_BO_MMAP_OFFSET \
|
||||
DRM_IOCTL_PANTHOR(WR, BO_MMAP_OFFSET, bo_mmap_offset)
|
||||
#define DRM_IOCTL_PANTHOR_GROUP_CREATE \
|
||||
DRM_IOCTL_PANTHOR(WR, GROUP_CREATE, group_create)
|
||||
#define DRM_IOCTL_PANTHOR_GROUP_DESTROY \
|
||||
DRM_IOCTL_PANTHOR(WR, GROUP_DESTROY, group_destroy)
|
||||
#define DRM_IOCTL_PANTHOR_GROUP_SUBMIT \
|
||||
DRM_IOCTL_PANTHOR(WR, GROUP_SUBMIT, group_submit)
|
||||
#define DRM_IOCTL_PANTHOR_GROUP_GET_STATE \
|
||||
DRM_IOCTL_PANTHOR(WR, GROUP_GET_STATE, group_get_state)
|
||||
#define DRM_IOCTL_PANTHOR_TILER_HEAP_CREATE \
|
||||
DRM_IOCTL_PANTHOR(WR, TILER_HEAP_CREATE, tiler_heap_create)
|
||||
#define DRM_IOCTL_PANTHOR_TILER_HEAP_DESTROY \
|
||||
DRM_IOCTL_PANTHOR(WR, TILER_HEAP_DESTROY, tiler_heap_destroy)
|
||||
|
||||
/**
|
||||
* DOC: IOCTL arguments
|
||||
*/
|
||||
|
||||
/**
|
||||
* struct drm_panthor_obj_array - Object array.
|
||||
*
|
||||
* This object is used to pass an array of objects whose size is subject to changes in
|
||||
* future versions of the driver. In order to support this mutability, we pass a stride
|
||||
* describing the size of the object as known by userspace.
|
||||
*
|
||||
* You shouldn't fill drm_panthor_obj_array fields directly. You should instead use
|
||||
* the DRM_PANTHOR_OBJ_ARRAY() macro that takes care of initializing the stride to
|
||||
* the object size.
|
||||
*/
|
||||
struct drm_panthor_obj_array {
|
||||
/** @stride: Stride of object struct. Used for versioning. */
|
||||
__u32 stride;
|
||||
|
||||
/** @count: Number of objects in the array. */
|
||||
__u32 count;
|
||||
|
||||
/** @array: User pointer to an array of objects. */
|
||||
__u64 array;
|
||||
};
|
||||
|
||||
/**
|
||||
* DRM_PANTHOR_OBJ_ARRAY() - Initialize a drm_panthor_obj_array field.
|
||||
* @cnt: Number of elements in the array.
|
||||
* @ptr: Pointer to the array to pass to the kernel.
|
||||
*
|
||||
* Macro initializing a drm_panthor_obj_array based on the object size as known
|
||||
* by userspace.
|
||||
*/
|
||||
#define DRM_PANTHOR_OBJ_ARRAY(cnt, ptr) \
|
||||
{ .stride = sizeof((ptr)[0]), .count = (cnt), .array = (__u64)(uintptr_t)(ptr) }
|
||||
|
||||
/**
|
||||
* enum drm_panthor_sync_op_flags - Synchronization operation flags.
|
||||
*/
|
||||
enum drm_panthor_sync_op_flags {
|
||||
/** @DRM_PANTHOR_SYNC_OP_HANDLE_TYPE_MASK: Synchronization handle type mask. */
|
||||
DRM_PANTHOR_SYNC_OP_HANDLE_TYPE_MASK = 0xff,
|
||||
|
||||
/** @DRM_PANTHOR_SYNC_OP_HANDLE_TYPE_SYNCOBJ: Synchronization object type. */
|
||||
DRM_PANTHOR_SYNC_OP_HANDLE_TYPE_SYNCOBJ = 0,
|
||||
|
||||
/**
|
||||
* @DRM_PANTHOR_SYNC_OP_HANDLE_TYPE_TIMELINE_SYNCOBJ: Timeline synchronization
|
||||
* object type.
|
||||
*/
|
||||
DRM_PANTHOR_SYNC_OP_HANDLE_TYPE_TIMELINE_SYNCOBJ = 1,
|
||||
|
||||
/** @DRM_PANTHOR_SYNC_OP_WAIT: Wait operation. */
|
||||
DRM_PANTHOR_SYNC_OP_WAIT = 0 << 31,
|
||||
|
||||
/** @DRM_PANTHOR_SYNC_OP_SIGNAL: Signal operation. */
|
||||
DRM_PANTHOR_SYNC_OP_SIGNAL = (int)(1u << 31),
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_sync_op - Synchronization operation.
|
||||
*/
|
||||
struct drm_panthor_sync_op {
|
||||
/** @flags: Synchronization operation flags. Combination of DRM_PANTHOR_SYNC_OP values. */
|
||||
__u32 flags;
|
||||
|
||||
/** @handle: Sync handle. */
|
||||
__u32 handle;
|
||||
|
||||
/**
|
||||
* @timeline_value: MBZ if
|
||||
* (flags & DRM_PANTHOR_SYNC_OP_HANDLE_TYPE_MASK) !=
|
||||
* DRM_PANTHOR_SYNC_OP_HANDLE_TYPE_TIMELINE_SYNCOBJ.
|
||||
*/
|
||||
__u64 timeline_value;
|
||||
};
|
||||
|
||||
/**
|
||||
* enum drm_panthor_dev_query_type - Query type
|
||||
*
|
||||
* Place new types at the end, don't re-order, don't remove or replace.
|
||||
*/
|
||||
enum drm_panthor_dev_query_type {
|
||||
/** @DRM_PANTHOR_DEV_QUERY_GPU_INFO: Query GPU information. */
|
||||
DRM_PANTHOR_DEV_QUERY_GPU_INFO = 0,
|
||||
|
||||
/** @DRM_PANTHOR_DEV_QUERY_CSIF_INFO: Query command-stream interface information. */
|
||||
DRM_PANTHOR_DEV_QUERY_CSIF_INFO,
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_gpu_info - GPU information
|
||||
*
|
||||
* Structure grouping all queryable information relating to the GPU.
|
||||
*/
|
||||
struct drm_panthor_gpu_info {
|
||||
/** @gpu_id : GPU ID. */
|
||||
__u32 gpu_id;
|
||||
#define DRM_PANTHOR_ARCH_MAJOR(x) ((x) >> 28)
|
||||
#define DRM_PANTHOR_ARCH_MINOR(x) (((x) >> 24) & 0xf)
|
||||
#define DRM_PANTHOR_ARCH_REV(x) (((x) >> 20) & 0xf)
|
||||
#define DRM_PANTHOR_PRODUCT_MAJOR(x) (((x) >> 16) & 0xf)
|
||||
#define DRM_PANTHOR_VERSION_MAJOR(x) (((x) >> 12) & 0xf)
|
||||
#define DRM_PANTHOR_VERSION_MINOR(x) (((x) >> 4) & 0xff)
|
||||
#define DRM_PANTHOR_VERSION_STATUS(x) ((x) & 0xf)
|
||||
|
||||
/** @gpu_rev: GPU revision. */
|
||||
__u32 gpu_rev;
|
||||
|
||||
/** @csf_id: Command stream frontend ID. */
|
||||
__u32 csf_id;
|
||||
#define DRM_PANTHOR_CSHW_MAJOR(x) (((x) >> 26) & 0x3f)
|
||||
#define DRM_PANTHOR_CSHW_MINOR(x) (((x) >> 20) & 0x3f)
|
||||
#define DRM_PANTHOR_CSHW_REV(x) (((x) >> 16) & 0xf)
|
||||
#define DRM_PANTHOR_MCU_MAJOR(x) (((x) >> 10) & 0x3f)
|
||||
#define DRM_PANTHOR_MCU_MINOR(x) (((x) >> 4) & 0x3f)
|
||||
#define DRM_PANTHOR_MCU_REV(x) ((x) & 0xf)
|
||||
|
||||
/** @l2_features: L2-cache features. */
|
||||
__u32 l2_features;
|
||||
|
||||
/** @tiler_features: Tiler features. */
|
||||
__u32 tiler_features;
|
||||
|
||||
/** @mem_features: Memory features. */
|
||||
__u32 mem_features;
|
||||
|
||||
/** @mmu_features: MMU features. */
|
||||
__u32 mmu_features;
|
||||
#define DRM_PANTHOR_MMU_VA_BITS(x) ((x) & 0xff)
|
||||
|
||||
/** @thread_features: Thread features. */
|
||||
__u32 thread_features;
|
||||
|
||||
/** @max_threads: Maximum number of threads. */
|
||||
__u32 max_threads;
|
||||
|
||||
/** @thread_max_workgroup_size: Maximum workgroup size. */
|
||||
__u32 thread_max_workgroup_size;
|
||||
|
||||
/**
|
||||
* @thread_max_barrier_size: Maximum number of threads that can wait
|
||||
* simultaneously on a barrier.
|
||||
*/
|
||||
__u32 thread_max_barrier_size;
|
||||
|
||||
/** @coherency_features: Coherency features. */
|
||||
__u32 coherency_features;
|
||||
|
||||
/** @texture_features: Texture features. */
|
||||
__u32 texture_features[4];
|
||||
|
||||
/** @as_present: Bitmask encoding the number of address-space exposed by the MMU. */
|
||||
__u32 as_present;
|
||||
|
||||
/** @shader_present: Bitmask encoding the shader cores exposed by the GPU. */
|
||||
__u64 shader_present;
|
||||
|
||||
/** @l2_present: Bitmask encoding the L2 caches exposed by the GPU. */
|
||||
__u64 l2_present;
|
||||
|
||||
/** @tiler_present: Bitmask encoding the tiler units exposed by the GPU. */
|
||||
__u64 tiler_present;
|
||||
|
||||
/* @core_features: Used to discriminate core variants when they exist. */
|
||||
__u32 core_features;
|
||||
|
||||
/* @pad: MBZ. */
|
||||
__u32 pad;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_csif_info - Command stream interface information
|
||||
*
|
||||
* Structure grouping all queryable information relating to the command stream interface.
|
||||
*/
|
||||
struct drm_panthor_csif_info {
|
||||
/** @csg_slot_count: Number of command stream group slots exposed by the firmware. */
|
||||
__u32 csg_slot_count;
|
||||
|
||||
/** @cs_slot_count: Number of command stream slots per group. */
|
||||
__u32 cs_slot_count;
|
||||
|
||||
/** @cs_reg_count: Number of command stream registers. */
|
||||
__u32 cs_reg_count;
|
||||
|
||||
/** @scoreboard_slot_count: Number of scoreboard slots. */
|
||||
__u32 scoreboard_slot_count;
|
||||
|
||||
/**
|
||||
* @unpreserved_cs_reg_count: Number of command stream registers reserved by
|
||||
* the kernel driver to call a userspace command stream.
|
||||
*
|
||||
* All registers can be used by a userspace command stream, but the
|
||||
* [cs_slot_count - unpreserved_cs_reg_count .. cs_slot_count] registers are
|
||||
* used by the kernel when DRM_PANTHOR_IOCTL_GROUP_SUBMIT is called.
|
||||
*/
|
||||
__u32 unpreserved_cs_reg_count;
|
||||
|
||||
/**
|
||||
* @pad: Padding field, set to zero.
|
||||
*/
|
||||
__u32 pad;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_dev_query - Arguments passed to DRM_PANTHOR_IOCTL_DEV_QUERY
|
||||
*/
|
||||
struct drm_panthor_dev_query {
|
||||
/** @type: the query type (see drm_panthor_dev_query_type). */
|
||||
__u32 type;
|
||||
|
||||
/**
|
||||
* @size: size of the type being queried.
|
||||
*
|
||||
* If pointer is NULL, size is updated by the driver to provide the
|
||||
* output structure size. If pointer is not NULL, the driver will
|
||||
* only copy min(size, actual_structure_size) bytes to the pointer,
|
||||
* and update the size accordingly. This allows us to extend query
|
||||
* types without breaking userspace.
|
||||
*/
|
||||
__u32 size;
|
||||
|
||||
/**
|
||||
* @pointer: user pointer to a query type struct.
|
||||
*
|
||||
* Pointer can be NULL, in which case, nothing is copied, but the
|
||||
* actual structure size is returned. If not NULL, it must point to
|
||||
* a location that's large enough to hold size bytes.
|
||||
*/
|
||||
__u64 pointer;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_vm_create - Arguments passed to DRM_PANTHOR_IOCTL_VM_CREATE
|
||||
*/
|
||||
struct drm_panthor_vm_create {
|
||||
/** @flags: VM flags, MBZ. */
|
||||
__u32 flags;
|
||||
|
||||
/** @id: Returned VM ID. */
|
||||
__u32 id;
|
||||
|
||||
/**
|
||||
* @user_va_range: Size of the VA space reserved for user objects.
|
||||
*
|
||||
* The kernel will pick the remaining space to map kernel-only objects to the
|
||||
* VM (heap chunks, heap context, ring buffers, kernel synchronization objects,
|
||||
* ...). If the space left for kernel objects is too small, kernel object
|
||||
* allocation will fail further down the road. One can use
|
||||
* drm_panthor_gpu_info::mmu_features to extract the total virtual address
|
||||
* range, and chose a user_va_range that leaves some space to the kernel.
|
||||
*
|
||||
* If user_va_range is zero, the kernel will pick a sensible value based on
|
||||
* TASK_SIZE and the virtual range supported by the GPU MMU (the kernel/user
|
||||
* split should leave enough VA space for userspace processes to support SVM,
|
||||
* while still allowing the kernel to map some amount of kernel objects in
|
||||
* the kernel VA range). The value chosen by the driver will be returned in
|
||||
* @user_va_range.
|
||||
*
|
||||
* User VA space always starts at 0x0, kernel VA space is always placed after
|
||||
* the user VA range.
|
||||
*/
|
||||
__u64 user_va_range;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_vm_destroy - Arguments passed to DRM_PANTHOR_IOCTL_VM_DESTROY
|
||||
*/
|
||||
struct drm_panthor_vm_destroy {
|
||||
/** @id: ID of the VM to destroy. */
|
||||
__u32 id;
|
||||
|
||||
/** @pad: MBZ. */
|
||||
__u32 pad;
|
||||
};
|
||||
|
||||
/**
|
||||
* enum drm_panthor_vm_bind_op_flags - VM bind operation flags
|
||||
*/
|
||||
enum drm_panthor_vm_bind_op_flags {
|
||||
/**
|
||||
* @DRM_PANTHOR_VM_BIND_OP_MAP_READONLY: Map the memory read-only.
|
||||
*
|
||||
* Only valid with DRM_PANTHOR_VM_BIND_OP_TYPE_MAP.
|
||||
*/
|
||||
DRM_PANTHOR_VM_BIND_OP_MAP_READONLY = 1 << 0,
|
||||
|
||||
/**
|
||||
* @DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC: Map the memory not-executable.
|
||||
*
|
||||
* Only valid with DRM_PANTHOR_VM_BIND_OP_TYPE_MAP.
|
||||
*/
|
||||
DRM_PANTHOR_VM_BIND_OP_MAP_NOEXEC = 1 << 1,
|
||||
|
||||
/**
|
||||
* @DRM_PANTHOR_VM_BIND_OP_MAP_UNCACHED: Map the memory uncached.
|
||||
*
|
||||
* Only valid with DRM_PANTHOR_VM_BIND_OP_TYPE_MAP.
|
||||
*/
|
||||
DRM_PANTHOR_VM_BIND_OP_MAP_UNCACHED = 1 << 2,
|
||||
|
||||
/**
|
||||
* @DRM_PANTHOR_VM_BIND_OP_TYPE_MASK: Mask used to determine the type of operation.
|
||||
*/
|
||||
DRM_PANTHOR_VM_BIND_OP_TYPE_MASK = (int)(0xfu << 28),
|
||||
|
||||
/** @DRM_PANTHOR_VM_BIND_OP_TYPE_MAP: Map operation. */
|
||||
DRM_PANTHOR_VM_BIND_OP_TYPE_MAP = 0 << 28,
|
||||
|
||||
/** @DRM_PANTHOR_VM_BIND_OP_TYPE_UNMAP: Unmap operation. */
|
||||
DRM_PANTHOR_VM_BIND_OP_TYPE_UNMAP = 1 << 28,
|
||||
|
||||
/**
|
||||
* @DRM_PANTHOR_VM_BIND_OP_TYPE_SYNC_ONLY: No VM operation.
|
||||
*
|
||||
* Just serves as a synchronization point on a VM queue.
|
||||
*
|
||||
* Only valid if %DRM_PANTHOR_VM_BIND_ASYNC is set in drm_panthor_vm_bind::flags,
|
||||
* and drm_panthor_vm_bind_op::syncs contains at least one element.
|
||||
*/
|
||||
DRM_PANTHOR_VM_BIND_OP_TYPE_SYNC_ONLY = 2 << 28,
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_vm_bind_op - VM bind operation
|
||||
*/
|
||||
struct drm_panthor_vm_bind_op {
|
||||
/** @flags: Combination of drm_panthor_vm_bind_op_flags flags. */
|
||||
__u32 flags;
|
||||
|
||||
/**
|
||||
* @bo_handle: Handle of the buffer object to map.
|
||||
* MBZ for unmap or sync-only operations.
|
||||
*/
|
||||
__u32 bo_handle;
|
||||
|
||||
/**
|
||||
* @bo_offset: Buffer object offset.
|
||||
* MBZ for unmap or sync-only operations.
|
||||
*/
|
||||
__u64 bo_offset;
|
||||
|
||||
/**
|
||||
* @va: Virtual address to map/unmap.
|
||||
* MBZ for sync-only operations.
|
||||
*/
|
||||
__u64 va;
|
||||
|
||||
/**
|
||||
* @size: Size to map/unmap.
|
||||
* MBZ for sync-only operations.
|
||||
*/
|
||||
__u64 size;
|
||||
|
||||
/**
|
||||
* @syncs: Array of struct drm_panthor_sync_op synchronization
|
||||
* operations.
|
||||
*
|
||||
* This array must be empty if %DRM_PANTHOR_VM_BIND_ASYNC is not set on
|
||||
* the drm_panthor_vm_bind object containing this VM bind operation.
|
||||
*
|
||||
* This array shall not be empty for sync-only operations.
|
||||
*/
|
||||
struct drm_panthor_obj_array syncs;
|
||||
|
||||
};
|
||||
|
||||
/**
|
||||
* enum drm_panthor_vm_bind_flags - VM bind flags
|
||||
*/
|
||||
enum drm_panthor_vm_bind_flags {
|
||||
/**
|
||||
* @DRM_PANTHOR_VM_BIND_ASYNC: VM bind operations are queued to the VM
|
||||
* queue instead of being executed synchronously.
|
||||
*/
|
||||
DRM_PANTHOR_VM_BIND_ASYNC = 1 << 0,
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_vm_bind - Arguments passed to DRM_IOCTL_PANTHOR_VM_BIND
|
||||
*/
|
||||
struct drm_panthor_vm_bind {
|
||||
/** @vm_id: VM targeted by the bind request. */
|
||||
__u32 vm_id;
|
||||
|
||||
/** @flags: Combination of drm_panthor_vm_bind_flags flags. */
|
||||
__u32 flags;
|
||||
|
||||
/** @ops: Array of struct drm_panthor_vm_bind_op bind operations. */
|
||||
struct drm_panthor_obj_array ops;
|
||||
};
|
||||
|
||||
/**
|
||||
* enum drm_panthor_vm_state - VM states.
|
||||
*/
|
||||
enum drm_panthor_vm_state {
|
||||
/**
|
||||
* @DRM_PANTHOR_VM_STATE_USABLE: VM is usable.
|
||||
*
|
||||
* New VM operations will be accepted on this VM.
|
||||
*/
|
||||
DRM_PANTHOR_VM_STATE_USABLE,
|
||||
|
||||
/**
|
||||
* @DRM_PANTHOR_VM_STATE_UNUSABLE: VM is unusable.
|
||||
*
|
||||
* Something put the VM in an unusable state (like an asynchronous
|
||||
* VM_BIND request failing for any reason).
|
||||
*
|
||||
* Once the VM is in this state, all new MAP operations will be
|
||||
* rejected, and any GPU job targeting this VM will fail.
|
||||
* UNMAP operations are still accepted.
|
||||
*
|
||||
* The only way to recover from an unusable VM is to create a new
|
||||
* VM, and destroy the old one.
|
||||
*/
|
||||
DRM_PANTHOR_VM_STATE_UNUSABLE,
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_vm_get_state - Get VM state.
|
||||
*/
|
||||
struct drm_panthor_vm_get_state {
|
||||
/** @vm_id: VM targeted by the get_state request. */
|
||||
__u32 vm_id;
|
||||
|
||||
/**
|
||||
* @state: state returned by the driver.
|
||||
*
|
||||
* Must be one of the enum drm_panthor_vm_state values.
|
||||
*/
|
||||
__u32 state;
|
||||
};
|
||||
|
||||
/**
|
||||
* enum drm_panthor_bo_flags - Buffer object flags, passed at creation time.
|
||||
*/
|
||||
enum drm_panthor_bo_flags {
|
||||
/** @DRM_PANTHOR_BO_NO_MMAP: The buffer object will never be CPU-mapped in userspace. */
|
||||
DRM_PANTHOR_BO_NO_MMAP = (1 << 0),
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_bo_create - Arguments passed to DRM_IOCTL_PANTHOR_BO_CREATE.
|
||||
*/
|
||||
struct drm_panthor_bo_create {
|
||||
/**
|
||||
* @size: Requested size for the object
|
||||
*
|
||||
* The (page-aligned) allocated size for the object will be returned.
|
||||
*/
|
||||
__u64 size;
|
||||
|
||||
/**
|
||||
* @flags: Flags. Must be a combination of drm_panthor_bo_flags flags.
|
||||
*/
|
||||
__u32 flags;
|
||||
|
||||
/**
|
||||
* @exclusive_vm_id: Exclusive VM this buffer object will be mapped to.
|
||||
*
|
||||
* If not zero, the field must refer to a valid VM ID, and implies that:
|
||||
* - the buffer object will only ever be bound to that VM
|
||||
* - cannot be exported as a PRIME fd
|
||||
*/
|
||||
__u32 exclusive_vm_id;
|
||||
|
||||
/**
|
||||
* @handle: Returned handle for the object.
|
||||
*
|
||||
* Object handles are nonzero.
|
||||
*/
|
||||
__u32 handle;
|
||||
|
||||
/** @pad: MBZ. */
|
||||
__u32 pad;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_bo_mmap_offset - Arguments passed to DRM_IOCTL_PANTHOR_BO_MMAP_OFFSET.
|
||||
*/
|
||||
struct drm_panthor_bo_mmap_offset {
|
||||
/** @handle: Handle of the object we want an mmap offset for. */
|
||||
__u32 handle;
|
||||
|
||||
/** @pad: MBZ. */
|
||||
__u32 pad;
|
||||
|
||||
/** @offset: The fake offset to use for subsequent mmap calls. */
|
||||
__u64 offset;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_queue_create - Queue creation arguments.
|
||||
*/
|
||||
struct drm_panthor_queue_create {
|
||||
/**
|
||||
* @priority: Defines the priority of queues inside a group. Goes from 0 to 15,
|
||||
* 15 being the highest priority.
|
||||
*/
|
||||
__u8 priority;
|
||||
|
||||
/** @pad: Padding fields, MBZ. */
|
||||
__u8 pad[3];
|
||||
|
||||
/** @ringbuf_size: Size of the ring buffer to allocate to this queue. */
|
||||
__u32 ringbuf_size;
|
||||
};
|
||||
|
||||
/**
|
||||
* enum drm_panthor_group_priority - Scheduling group priority
|
||||
*/
|
||||
enum drm_panthor_group_priority {
|
||||
/** @PANTHOR_GROUP_PRIORITY_LOW: Low priority group. */
|
||||
PANTHOR_GROUP_PRIORITY_LOW = 0,
|
||||
|
||||
/** @PANTHOR_GROUP_PRIORITY_MEDIUM: Medium priority group. */
|
||||
PANTHOR_GROUP_PRIORITY_MEDIUM,
|
||||
|
||||
/** @PANTHOR_GROUP_PRIORITY_HIGH: High priority group. */
|
||||
PANTHOR_GROUP_PRIORITY_HIGH,
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_group_create - Arguments passed to DRM_IOCTL_PANTHOR_GROUP_CREATE
|
||||
*/
|
||||
struct drm_panthor_group_create {
|
||||
/** @queues: Array of drm_panthor_queue_create elements. */
|
||||
struct drm_panthor_obj_array queues;
|
||||
|
||||
/**
|
||||
* @max_compute_cores: Maximum number of cores that can be used by compute
|
||||
* jobs across CS queues bound to this group.
|
||||
*
|
||||
* Must be less or equal to the number of bits set in @compute_core_mask.
|
||||
*/
|
||||
__u8 max_compute_cores;
|
||||
|
||||
/**
|
||||
* @max_fragment_cores: Maximum number of cores that can be used by fragment
|
||||
* jobs across CS queues bound to this group.
|
||||
*
|
||||
* Must be less or equal to the number of bits set in @fragment_core_mask.
|
||||
*/
|
||||
__u8 max_fragment_cores;
|
||||
|
||||
/**
|
||||
* @max_tiler_cores: Maximum number of tilers that can be used by tiler jobs
|
||||
* across CS queues bound to this group.
|
||||
*
|
||||
* Must be less or equal to the number of bits set in @tiler_core_mask.
|
||||
*/
|
||||
__u8 max_tiler_cores;
|
||||
|
||||
/** @priority: Group priority (see enum drm_panthor_group_priority). */
|
||||
__u8 priority;
|
||||
|
||||
/** @pad: Padding field, MBZ. */
|
||||
__u32 pad;
|
||||
|
||||
/**
|
||||
* @compute_core_mask: Mask encoding cores that can be used for compute jobs.
|
||||
*
|
||||
* This field must have at least @max_compute_cores bits set.
|
||||
*
|
||||
* The bits set here should also be set in drm_panthor_gpu_info::shader_present.
|
||||
*/
|
||||
__u64 compute_core_mask;
|
||||
|
||||
/**
|
||||
* @fragment_core_mask: Mask encoding cores that can be used for fragment jobs.
|
||||
*
|
||||
* This field must have at least @max_fragment_cores bits set.
|
||||
*
|
||||
* The bits set here should also be set in drm_panthor_gpu_info::shader_present.
|
||||
*/
|
||||
__u64 fragment_core_mask;
|
||||
|
||||
/**
|
||||
* @tiler_core_mask: Mask encoding cores that can be used for tiler jobs.
|
||||
*
|
||||
* This field must have at least @max_tiler_cores bits set.
|
||||
*
|
||||
* The bits set here should also be set in drm_panthor_gpu_info::tiler_present.
|
||||
*/
|
||||
__u64 tiler_core_mask;
|
||||
|
||||
/**
|
||||
* @vm_id: VM ID to bind this group to.
|
||||
*
|
||||
* All submission to queues bound to this group will use this VM.
|
||||
*/
|
||||
__u32 vm_id;
|
||||
|
||||
/**
|
||||
* @group_handle: Returned group handle. Passed back when submitting jobs or
|
||||
* destroying a group.
|
||||
*/
|
||||
__u32 group_handle;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_group_destroy - Arguments passed to DRM_IOCTL_PANTHOR_GROUP_DESTROY
|
||||
*/
|
||||
struct drm_panthor_group_destroy {
|
||||
/** @group_handle: Group to destroy */
|
||||
__u32 group_handle;
|
||||
|
||||
/** @pad: Padding field, MBZ. */
|
||||
__u32 pad;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_queue_submit - Job submission arguments.
|
||||
*
|
||||
* This is describing the userspace command stream to call from the kernel
|
||||
* command stream ring-buffer. Queue submission is always part of a group
|
||||
* submission, taking one or more jobs to submit to the underlying queues.
|
||||
*/
|
||||
struct drm_panthor_queue_submit {
|
||||
/** @queue_index: Index of the queue inside a group. */
|
||||
__u32 queue_index;
|
||||
|
||||
/**
|
||||
* @stream_size: Size of the command stream to execute.
|
||||
*
|
||||
* Must be 64-bit/8-byte aligned (the size of a CS instruction)
|
||||
*
|
||||
* Can be zero if stream_addr is zero too.
|
||||
*/
|
||||
__u32 stream_size;
|
||||
|
||||
/**
|
||||
* @stream_addr: GPU address of the command stream to execute.
|
||||
*
|
||||
* Must be aligned on 64-byte.
|
||||
*
|
||||
* Can be zero is stream_size is zero too.
|
||||
*/
|
||||
__u64 stream_addr;
|
||||
|
||||
/**
|
||||
* @latest_flush: FLUSH_ID read at the time the stream was built.
|
||||
*
|
||||
* This allows cache flush elimination for the automatic
|
||||
* flush+invalidate(all) done at submission time, which is needed to
|
||||
* ensure the GPU doesn't get garbage when reading the indirect command
|
||||
* stream buffers. If you want the cache flush to happen
|
||||
* unconditionally, pass a zero here.
|
||||
*/
|
||||
__u32 latest_flush;
|
||||
|
||||
/** @pad: MBZ. */
|
||||
__u32 pad;
|
||||
|
||||
/** @syncs: Array of struct drm_panthor_sync_op sync operations. */
|
||||
struct drm_panthor_obj_array syncs;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_group_submit - Arguments passed to DRM_IOCTL_PANTHOR_GROUP_SUBMIT
|
||||
*/
|
||||
struct drm_panthor_group_submit {
|
||||
/** @group_handle: Handle of the group to queue jobs to. */
|
||||
__u32 group_handle;
|
||||
|
||||
/** @pad: MBZ. */
|
||||
__u32 pad;
|
||||
|
||||
/** @queue_submits: Array of drm_panthor_queue_submit objects. */
|
||||
struct drm_panthor_obj_array queue_submits;
|
||||
};
|
||||
|
||||
/**
|
||||
* enum drm_panthor_group_state_flags - Group state flags
|
||||
*/
|
||||
enum drm_panthor_group_state_flags {
|
||||
/**
|
||||
* @DRM_PANTHOR_GROUP_STATE_TIMEDOUT: Group had unfinished jobs.
|
||||
*
|
||||
* When a group ends up with this flag set, no jobs can be submitted to its queues.
|
||||
*/
|
||||
DRM_PANTHOR_GROUP_STATE_TIMEDOUT = 1 << 0,
|
||||
|
||||
/**
|
||||
* @DRM_PANTHOR_GROUP_STATE_FATAL_FAULT: Group had fatal faults.
|
||||
*
|
||||
* When a group ends up with this flag set, no jobs can be submitted to its queues.
|
||||
*/
|
||||
DRM_PANTHOR_GROUP_STATE_FATAL_FAULT = 1 << 1,
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_group_get_state - Arguments passed to DRM_IOCTL_PANTHOR_GROUP_GET_STATE
|
||||
*
|
||||
* Used to query the state of a group and decide whether a new group should be created to
|
||||
* replace it.
|
||||
*/
|
||||
struct drm_panthor_group_get_state {
|
||||
/** @group_handle: Handle of the group to query state on */
|
||||
__u32 group_handle;
|
||||
|
||||
/**
|
||||
* @state: Combination of DRM_PANTHOR_GROUP_STATE_* flags encoding the
|
||||
* group state.
|
||||
*/
|
||||
__u32 state;
|
||||
|
||||
/** @fatal_queues: Bitmask of queues that faced fatal faults. */
|
||||
__u32 fatal_queues;
|
||||
|
||||
/** @pad: MBZ */
|
||||
__u32 pad;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_tiler_heap_create - Arguments passed to DRM_IOCTL_PANTHOR_TILER_HEAP_CREATE
|
||||
*/
|
||||
struct drm_panthor_tiler_heap_create {
|
||||
/** @vm_id: VM ID the tiler heap should be mapped to */
|
||||
__u32 vm_id;
|
||||
|
||||
/** @initial_chunk_count: Initial number of chunks to allocate. */
|
||||
__u32 initial_chunk_count;
|
||||
|
||||
/** @chunk_size: Chunk size. Must be a power of two at least 256KB large. */
|
||||
__u32 chunk_size;
|
||||
|
||||
/** @max_chunks: Maximum number of chunks that can be allocated. */
|
||||
__u32 max_chunks;
|
||||
|
||||
/**
|
||||
* @target_in_flight: Maximum number of in-flight render passes.
|
||||
*
|
||||
* If the heap has more than tiler jobs in-flight, the FW will wait for render
|
||||
* passes to finish before queuing new tiler jobs.
|
||||
*/
|
||||
__u32 target_in_flight;
|
||||
|
||||
/** @handle: Returned heap handle. Passed back to DESTROY_TILER_HEAP. */
|
||||
__u32 handle;
|
||||
|
||||
/** @tiler_heap_ctx_gpu_va: Returned heap GPU virtual address returned */
|
||||
__u64 tiler_heap_ctx_gpu_va;
|
||||
|
||||
/**
|
||||
* @first_heap_chunk_gpu_va: First heap chunk.
|
||||
*
|
||||
* The tiler heap is formed of heap chunks forming a single-link list. This
|
||||
* is the first element in the list.
|
||||
*/
|
||||
__u64 first_heap_chunk_gpu_va;
|
||||
};
|
||||
|
||||
/**
|
||||
* struct drm_panthor_tiler_heap_destroy - Arguments passed to DRM_IOCTL_PANTHOR_TILER_HEAP_DESTROY
|
||||
*/
|
||||
struct drm_panthor_tiler_heap_destroy {
|
||||
/** @handle: Handle of the tiler heap to destroy */
|
||||
__u32 handle;
|
||||
|
||||
/** @pad: Padding field, MBZ. */
|
||||
__u32 pad;
|
||||
};
|
||||
|
||||
#if defined(__cplusplus)
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* _PANTHOR_DRM_H_ */
|
||||
Loading…
Add table
Add a link
Reference in a new issue