IllusionMods/HSPlugins
Archived
1
0
Fork 2
Code Issues 2 Pull requests 1 Projects Releases 1 Packages Wiki Activity Actions

feature/ISSUE-1-The-recording-method-is-very-inefficient #178

Open
SangMan_LINUX wants to merge 45 commits from SangMan_LINUX/HSPlugins:feature/ISSUE-1-The-recording-method-is-very-inefficient into main
pull from: SangMan_LINUX/HSPlugins:feature/ISSUE-1-The-recording-method-is-very-inefficient
merge into: IllusionMods:main
Conversation 4 Commits 45 Files changed 31 +1715 -323
SangMan_LINUX commented 2025-12-11 23:03:51 +00:00
First-time contributor
Copy link

Hello there,

By using in-memory pipeline technique, encoding time has been drastically reduced.

Bypassing GC occurrences except for KK, we pushed it to the limit as much as possible.

We have maintained maximum compatibility.

We have updated the required libraries.

In addition, we have supplemented the shortcomings.

Sincerely,

Hello there, By using in-memory pipeline technique, encoding time has been drastically reduced. Bypassing GC occurrences except for KK, we pushed it to the limit as much as possible. We have maintained maximum compatibility. We have updated the required libraries. In addition, we have supplemented the shortcomings. Sincerely, 
- VideoExport.Core\Extensions\GIFExtension.cs, Modified ffmpeg based gif code.
- VideoExport.Core\Extensions\GIFExtension.cs, Removed unnecessary comments.
- Regarding videoexport, updated ffmpeg (7.1.3-14) and gifski (1.32.0).
- Replaced with gifski which has ffmpeg built-in.
- VideoExport.Core\Extensions\WEBPExtension.cs, Have adapted the code for the webp format.
- VideoExport.Core\Extensions\GIFExtension.cs, Addressed the changed video filter format. Removed unnecessary frame values.
- VideoExport.Core\Extensions\MOVExtension.cs, Addressed the changed video filter format.
- VideoExport.Core\Extensions\MP4Extension.cs, Addressed the changed video filter format.
- VideoExport.Core\Resources\English.xml, Added parameters to gifski.
- VideoExport.Core\VideoExport.cs, Fixed an issue that occurred when there was no resize in gifski.
- FFmpeg, Replaced with the final custom build and also replaced the build options file accordingly.
ManlyMarco changed title from feature/ISSUE-1-The-recording-method-is-very-inefficient to WIP: feature/ISSUE-1-The-recording-method-is-very-inefficient 2025-12-13 19:42:09 +00:00
mrspuff commented 2025-12-14 14:33:00 +00:00
Contributor
Copy link

I've done something similar with raw textures, on a way smaller scale, on my local version of VE, and I've encountered the same issue with the GC. I haven't tried this PR, but if it works in the same way, it's basically half second of encoding and half second of running the GC, so you get half of the potential speed.

Seeing this PR, i thought finally someone had managed to work around the horrible GetRawTextureData in Unity 5, but unfortunately it looks like you hit the same wall.

So, since now it's possible that the official plugin will start supporting raw textures in the encoding pipeline, i tried again to look for a workaround, and I finally found one at https://meetemq.com/2023/01/14/using-pointers-in-c-unity/.

It's quite hacky and involves following pointers in memory, basically what GetRawTextureData does itself, but it allows a similar handling to the one used for the newer NativeArray-version of GetRawTextureData, like you're doing in GetNativeRawData with _frameDataBuffer.

If you have the time, you could integrate this handling for the KK case, otherwise I'll try to add it once this PR gets merged.

It shouldn't matter here since KK is the only one without native handling, but it's an important note: this is not guaranteed to work with other games/Unity versions, as I used the decompilation of GetRawTextureData from CharaStudio (Unity 5.6.2f1) as reference.

This should be all you need:

Put in a standalone class:

public static class PointerLib //https://meetemq.com/2023/01/14/using-pointers-in-c-unity/
{
    [MethodImpl(256)] //https://stackoverflow.com/a/43060488/23244567
    public unsafe static void* GetInternalPointer(this object obj)
    {
        return *(void**)((ulong)*(IntPtr*)&obj + (ulong)(sizeof(IntPtr) * 2));
    }
}

Put in VideoExport or somewhere else where Logger is accessible:

[StructLayout(LayoutKind.Sequential)]
unsafe struct TextureAccess
{
    public fixed byte otherData[80];
    public TextureData* texData;
}

[StructLayout(LayoutKind.Sequential)]
unsafe struct TextureData
{
    public fixed byte monoHeader[16];
    public byte* tex;
    public fixed byte otherData[40];
    public ulong size;
}


public static unsafe byte[] GetRawTextureData(Texture2D tex, byte[] textureBytes)
{
    //pointer access can be flaky sometimes, so we try multiple times
    const int maxAttempts = 10;
    if (tex == null)
        return null;

    for (int attempt = 0; attempt < maxAttempts; attempt++)
    {
        try
        {
            byte[] result = GetRawTextureDataInternal(tex, textureBytes);
            if (result != null)
                return result;
        }
        catch (Exception e)
        {
            Logger.LogError("Exception in GetRawTextureDataInternal: " + e);
        }
    }

    Logger.LogError("GetRawTextureDataInternal failed after " + maxAttempts + " attempts.");
    return null;
}

private static unsafe byte[] GetRawTextureDataInternal(Texture2D tex, byte[] textureBytes)
{
    var texPtr = (TextureAccess*)PointerLib.GetInternalPointer(tex);

    if (texPtr == null)
        return null;

    TextureData* data = texPtr->texData;
    if (data == null)
    {
        Logger.LogError("TextureData pointer is null.");
        return null;
    }

    byte* nativeBuffer = data->tex;
    if (nativeBuffer == null)
    {
        Logger.LogError("Native texture buffer pointer is null.");
        return null;
    }

    ulong nativeSize = data->size;
    if (nativeSize == 0 || nativeSize > int.MaxValue)
    {
        Logger.LogError($"Invalid native texture size: {nativeSize}");
        return null;
    }

    if(textureBytes == null || textureBytes.Length != (int)nativeSize)
    {
        textureBytes = new byte[nativeSize];
    }
    Marshal.Copy(new IntPtr(nativeBuffer), textureBytes, 0, (int)nativeSize);
    return textureBytes;
}

For completion, here's the pseudocode from ghidra for GetRawTextureData, with a minimal set of renames needed to understand a little more:


undefined8 GetRawTextureData(longlong texAddr)

{
  code *pcVar1;
  char cVar2;
  longlong lVar3;
  void *_Dst;
  undefined8 uVar4;
  undefined4 uVar5;
  size_t _Size;
  void *_Src;
  undefined8 local_res8;
  longlong *texture;
  
  if ((texAddr == 0) || (*(longlong *)(texAddr + 16) == 0)) {
    ThrowNullReferenceException();
    pcVar1 = (code *)swi(3);
    uVar4 = (*pcVar1)();
    return uVar4;
  }
  lVar3 = *(longlong *)(*(longlong *)(texAddr + 16) + 80);
  if (lVar3 == 0) {
    _Size = 0;
  }
  else {
    _Size = *(size_t *)(lVar3 + 64);
  }
  lVar3 = FUN_1408398a0();
  NewMonoArray(&local_res8,*(undefined8 *)(lVar3 + 256),1,(longlong)(int)_Size);
  texture = *(longlong **)(texAddr + 16);
  if (texture != (longlong *)0) {
    _Src = (void *)texture[10];
    if ((_Src == (void *)0) || (_Src = *(void **)((longlong)_Src + 16), _Src == (void *)0)) {
      if (texture == (longlong *)0) {
        ThrowNullReferenceException(texAddr);
        pcVar1 = (code *)swi(3);
        uVar4 = (*pcVar1)();
        return uVar4;
      }
      cVar2 = (**(code **)(*texture + 320))();
      if (cVar2 == '\0') {
        if (*(longlong *)(texAddr + 16) == 0) {
          uVar5 = 0;
        }
        else {
          uVar5 = *(undefined4 *)(*(longlong *)(texAddr + 16) + 8);
        }
        FUN_1404a2800("Texture needs to be marked as Read/Write to be able to GetRawTextureData in player"
                      ,0,&DAT_1410bc26c,512,1,uVar5,0,0);
        return local_res8;
      }
    }
    _Dst = (void *)GetPointerToMonoArrayElement(local_res8,0,1);
    memcpy(_Dst,_Src,_Size);
    return local_res8;
  }
  ThrowNullReferenceException(texAddr);
  pcVar1 = (code *)swi(3);
  uVar4 = (*pcVar1)();
  return uVar4;
}
I've done something similar with raw textures, on a way smaller scale, on my local version of VE, and I've encountered the same issue with the GC. I haven't tried this PR, but if it works in the same way, it's basically half second of encoding and half second of running the GC, so you get half of the potential speed. Seeing this PR, i thought finally someone had managed to work around the horrible GetRawTextureData in Unity 5, but unfortunately it looks like you hit the same wall. So, since now it's possible that the official plugin will start supporting raw textures in the encoding pipeline, i tried again to look for a workaround, and I finally found one at https://meetemq.com/2023/01/14/using-pointers-in-c-unity/. It's quite hacky and involves following pointers in memory, basically what GetRawTextureData does itself, but it allows a similar handling to the one used for the newer NativeArray-version of GetRawTextureData, like you're doing in GetNativeRawData with _frameDataBuffer. If you have the time, you could integrate this handling for the KK case, otherwise I'll try to add it once this PR gets merged. It shouldn't matter here since KK is the only one without native handling, but it's an important note: **this is not guaranteed to work with other games/Unity versions**, as I used the decompilation of GetRawTextureData from CharaStudio (Unity 5.6.2f1) as reference. This should be all you need: Put in a standalone class: ```csharp public static class PointerLib //https://meetemq.com/2023/01/14/using-pointers-in-c-unity/ { [MethodImpl(256)] //https://stackoverflow.com/a/43060488/23244567 public unsafe static void* GetInternalPointer(this object obj) { return *(void**)((ulong)*(IntPtr*)&obj + (ulong)(sizeof(IntPtr) * 2)); } } ``` Put in VideoExport or somewhere else where Logger is accessible: ```csharp [StructLayout(LayoutKind.Sequential)] unsafe struct TextureAccess { public fixed byte otherData[80]; public TextureData* texData; } [StructLayout(LayoutKind.Sequential)] unsafe struct TextureData { public fixed byte monoHeader[16]; public byte* tex; public fixed byte otherData[40]; public ulong size; } public static unsafe byte[] GetRawTextureData(Texture2D tex, byte[] textureBytes) { //pointer access can be flaky sometimes, so we try multiple times const int maxAttempts = 10; if (tex == null) return null; for (int attempt = 0; attempt < maxAttempts; attempt++) { try { byte[] result = GetRawTextureDataInternal(tex, textureBytes); if (result != null) return result; } catch (Exception e) { Logger.LogError("Exception in GetRawTextureDataInternal: " + e); } } Logger.LogError("GetRawTextureDataInternal failed after " + maxAttempts + " attempts."); return null; } private static unsafe byte[] GetRawTextureDataInternal(Texture2D tex, byte[] textureBytes) { var texPtr = (TextureAccess*)PointerLib.GetInternalPointer(tex); if (texPtr == null) return null; TextureData* data = texPtr->texData; if (data == null) { Logger.LogError("TextureData pointer is null."); return null; } byte* nativeBuffer = data->tex; if (nativeBuffer == null) { Logger.LogError("Native texture buffer pointer is null."); return null; } ulong nativeSize = data->size; if (nativeSize == 0 || nativeSize > int.MaxValue) { Logger.LogError($"Invalid native texture size: {nativeSize}"); return null; } if(textureBytes == null || textureBytes.Length != (int)nativeSize) { textureBytes = new byte[nativeSize]; } Marshal.Copy(new IntPtr(nativeBuffer), textureBytes, 0, (int)nativeSize); return textureBytes; } ``` For completion, here's the pseudocode from ghidra for GetRawTextureData, with a minimal set of renames needed to understand a little more: ```cpp undefined8 GetRawTextureData(longlong texAddr) { code *pcVar1; char cVar2; longlong lVar3; void *_Dst; undefined8 uVar4; undefined4 uVar5; size_t _Size; void *_Src; undefined8 local_res8; longlong *texture; if ((texAddr == 0) || (*(longlong *)(texAddr + 16) == 0)) { ThrowNullReferenceException(); pcVar1 = (code *)swi(3); uVar4 = (*pcVar1)(); return uVar4; } lVar3 = *(longlong *)(*(longlong *)(texAddr + 16) + 80); if (lVar3 == 0) { _Size = 0; } else { _Size = *(size_t *)(lVar3 + 64); } lVar3 = FUN_1408398a0(); NewMonoArray(&local_res8,*(undefined8 *)(lVar3 + 256),1,(longlong)(int)_Size); texture = *(longlong **)(texAddr + 16); if (texture != (longlong *)0) { _Src = (void *)texture[10]; if ((_Src == (void *)0) || (_Src = *(void **)((longlong)_Src + 16), _Src == (void *)0)) { if (texture == (longlong *)0) { ThrowNullReferenceException(texAddr); pcVar1 = (code *)swi(3); uVar4 = (*pcVar1)(); return uVar4; } cVar2 = (**(code **)(*texture + 320))(); if (cVar2 == '\0') { if (*(longlong *)(texAddr + 16) == 0) { uVar5 = 0; } else { uVar5 = *(undefined4 *)(*(longlong *)(texAddr + 16) + 8); } FUN_1404a2800("Texture needs to be marked as Read/Write to be able to GetRawTextureData in player" ,0,&DAT_1410bc26c,512,1,uVar5,0,0); return local_res8; } } _Dst = (void *)GetPointerToMonoArrayElement(local_res8,0,1); memcpy(_Dst,_Src,_Size); return local_res8; } ThrowNullReferenceException(texAddr); pcVar1 = (code *)swi(3); uVar4 = (*pcVar1)(); return uVar4; } ```
SangMan_LINUX commented 2025-12-14 20:09:42 +00:00
Author
First-time contributor
Copy link

Hello there, @mrspuff .

First of all, thank you for sharing your valuable knowledge with me, even though my brain cannot grasp it.

The unfortunate thing is that I don't have KK so I have no way to test it.

Also more importantly, as of now, I am not confident that I can handle "unsafe" keyword with my abilities.

So, I would like to prioritize maintaining the status quo in PR right now.

Sincerely,

Hello there, @mrspuff . First of all, thank you for sharing your valuable knowledge with me, even though my brain cannot grasp it. The unfortunate thing is that I don't have KK so I have no way to test it. Also more importantly, as of now, I am not confident that I can handle "unsafe" keyword with my abilities. So, I would like to prioritize maintaining the status quo in PR right now. Sincerely,
mrspuff commented 2025-12-14 21:47:05 +00:00
Contributor
Copy link

No problem, I'll wait until this gets merged and handle it myself when I can. Thanks for the great work!

No problem, I'll wait until this gets merged and handle it myself when I can. Thanks for the great work!
- VideoExport.Core\ScreenshotPlugins\ReshadePlugin.cs, Refactored Reshade to fit the current code structure.
- VideoExport.Core\VideoExport.cs, Fixed an issue where animation on the first frame would break when timeline constraints were applied. Added code corresponding to the reshade code.
- VideoExport.Core\TextureEncoder.cs, Corresponded to the format in which PNG is saved according to transparency.
- VideoExport.Core\VideoExport.cs,Corresponded to the format in which PNG is saved according to transparency.
- VideoExport.Core\Extensions\MKVExtension.cs, Added mkv container corresponding to FFV1 codec.
- VideoExport.Core\Resources\English.xml, Added text corresponding to the FFV1 codec.
- VideoExport.Core\VideoExport.cs, Added mkv container corresponding to FFV1 codec. Disabled parallelScreenshotEncoding in UI.
- FFmpeg, A custom building ffmpeg for FFV1 and preserved the build options in a file.
- VideoExport.Core\VideoExport.Core.projitems, Added MKV extension file.
- VideoExport.Core\VideoExport.cs, Upgraded the version.
- VideoExport.Core\Extensions\GIFExtension.cs, Modified the code to accommodate the introduction of AsyncGPUReadback.
- VideoExport.Core\Extensions\MKVExtension.cs, Modified the code to accommodate the introduction of AsyncGPUReadback.
- VideoExport.Core\Extensions\MOVExtension.cs, Modified the code to accommodate the introduction of AsyncGPUReadback.
- VideoExport.Core\Extensions\MP4Extension.cs, Modified the code to accommodate the introduction of AsyncGPUReadback.
- VideoExport.Core\Extensions\WEBMExtension.cs, Modified the code to accommodate the introduction of AsyncGPUReadback.
- VideoExport.Core\Extensions\WEBPExtension.cs, Modified the code to accommodate the introduction of AsyncGPUReadback.
- VideoExport.Core\ScreenshotPlugins\Builtin.cs, Modified the code to accommodate the introduction of AsyncGPUReadback.
- VideoExport.Core\ScreenshotPlugins\IScreenshotPlugin.cs, Modified the code to accommodate the introduction of AsyncGPUReadback.
- VideoExport.Core\ScreenshotPlugins\ReshadePlugin.cs, Modified the code to accommodate the introduction of AsyncGPUReadback.
- VideoExport.Core\ScreenshotPlugins\ScreencapPlugin.cs, Modified the code to accommodate the introduction of AsyncGPUReadback.
- VideoExport.Core\ScreenshotPlugins\Win32Plugin.cs, Modified the code to accommodate the introduction of AsyncGPUReadback.
- VideoExport.Core\VideoExport.cs, Modified the code to accommodate the introduction of AsyncGPUReadback.
- VideoExport.Core\VideoExport.cs, Fixed deadlock issue in KK version.
- VideoExport.Core\Extensions\AVIFExtension.cs, Added code to flexibly respond to pixel formats for all game environments.
- VideoExport.Core\Extensions\GIFExtension.cs, Added code to flexibly respond to pixel formats for all game environments.
- VideoExport.Core\Extensions\IExtention.cs, Added code to flexibly respond to pixel formats for all game environments.
- VideoExport.Core\Extensions\MKVExtension.cs, Added code to flexibly respond to pixel formats for all game environments.
- VideoExport.Core\Extensions\MOVExtension.cs, Added code to flexibly respond to pixel formats for all game environments.
- VideoExport.Core\Extensions\MP4Extension.cs, Added code to flexibly respond to pixel formats for all game environments.
- VideoExport.Core\Extensions\WEBMExtension.cs, Added code to flexibly respond to pixel formats for all game environments.
- VideoExport.Core\Extensions\WEBPExtension.cs, Added code to flexibly respond to pixel formats for all game environments.
- VideoExport.Core\VideoExport.cs, Updated version. Added code to flexibly respond to pixel formats for all game environments. In KK, have temporarily addressed the issue of the game freezing when recording video.
Reviewed-on: SangMan_LINUX/HSPlugins#1
Reviewed-by: SangMan_LINUX <sangman_linux@noreply.localhost>
- VideoExport.Core\Styles.cs, Temporarily fixed UI compatibility issues in KK.
- VideoExport.Core\VideoExport.Core.projitems, Removed unnecessary items.
- VideoExport.Core\VideoExport.cs, Due to the changed UI, the feature to automatically delete photos after export is complete has been removed.
- VideoExport.Core\VideoExport.cs, Upgraded the version. Simplify the progress indicator. Implemented buffer pool and flow control. Decouple the ffmpeg input thread from the Unity main thread.
- VideoExport.Core\VideoExport.cs, Upgraded the version.
- VideoExport.Core\Resources\English.xml, Implemented to allow selection of hardware codec.
- VideoExport.Core\VideoExport.cs, Fixed a bug with the indicator progress.
- VideoExport.Core\Extensions\GIFExtension.cs, Fixed an issue where Bad file descriptor related errors were output to the log.
- VideoExport.Core\Extensions\MKVExtension.cs, Fixed an issue where Bad file descriptor related errors were output to the log.
- VideoExport.Core\Extensions\MOVExtension.cs, Fixed an issue where Bad file descriptor related errors were output to the log.
- VideoExport.Core\Extensions\MP4Extension.cs, Fixed an issue where Bad file descriptor related errors were output to the log.
- VideoExport.Core\Extensions\WEBMExtension.cs, Fixed an issue where Bad file descriptor related errors were output to the log.
- VideoExport.Core\Extensions\WEBPExtension.cs, Fixed an issue where Bad file descriptor related errors were output to the log.
- VideoExport.Core\VideoExport.cs, Updated version.
Reviewed-on: SangMan_LINUX/HSPlugins#2
Reviewed-by: SangMan_LINUX <sangman_linux@noreply.localhost>
- VideoExport.Core\Resources\English.xml, Implemented AV1 codec in MP4 container to support both software and hardware encoders.
- VideoExport.Core\VideoExport.cs, Implemented AV1 codec in MP4 container to support both software and hardware encoders. The output of the libsvtav1 codec has been moved from error to info, as intended.
Moved plugin, currentSIze and targetSize outside so we can reuse it.
We are now logging start frame capture with stats about the capture.
Added warnings for not returned textures from screenshot plugins.
Removed duplicate log for screenshot and video creation since it printed the same time.
We are now logging stats about the render after its completion in both async and sync mode.
Don't try to delete the frame directory if _autoGenerateVideo is false.
Adedd a function to display padded tables in console.
Reviewed-on: SangMan_LINUX/HSPlugins#3
Reviewed-by: SangMan_LINUX <sangman_linux@noreply.localhost>
SangMan_LINUX changed title from WIP: feature/ISSUE-1-The-recording-method-is-very-inefficient to feature/ISSUE-1-The-recording-method-is-very-inefficient 2025-12-26 16:17:29 +00:00
SangMan_LINUX force-pushed feature/ISSUE-1-The-recording-method-is-very-inefficient from d8514c9469 to a1a10732b1 2025-12-30 14:14:01 +00:00 Compare
ManlyMarco commented 2025-12-30 23:27:19 +00:00
Owner
Copy link

Discussion moved to https://github.com/IllusionMods/HSPlugins/pull/180

Discussion moved to https://github.com/IllusionMods/HSPlugins/pull/180
Commenting is not possible because the repository is archived.
Reviewers
No reviewers
Labels
Clear labels   
bug

Something isn't working

  
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
enhancement

New feature or request

  
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
invalid

This doesn't seem right

  
question

Further information is requested

  
wontfix

This will not be worked on

No labels
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
IllusionMods/HSPlugins!178
No description provided.