Topic: Encode error  (Read 3719 times)

Encode error
« on: March 09, 2018, 06:52:55 PM »

0scar

  • Code Baker
  • Chef
  • ***
  • Date Registered: Jan 2013
  • Posts: 309
System.OutOfMemoryException error while encode.
The 7z files  67MB (200MB decompressed) seems to be too large.
Is there a max. file size?

Code: [Select]
<Errors>
- [1] Error in Script [Encode files.7z] (Win7PESE\0scars\Apps\Supplementary\MultiMedia\SMplayerPortable\7z\files7z.script)
[Error] Encode - Exception of type 'System.OutOfMemoryException' was thrown. (Encode,%ScriptDir%\%ScriptNew%,carpeta,%ScriptDir%\%SetupFilex86%) (Line 26)


Re: Encode error
« Reply #1 on: March 09, 2018, 07:54:14 PM »

Atari800xl

  • Code Baker
  • Sr. Chef
  • ****
  • Date Registered: Feb 2013
  • Posts: 827
Thanks for reporting, Oscar, I notified the developer on GitHub.

Re: Encode error
« Reply #2 on: March 10, 2018, 02:50:05 AM »

Homes32

  • Code Baker
  • Chef
  • ***
  • Date Registered: Dec 2010
  • Posts: 177
how much RAM does your computer have and are you running a x32 or x64 OS?

I've tested with files up to 800MB in size with no issues, but I have a pretty beefy rig.

Re: Encode error
« Reply #3 on: March 10, 2018, 09:04:27 AM »

ied206

  • Moderator,
  • Chef
  • *****
  • Location: South Korea
  • Date Registered: Jan 2014
  • Posts: 257
0scar, does your system have low memory?

PEBakery roughly takes three steps when encoding files:
1. Compress file into ZLibStream (Similar to zip compress)
2. Encode ZLibStream into base64 string
3. Split base64 string into fit in .script multiline format (WBZip)

Let's say a file's size is n Byte. Each step requires a buffer able to hold specific size.
Step1 : n or less (since it is compressed)
Step2 : 3n (Base64 have a overhead of 33% and .Net stores string in UTF16, doubling the size. 1.33n * 2 => 2.7n)
Step3 : 3n (Same with step2)

So in theory, PEBakery requires roughly memory 7 times of file size.
I always wanted to optimize this, but handling WBZip format is tricky and was delayed since other features had more priority.
« Last Edit: March 10, 2018, 09:05:50 AM by ied206, Reason: grammar »

Re: Encode error
« Reply #4 on: March 10, 2018, 03:00:47 PM »

0scar

  • Code Baker
  • Chef
  • ***
  • Date Registered: Jan 2013
  • Posts: 309
My computer has 4GB 

Task manager memory numbers (W7, 32bits):
Total 3300
Cached 1700
Available 2600
Free  +-600 (this is the problem, it goes to cero fast when the plugin runs )

With w10 the memory numbers are different, there is no Free section and it is not easy to see what part goes wrong.

Re: Encode error
« Reply #5 on: March 13, 2018, 12:41:30 PM »

0scar

  • Code Baker
  • Chef
  • ***
  • Date Registered: Jan 2013
  • Posts: 309
I have found another issue with Encode.
Encode does not work with scripts (Unable to load script error) located in folders above the folder PEBakery is, but it can load the script fine if it is in any folder under the folder PEBakery is.


Code: [Select]
<Errors>
- [1] Error in Script [Encode test file.7z] (Win8.1SE\0scars\Utils\oscar packed\files7.script)
[Error] Encode - Unable to load script [E:\test\\new.script] (Encode,%FileBox2%\%ScriptName%,Folder,%FileBox2%\%SetupFilex86%) (Line 97)

Re: Encode error
« Reply #6 on: March 15, 2018, 06:39:45 AM »

ied206

  • Moderator,
  • Chef
  • *****
  • Location: South Korea
  • Date Registered: Jan 2014
  • Posts: 257
I am optimizing memory requirement of Encode/Decode command.

Code: [Select]
Encode does not work with scripts (Unable to load script error) located in folders above the folder PEBakery is, but it can load the script fine if it is in any folder under the folder PEBakery is.
PEBakery only accepts .script file under %BaseDir% in Run/Exec command, it seems that limitation also applied into Encode/Decode without notice.

Re: Encode error
« Reply #7 on: March 17, 2018, 03:44:42 PM »

ied206

  • Moderator,
  • Chef
  • *****
  • Location: South Korea
  • Date Registered: Jan 2014
  • Posts: 257
I have optimized Encode command's memory usage, please try lastest nightly build.

Re: Encode error
« Reply #8 on: March 17, 2018, 06:34:37 PM »

0scar

  • Code Baker
  • Chef
  • ***
  • Date Registered: Jan 2013
  • Posts: 309
All old issues solved, no problem with 7z file size when encoding  and Pencil edits files or opens all folders fine.  :thumbsup:
But there is a new issue:
Code: [Select]
<Errors>
- [1] Error in Script [SMplayer 7z] (Win10PESE\MyPlugins_Direct\0scars\Apps\Supplementary\MultiMedia\SMplayerPortable\SMplayer.script)
[Error] ExtractFile - Invalid length for a Base-64 char array or string. (ExtractFile,%ScriptFile%,Folder,%SetupFile%,%TargetDir%\Program Files\%ProgramFolder%) (Line 38)

Request:
May I suggest you to change the Pencil tootip to Edit Script or Open Folder?
Though I think a new dedicated button Open Script Folder  might be better.

Re: Encode error
« Reply #9 on: March 18, 2018, 11:38:42 AM »

ied206

  • Moderator,
  • Chef
  • *****
  • Location: South Korea
  • Date Registered: Jan 2014
  • Posts: 257
Can you upload a file SMPlayer.7z so I can investigate the issue?

Request:
May I suggest you to change the Pencil tootip to Edit Script or Open Folder?
Though I think a new dedicated button Open Script Folder  might be better.
I changed small buttons' behavior when a folder is selected. Please download lastest version again and test.

Re: Encode error
« Reply #10 on: March 18, 2018, 02:03:15 PM »

0scar

  • Code Baker
  • Chef
  • ***
  • Date Registered: Jan 2013
  • Posts: 309

Re: Encode error
« Reply #11 on: March 18, 2018, 03:04:54 PM »

ied206

  • Moderator,
  • Chef
  • *****
  • Location: South Korea
  • Date Registered: Jan 2014
  • Posts: 257
I found a line 4096 of attachment is missing:
ila_rendered
Strange enough, it gets properly extracted with WinBuilder.

EDIT
Line 8196 and 12288 is empty too, I guess it is done by purpose.
« Last Edit: March 18, 2018, 03:17:04 PM by ied206 »

Re: Encode error
« Reply #12 on: March 18, 2018, 03:24:37 PM »

ied206

  • Moderator,
  • Chef
  • *****
  • Location: South Korea
  • Date Registered: Jan 2014
  • Posts: 257
When I encode this file in WB082, there are no empty lines in attachment.
Which version of WinBuilder was used to create this .script file?

Re: Encode error
« Reply #13 on: March 18, 2018, 03:33:42 PM »

0scar

  • Code Baker
  • Chef
  • ***
  • Date Registered: Jan 2013
  • Posts: 309
it gets properly extracted with PEBakery  Beta3 too
The script was created with  your last nightly build
I have created it with my own tool but if created with Pc Packed it has the same error.

Re: Encode error
« Reply #14 on: March 18, 2018, 03:59:44 PM »

ied206

  • Moderator,
  • Chef
  • *****
  • Location: South Korea
  • Date Registered: Jan 2014
  • Posts: 257
Then it should be a regression caused by recent optimization.

Re: Encode error
« Reply #15 on: March 18, 2018, 05:51:22 PM »

ied206

  • Moderator,
  • Chef
  • *****
  • Location: South Korea
  • Date Registered: Jan 2014
  • Posts: 257
This issue only appears if a compressed file is larget than about 16MB.
Please re-encode the file and try extracting with new nightly build. It should have fixed the issue.
It also improved error checking in ExtractFile, so it should log "Encoded lines are malformed" error in corrupted script (one created with last nightly bulid)


Re: Encode error
« Reply #16 on: March 18, 2018, 07:39:36 PM »

0scar

  • Code Baker
  • Chef
  • ***
  • Date Registered: Jan 2013
  • Posts: 309
Success   :worship:



But now...
Code: [Select]
<Errors>
- [1] Error in Script [9 - Autorun] (Win10PESE\Build\9-Autorun.script)
[Error] ExtractAllFiles - Encoded lines are malformed (ExtractAllFiles,%ScriptFile%,%SourceArch%,%Target_Sys%) (Line 51)
« Last Edit: March 18, 2018, 10:27:48 PM by 0scar »

Re: Encode error
« Reply #17 on: March 20, 2018, 01:42:12 PM »

ied206

  • Moderator,
  • Chef
  • *****
  • Location: South Korea
  • Date Registered: Jan 2014
  • Posts: 257
Strange enough, encoded files of '9-Autorun.script' have 1024 chars per line (opposed to 4090 chars of others).
I tried to fix this, test agian with lastest nightly build.
« Last Edit: March 20, 2018, 03:07:55 PM by ied206 »

Re: Encode error
« Reply #18 on: March 20, 2018, 06:41:04 PM »

0scar

  • Code Baker
  • Chef
  • ***
  • Date Registered: Jan 2013
  • Posts: 309
Solved!  :thumbsup:

Re: Encode error
« Reply #19 on: March 24, 2018, 03:31:34 PM »

0scar

  • Code Baker
  • Chef
  • ***
  • Date Registered: Jan 2013
  • Posts: 309

Code: [Select]
Encode does not work with scripts (Unable to load script error) located in folders above the folder PEBakery is, but it can load the script fine if it is in any folder under the folder PEBakery is.
PEBakery only accepts .script file under %BaseDir% in Run/Exec command, it seems that limitation also applied into Encode/Decode without notice.

ExtractAllFiles has the same limitation:

Code: [Select]
<Errors>
- [1] Error in Script [Extract From Scripts] (Win10PESE\MyPlugins_Direct\0scars\Utils\oscar packed\extract.script)
[Error] ExtractAllFiles - Unable to load script [E:\Win81PE\Projects\Win8.1SE\Apps\HD Tasks\MBR\BootIce.script] (ExtractAllFiles,%FileBox3%,Folder,%FileBox2%) (Line 26)

 

Powered by EzPortal