Chief PluginCache

Available at
Win10PESE\Build\'Chief PluginCache'
Gena\Basic\'Chief PluginCache'

'Chief PluginCache' is a plugin (+2)
 That controls Cache for Basic and Components plugins.

Naturally, end user do not much change settings on
 Basic and Components plugins
    during the creation of PE .iso builds. 


Max speed so far with Default Builds:
Win10PESE - 2 minutes 21 seconds
Gena - 4 minutes 29 seconds


Max speed so far with Basic Builds:
Win10PESE - 1 minute 9 seconds
Gena - 55 seconds

Basic Builds: Use Basic Project plugin: http://TheOven.org/index.php?topic=1876.0

Junction plugin used with SoftPerfect RamDrive http://TheOven.org/index.php?topic=1198


Reminder:
 Currently use Utils\Update XXX (Exact and Secure)-->Update button
   to update all plugins from servers,
     'Chief PluginCache' only works fine with latest plugins.

More Info.

Many plugins already have internal cache,
most noticable ones:

SE:
Build\Retrieve Tools
Build\'1 - Files' (new)
Build\'2 - Shell & Config'

Gena:
Basic\Build\: 'saydin77 Core'
Drivers\3 Network\'DPs LAN Drivers'
Drivers\3 Network\'DPs WLAN Drivers'
Drivers\4 Storage\'Build DPs Mass Storage Drivers Plugin'

Also reminding, mkisof use cache to speed up .iso creation.



As a result,
(normal disk - no junction -> any new user (AV disabled or D:\Oven\ folder added exclusion list))

First Default Build takes time,
Win10PESE - 15 minutes 5 seconds
Gena - 19 minutes 5 seconds

Second Build Takes less time
Win10PESE - 4 minutes 38 seconds
Gena - 5 minutes 28 seconds



Further with 2nd and later builds:

When Cache disabled on 2 plugins ('Chief PluginCache':-Basic - Components ---- - '1 files'/'saydin77 Core': disable cache):
Win10PESE - 10 minutes 47 seconds
Gena - 8 minutes 59 seconds

-----> regular or new users gain ~ x 2 speed out of box with 'Chief PluginCache' 

With Junction plugin to RamDrive:
Win10PESE - 2 minutes 21 seconds
Gena - 4 minutes 29 seconds

-----> Junction plugin users gain SE: ~ x 2 speed / Gena: ~ x 0.2 speed with 'Chief PluginCache' 


Tip / More Info :
Junction plugin speed gain only at PostConfig/CreateISO AND  'Chief PluginCache' backup/restore
( Gena also with RF special builds at virtualization step)

*
+ mkisof cache can be broken in time - with Junction it can never get broken 
--> This makes PostConfig/CreateISO steps always work fastest ignoring mkisof cache broken.
------> good when building many .iso on trials.
+ Also this helps other PostConfig/CreateISO options (compress higher, cdimage, oscdimg) works fastest possible.

===> so Total Time gain with Junction plugin is much more than documented here 
 which is very important for limited hobby time.


SE Default : 4 minutes 38 seconds  --> 2 minutes 21 seconds :
   Junction plugin gain 2 minutes, rest is related to amount of post Components plugins and size of build.
     Speed gain comes from big size of SE + small number of post Components plugins, compared to Gena
Gena Default: 5 minutes 28 seconds --> 4 minutes 29 seconds
   Junction plugin gain 1 minute, rest is related to amount of post Components plugins and size of build.
     Gena have many plugins after Components compared to SE
      and smaller size compared to SE have less effect on Junction plugin.
        Still Speed gain %20 is quite noticable.


Here
Junction plugin used with SoftPerfect RamDrive http://TheOven.org/index.php?topic=1198


Basic Builds:
(Junction plugin used)
Basic Builds: Use Basic Project plugin: http://TheOven.org/index.php?topic=1876.0

Win10PESE - 1 minute 23 seconds
Gena - 1 minute 8 seconds

Basic Builds + ('Chief PluginCache' -> Disable only Components Cache) :
Win10PESE - 1 minute 9 seconds
Gena - 55 seconds


 

Reminder:
Basic Builds helps
+ development of Post Basic plugins (Apps, Components, Drivers,.... plugins and etc.) , limited hobby time....
+ Faster test result to get from end users on a case of debug.





****
More Info:
'Chief PluginCache' by its nature also helps on winbuilder ram bug that cause winbuilder slow down during build time line.
'Chief PluginCache' also cause smaller log file, which will open on browser faster, this also helps developing Post Basic and Post Components plugins.
'Chief PluginCache' would work faster when project folder is at Fast SSD drive, not tested how significant but due to winbuilder bugs I do not expect big difference compared to Junction plugin builds.
Still SSD would be faster, I wonder how much faster. 2nd Default and Basic builds results would be good.   

Life is Fast, Have fun ...

'Chief PluginCache' Currently available at
Gena
Win10PESE
Win8.1SE
Win8PESE
Win7PESE

Do not try this plugin at other projects till it is available by Project Admin, results would be unexpected.

UNDER CONSTRUCTION - reserved

Win7PESE test results:

*
- Source Win7 Ult x64
- Junction to RAM Drive NOT used
- Extracted Wim Folders used
- Only default Basic and Components plugins used + PostConfig + CreateISO

https://www.dropbox.com/s/abc1b0sgo3m0ive/Test_20170131_1439.7z?dl=0

9 min 18 Sec --> 3 min 52 Sec

***
with junction (I guess equivalent with ssd ! junction is faster.)

- Source Win7 Ult x64
- Junction to RAM Drive YES used
- Extracted Wim Folders used
- Only default Basic and Components plugins used + PostConfig + CreateISO

5 min 28 Sec --> 1 min 21 sec

***
with junction (I guess equivalent with ssd ! ) + only Basic plugins

- Source Win7 Ult x64
- Junction to RAM Drive YES used
- Extracted Wim Folders used
- Only default Basic plugins used + PostConfig + CreateISO

--> 1 min 10 sec








*
SSD test result from Bob.Omb:

New Updated PESE default build - w/SSD - build creation times:

Use extracted WIM is now default setting

With Chief Plugin Cache enabled....

1st Build: 13 minutes 2 seconds (Extraction of WIM for future builds)

2nd Build: 2 minutes 17 seconds

Junction test result from Lancelot (2018.11.15):
1st: 17 minutes, 29 seconds and 683 milliseconds
2nd: 3 minutes, 37 seconds and 163 milliseconds
Source Win10_1809_English_x64.iso (I believe Bob.Omb use same source)

How does the plugin cache work?  I know if we add the two lines at the beginning of process section we can add it like I did in IE plugin..for build, shell, and component sections

Code: [Select]

[Process]
If,ExistFile,%ProjectTemp%\Components_PluginCache_Extract.ini,Echo,"Components PluginCache"
If,Not,ExistFile,%ProjectTemp%\Components_PluginCache_Extract.ini,Run,%ScriptFile%,Process_Normal

[Process_Normal]
*YourPluginHere

Magic 

The reason I am asking is, can we make a cache for AppsMy to speed build times up as our projects increase in size?  Trying to figure out how it works, does it check to see if the plugin has been changed? And if not uses the cached version and if so lets the plugin run?

If it is only checking to see if a cached version exists, can we change it to make it save the plugin itself to the cache as well as the post run content - then -compare- that compressed cached version of the actual plugin against the current version for any changes, and if there are changes then delete the cached version of the plugin and run the current plugin instead...?? (If this is how it already works then great  we should make an AppsMy version  )

Maybe add all the plugins that are a part of the cache to a single 7z separately then extract and compare at the beginning of cache plugin before build starts...

How does the plugin cache work?

I put Info button to "Chief PluginCache" plugin.
On server (you can use green download button on plugin)

I know if we add the two lines at the beginning of process section we can add it like I did in IE plugin..for build, shell, and component sections

On plugin side, if no "provide" option -> yes , but you also have to update "Chief PluginCache"

Code: [Select]

History070=Lancelot Add Components\Internet_Explorer_11.Script - Date:2018.10.15

+
To process faster during build time, "Chief PluginCache" holds info for relevant \Basic\ and \Components\ plugins

*
On plugin side, if there is "provide option"
plugin should also include more lines.
eg:
2 - Shell & Config -> newly added provide option

Code: [Select]

History073=Lancelot 2-Config plugin now have custom option - Bob.Omb Reply3 and later http://TheOven.org/index.php?topic=2613.msg28655#msg28655 - Date:2018.10.28

*

Magic 

All magic happens at "Chief PluginCache"
Maintaining (Add/Remove new plugins, or provide options) is very easy now.


*

The reason I am asking is, can we make a cache for AppsMy to speed build times up as our projects increase in size?  Trying to figure out how it works, does it check to see if the plugin has been changed? And if not uses the cached version and if so lets the plugin run?

Info button may help.
ps: Feel free to improve things written inside Info
You will notice Info button opens info at [Info] which is easier to update. 

--------------------------------------------------
History may help to understand better how it works.
+
Historically: Chief PluginCache is the continuation of "Backup Restore" plugin around the year 2011
 http://theoven.org/index.php?topic=87.0
The difference is,
 "Backup Restore" plugin need the end user to click "Restore" button,
 "Chief PluginCache" works automatically (that is the reason of "Cache" name)

+
"Backup Restore" plugin compatible to SE, available at Yomi.
   Sometimes It is more useful to test things for "Post Basic" Plugins.
    "Basic Project" have special features to use "Backup Restore" plugin to test "Post Basic" plugins.

--------------------------------------------------

*
Theoretically, we can make all plugins cache with the same method,
 but with ALL current builders, it is not possible for various reasons.

+
I had worked on another general way for "plugin standalone cache" 8 months ago I guess,
 but I see It is only useful for big packages and there are already solutions for that.
+
Again Theoretically, we can make "plugin standalone cache",
 but with ALL current builders it is not possible for various reasons.


*

If it is only checking to see if a cached version exists, can we change it to make it save the plugin itself to the cache as well as the post run content - then -compare- that compressed cached version of the actual plugin against the current version for any changes, and if there are changes then delete the cached version of the plugin and run the current plugin instead...?? (If this is how it already works then great  we should make an AppsMy version  )

We could, but scanning and checking all plugins will take a lot much time with Chief PluginCache.

eg. Gena have more \Basic\ and \Components\ plugins and takes a lot more time.

eg. Utils\Plugin Manager -> spends a lot time to scan all plugins to save their settings
--> and keep in mind It will be much slower if Plugin Manager written with wb, still with AutoIT it is slow...
-----> (That is the reason "Chief PluginCache" hold list of plugins)




*
After reading all current long post, It is now fun time for you I guess:

Maybe add all the plugins that are a part of the cache to a single 7z separately then extract and compare at the beginning of cache plugin before build starts...

Already available,
I add examples to \Components\"Additional Files" plugin which is cached.

when you make your personal \Components\"Additional Files" with shortcuts and requirements and adding files to folder (no need 7z)
you will see It will not be processed again.

I feel you will have a good time playing with \Components\"Additional Files"
Share your final results 

ps:
Feel free to improve examples inside plugin.
(I added quickly without testing)


***
I hope post clears all your questions. 

New Updated PESE default build - w/SSD - build creation times:

Use extracted WIM is now default setting

With Chief Plugin Cache enabled....

1st Build: 13 minutes 2 seconds (Extraction of WIM for future builds)

2nd Build: 2 minutes 17 seconds

Hi Bob.Omb

Thanks for the SSD results.

Here: Source Win10_1809_English_x64.iso
Using Junction plugin:
1st: 17 minutes, 29 seconds and 683 milliseconds
2nd: 3 minutes, 37 seconds and 163 milliseconds

SSD is faster. 


After 2 years, It is also good to see

The 17000 line Macro Library (ML) do NOT have a significant effect on total build speed. 



edit: 2019.03.01 test results:
Extracted wim folders - Win10 1809 x64 enu:
with normal hd:
29 minutes, 35 seconds and 344 milliseconds
9 minutes, 33 seconds and 829 milliseconds
with junction plugin:
15 minutes, 654 milliseconds
5 minutes, 15 seconds and 905 milliseconds

In passing time number and size of plugins increase a lot, speed gain still very good.