Topic: Writing 'WB Compatible Builder', questions about WinBuilder language (Read 9058 times)

Is there any opinion about language improvement?

A looooot.

To me most critical,
winbuilder use wrong kind of memory management,
very wrong log file which cause big log and + with bad memory management cause winbuilder hang ...

also winbuilder can not ignore encoded files during read of a plugin + bad memory management = cause slow down of big sized plugins during execution and during interface edits.
=> we use workaround with File Container plugins.
===> but initial execution of winbuilder cause too much time with increasing number of plugins following bad plugin management with bad memory management.

*

I think some commands should be deprecated or redesigned.

Some commands already deprecated.

When you search current project files, you will notice some commands (and some variables) never used.

http://theoven.org/index.php?topic=1465.0

ps:
I believe
'System,FILEREDIRECT,{OFF|ON}'
can also be depricated.

*

Any information and opinion about variables system is also helpful.

It is very same like cmd.exe

+ There are internal variables (int1)
you can see them at log file. (side by side with script.project variables)
--> lots of them deprecated since they are redundant.
---> search project files, you will notice lots of internal variables never used.
-----> design bug of winbuilder, developers add so many variables they cause slow down of process. (redundancy)

+ There are internal variables during execution of a plugin (int2)
%ScriptFile% %ScriptTitle% %ScriptDir% -> after executing a plugin automatically created
%RegDataType% %ExitCode% -> during some operations (%RegDataType% not used inside plugins as far as I know)

+ and other variables up to projects, some stored inside script.project and Macro Library.

....... Lots of to write ....

=====>
Keeping in mind, you are creating a batch engine like cmd.exe, rest is long to write but simple:
batch.exe starts and gets internal variables (int1)
when you select a project at left tree, batch.exe gets variables from script.project
when you big blue, all batch files executed one by one following execution order.
during batch file execution
-> for each batch file internal variables created (int2)
-> for each batch file [Variables] and than [Interface] values add to variables temporarly
-> than start at [Process]
-> after batch finishes, all batch variables cleans from memory,
-> at start of next plugin there is (int1) + script.project variables (+global variables if used inside project)

Above is simple summary,
there is global variables (only used by MacroLibrary) declared at script.project
there is also plugin Macros defined side by side with variables
...

I hope summary gives you a better picture on your design.

*

1. Button control
According to WB document, button control has operands up to 11.
However, in production, it has various length of operands.

documentation not updated regularly in past.
some interface elements have tooltips ,
executing a section to button added (only used at SE special plugins due to incompatibility)
and winbuilder automatically adds empty values.

You need to test things to figure out.

ps: during execution of a plugin, not all [Interface] elements adds to variables, only the useful ones.

*

2. UI Controls to be deprecated
I cannot find usage of CheckList in Win10PESE.
If there is no demand for this, can this be deprecated?

Ideas about new UI control is also good.

CheckList very rarely used,
here is last time I see:
Downloads\AppYGS\HD Tasks\Runtime Software ALL

*

Quote from: ied206 on March 26, 2017, 08:47:18 AM

3. Encoded File Format

as long as you can decode, I do not think there will be problem for compatibility.
during evolution lots of formats tested,
at that time there were programmer contributers, but like project developers they were kicked out from development.

****
JFX and Galapo may help better during internal development of builder since they know better about internals and annoyed about bugs.

And at last JFX can better touch plugins with new builder...

It would be good to see a good working open source builder with a good evolution to better.
Something we sadly miss and mostly workaround during development of projects.
At least we designed projects and plugins in a way to be compatible to a new builder which will help to a new builder development.

I wish my very best luck.

« **Reply #2 on:** March 26, 2017, 08:47:18 AM »

Thanks for your opinion, Lancelot.

Your suggestions are vital, and here is some ideas about improvements:

1. Logging

Quote

To me most critical,
winbuilder use wrong kind of memory management,
very wrong log file which cause big log and + with bad memory management cause winbuilder hang ...

For logging, I intend to use SQLite database.
This way, logging won't explode PEBakery, and provide more reliable and reconstructable log.

2. Lag, Lag and Lags

Quote

also winbuilder can not ignore encoded files during read of a plugin + bad memory management = cause slow down of big sized plugins during execution and during interface edits.
=> we use workaround with File Container plugins.
===> but initial execution of winbuilder cause too much time with increasing number of plugins following bad plugin management with bad memory management.

At my laptop, WB082 took 35sec to load, while Windows nagging like "No responce. Can I shut it down?".
So I put progress bar at PEBakery, no more Windows' nagging.

Currently PEBakery load only [Main], [Interface], [Variables], [Process], etc at load time.
Encoded file parts are lazy loaded at ondemand, and also utilize multicore for performance.
Because of this, PEBakery loads at 10sec / 100MB while WB082 took 35sec / 20MB to load.
Slower than 4sec and takes more mem than 20MB, but it seems to be successful tradeoff.

3. Wow64 redirection

Quote

I believe
'System,FILEREDIRECT,{OFF|ON}'
can also be depricated.

Since PEBakery is C# application, it support x86 / x64 both with one binary.
System,FileRedirect can be safely removed.

4. Future of language specification

Quote

as long as you can decode, I do not think there will be problem for compatibility.
during evolution lots of formats tested,
at that time there were programmer contributers, but like project developers they were kicked out from development

Quote

It would be good to see a good working open source builder with a good evolution to better.
Something we sadly miss and mostly workaround during development of projects.
At least we designed projects and plugins in a way to be compatible to a new builder which will help to a new builder development.

This is what I am planning:
1) Implement WB082 compatible builder <- Current status
2) Propose new PEBakery file format and grammar
3) Write converter, and convert all plugin into new format

Process 2 and 3 will take years, but if all goes well, I hope bad legacy to be removed.

5. Tirivial things
- Fast UI rendering
Since WPF utilize GPU accleration, no more WB's slooooooooow UI rendering.
No more facepalm in while opening more options in '7-Zip File Manager' plugin.

- Unicode Support
If I use CJK characters in WB, WB just freezes.
PEBakery already supports UTF8 and UTF16.

- HiDPI
WinBuilder do not support even System Aware DPI awareness.
Starting from Windows 10 v1607 and .Net Framework 4.6.2, WPF apps support Per Monitor DPI out-of-the-box.
The only problem is plugin embedded images - they need to be changed to higher resolution.

- Long file path (<32k)
We had problems with putting .Net Framework into PE because of 260B path restriction.
PEBakery support long file path up to 32765B out-of-the-box, powered by .Net Framework 4.6.2.

« **Reply #3 on:** March 26, 2017, 09:24:11 AM »

So far sounds good.

For logging, I intend to use SQLite database.

*
I believe with main log file as SQLite database,
this also can be converted to simpler and smaller html or txt .

*
Another idea is:

**
when PEBakery.WPF.exe starts,
it creates list - order of plugins

when you add new plugin you need to use refresh button
---> with increasing number of plugins, it becomes a bad idea.

idea:
when PEBakery.WPF.exe starts,
it creates list - order of plugins
and save at a database (SQLite database or txt or ini or whatever)

when PEBakery.WPF.exe again starts, it reads list from databse
=> PEBakery.WPF.exe startsup faster, refresh is required rarely,

*
Also keep in mind.
we will need a syntax to add new plugin to available list:
ex:
- copy a new plugin to project folder
- than inside codebox or plugin a command like:
Add,PluginToList,%ProjectDir%\Newplugin.....
- will add new plugin to list quickly (without need to Refresh)

******

Quote from: ied206 on March 26, 2017, 08:47:18 AM

This is what I am planning:
1) Implement WB082 compatible builder <- Current status
2) Propose new PEBakery file format and grammar
3) Write converter, and convert all plugin into new format

There can be easier 1b) in future.

and 1) does not have to be full, enough to run current SE projects is good.

We already do not use many features, you do not need to implement.

and feel free adding new syntax to 1)

**
I wish my best luck, sad I do not have any free time.

I hope JFX , Galapo , ChrisR and others have free time to give more hand.

See You

« **Reply #4 on:** March 27, 2017, 08:52:55 AM »

Hi ied206,

hoping you have a good internet connection without size limit :

Test PEBakery with full Yomi server side by side with Win10PESE :
on wb:
- Win10PESE\Downloads\Yomi -> Download button
- Win10PESE\Downloads\UtilsY\Update Yomi -> Download button
- Refresh on wb
- Yomi\UtilsY\Update Yomi -> Update button

===>
This will download full Yomi which is 3.71 GB

* ps: Yomi server contains all shared plugins between projects.

- After download, start PEBakery , see how parsing goes.

************************************

* ps: You can also add other 4 projects (Win7PESE Win8PESE Win8.1SE Gena) to same base folder and see how parsing goes with PEBakery
See
wb -> Win10PESE\Downloads\Download Other Projects

This way you can make full parsing test with plugins available on 6 servers. (5 project , 1 plugin server)

« **Reply #5 on:** March 30, 2017, 02:22:52 PM »

Plugin interface rendering nearly done, here is test build:

PEBakery.WPF.20170330.zip (1199.57 kB - downloaded 136 times.)

- All interface control implemented (with the exception of CheckList)
- Button parsing and rendering improved
- More image type supported (jpg, png, gif, bmp, ico, svg)
Limitation : White color is not considered as transparent, can be substituted by png with alpha channels.
- Clicking image (open URL or show with external viewer) implemented
- Wrong font point-to-pixel conversion fixed
As a result, text layout is less malformed.

There was a problem that interface is too small, so in the Setting dialog, scale factor option was added.
(Currently this value is not saved so you have to set it every time)

Scale of 100%

Scale of 200%

Yet there is long way to go...
- WPF's image rendering system considers image's native DPI, causing plugin logo to be shrinked.
Ex) Win10PESE/Tweaks/My Computer - Name : Logo's DPI is not 96.
Solution = Change logo into strict 96DPIm or just ignore DPI.
(In fact, I tried ignoring DPI, but failed.)

- Scale factor system works nearly fine, with the exception of blurry image.
Solution = Must provide image of higher resolution or DPI.

« **Reply #6 on:** April 02, 2017, 06:13:40 AM »

*
Scale works very very very nice, Cool

*

Quote from: ied206 on March 30, 2017, 02:22:52 PM

Limitation : White color is not considered as transparent, can be substituted by png with alpha channels.

No problem

We were forced to use only .bmp on interface, we can replace them with .ico in time.

....

*
CheckList can wait for a loooong time, we do not use checklist on project build

*
I just update "My Computer - Name" plugin with a 48x48 jpg logo.

We can update plugins with Custom resolution logo to more standart resolution logo when required.

Quote from: ied206 on March 30, 2017, 02:22:52 PM

Yet there is long way to go...

Slowly slowly , always forward.

« **Reply #7 on:** April 17, 2017, 08:51:47 AM »

WinBuilder is slow and has memory problems. LiveSystem Pro is fast, and almost bugfree (after helping Kare remove countless bugs).
But the grammar (syntax) for both builders sucks big time. I'm tired of all those comma's and return values as arguments.

Why not write something that has a syntax like a normal language ? So I am writing this right now.
Lexer is done, parser is 95% done. It has full expression support. Syntax is sort of PureBasic like (which I'm using to write it in).

This is a test script that works:

[Process]
// Every variable is a STRING !!!
Set %Var1% = 153
Set %Var2% = 3
Set %Var3% = %Var1%

// SET is optional
%Var4% = %Var2%
%Var5% = "String"
%Var6% = -100
%Var7% = 1 + 3 : // 13 (String +)
%Var8% = Math(1 + 3) : // 4 (Math +)

// String functions
%String1% = Left(%Var5%, %Var2%)
%String2% = Right(%Var5%, %Var2%)
%String3% = Mid(%Var5%, %Var2%, 2)
%String4% = Mid("Dit is een lange string", 1, 11)
%String5% = Len("Dit is een lange string")

// Expressions !!
%Expr1% = Left("Lange String", Len("12345"))
%Expr2% = %String4% + %Var5% + " " + "YES!!!!!"

// Multiple statements per line
%Mul1% = 100 : %Mul2% = 101

// Line continuation
%Linecont% = "Test String \
line continuation"

// Math (divide/multiply/modulus)
%Math1% = Math(8/4) : // 2
%Math2% = Math(5/2) : // 2.5
%Math3% = Math(100 /2.4) : // 41.7
%Math4% = Math(100 %% 23) : // 8
%Math5% = Math(-100 %% 23) : // -8
%Math6% = Math(-100 %% 23.3) : // -6.8
// Operator precedence (* before +)
%Math7% = Math(2.5 + 3 * 4) : // 14.5
%Math8% = Math(2.5 - 3 * 4) : // -9.5
%Math9% = Math(2.5 - 8 / 4) : // 0.5
%Math10% = Math(2 + 3) + Math(3 + 3) : // 56 ! (String +)
//%Math11% = Math(2 + 3) - Math(3 + 3) : // Parser Error (cannot substract strings)
%Math12% = Math(Math(2 + 3) + Math(3 + 3)) : // 11 ! (Math +)
%Math13% = Math(2 + 3 + 3 + 3) : // 11 ! (Math +)
// Relation (< > <= >= = <>)
%Math14% = Math(%Var1% < %Var2%) : // 0
%Math15% = Math(%Var1% > %Var2%) : // 1
%Math16% = Math(%Var1% <> %Var2%) : // 1

// If Else EndIf nested unlimited
If Math(%Var1% < %Var2%)
%IF1% = "YES"
Else
%IF1% = "NO"
EndIf

// Just a comment

Will need for/next, while/wend and repeat/until loops nested. But it is a start.

So if someone is writing a replacement builder, why not change the syntax as well. Sure, it will break everybodies projects including mine, but it also gives a lot of new and better commands.

Just my 2 cents.

« **Reply #8 on:** April 17, 2017, 07:24:28 PM »

Quote from: Martt on April 17, 2017, 08:51:47 AM

So if someone is writing a replacement builder, why not change the syntax as well. Sure, it will break everybodies projects including mine, but it also gives a lot of new and better commands.

Just my 2 cents.

Totally agree. The syntax of WB is also one of it's main problems.
LSP mostly inherit this style and soon there will be another builder with this silly synatx

« **Reply #9 on:** April 18, 2017, 07:35:37 AM »

Hi Martt,

Time ( 8 years) prove your suggestion simply sabotage any existing development

Instead we develop projects and improve to better since 8 years time instead of starting everything from begining.

Development forward gains big free time than development circle.

Still, javawinbuilder, LSP goes that path having their own foreseen troubles....
There is also builder independent batch files (Powershell , CMD) on internet, or project special builder I saw in the past. (closed source exe)

*
To me as written before:

Free Limited Time Evolution should go step by step since there is no programmer team that can transfer all projects and all plugins with new syntax.

or give me big money, I can do all in 6 months on Silicon Valley by hiring a set of programmers at a newly created company.

So natural path should be first rescue with compatible builder (partial 082.0.0.15 compatible is enough as written on current topic),
than this will open big gate
+ to develop new syntaxes even with new extensions (than your suggesions may count by developer)
+ or even support using known extensions (.cmd, .ps) out of box...
+ and many things....

**
edit:
Kare (CD-RW) Spam Advertisement moved here http://theoven.org/index.php?topic=2101.0

**
Anyway, too many things to do with Free Limited time

Let's see what ied206 will do in time.

« **Reply #10 on:** April 18, 2017, 09:31:08 AM »

Quote from: Martt on April 17, 2017, 08:51:47 AM

WinBuilder is slow and has memory problems. LiveSystem Pro is fast, and almost bugfree (after helping Kare remove countless bugs).
But the grammar (syntax) for both builders sucks big time. I'm tired of all those comma's and return values as arguments.

Why not write something that has a syntax like a normal language ? So I am writing this right now.
Lexer is done, parser is 95% done. It has full expression support. Syntax is sort of PureBasic like (which I'm using to write it in).

Quote from: Martt on April 17, 2017, 08:51:47 AM

So if someone is writing a replacement builder, why not change the syntax as well. Sure, it will break everybodies projects including mine, but it also gives a lot of new and better commands.

Quote from: JFX on April 17, 2017, 07:24:28 PM

Totally agree. The syntax of WB is also one of it's main problems.
LSP mostly inherit this style and soon there will be another builder with this silly synatx

I agree with Martt and JFX's opinion about WB's syntax, it is total mess.

Quote from: Lancelot on April 18, 2017, 07:35:37 AM

Free Limited Time Evolution should go step by step since there is no programmer team that can transfer all projects and all plugins with new syntax.

However, we don't have time to convert current SE projects by hand, as Lancelot described.

Quote from: ied206 on March 26, 2017, 08:47:18 AM

This is what I am planning:
1) Implement WB082 compatible builder <- Current status
2) Propose new PEBakery file format and grammar
3) Write converter, and convert all plugin into new format

Process 2 and 3 will take years, but if all goes well, I hope bad legacy to be removed.

That is why I have process 2 and 3 in roadmap.
WB082 syntax support is needed for current SE projects' codes, while we need new, clean syntax for the future of SE projects.
To transition, we need to support old grammar, new grammar, and converter.

In my opinion, Martt's grammar seems intuitive, while easy to convert from WB082's syntax.
Martt, are you willing to merge our work?
I can provide GUI and WB082 syntax implementation, while you can provide new grammar and parser.
After our work becomes stable, we can convert all SE plugins into Martt's new grammar, and everyone will be happy.

« **Reply #11 on:** April 18, 2017, 11:56:19 AM »

Quote

So if someone is writing a replacement builder, why not change the syntax as well. Sure, it will break everybodies projects including mine, but it also gives a lot of new and better commands.

I had written simple batch+vbscript builder for myself to create PE with simple additems.txt, origin *.reg files.
I don't know why we need write scripts(plugins) for just adding/deleting files and update registry.

The above *MATH* things are realy necessary?

maybe need some *If* for dealing with different version OS, or arch,
maybe need some *String/File* things for updating pecmd.ini,
that's all.

If you recreate one, make a better easy-to-use builder rather than a WB compatible builder.

« **Reply #12 on:** April 18, 2017, 12:43:00 PM »

ied206: It is no problem to work with you. Happy too.

slore: Hmm, it is something I am struggling with right now. ALL vars are strings, so doing expressions is a problem. That it why the math is needed.

Example:

%Var1% = 153
%Var2% = 3

If %Var1% < %Var2% ==> True, string compare
If Math(%Var1% < %Var2%) ==> False, numeric compare

The alternative is a system with different vars, integer and string for minimum, maybe more. It can be done.

The problem, as allways, is *what* do you want to achieve? The goal should be to write an easy to use builder, not a full language interpreter. Or not ?

I do not agree with you on the batch+vbs solution. It will work. But on a single project that supports multiple Windows version you do need a lot more.

« **Reply #13 on:** April 18, 2017, 02:02:30 PM »

Hi, Martt

Quote

%Var1% = 153
%Var2% = 3

If %Var1% < %Var2% ==> True, string compare
If Math(%Var1% < %Var2%) ==> False, numeric compare

even simple equal expression will need write with a function(*MATH*)?

why %Var1% = "153" <- Var1 is string type, %Var2% = 153 <- Var2 is integer type,
more flexible, supports to_i, to_s for converting them by userself in some case.
%Var2% = %Var1%.to_i => Var2 is integer type
%Var2%.to_s => be string object

"10.0" + 16053 => error? 16063.0?
"10.0." + 16053 = error or auto "10.0.16053"
"10.0.".to_i + 16053 = 16063.0
"10.0" + 16053.to_s => 10.016053

Quote

I do not agree with you on the batch+vbs solution. It will work. But on a single project that supports multiple Windows version you do need a lot more.

if i need supoorts multiple windows version, my builder is use the origin batch var(%environment variable%),if,calculator(set /a) for that.
(I think most people will use batch file the autoit, the extend batch command is be written by themself.)

also the builder self is written by batch file, any one can extend with some program languge by themself.(python,ruby,autoit)
the 90% code just was written in a weekend, development and maintenance is easy and fast with dynamic script.

wimbuilder:
http://theoven.org/index.php?topic=1969.0

vbscript is do the convert *.reg's orgin HKLM => HKLM\PE_SOFT, HKU => HKLM\PE_DEFUALT,
or some thing is easier(deal with +MUI in additems.txt which will auto copy the mui file) than batch.

« **Reply #14 on:** April 18, 2017, 04:59:48 PM »

More or less like PureBasic? A.s = string, B.i = integer. Or some variation on that ? Floats also needed. Like 13.4.
Or %Var1%.i, so suffix. Or a declare %Var1%.s first once, followed by a %Var1% everywhere else in the script.

It would eliminate the need for the ugly Math all over the script.

Sofar I have everything working, so:

- Factor (Expressions, all functions, values)
- Term (Multiply, divide, modulus)
- Expression (Plus, minus, bitwise and, bitwise or)
- Relation (<, >, =, <>, <=, >=)
- Logical (Logical Or, Logical And)

Still need Math() for numbers.

Will think about the var stuff.

« **Reply #15 on:** April 19, 2017, 07:53:17 AM »

I am new to this Forum, but I still would like to give my opinions because I am long interested in WinPE, even since boot-land.net.

1. It would be nice to have faster and more reliable builder compatible with all existing plugins for the start.

2. I very like slore's idea with VBscript as a programing language for a new syntax.

3. I don't like .NET dependency, I would stay at least to 3.5 version included in Win7 if it is possible at all.

« **Reply #16 on:** April 19, 2017, 12:13:16 PM »

Hello, ZYX! Welcome to the forum.

1. It would be nice to have faster and more reliable builder compatible with all existing plugins for the start.

I agree with you.

2. I very like slore's idea with VBscript as a programing language for a new syntax.

I haven't used VBScript, what is the advantages of using VBScript's syntax?
Are there any customizable parser and runtime exists?
To support log functionability and others, I think customizing runtime is inevitable.

We also have to consider about easiness of converting WB082 syntax into new syntax.

3. I don't like .NET dependency, I would stay at least to 3.5 version included in Win7 if it is possible at all.

I understand about your opinion, but I think using C#/WPF with recent .Net Framework is the best choice.
I had also considered about using C++/Qt, but it did have some limitations compared with C#/WPF.

First, using C# shortens developing time.
Since I don't have much free time, longer developing time is burden to me.
Thinking about .Net Framework version, C# in .Net Framework 3.5 does lack useful features.
Moreover, starting from Windows 8, Microsoft pre-install .Net Framework 4.x, while 3.5 is not.
Thinking of Windows 7's support will be dropped around 2019, I think using .Net Framework 4.x is better.

Second, .Net Framework 4.6.2, supports long file name (up to 32K) out-of-the-box.
As you know, 255byte file length limit of WB082 sometimes wreck PE building.
Of course, supporting long file name can be done in native Windows API, but it is quite complex.

Third, WPF supports GPU-acclerated rendering and HiDPI support out-of-the-box.
WB082's layout rendering is too slow, you can try with clicking 'More Options' button in '7-zip' plugin.
Sadly, Per-Monitor HiDPI support is supported starting from .Net Framework 4.6.2 (and with Win 10 v1607)
In WPF, GPU acclearation and HiDPI support comes with no cost.

« **Reply #17 on:** April 19, 2017, 05:14:09 PM »

Hello, ZYX! Welcome to the forum.

Thanks :)

I haven't used VBScript, what is the advantages of using VBScript's syntax?

If you are comparing it to the batch language used in WinBuilder, the advantages are many:
it is standardized language based on Visual Basic, object oriented, it supports several different variable types including strings, integers, double precision reals, arrays, collections (using objects) and many another. It has If Else End If, For Next, Do Loop, Select Case End Select and other structures like any normal programing language.

Are there any customizable parser and runtime exists?

I am not sure that understand what you mean by customizable parser and runtime. The VBscript interpreter (runtime module) is included with every version of Windows. There are many programs that have plugins/addons based on VBscript, so I suppose they use the mentioned Windows interpreter as a parser.

To support log functionability and others, I think customizing runtime is inevitable.

Again, I am not sure that understand what you are talking. You could write logger in VBscript quite easily. It supports COM objects, so you could import e.g. FileSystemObject to work with files/folders.

As I said, VBscript is interpreter language, you could write and modify scripts and you could run them immediately after that without waiting for some kind of compilation, if that is what you mean by runtime.

We also have to consider about easiness of converting WB082 syntax into new syntax.

VBscript supports RegEx object, so you could do any kind of replacements. Actually, I already made some simple parsers with it.

By the way, with the appropriate COM object, you could do any kind of database manipulations, even with SQLite.