Topic: Handling UTF-8 strings in plugins: workarounds, hacks?  (Read 1016 times)

Handling UTF-8 strings in plugins: workarounds, hacks?
« on: September 03, 2019, 10:36:53 AM »

Mikka

  • Code Baker
  • Chef
  • ***
  • Location: Germany
  • Date Registered: May 2012
  • Posts: 256
The Classic WinBuilder is an aged programme by now, but still in use for current projects.
(I know, there are successors, but as I did not test them I cannot say anything about them.)
Even if the ML module has polished WB quite up, there are limitations plugin authors sometimes encounter.

For instance, more and more applications (like Mozilla Firefox) deal with utf-8 data.
As long as we're using external scripts doing stuff we're safe, but importing it in WB is difficult:
E.g. instead of reading in the string "Wikipédia (fr)" we'll get something like "Wikipédia (fr)"—let alone strings like "Википедия".

One workaround would be to largely avoid the detour via WB variables and to realise these information flows externally.
I'm looking into a case (i.e. unreleased plugin version) right now whether it'll be feasible...

How do you solve the problem?
Or will it vanish with a WB successor?
« Last Edit: September 03, 2019, 10:38:53 AM by Mikka, Reason: For the sake of completeness a link was added »

Re: Handling UTF-8 strings in plugins: workarounds, hacks?
« Reply #1 on: September 03, 2019, 10:59:04 AM »

CD-RW

  • LiveSystem pro
  • Chef
  • ***
  • Date Registered: Oct 2010
  • Posts: 168
Hello
why do not you just take the LiveSystempro Builder?
He can handle all popular formats like Ansi, Unicode and Utf8.
For this he is actively developed further. The program has been developed for more than 10 years, and therefore no mayflies.
Many developers have come and gone in the last 14 years.

Kare
--

Re: Handling UTF-8 strings in plugins: workarounds, hacks?
« Reply #2 on: September 03, 2019, 02:52:25 PM »

James

  • Grand Chef
  • *****
  • Location: USA
  • Date Registered: Dec 2017
  • Posts: 2272
RegFileImport

Re: Handling UTF-8 strings in plugins: workarounds, hacks?
« Reply #3 on: September 03, 2019, 03:25:03 PM »

Mikka

  • Code Baker
  • Chef
  • ***
  • Location: Germany
  • Date Registered: May 2012
  • Posts: 256
why do not you just take the LiveSystempro Builder?
He can handle all popular formats like Ansi, Unicode and Utf8.
I'd rather check out a WB replacement which is syntax compliant, i.e. that offers the full command set including ML.
As far as I know, this isn't the case with the—syntactically simplified—Live System Pro Builder, so that's no realistic option.

RegFileImport—hm. That's somehow like the Inifile workaround.
Although I can imagine suitable scenarios, I am not fully convinced by this "hack"...

Re: Handling UTF-8 strings in plugins: workarounds, hacks?
« Reply #4 on: September 03, 2019, 10:13:33 PM »

ChrisR

  • XPE Baker
  • Grand Chef
  • *****
  • Date Registered: Mar 2011
  • Posts: 3494
the Other builders (LSP, Pebakery, wimbuilder2) seem to support the most common encoding formats: unicode, utf-8, utf-8 with bom and Ansi  :thumbsup:
On its side, WB does not support unicode, but seems to support UTF-8.

With the file encoded in Utf-8, TXTReplace or TXTAddLine seems to do the job 
Code: [Select]
TXTReplace,%Target_Prog%\%ProgramFolder%\distribution\policies.json,"#$qDisablePocket#$q: true#$c","#$qDisablePocket#$q: false#$c"
or TXTAddLine,%Target_Prog%\%ProgramFolder%\distribution\policies.json,xxxxxx,Append

In running Firefox Quantum and using policies.json encoded in UTF-8, "Wikipédia (fr)" is well recognized and properly removed from the list of search engines
« Last Edit: September 04, 2019, 04:12:17 PM by ChrisR »

Re: Handling UTF-8 strings in plugins: workarounds, hacks?
« Reply #5 on: September 04, 2019, 04:16:45 PM »

ChrisR

  • XPE Baker
  • Grand Chef
  • *****
  • Date Registered: Mar 2011
  • Posts: 3494
I don't know what my tests were last night but they weren't good.
I edited my previous post.

Re: Handling UTF-8 strings in plugins: workarounds, hacks?
« Reply #6 on: September 04, 2019, 04:50:27 PM »

Mikka

  • Code Baker
  • Chef
  • ***
  • Location: Germany
  • Date Registered: May 2012
  • Posts: 256
Quote from: ChrisR link=topic=2928.msg34597#msg34597
the Other builders (LSP, Pebakery, wimbuilder2) seem to support the most common encoding formats: unicode, utf-8, utf-8 with bom and Ansi  :thumbsup:
That seems like a good reason for switching to one of them... PEBakery perhaps?
Are there any difficulties concerning syntax compatibility (incl. external commands/ML)?

Quote from: ChrisR
On its side, WB does not support unicode, but seems to support UTF-8.

With the file encoded in Utf-8, TXTReplace or TXTAddLine seems to do the job 
Code: [Select]
TXTReplace,%Target_Prog%\%ProgramFolder%\distribution\policies.json,"#$qDisablePocket#$q: true#$c","#$qDisablePocket#$q: false#$c"
or TXTAddLine,%Target_Prog%\%ProgramFolder%\distribution\policies.json,xxxxxx,Append
I recently added one TxtReplace line to handle the XPE audio topic, for the .cfg file (untested yet).

For files like policies.json I use the parser jq* (in a cmd session), so if I write the output of jq into a file, it will be utf-8 encoded.
The current problem I'm facing is getting a utf-8 encoded string into WB, IniRead etc. won't do. (Cannot tell another method.)
That's why I thought of circumventing the problem by entirely generating the intended parsing command externally...
:undecided:

* Parsing example:
Code: [Select]
<policies.json jq ".policies.DisablePocket=false" >policies.new && copy/y policies.new policies.json
« Last Edit: September 04, 2019, 05:35:57 PM by Mikka, Reason: jq example added »

Re: Handling UTF-8 strings in plugins: workarounds, hacks?
« Reply #7 on: September 04, 2019, 06:50:32 PM »

sandy

  • Jr. Chef
  • **
  • Date Registered: Oct 2018
  • Posts: 49
I use latest develop branch version of pebakery for win10pese. It has continuous development on the develop branch. Master branch seems to be largely ignored by the developer.

Re: Handling UTF-8 strings in plugins: workarounds, hacks?
« Reply #8 on: September 05, 2019, 10:38:13 AM »

APT

  • Grand Chef
  • *****
  • winpe enthusiast
  • Location: UK
  • Date Registered: Nov 2012
  • Posts: 1150
Hi sandy

Quote
Master branch seems to be largely ignored by the developer.

odd, and incorrect thing to say

when ied206 has completed his development goals for the next release - beta6, that would be updated, as would be perfectly normal :thumbsup:
and hopefully, after beta6 we will go to Release 1.0
« Last Edit: September 05, 2019, 01:49:28 PM by APT »

Re: Handling UTF-8 strings in plugins: workarounds, hacks?
« Reply #9 on: September 07, 2019, 12:15:10 PM »

Mikka

  • Code Baker
  • Chef
  • ***
  • Location: Germany
  • Date Registered: May 2012
  • Posts: 256
I worked around the encoding problem by remodeling the code and made sure that the processing takes place independently from WinBuilder.
It's a major drawback of WinBuilder that it can't handle UTF-8 properly. I will upload the current plugin in the corresponding thread soon.

@sandy & APT:
Out of curiosity, did you use one of my recent plugins—Firefox Quantum ESR 68 (x64) or Notepad++ x64—with PEBakery yet?
« Last Edit: September 07, 2019, 01:13:09 PM by Mikka, Reason: typo »

Re: Handling UTF-8 strings in plugins: workarounds, hacks?
« Reply #10 on: September 07, 2019, 02:11:04 PM »

sandy

  • Jr. Chef
  • **
  • Date Registered: Oct 2018
  • Posts: 49
Was not trying to be critical about pebakery. In fact I see no reason not to use it for the currently active projects.

Re: Handling UTF-8 strings in plugins: workarounds, hacks?
« Reply #11 on: September 09, 2019, 12:45:05 PM »

APT

  • Grand Chef
  • *****
  • winpe enthusiast
  • Location: UK
  • Date Registered: Nov 2012
  • Posts: 1150
Hi Mikka

Quote
Out of curiosity, did you use one of my recent plugins—Firefox Quantum ESR 68 (x64) or Notepad++ x64—with PEBakery yet?

Yes, but not extensively, only to check they both run up to working apps in a VM on the x64 build I tested on, with latest nightly PEB

both have minor syntax errors, but nothing to stop build

nothing to do with PEB, but is it ness for Firefox to download source file every run? - quite annoying when running test builds

anyway, thanks for the plugin updates  :thumbsup:
APT

ps I think we are wandering a bit off topic here!
« Last Edit: September 09, 2019, 12:59:36 PM by APT »

 

Powered by EzPortal