As a thought, if you resize the Edit Database Cell window/pane, is it possible to show the full contents correctly?

I'm kind of thinking maybe the left side of that window pane might be fixed width atm.

With the binary detection, this looks like the code which does it:

Hmmm, seems to be the isTextOnly() call in there which is relevant:

As a thought, if you resize the Edit Database Cell window/pane, is it possible to show the full contents correctly?

Yes problem is that my windows is 27 in wide, the database cell occupies 6in of that, and I'm still scrolling the row data. The horizontal scrollbar on the edit cell should work...

I'm kind of thinking maybe the left side of that window pane might be fixed width atm.

Yes, sounds right because of the dump. But still should be able to scroll to see the right half...

Good point. That scroll bar should definitely be shown when the window size isn't big enough. 😄

Hmmm, seems to be the isTextOnly() call in there which is relevant:

That's a strange approach (seeing if round trip through unicode results in no change).

I think after the BOM check, you can simply do a memchr() and look for 0. Or a loop and count the zeros vs length....

UTF-8 is a superset of Latin-1, so I'm not sure why my accented character is being rejected.

FWIW, the data in that cell is originally Latin-1 - Perl encodes it into its internal representation, and what happens inside the DBI/SQLite internals is a mystery to me at the moment.

From rough memory, I think we did used to look for 0x0 characters, but we switched to the current approach for some reason. Not remembering exactly why off the top of my head though.

Hmmm, I wonder if it's possible that the 512 byte boundary the data is split on (for performance) could be leaving a malformed utf8 character, which is then capable of causing an incorrect result for the data type check.

Could happen that way. But not in this case, as shown in the screenshot, the data is only 90 bytes here.

It is true that if you have reason to know that a field should be UTF-8, and it's malformed, calling it binary is reasonable. But you have no way to know. And in this case, the data is valid ISO Latin-1 (and so valid UTF-8).

Not remembering exactly why off the top of my head though.

git blame.

(Which should have been called 'git credit', but Linus is rarely politically correct.)

@tlhackque I made a commit yesterday that fixes the problem of the hex editor being disabled, instead of read-only.

That's a strange approach (seeing if round trip through unicode results in no change).

Regarding the binary detection, I think the application is doing right. Of course, the term binary is a diffuse concept, but we follow a simple approach: whatever is not fully correct text in the selected encoding is treated as binary (but only 512 bytes are tested for performance). Would you say this is binary?:
FF FF FF FF
In UTF-8 is meaningless, but in Latin-1 it is a correct text string: ÿÿÿÿ

The default encoding is UTF-8 and consequently FF FF FF FF is considered a BLOB. But if you change it the encoding to Latin-1, you will see it treated as text and shown: ÿÿÿÿ.

UTF-8 is a superset of Latin-1, so I'm not sure why my accented character is being rejected.

That's only true when you considere the set of characters (we should say Unicode in that sense, I think), but not in representation terms. Only US-ASCII is a common subset of UTF-8 and Latin-1. All the extended characters (over 7 bits) are represented differently.

E4 is "small a dieresis or umlaut"

E4 is not always 'ä'. For example, in UTF-8 E4 by itself is meaningless. On the other hand, 'ä' in UTF-8 is the two byte sequence: C3 A4.

A very common (and simple) test for "binary" is the presence of a 0 (NUL) byte.

Once again that depends on encoding. 'ä' in UTF-16 is 0x00E4 and text in UTF-16 shouldn't be considered binary if you know the correct encoding.

The way to tell DB4S which encoding to use is clicking with the right mouse button over the column header and choosing 'Set encoding'.

In my opinion, the only sensible enhancement in this regard would be detecting the more usual text encodings, like ISO-8859-?, and applying them automatically or ask the user whether to apply them.

I made a commit yesterday that fixes the problem of the hex editor being disabled, instead of read-only.
Thanks! I'll look for that the next time I pull a nightly build.

I agree that encoding vs. codepoints can be confusing. I should have been more careful. Still, it is the case that even in Unicode, 00 (C0 NUL) is a good marker for binary data.

I'm not sure if autodetection of "If not Unicode, then ..." is feasible in the general case, but there are certainly some heuristics that could be tried. But...

The way to tell DB4S which encoding to use is clicking with the right mouse button over the column header and choosing 'Set encoding'.

Another great. but hidden feature. Should be available in "Edit Database Cell" - at least, that's where I looked for it. Or right-clicking on a cell, which is the second place I looked.

How about a pulldown next to Mode for Encoding? If that had been there, I'd never have started this thread :-)

That's an extensive collection of encodings - nice job on getting them all in. Is there a reason for the the ordering in the dropdown menu? I guess I understand putting the most frequently used at the top, but the rest seem to be in random order. I'd list the frequent (by some measure) ones at the top of the list, then a separator, then all in alphabetical order...

That's only true when you consider the set of characters (we should say Unicode in that sense, I think), but not in representation terms. Only US-ASCII is a common subset of UTF-8 and Latin-1. All the extended characters (over 7 bits) are represented differently.

Depends on if you are thinking encoding or code points. UTF-8 (which I restricted my comments to) uses the same codepoints as 8859-1 - for 0x00-0xFF. See unicode.org P37:

ASCII and Latin-1 Compatibility Area. For compatibility with the ASCII and ISO 8859-1,
Latin-1 standards, this area contains the same repertoire and ordering as Latin-1. Accordingly,
it contains the basic Latin alphabet, European digits, and then the same collection of
miscellaneous punctuation, symbols, and additional Latin letters as are found in Latin-1.

And even the Latin-* control codes have matching codeponts - unicode.org P 544

There are 65 code points set aside in the Unicode Standard for compatibility with the C0
and C1 control codes defined in the ISO/IEC 2022 framework. The ranges of these code
points are U+0000..U+001F, U+007F, and U+0080..U+009F, which correspond to the 8-bit
controls 0016 to 1F16 (C0 controls), 7F16 (delete), and 8016 to 9F16 (C1 controls), respectively.
For example, the 8-bit legacy control code character tabulation (or tab) is the byte
value 0916; the Unicode Standard encodes the corresponding control code at U+0009.
The Unicode Standard provides for the intact interchange of these code points, neither
adding to nor subtracting from their semantics. The semantics of the control codes are generally
determined by the application with which they are used. However, in the absence of
specific application uses, they may be interpreted according to the control function semantics
specified in ISO/IEC 6429:1992.

On the other hand, the encoding uses the MSB set to indicate that the next few bits encode multi-byte strings. unicode.org P95 So at that level, yes, it's trickier to distinguish Unicode from Latin-1.

I appreciate all the effort you've been putting in to responding to my experience. Many thanks!

Another great. but hidden feature.

Yeah, this is one of the things which has bugged me for a while. It's reasonably common for us to implement new things and requests, but they're often very hmmm... hard to discover (?) unless the person already knows about them.

We seem to struggle in figuring out how to implement the "make this easily discoverable by new users". Suggestions welcome of course. 😄

How about a pulldown next to Mode for Encoding?

Awesome, good idea!

Another great. but hidden feature. Should be available in "Edit Database Cell" - at least, that's where I looked for it. Or right-clicking on a cell, which is the second place I looked.

Yes, I know it's hidden. Indeed I only knew about it when it was mentioned in some issue 😄 I discovered then all the menu options in the contextual menu of the column header.

How about a pulldown next to Mode for Encoding?

Awesome, good idea!

Well, the space is limited there. What about another entry in the future Tools entry in the menu bar. At least the "Set encoding for all tables" option. "Set encoding" for this table has a context problem. Only in the Data Browse tab would make sense.

Another option for both entries would be in the contextual menu of the Edit DB Cell dock.

Well, the space is limited there.

I meant here, where I seem to have space.

Or, if that's a problem on smaller screens, the "Mode label" could become something like "View", that cascades to Mode and Encoding. E.g.

VIEW
| -- Mode --> Text
              Binary
              Image
              JSON
              XML
| -- Encoding
            UTF-8
            ISO-8859-15
             ...

That would be expandable, for when someone comes up with "Writing mode LeftToRight/RightToLeft", "Suppress non-printables" - or whatever else the future brings. (No, I'm not suggesting either of these right now, just observing that the future usually brings "more".)