UTF-8 collumn names written as ascii

[sehHeiden]

I tried to open ESRI Shape files offered by the OpenData website of TW (https://data.gov.tw/en/datasets/all). For example in a file for smaller Taichung City. Column names that are saved as UTF-8 characters and can be display in QGIS (my version is 3.20), are displayed by a sort ASCII 8-bit? characters, when I open these with GeoDataFrames.jl.
圖名代碼 ->\xb9ϦW\xa5N\xbdX

[visr]

I downloaded one of the shapefiles from that website, but it looks like the DBF files with the column names are not encoded in UTF-8, but BIG5. We assume UTF-8 in DBFTables.jl, by converting the bytes to String, which is UTF-8.

https://github.com/JuliaData/DBFTables.jl/blob/6b4ef1ab5843225a0e0fae04abbc3bbb44fcac44/src/DBFTables.jl#L69

If the same bytes are decoded as BIG5 however the result seems fine:

using StringEncodings
bytes = UInt8[0xb9, 0xcf, 0xa6, 0x57, 0xa5, 0x4e, 0xbd, 0x58]
decode(bytes, "BIG5")  # -> "圖名代碼"

It is not supported by this package, but often a .cpg file is added that specifies the encoding:

.cpg—An optional file that can be used to specify the codepage for identifying the characterset to be used.

I'm impressed that GDAL (used by GeoDataFrames and QGIS) seems to correctly guess the encoding here. If I export the file from QGIS, it encodes it in UTF-8 and writes a .cpg file with UTF-8 inside.

Looking at these links

• https://github.com/OSGeo/gdal/blob/a96dab1d4c35132d4b64cf61ebb244fa4de0533d/ogr/ogrsf_frmts/shape/ogrshapelayer.cpp#L343
• https://gdal.org/drivers/vector/shapefile.html#encoding
• Language Driver ID in dbf file of new shapefile qgis/QGIS#15355
• https://stackoverflow.com/questions/52590941/how-to-interpret-the-language-driver-name-in-a-dbase-dbf-file

it seems there is a Language Driver ID in some DBF headers, that can be used for this as well. I suppose that is how GDAL figured out the encoding. I suppose that DBFTables could take a dependency on StringEncodings and add support for both CPG files and Language Driver IDs, though that would require some effort.

Probably your best bet for now is to just use GeoDataFrames.jl. GDAL in general is better at reading a wide variety of shapefiles compared to this package. Another option is to pull this file through GDAL's ogr2ogr utility, and since it will be UTF-8 you can use this package on the result, see the snippet here: #53 (comment).