UnicodeDecodeError parsing results

I get the following error when piping to fpp, apparently some files are binary and it's breaking

Traceback (most recent call last):
  File "/home/tulsi/PathPicker/src/processInput.py", line 84, in <module>
    doProgram(flags)
  File "/home/tulsi/PathPicker/src/processInput.py", line 53, in doProgram
    lineObjs = getLineObjs(flags)
  File "/home/tulsi/PathPicker/src/processInput.py", line 20, in getLineObjs
    inputLines = sys.stdin.readlines()
  File "/usr/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc1 in position 1191: invalid start byte

python version: 2.7.17

Can you try python3? From: AndresQ <notifications@github.com> Reply-To: facebook/PathPicker <reply@reply.github.com> Date: Wednesday, January 22, 2020 at 7:43 AM To: facebook/PathPicker <PathPicker@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: [facebook/PathPicker] UnicodeDecodeError parsing results (#312) I get the following error when piping to fpp, apparently some files are binary and it's breaking Traceback (most recent call last): File "/home/tulsi/PathPicker/src/processInput.py", line 84, in <module> doProgram(flags) File "/home/tulsi/PathPicker/src/processInput.py", line 53, in doProgram lineObjs = getLineObjs(flags) File "/home/tulsi/PathPicker/src/processInput.py", line 20, in getLineObjs inputLines = sys.stdin.readlines() File "/usr/lib/python3.7/codecs.py", line 322, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc1 in position 1191: invalid start byte python version: 2.7.17 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#312>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAIVDH4ZOTS722FB42HF77LQ7BLPNANCNFSM4KKG6GMA>.

Can you try python3?

Same error :(

$ python --version
Python 3.7.6

Traceback (most recent call last):
File "/home/tulsi/PathPicker/src/processInput.py", line 84, in
doProgram(flags)
File "/home/tulsi/PathPicker/src/processInput.py", line 53, in doProgram
lineObjs = getLineObjs(flags)
File "/home/tulsi/PathPicker/src/processInput.py", line 20, in getLineObjs
inputLines = sys.stdin.readlines()
File "/usr/lib/python3.7/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 4397: invalid start byte

Ah ok that's a helpful piece of context. Do you mind adding the most minimal stdout that repros the issue? I presume its some kind of binary output that's piped straight into fpp?

Indeed it's binary ouput being piped directly. I don't know if this is useful, but it's a use case

$ grep dii
95203:�}X�d�ܗ;���5���-\0{����e�D�8�`�+ڒ�K���"�NX�)�
�R��@�s}W�����s}ldii�tڑ�v6x&EZa����ž��S�=a�x}
218194:���wxC�O]��n��'5�{����	��
�y�-�^떟����`��B�m�->8\���~����б����t}�����<D6@,��̞3a���q�pQ�$���~�9<�Vz/흩����.SX2��켨�L'�=ڒ��_��􄢛cj���;�w��o�n����2sY�H��1��g��G̤c	뎱��K�V?A�)ɯ�/2R'��bng��ݚ4H
                                   AeKJ�1V�K�0�ޗz��``�s|�DT`R�{>:ń�]��x�v�	�za��~9�ұ/0oR�����Y�R�MX:ؤ���L�)�v���m���=[oQ�-U	2ex=ֵ�dV
               g/I;�^Ъƺ�mϭ�����1�n�����z�      �k#p�I� ����ĭ�xT�2'��A�	�/�Ӽa�<@dii�Q����@,H������.sɰi2A9u��t�N�[�ʡ1c@����vk�A4��%�����~1��_O\��BB�o�9ΈB�����?�0h���]j>$3/��lu7��y����W�����àa&��$aQ1�,3u�(�K�q�-�O�
                                                                                        �w;=���*'է53?1$LTë��9��{�]m����D7|G;�.O2p9K%�H8�����b0EOq
                     C�t���O�

same output if piped to fpp throws the previously mentioned error

Hrm I wasn't able to reproduce, but I imagine some of the binary data got lost between our pastes:

[pcottle:~:]$ cat foo.txt 
95203:�}X�d�ܗ;���5���-\0{����e�D�8�`�+ڒ�K���"�NX�)�
�R��@�s}W�����s}ldii�tڑ�v6x&EZa����ž��S�=a�x}
218194:���wxC�O]��n��'5�{����	��
�y�-�^떟����`��B�m�->8\���~����б����t}�����<D6@,��̞3a���q�pQ�$���~�9<�Vz/흩����.SX2��켨�L'�=ڒ��_��􄢛cj���;�w��o�n����2sY�H��1��g��G̤c	뎱��K�V?A�)ɯ�/2R'��bng��ݚ4H
                                   AeKJ�1V�K�0�ޗz��``�s|�DT`R�{>:ń�]��x�v�	�za��~9�ұ/0oR�����Y�R�MX:ؤ���L�)�v���m���=[oQ�-U	2ex=ֵ�dV
               g/I;�^Ъƺ�mϭ�����1�n�����z�      �k#p�I� ����ĭ�xT�2'��A�	�/�Ӽa�<@dii�Q����@,H������.sɰi2A9u��t�N�[�ʡ1c@����vk�A4��%�����~1��_O\��BB�o�9ΈB�����?�0h���]j>$3/��lu7��y����W�����àa&��$aQ1�,3u�(�K�q�-�O�
                                                                                        �w;=���*'է53?1$LTë��9��{�]m����D7|G;�.O2p9K%�H8�����b0EOq
                     C�t���O�[pcottle:~:]$ cat foo.txt  | fpp
No lines matched!!

Yes, there are surely many non-printable characters.

I was able to create a simple file, since the error states that it can parse the char 0x8b

See attached a.txt file

cat a.txt | fpp

reproduces the error

Hrm the file input has an errors parameter:
https://stackoverflow.com/questions/35028683/python3-unicodedecodeerror-with-readlines-method/41652865

do you know if there's an equivalent for sys.stdin.readlines? Seems like that would solve our issues

I know very little python, apparently the trick is to read from stdin in binary mode, like this

Yep but then we’ll have the same issue just later on when converting from raw bytes to utf8 encoding… From: AndresQ <notifications@github.com> Reply-To: facebook/PathPicker <reply@reply.github.com> Date: Wednesday, January 29, 2020 at 12:45 PM To: facebook/PathPicker <PathPicker@noreply.github.com> Cc: Peter Cottle <pcottle@fb.com>, Comment <comment@noreply.github.com> Subject: Re: [facebook/PathPicker] UnicodeDecodeError parsing results (#312) I know very little python, apparently the trick is to read from stdin in binary mode, like this<https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_32282448_read-2Dstdin-2Das-2Dbinary&d=DwMCaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=Cf5FGgmOwb6SJmWKfL-IoQ&m=F7t_r4P3qJYYrYyljjAQTWo9mLcMuHo-OrO4aIwjiIY&s=M0ukZcWV7lh6F-RbcW3KazjVgLmvuKmHUYJM3ih8-WQ&e=> — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#312>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAIVDHZ77CY5VGDHH5RK5HTRAHTFFANCNFSM4KKG6GMA>.

Navigation Menu

Repository navigation

UnicodeDecodeError parsing results #312

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

UnicodeDecodeError parsing results #312

Description

Activity

pcottle commented on Jan 24, 2020

tulsidas commented on Jan 24, 2020

pcottle commented on Jan 24, 2020

tulsidas commented on Jan 24, 2020

pcottle commented on Jan 27, 2020

tulsidas commented on Jan 27, 2020

pcottle commented on Jan 29, 2020

tulsidas commented on Jan 29, 2020

pcottle commented on Jan 29, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions