Bdtjtk

I'm doing a fairly simple code that transforms European Portuguese input into Brazilian Portuguese -- so there are a lot of accented characters such as á,é,À,ç, etc.

Basically, the goal is to find words in the text from a list and replace them with the BR words from a second list.

Here's the code:

#-*- coding: latin-1 -*-



listapt=["gestão","utilizador","telemóvel"]

listabr=["gerenciamento", "usuário", "celular"]



while True:



    #this is all because I need to be able to input multiple lines of text, seems to be working fine 



    print ("Insert text")

    lines = 



    while True:

        line = raw_input()

        if line != "FIM":

            lines.append(line)

        else:

            break

    text = 'n'.join(lines)    



    for word in listapt:

        if word in text:

            num = listapt.index(word)

            wordbr = listabr[num]

            print(word + " --> " + wordbr) #just to show what changes were made

            text = text.replace(word, wordbr)



    print(text)

I run the code on Windows using IDLE and by double-clicking on the .py file.
The code works fine when using IDLE, but does not match and replace characters when double-clicking the .py file.

Here's why the code works as expected in IDLE but not from CMD or by doubleclicking:

1. Your code is UTF-8 encoded, not latin-1 encoded


2. IDLE always works in UTF-8 "input/output" mode.


3. On Windows, CMD/Doubleclicking will use a non-UTF-8 8bit locale.


4. When your code compares the input to the hardcoded strings it's doing so at a byte level. On IDLE, it's comparing UTF-8 to hardcoded UTF-8. On CMD, it's comparing non-UTF-8 8bit to hardcoded UTF-8 (If you were on a stock MacOS, it would also work).

The way to fix this is to make sure you're comparing "apples with apples". You could do this by converting everything to the same encoding. E.g. Convert the input read to UTF-8 so it matches the hardcoded strings. The better solution is to convert all [byte] strings to Unicode strings (Strings with no encoding). If you were on Python 3, this would be all automatic.

On Python 2.x, you need to do three things:

1. 
   

   Prefix all sourcecode strings with u to make them Unicode strings:

   
   
   

   listapt=[u"gestão",u"utilizador",u"telemóvel"]
   
   listabr=[u"gerenciamento",u"usuário", u"celula]
   
   ...
   
   if line != u"FIM":
   
   

   
   
   

   Alternatively, add from __future__ import unicode_literals to avoid changing all your code.

   
   


2. 
   

   Use the correct coding header for the encoding of your file. I suspect your header should read utf-8. E.g.

   
   
   

   #-*- coding: utf-8 -*-
   
   

   
   


3. 
   

   Convert the result of raw_input to Unicode. This must be done with the detected encoding of the standard input:

   
   
   

   import sys
   
   line = raw_input().decode(sys.stdin.encoding) 
   
   

   
   

By the way, the better way to model list of words to replace it to use a dict. The keys are the original word, the value is the replacement. E.g.

words = { u"telemóvel": u"celula"}

I don't see that problem over here.

Based on your use of raw_input, it seems like you're using Python 2.x

This may be because I'm copypasting off of stack overflow, and have a different dev environment to you.

Try running your script under the latest Python 3 interpreter, as well as removing the "#-*- coding:" line.

This should either hit UnicodeDecodeError issues a lot sooner in your code, or work fine.

The problem you have here is Python 2.x getting confused at some point while trying to translate between byte sequences (what Python 2.x strings contain, eg binary file contents), and human-meaningful text (unicode, eg for things like user informational display of chinese characters), because it makes incorrect assumptions about how human-readable text was encoded into the byte sequence seen in the Python strings.

It's a detail that Python 3 attempts to address a lot better/less ambiguously.

First try executing the code below, it should resolve the issue:

# -*- coding: latin-1 -*-



listapt=[u"gestão",u"utilizador",u"telemóvel"]

listabr=[u"gerenciamento",u"usuário", u"celular"]



lines=

line = raw_input()

line = line.decode('latin-1')

if line != "FIM":

    lines.append(line)



text = u'n'.join(lines)    



for word in listapt:

    if word in text:

        print("Hello")

        num = listapt.index(word)

        print(num)

        wordbr = listabr[num]

        print(wordbr)