CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Advanced:
Perl and unicode fun

 



alex_v2
User

May 11, 2009, 4:27 PM

Post #1 of 5 (1761 views)
Perl and unicode fun Can't Post

So I have pulled out what little hair I have left and reading and searching but have yet to find an answer to my current dilemma.

my $test = "Man\x{C3}\x{9F}earpig";
$test = decode("utf8", $test);
print qq($test\n);

simple right... no not really. This code for me will execute just fine and out put exactly what i want. the problem comes in when I try to warn $test or insert $test into a database. Doing either of these things will give me something I can not use and don't desire.

I have tried all of the following:
binmode DATA, ":utf8";
binmode STDOUT, ":utf8";
binmode STDERR, ":utf8";

also

binmode DATA, ":latin1";
binmode STDOUT, ":latin1";
binmode STDERR, ":latin1";

Oddly enough I have to decode with utf8 but STDOUT in latin 1 to get it to display right.

I have tried both utf8 and latin1 on my DB and the result is the same. They both give me an odd character I dont want not the odd character I do want.

I am getting Man├earpig from warn and in the DB which is bad.
i am getting Man▀earpig from print which is right.

this seems like something that should be really simple but have been trying for about 5 days now and have hit a brick wall with ideas and thoughts.

All my i18n files are right and like I said have run out of ideas. any feed back on this would be greatly appreciated. Thanks.

-Alexander

Edit: Alright so I was a bit tired and have realized that the:
$test = decode("utf8", $test);
is doing nothing. It is something perl internally is doing. if I change:
my $test = "Man\x{C3}\x{9F}earpig";
to:
my $test = 'Man\x{C3}\x{9F}earpig';
The code no longer works. So it is in perls interprelation that it is making the follow string good. SO if anyone know what it is perl is doing that would be so helpful to me. Thanks again.


(This post was edited by alex_v2 on May 11, 2009, 4:48 PM)


KevinR
Veteran


May 11, 2009, 5:21 PM

Post #2 of 5 (1756 views)
Re: [alex_v2] Perl and unicode fun [In reply to] Can't Post

Your question is out of my range of perl experience and knowledge. If you get no advise here, try over at www.perlmonks.com
-------------------------------------------------


alex_v2
User

May 11, 2009, 5:23 PM

Post #3 of 5 (1755 views)
Re: [KevinR] Perl and unicode fun [In reply to] Can't Post

I will try there as well Kevin thanks.

-Alexander


alex_v2
User

May 11, 2009, 6:12 PM

Post #4 of 5 (1752 views)
Re: [alex_v2] Perl and unicode fun [In reply to] Can't Post

Update #1

So I have now also tried the following with the same results.

#!/usr/bin/perl

use URI::Escape;
use Encode;
use strict;
use DBI;

my $dbh = DBI->connect(*************);

my $url = 'Magn%C3%BCs';

$url = uri_unescape($url);
$url = decode("utf8", $url);

print qq($url\n);
warn qq($url);

my $sql = qq(insert into char_queue (char_name, realm) values('$url', 'blah'));
$dbh->do($sql);

$dbh->disconnect();

The print will give me the desired output of MagnŘs.
The warn line and mysql insert give me the following output Magn├╝s.

So I am still at the drawing board reading and trying to find out how this seemingly easy conversion is hating me =)

-Alexander


alex_v2
User

May 11, 2009, 7:41 PM

Post #5 of 5 (1748 views)
Re: [alex_v2] Perl and unicode fun [In reply to] Can't Post

So after many more hours of digging and reading I found the answer in an email archive that is over 6 years old. The following code outputs the desired variable in print warn and database.

#!/usr/bin/perl

use URI::Escape;
use Encode;
use strict;
use DBI;

my $dbh = DBI->connect(***********);

my $url = 'Magn%C3%BCs';

$url =~ s/%([0-9a-f]{2})/chr(hex($1))/egi;

if ( $url =~ m/[\x80-\xff]/ ) {
$url = encode( "iso-8859-1", decode_utf8( $url ));
}

print qq($url\n);
warn qq($url);

my $sql = qq(insert into char_queue (char_name, realm) values('$url', 'blahlblah'));
$dbh->do($sql);

$dbh->disconnect();

I hope this may help someone else and save them the hair pulling I had done.

-Alexander

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives