编程小贴士

给你的编程提供小点子


利用java.nio.charset.CharsetDecoder自动识别字符集

研究了在网上能找到的自动识别字符集的办法,有效的就是利用第三方类库jchardet。也有用cpdetector,其实也是利用jchardet。偶然发现jdk的java.nio.charset.CharsetDecoder可以用来识别字符集。

一、原理

一般用两种方法构建InputStreamReader:

InputStreamReader reader = new InputStreamReader(in, charsetName);

或者

InputStreamReader reader = new InputStreamReader(in, charset);

如果charset不匹配,则输出乱码。

还有一种构建方法,即利用CharsetDecoder:

CharsetDecoder cd = charset.newDecoder();
InputStreamReader reader = new InputStreamReader(in, cd);

这时如果不匹配,则抛出异常:

java.nio.charset.MalformedInputException: Input length = 1
    at java.nio.charset.CoderResult.throwException(CoderResult.java:277)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:338)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
        ....

这样,就可以用作字符集探测。

二、AutoCharsetReader的使用

AutoCharsetReader是根据上述原理,参考InputStreamReader而写成的类,继承Reader,可以看作为Charset自适应的InputStreamReader。

AutoCharsetReader ar= new AutoCharsetReader(in);
char c = ar.read();
...
char[] cbuf = new char[2000];
ar.read(cbuf);
...
BufferedReader br = new BufferedReader(ar);
br.readLine();
...

再比如Lucene创建全文索引的TextField需要Reader参数,可以直接利用这个类:

Field field = new TextField("content", new AutoCharsetReader(file));

读完文件之后,可以得到文件的charset。注意,是读完之后。

Charset charset = ar.charset();

三、备选字符集

因为采用多次尝试的办法来最终确定字符集,所以要提供备选。当前代码提供的默认备选字符集如下:

    private final static String[] _defaultCharsets = {        
            "US-ASCII",
            "UTF-8",
            "GB2312", 
            "BIG5",
            "GBK",
            "GB18030",                
            "UTF-16BE", 
            "UTF-16LE", 
            "UTF-16",
            "UNICODE"};

也提供了更改备选字符集的方法。比如:

AutoCharsetReader ar = new AutoCharsetReader(in).setCharset("ascii", "utf-8", "gbk");

先后顺序会影响探测结果。比如,如果GBK在GB2312之前,则检测结果只能是GBK,不会是GB2312,因为GBK包含GB2312。

四、只作字符集检测

可以只用作字符集检测:

charset = AutoCharsetReader.quickDetect(file.toURI().toURL(), charsets);
or:
charset = AutoCharsetReader.deepDetect(file.toURI().toURL(), charsets, stops);

quickDetect只读一个字符,适用于单字符集文件。对于html,可能需要全部读完才知道charset,则使用deepDetect。其中参数charsets可以为null。

如果一组文件,已知可能的字符集有“ascii”,“utf-8”,“gb2312”,和“gbk”,当检测得知一个文件的字符集为“utf-8”或”gbk”的时候,可以马上返回结果,无需继续读文件。这时可以把stops参数赋值为{“utf-8″, “gbk”}。为null则需全部读完。

五、其他

为提高效率,本类设有buffer,初选的字符集解码失败,不必重新读取io。buffer大小默认为8192,对象构建时可以自定义buffer大小,若参数小于16,则设为16。

 

97 Responses to “ 利用java.nio.charset.CharsetDecoder自动识别字符集 ”

  1. 域名说道:

    时间真快,又到年底!正好有空,到这里看看!

  2. chocopie说道:

    5nwv9I Way cool! Some very valid points! I appreciate you penning this write-up and the rest of the site is also really good.

  3. i love pron说道:

    gk79Hj Thanks for sharing, this is a fantastic blog article.Really looking forward to read more. Really Cool.

  4. see pron说道:

    kboff0 Thanks-a-mundo for the article post.Thanks Again. Much obliged.

  5. best pron说道:

    xipXRF Wow, superb weblog format! How lengthy have you been blogging for? you made running a blog glance easy. The overall glance of your website is fantastic, let alone the content material!

  6. suba suba说道:

    mkMG0U Thanks for the article.Much thanks again. Awesome.

  7. me suba说道:

    HULC6O yours and my users would really benefit from some of

  8. tra gung说道:

    Way cool! Some very valid points! I appreciate you writing this write-up plus the rest of the site is extremely good.

  9. Thank you for the good writeup. It in fact was a amusement account it. Look advanced to more added agreeable from you! By the way, how could we communicate?

  10. Really enjoyed this blog post.Really looking forward to read more. Awesome.

  11. Lung Detox说道:

    Say, you got a nice post.Really looking forward to read more. Keep writing.

  12. Thankyou for helping out, excellent information.

  13. Thanks so much for the blog article.Really looking forward to read more. Really Cool.

  14. I went over this website and I believe you have a lot of good info, saved to fav (:.

  15. I will not speak about your competence, the post simply disgusting

  16. W88说道:

    Website We Recommend You made some fine points there. I did a search on the matter and found most people will consent with your blog.

  17. There is perceptibly a bunch to identify about this. I suppose you made some good points in features also.

  18. Some really quality articles on this web site , bookmarked.

  19. pre booked说道:

    Wow, amazing blog layout! How long have you been blogging for? you make blogging look easy. The overall look of your website is magnificent, let alone the content!. Thanks For Your article about sex.

  20. stuff prior to and you are just extremely fantastic. I actually like what you ave received

  21. Pretty! This was an extremely wonderful post. Many thanks for supplying these details.

  22. Very neat article.Thanks Again. Really Great.

  23. microblog说道:

    Say, you got a nice blog post.Much thanks again. Want more.

  24. visit website说道:

    This will most certainly increase your chances of conversion.

  25. Thanks so much for the blog article.Really looking forward to read more. Will read on

  26. H1B说道:

    Thanks so much for the article. Awesome.

  27. useful source说道:

    What Is The Best Way To Import MySpace Blogs To Facebook?

  28. Nwokolo说道:

    whether this post is written by him as nobody else know such detailed about my difficulty.

  29. klm coupon code说道:

    Say, you got a nice article.Really thank you! Will read on

  30. I was recommended this website by my cousin. I am not sure whether this post is written by him as no one else know such detailed about my problem. You are amazing! Thanks!

  31. This is how to get your foot in the door.

  32. junk removala说道:

    I wanted to thank you for this excellent read!! I absolutely loved every little bit of it. I have you book-marked to look at new stuff you post

  33. Beyonce说道:

    Link exchange is nothing else but it is just placing the other person as blog link on your page at appropriate place and other person will also do similar for you.

  34. Souls in the Waves Great Morning, I just stopped in to go to your web site and thought I ad say I liked myself.

  35. popeyes survey说道:

    It as really a nice and useful piece of information. I am glad that you shared this helpful info with us. Please keep us up to date like this. Thank you for sharing.

  36. mens fashion说道:

    I really liked your blog.Really thank you! Keep writing.

  37. Really enjoyed this article.Really thank you! Awesome.

  38. seo tips说道:

    Luo the wood spoke the thing that he or she moreover need to

  39. womensfit说道:

    information with us. Please stay us up to date like this.

  40. When are you going to post again? You really entertain a lot of people!

  41. Perfectly written content, thanks for selective information.

  42. It as not that I want to replicate your web site, but I really like the style. Could you let me know which theme are you using? Or was it tailor made?

  43. This is an spectacular entry. Thank you very much for the supreme post provided! I was looking for this entry for a long time, but I wasn at able to find a authentic source.

  44. local说道:

    Wow, amazing blog layout! How long have you been blogging for? you made blogging look easy. The overall look of your web site is fantastic, let alone the content!

  45. more information说道:

    Im obliged for the blog article.Really looking forward to read more. Fantastic.

  46. Waec说道:

    Im grateful for the article post. Much obliged.

  47. Really appreciate you sharing this article. Will read on

  48. Very good article! We are linking to this great content on our site. Keep up the good writing.

  49. Very good info can be found on weblog.

  50. click here说道:

    the near future. Anyway, should you have any suggestions or techniques for new blog owners please

  51. I really liked your post.Thanks Again. Will read on

  52. Open heaven说道:

    Very informative article.Much thanks again. Really Cool.

  53. woh I love your content , saved to my bookmarks !.

  54. Very very good publish, thank that you simply lot pertaining to sharing. Do you happen to have an RSS feed I can subscribe to be able to?

  55. It’а†s actually a cool and helpful piece of info. I am satisfied that you just shared this useful information with us. Please keep us up to date like this. Thank you for sharing.

  56. tips de ventas说道:

    When someone writes an post he/she retains the idea of a

  57. Ridiculous quest there. What occurred after? Take care!

  58. There as definately a lot to find out about this subject. I love all of the points you made.

  59. Real estate说道:

    Remarkable! Its in fact amazing article, I have got much clear idea on the topic of from this paragraph.

  60. entclass说道:

    It’а†s really a cool and helpful piece of info. I am glad that you simply shared this useful information with us. Please keep us up to date like this. Thank you for sharing.

  61. travel说道:

    I value the article.Much thanks again. Awesome.

  62. I will right away clutch your rss feed as I can not find your e-mail subscription hyperlink or e-newsletter service. Do you ave any? Kindly let me recognize in order that I could subscribe. Thanks.

  63. you could have a great blog here! would you prefer to make some invite posts on my weblog?

  64. I visited various blogs but the audio quality for

  65. wow, awesome blog article. Much obliged.

  66. I visit every day a few sites and information sites to read articles or reviews, however this webpage provides feature based writing.

  67. personal injury说道:

    Wonderful blog! I found it while surfing around on Yahoo News. Do you have any suggestions on how to get listed in Yahoo News? I ave been trying for a while but I never seem to get there! Thank you

  68. Wohh precisely what I was searching for, thanks for putting up.

  69. You completed several good points there. I did a search on the matter and found mainly folks will agree with your blog.

  70. Supporters of Manchester United, City, Chelsea and Arsenal. He was also named the World Soccer Player of the Year.

  71. incredibly excellent post, i absolutely actually like this exceptional internet site, carry on it

  72. I value the blog post.Much thanks again. Want more.

  73. Berniece说道:

    Inspiring quest there. What happened after? Good luck!

  74. go to说道:

    Normally I do not learn post on blogs, but I wish to say that this write-up very forced me to take a look at and do so! Your writing taste has been surprised me. Thanks, very nice post.

  75. free guest post说道:

    My brother suggested I might like this website. He was entirely right. This post actually made my day. You cann at imagine simply how much time I had spent for this information! Thanks!

  76. otlivy说道:

    I think this is a real great blog article.Thanks Again. Will read on

  77. studiya dizayna说道:

    Thanks-a-mundo for the blog post.Really looking forward to read more. Keep writing.

  78. have a look at说道:

    Very nice blog post. I absolutely love this site. Thanks!

  79. Well I really enjoyed reading it. This subject provided by you is very constructive for good planning.

  80. Wow, awesome blog layout! How long have you been blogging for? you made blogging look easy. The overall look of your web site is fantastic, let alone the content!. Thanks For Your article about sex.

  81. Share说道:

    You made some decent points there. I looked online for that problem and located most individuals will go coupled with in conjunction with your web internet site.

  82. twitter说道:

    Skillful Plan Developing I consider something genuinely special in this website.

  83. Well I really enjoyed studying it. This article offered by you is very practical for proper planning.

  84. premium ads说道:

    Really enjoyed this blog article.Much thanks again. Keep writing.

  85. This website was how do I say it? Relevant!! Finally I have found something which helped me. Thanks!

  86. Preferably, any time you gain understanding, are you currently in a position to thoughts updating your internet site with an increase of info? It as pretty ideal for me.

  87. visit website说道:

    Its like you read my mind! You seem to know so much about this, like you wrote

  88. you have got an incredible blog right here! would you wish to make some invite posts on my weblog?

  89. Emanuel Capuchin说道:

    It is rare to discover a specialist in whom you might have some faith. In the world these days, nobody genuinely cares about showing others exactly how in this issue. How fortunate I am to have now found such a wonderful website as this. It is people like you that make a genuine difference currently through the suggestions they discuss.

  90. This blog is no doubt entertaining and besides factual. I have picked up a bunch of helpful stuff out of this amazing blog. I ad love to visit it every once in a while. Thanks a bunch!

  91. conveyancing说道:

    It’s a shame you don’t have a donate button! I’d certainly donate to this fantastic blog!

  92. link说道:

    Really appreciate you sharing this article.Thanks Again. Great.

  93. Thanks-a-mundo for the blog article.Really thank you! Want more.

  94. Say, you got a nice blog.Thanks Again. Want more.

  95. This is a really great examine for me, Must admit that you are a single of the best bloggers I ever saw.Thanks for posting this informative article.

  96. Very good post. I absolutely love this site. Continue the good work!

  97. informative post说道:

    Appreciate you sharing, great article.Really thank you! Great.

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>