编程小贴士

给你的编程提供小点子


awk命令的详细学习

简介

awk是一个强大的文本分析工具,相对于grep的查找,sed的编辑,awk在其对数据分析并生成报告时,显得尤为强大。简单来说awk就是把文件逐行的读入,以空格为默认分隔符将每行切片,切开的部分再进行各种分析处理。

awk有3个不同版本: awk、nawk和gawk,未作特别说明,一般指gawk,gawk 是 AWK 的 GNU 版本。

awk其名称得自于它的创始人 Alfred Aho 、Peter Weinberger 和 Brian Kernighan 姓氏的首个字母。实际上 AWK 的确拥有自己的语言: AWK 程序设计语言 , 三位创建者已将它正式定义为“样式扫描和处理语言”。它允许您创建简短的程序,这些程序读取输入文件、为数据排序、处理数据、对输入执行计算以及生成报 表,还有无数其他的功能。

 

使用方法

awk '{pattern + action}' {filenames}

尽管操作可能会很复杂,但语法总是这样,其中 pattern 表示 AWK 在数据中查找的内容,而 action 是在找到匹配内容时所执行的一系列命令。花括号({})不需要在程序中始终出现,但它们用于根据特定的模式对一系列指令进行分组。 pattern就是要表示的正则表达式,用斜杠括起来。

awk语言的最基本功能是在文件或者字符串中基于指定规则浏览和抽取信息,awk抽取信息后,才能进行其他文本操作。完整的awk脚本通常用来格式化文本文件中的信息。

通常,awk是以文件的一行为处理单位的。awk每接收文件的一行,然后执行相应的命令,来处理文本。

 

调用awk

有三种方式调用awk

1.命令行方式
awk [-F  field-separator]  'commands'  input-file(s)
其中,commands 是真正awk命令,[-F域分隔符]是可选的。 input-file(s) 是待处理的文件。
在awk中,文件的每一行中,由域分隔符分开的每一项称为一个域。通常,在不指名-F域分隔符的情况下,默认的域分隔符是空格。

2.shell脚本方式
将所有的awk命令插入一个文件,并使awk程序可执行,然后awk命令解释器作为脚本的首行,一遍通过键入脚本名称来调用。
相当于shell脚本首行的:#!/bin/sh
可以换成:#!/bin/awk

3.将所有的awk命令插入一个单独文件,然后调用:
awk -f awk-script-file input-file(s)
其中,-f选项加载awk-script-file中的awk脚本,input-file(s)跟上面的是一样的。

本章重点介绍命令行方式。

 

入门实例

假设last -n 5的输出如下

[root@www ~]# last -n 5 <==仅取出前五行
root     pts/1   192.168.1.100  Tue Feb 10 11:21   still logged in
root     pts/1   192.168.1.100  Tue Feb 10 00:46 - 02:28  (01:41)
root     pts/1   192.168.1.100  Mon Feb  9 11:41 - 18:30  (06:48)
dmtsai   pts/1   192.168.1.100  Mon Feb  9 11:41 - 11:41  (00:00)
root     tty1                   Fri Sep  5 14:09 - 14:10  (00:01)

如果只是显示最近登录的5个帐号

#last -n 5 | awk  '{print $1}'
root
root
root
dmtsai
root

awk工作流程是这样的:读入有’\n’换行符分割的一条记录,然后将记录按指定的域分隔符划分域,填充域,$0则表示所有域,$1表示第一个域,$n表示第n个域。默认域分隔符是”空白键” 或 “[tab]键”,所以$1表示登录用户,$3表示登录用户ip,以此类推。

 

如果只是显示/etc/passwd的账户

#cat /etc/passwd |awk  -F ':'  '{print $1}'  
root
daemon
bin
sys

这种是awk+action的示例,每行都会执行action{print $1}。

-F指定域分隔符为':’。

 

如果只是显示/etc/passwd的账户和账户对应的shell,而账户与shell之间以tab键分割

#cat /etc/passwd |awk  -F ':'  '{print $1"\t"$7}'
root    /bin/bash
daemon  /bin/sh
bin     /bin/sh
sys     /bin/sh

 

如果只是显示/etc/passwd的账户和账户对应的shell,而账户与shell之间以逗号分割,而且在所有行添加列名name,shell,在最后一行添加”blue,/bin/nosh”。

cat /etc/passwd |awk  -F ':'  'BEGIN {print "name,shell"}  {print $1","$7} END {print "blue,/bin/nosh"}'
name,shell
root,/bin/bash
daemon,/bin/sh
bin,/bin/sh
sys,/bin/sh
....
blue,/bin/nosh

awk工作流程是这样的:先执行BEGING,然后读取文件,读入有/n换行符分割的一条记录,然后将记录按指定的域分隔符划分域,填充域,$0则表示所有域,$1表示第一个域,$n表示第n个域,随后开始执行模式所对应的动作action。接着开始读入第二条记录······直到所有的记录都读完,最后执行END操作。

 

搜索/etc/passwd有root关键字的所有行

#awk -F: '/root/' /etc/passwd
root:x:0:0:root:/root:/bin/bash

这种是pattern的使用示例,匹配了pattern(这里是root)的行才会执行action(没有指定action,默认输出每行的内容)。

搜索支持正则,例如找root开头的: awk -F: ‘/^root/’ /etc/passwd

 

搜索/etc/passwd有root关键字的所有行,并显示对应的shell

# awk -F: '/root/{print $7}' /etc/passwd             
/bin/bash

这里指定了action{print $7}

 

awk内置变量

awk有许多内置变量用来设置环境信息,这些变量可以被改变,下面给出了最常用的一些变量。

ARGC               命令行参数个数
ARGV               命令行参数排列
ENVIRON            支持队列中系统环境变量的使用
FILENAME           awk浏览的文件名
FNR                浏览文件的记录数
FS                 设置输入域分隔符,等价于命令行 -F选项
NF                 浏览记录的域的个数
NR                 已读的记录数
OFS                输出域分隔符
ORS                输出记录分隔符
RS                 控制记录分隔符

此外,$0变量是指整条记录。$1表示当前行的第一个域,$2表示当前行的第二个域,……以此类推。

 

统计/etc/passwd:文件名,每行的行号,每行的列数,对应的完整行内容:

#awk  -F ':'  '{print "filename:" FILENAME ",linenumber:" NR ",columns:" NF ",linecontent:"$0}' /etc/passwd
filename:/etc/passwd,linenumber:1,columns:7,linecontent:root:x:0:0:root:/root:/bin/bash
filename:/etc/passwd,linenumber:2,columns:7,linecontent:daemon:x:1:1:daemon:/usr/sbin:/bin/sh
filename:/etc/passwd,linenumber:3,columns:7,linecontent:bin:x:2:2:bin:/bin:/bin/sh
filename:/etc/passwd,linenumber:4,columns:7,linecontent:sys:x:3:3:sys:/dev:/bin/sh

 

使用printf替代print,可以让代码更加简洁,易读

 awk  -F ':'  '{printf("filename:%10s,linenumber:%s,columns:%s,linecontent:%s\n",FILENAME,NR,NF,$0)}' /etc/passwd

 

print和printf

awk中同时提供了print和printf两种打印输出的函数。

其中print函数的参数可以是变量、数值或者字符串。字符串必须用双引号引用,参数用逗号分隔。如果没有逗号,参数就串联在一起而无法区分。这里,逗号的作用与输出文件的分隔符的作用是一样的,只是后者是空格而已。

printf函数,其用法和c语言中printf基本相似,可以格式化字符串,输出复杂时,printf更加好用,代码更易懂。

 

awk编程

变量和赋值

除了awk的内置变量,awk还可以自定义变量。

下面统计/etc/passwd的账户人数

awk '{count++;print $0;} END{print "user count is ", count}' /etc/passwd
root:x:0:0:root:/root:/bin/bash
......
user count is  40

count是自定义变量。之前的action{}里都是只有一个print,其实print只是一个语句,而action{}可以有多个语句,以;号隔开。

 

这里没有初始化count,虽然默认是0,但是妥当的做法还是初始化为0:

awk 'BEGIN {count=0;print "[start]user count is ", count} {count=count+1;print $0;} END{print "[end]user count is ", count}' /etc/passwd
[start]user count is  0
root:x:0:0:root:/root:/bin/bash
...
[end]user count is  40

 

统计某个文件夹下的文件占用的字节数

ls -l |awk 'BEGIN {size=0;} {size=size+$5;} END{print "[end]size is ", size}'
[end]size is  8657198

 

如果以M为单位显示:

ls -l |awk 'BEGIN {size=0;} {size=size+$5;} END{print "[end]size is ", size/1024/1024,"M"}' 
[end]size is  8.25889 M

注意,统计不包括文件夹的子目录。

 

条件语句

awk中的条件语句是从C语言中借鉴来的,见如下声明方式:

if (expression) {
    statement;
    statement;
    ... ...
}

if (expression) {
    statement;
} else {
    statement2;
}

if (expression) {
    statement1;
} else if (expression1) {
    statement2;
} else {
    statement3;
}

 

统计某个文件夹下的文件占用的字节数,过滤4096大小的文件(一般都是文件夹):

ls -l |awk 'BEGIN {size=0;print "[start]size is ", size} {if($5!=4096){size=size+$5;}} END{print "[end]size is ", size/1024/1024,"M"}' 
[end]size is  8.22339 M

 

循环语句

awk中的循环语句同样借鉴于C语言,支持while、do/while、for、break、continue,这些关键字的语义和C语言中的语义完全相同。

 

数组

因为awk中数组的下标可以是数字和字母,数组的下标通常被称为关键字(key)。值和关键字都存储在内部的一张针对key/value应用hash的表 格里。由于hash不是顺序存储,因此在显示数组内容时会发现,它们并不是按照你预料的顺序显示出来的。数组和变量一样,都是在使用时自动创建的,awk 也同样会自动判断其存储的是数字还是字符串。一般而言,awk中的数组用来从记录中收集信息,可以用于计算总和、统计单词以及跟踪模板被匹配的次数等等。

 

显示/etc/passwd的账户

awk -F ':' 'BEGIN {count=0;} {name[count] = $1;count++;}; END{for (i = 0; i < NR; i++) print i, name[i]}' /etc/passwd
0 root
1 daemon
2 bin
3 sys
4 sync
5 games
......

这里使用for循环遍历数组

77 Responses to “ awk命令的详细学习 ”

  1. see pron说道:

    RdZbdh Wow, incredible blog layout! How long have you been blogging for? you made blogging look easy. The overall look of your website is wonderful, as well as the content!

  2. me suba说道:

    GMY2RT You made various good points there. I did a search on the topic and located most people will have exactly the same opinion along with your weblog.

  3. Tattoo说道:

    Super-Duper website! I am loving it!! Will be real backside soon to interpret a number of extra. I am captivating your feeds also

  4. My brother suggested I might like this blog. He was entirely right. This post truly made my day. You cann at imagine simply how much time I had spent for this information! Thanks!

  5. casino说道:

    What a funny blog! I truly loved watching this comic video with my family unit as well as with my mates.

  6. Nice blog here! Also your site lots up fast! What web host are you the use of? Can I am getting your associate link in your host? I want my site loaded up as quickly as yours lol

  7. This unique blog is really educating and also diverting. I have chosen many handy advices out of this amazing blog. I ad love to go back again and again. Cheers!

  8. Download gclub说道:

    Thanks for the blog post.Really thank you! Awesome.

  9. I really liked your article post.Really looking forward to read more. Awesome.

  10. otel v arkadii说道:

    Thanks for sharing, this is a fantastic blog.Much thanks again. Really Cool.

  11. Judging by the way you compose, you seem like a professional writer.,.;*~

  12. The Best Supplements for Men To Take Your Regime Including to my diet regime has given a boost to my fitness routine

  13. or maybe guest writing a blog post or vice-versa? My website goes

  14. Yeah bookmaking this wasn at a high risk decision outstanding post!.

  15. crypto说道:

    It as not that I want to replicate your internet site, but I really like the style. Could you tell me which style are you using? Or was it especially designed?

  16. Way cool! Some very valid points! I appreciate you writing this post and the rest of the site is also really good.

  17. Incredible points. Outstanding arguments. Keep up the amazing spirit.

  18. Escorts Miami说道:

    Thanks for sharing this very good article. Very interesting ideas! (as always, btw)

  19. You made some nice points there. I did a search on the issue and found most guys will agree with your site.

  20. Thank you ever so for you article.Thanks Again. Really Cool.

  21. The Lost Ways说道:

    There is noticeably a lot to identify about this. I believe you made various nice points in features also.

  22. You have made some really good points there. I looked on the net for more information about the issue and found most people will go along with your views on this site.

  23. escort amsterdam说道:

    I truly appreciate this blog article.Much thanks again. Really Cool.

  24. This is very interesting, You are a very skilled blogger. I ave joined your feed and look forward to seeking more of your excellent post. Also, I have shared your website in my social networks!

  25. houston escorts说道:

    This website was how do you say it? Relevant!! Finally I ave found something that

  26. Really clear website , appreciate it for this post.

  27. I visit everyday some blogs and websites to read articles, except this website offers quality based articles.

  28. Thanks for the blog.Thanks Again. Much obliged.

  29. Looking around While I was browsing yesterday I saw a great post concerning

  30. buy mp3说道:

    pretty handy material, overall I think this is worthy of a bookmark, thanks

  31. Thanks so much for the article.Really looking forward to read more. Really Cool.

  32. buy music说道:

    Very neat blog.Really looking forward to read more. Cool.

  33. buy mp3说道:

    Really enjoyed this blog post.Really thank you! Keep writing.

  34. I really liked your article post.Really thank you! Really Cool.

  35. I think other web-site proprietors should take this web site as an model, very clean and great user genial style and design, as well as the content. You are an expert in this topic!

  36. I think this is a real great article post.Really looking forward to read more. Keep writing.

  37. Barcode说道:

    you might have a terrific weblog right here! would you wish to make some invite posts on my weblog?

  38. 토토사이트说道:

    This unique blog is obviously educating and also amusing. I have picked up many helpful tips out of this blog. I ad love to visit it over and over again. Thanks!

  39. what we do with them. User Demographics. struggling

  40. 먹튀 검증说道:

    You have touched some good points here. Any way keep up wrinting.

  41. 바다 이야기说道:

    Some truly interesting information, well written and broadly user pleasant.

  42. This unique blog is really interesting additionally diverting. I have found many helpful stuff out of this amazing blog. I ad love to go back over and over again. Thanks!

  43. 다잡아说道:

    You have made some decent points there. I checked on the internet for more information about the issue and found most people will go along with your views on this site.|

  44. Vancouver SEO说道:

    whoah this blog is fantastic i love reading your posts. Keep up the great work! You know, lots of people are looking around for this information, you can aid them greatly.

  45. Say, you got a nice article post.Much thanks again. Really Cool.

  46. 먹튀 폴리스说道:

    Major thankies for the blog.Really thank you! Will read on

  47. buy albuterol说道:

    Thank you for thw tipd. It hwlpwd mw a lot.

  48. 토토사이트说道:

    Looking forward to reading more. Great blog article.Really thank you! Keep writing.

  49. The Firefox updated tab comes up everytime i start firefox. What do i do to stop it?

  50. Regards for helping out, great information.

  51. Really informative blog article.Really looking forward to read more. Really Cool.

  52. Really informative post.Really thank you! Really Great.

  53. Usually I do not learn post on blogs, but I wish to say that this write-up very pressured me to check out and do it! Your writing style has been surprised me. Thanks, quite nice post.

  54. Red your site publish and beloved it. Have you at any time thought about visitor putting up on other associated weblogs comparable to your weblog?

  55. Just discovered this blog through Bing, what a way to brighten up my year!

  56. Wow! Thank you! I continually wanted to write on my site something like that. Can I implement a portion of your post to my blog?

  57. I value the post.Really looking forward to read more. Fantastic.

  58. tu van bhxh说道:

    Right now it seems like Movable Type is the best blogging platform available right now. (from what I ave read) Is that what you are using on your blog?

  59. Thanks again for the blog post.Much thanks again. Really Great.

  60. Your writing taste has been amazed me. Thanks, quite great post.

  61. Venus Factor说道:

    Really appreciate you sharing this post.Really thank you! Much obliged.

  62. I will right away grasp your rss feed as I can at in finding your email subscription hyperlink or newsletter service. Do you have any? Kindly permit me recognize in order that I may subscribe. Thanks.

  63. ban da lavabo说道:

    Way cool! Some extremely valid points! I appreciate you writing this write-up and also the rest of the website is also really good.

  64. Wow, marvelous blog structure! How lengthy have you ever been blogging for? you made blogging look easy. The whole look of your website is excellent, let alone the content material!

  65. 바다이야기说道:

    This very blog is without a doubt awesome and also amusing. I have discovered a bunch of interesting things out of it. I ad love to return every once in a while. Cheers!

  66. 다잡아说道:

    Outstanding post, I conceive people should acquire a lot from this website its rattling user genial. So much wonderful information on here .

  67. I think this is a real great article.Really thank you! Really Cool.

  68. It’а†s actually a great and useful piece of information. I am glad that you shared this useful info with us. Please keep us up to date like this. Thank you for sharing.

  69. Thanks so much for the blog post.Really thank you! Awesome.

  70. 토토 사이트说道:

    This page certainly has all the info I needed concerning this subject and didn at know who to ask.

  71. 토토사이트说道:

    I truly appreciate this post.Really looking forward to read more. Want more.

  72. Thanks so much for the blog post.Thanks Again. Really Great.

  73. 먹튀검증说道:

    I wanjt to encourage conttinue your reat work, have a nice holiday weekend!

  74. Tattoo Studio说道:

    You have observed very interesting details ! ps nice web site. I understand a fury in your words, But not the words. by William Shakespeare.

  75. 먹튀 폴리스说道:

    Very interesting subject , appreciate it for posting.

  76. 검증사이트说道:

    Thanks again for the blog.Really thank you!

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>