Unix and Linux Stack Exchange
Unix and Linux Stack Exchange

Questions tagged [awk]

A pattern-directed scanning and processing language.

Awk is a text processing tool with a far more expressive manipulation language than .
It has named scalar and array variables, functions, and common imperative control structures.

Implementations and documentation

Related tags

  • Text processing in general, when you don't know which tool is best
  • , , and other text processing tools
  • , for more general programming languages that can be used for text processing

Further reading

7608 questions
921
votes
10 answers

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub directories? replace only if the file name matches…
terdon
  • 220,769
  • 58
  • 415
  • 622
110
votes
7 answers

Is there a basic tutorial for grep, awk and sed?

I've been a Linux user for a while, and I've a pretty decent understanding of most the common command line utilities. However, ones that come up and up again in relation to programming are grep, awk, and sed. About the only thing I've used grep for…
Macha
  • 3,680
  • 7
  • 29
  • 34
108
votes
13 answers

Show sum of file sizes in directory listing

The Windows dir directory listing command has a line at the end showing the total amount of space taken up by the files listed. For example, dir *.exe shows all the .exe files in the current directory, their sizes, and the sum total of their sizes.…
MattDMo
  • 2,144
  • 3
  • 18
  • 28
102
votes
5 answers

Using awk to sum the values of a column, based on the values of another column

I am trying to sum certain numbers in a column using awk. I would like to sum just column 3 of the "smiths" to get a total of 212. I can sum the whole column using awk but not just the "smiths". I have: awk 'BEGIN {FS = "|"} ; {sum+=$3} END {print…
jake
  • 1,165
  • 2
  • 9
  • 9
98
votes
5 answers

Remove line containing certain string and the following line

I use this cat foo.txt | sed '/bar/d' to remove lines containing the string bar in the file. I would like however to remove those lines and the line directly after it. Preferably in sed, awk or other tool that's available in MinGW32. It's a kind of…
jakub.g
  • 2,943
  • 4
  • 19
  • 17
88
votes
6 answers

How to use a shell command to only show the first column and last column in a text file?

I need some help to figure out how to use the sed command to only show the first column and last column in a text file. Here is what I have so far for column 1: cat logfile | sed 's/\|/ /'|awk '{print $1}' My feeble attempt at getting the last…
user70573
  • 1,119
  • 1
  • 8
  • 7
87
votes
7 answers

How do you list number of lines of every file in a directory in human readable format.

I have a list of directories and subdirectories that contain large csv files. There are about 500 million lines in these files, each is a record. I would like to know How many lines are in each file. How many lines are in directory. How many…
Hexatonic
  • 1,095
  • 2
  • 8
  • 9
86
votes
7 answers

How to read first and last line from cat output?

I have text file. Task - get first and last line from file after $ cat file | grep -E "1|2|3|4" | commandtoprint $ cat file 1 2 3 4 5 Need this without cat output (only 1 and 5). ~$ cat file | tee >(head -n 1) >(wc -l) 1 2 3 4 5 5 1 Maybe awk…
dmgl
  • 993
  • 2
  • 9
  • 9
84
votes
13 answers

How to parse JSON with shell scripting in Linux?

I have a JSON output from which I need to extract a few parameters in Linux. This is the JSON output: { "OwnerId": "121456789127", "ReservationId": "r-48465168", "Groups": [], "Instances": [ { …
user3086014
  • 941
  • 1
  • 7
  • 6
80
votes
2 answers

Replacing string based on line number

I have a situation where i want to replace a particular string in many files Replace a string AAA with another string BBB but there are lot of strings starting with AAA or ending in AAA ,and i want to replace only one on line 34 and keep others…
krypto
  • 901
  • 1
  • 6
  • 5
76
votes
5 answers

Pass shell variable as a /pattern/ to awk

Having the following in one of my shell functions: function _process () { awk -v l="$line" ' BEGIN {p=0} /'"$1"'/ {p=1} END{ if(p) print l >> "outfile.txt" } ' } , so when called as _process $arg, $arg gets passed as $1, and used as a…
branquito
  • 997
  • 1
  • 8
  • 17
71
votes
1 answer

Difference between gawk vs. awk

Trying to understand the differences between the two functions gawk vs. awk? When would one use gawk vs awk? Or are they the same in terms of usage? Also, could one provide an example?
chrisjlee
  • 7,813
  • 14
  • 47
  • 52
67
votes
4 answers

Skip the first 6 lines/rows in a text file with awk

How can I skip the first 6 lines/rows in a text file (input.txt) and process the rest with awk? The format of my awk script (program.awk) is: BEGIN { } { process here } END { } My text file is like this: 0 3 5 0.1 4.3 2.0 1.5 1.5 3.0 0.3…
amatek
  • 783
  • 1
  • 5
  • 7
65
votes
4 answers

How does awk '!a[$0]++' work?

This one-liner removes duplicate lines from text input without pre-sorting. For example: $ cat >f q w e w r $ awk '!a[$0]++'
Alexander Shcheblikin
  • 1,653
  • 1
  • 14
  • 16
64
votes
4 answers

How to view all the content in an awk array?

In my understanding, awk array is something like python dict. So I write down the code bellow to explore it: awk '{my_dict[$1] = $2} END { print my_dict}' zen And I got: awk: can't read value of my_dict; it's an array name. As the first column…
Zen
  • 6,997
  • 17
  • 46
  • 69
1
2 3
…
99 100