{"id":3450,"date":"2017-07-29T14:02:41","date_gmt":"2017-07-29T06:02:41","guid":{"rendered":"https:\/\/ixyzero.com\/blog\/?p=3450"},"modified":"2017-07-29T14:02:41","modified_gmt":"2017-07-29T06:02:41","slug":"linux%e6%95%85%e9%9a%9c%e6%8e%92%e9%99%a4%e6%89%8b%e5%86%8c%ef%bc%9astracehtoplsoftcpdumpiftopsysdig","status":"publish","type":"post","link":"https:\/\/ixyzero.com\/blog\/archives\/3450.html","title":{"rendered":"Linux\u6545\u969c\u6392\u9664\u624b\u518c\uff1astrace\/htop\/lsof\/tcpdump\/iftop\/sysdig"},"content":{"rendered":"

=Start=<\/p>\n

\u7f18\u7531\uff1a<\/h4>\n

\u5b66\u4e60\u4e00\u4e0b\u5404\u79cd\u5e38\u7528Linux\u7cfb\u7edf\u6545\u969c\u6392\u9664\u5de5\u5177\u7684\u4f7f\u7528\u3002<\/p>\n

\u6b63\u6587\uff1a<\/h4>\n

\u53c2\u8003\u89e3\u7b54\uff1a<\/h5>\n

This Sysdig cheatsheet is a great guide of command-lines linux admins can use to get insights into their servers. Whether you\u2019ve been an admin for one month or 20 years you\u2019ve definitely used one if not all of these tools to troubleshoot an issue. Because we love Sysdig (naturally!) we also included a translation for each of these common operations into the sysdig command line or csysdig.<\/p>\n

Rather than attempt covering all options from manpages (which would have made for boring coverage of many esoteric, rarely-used switches), we\u2019ve started from examples referenced at the most popular web pages you\u2019d find when you search for terms like \u201cstrace examples\u201d, \u201chtop examples\u201d, and so forth.<\/p>\n

Do you have favorites that aren\u2019t listed here? Let us know and we\u2019ll include them in future articles.<\/p>\n

strace<\/h2>\n

There\u2019s one subtle difference between strace and sysdig that will be apparent in many of these side-by-side comparisons: Many of the simplest strace examples include command-lines that are executed and traced as a \u201cone-shot\u201d operation. On the other hand, Sysdig has a somewhat different philosophy, in that it either watches live events from afar as they happen, or analyzes capture data previously saved to a file. Thankfully, Sysdig\u2019s rich filtering options provide the knobs to watch for specific one-shot executions, as you\u2019ll soon see.<\/p>\n

\n
\n\n\n\n\n\n<\/colgroup>\n\n\n\n\n\n\n\n\n\n\n\n
\n
Operation<\/div>\n<\/th>\n
\n
strace<\/div>\n<\/th>\n
\n
sysdig<\/div>\n<\/th>\n
\n
Note<\/div>\n<\/th>\n<\/tr>\n<\/thead>\n
Trace the execution of a command<\/td>\nstrace who<\/code><\/td>\nsysdig proc.name=who<\/code><\/td>\nWhereas strace runs the who command shown here as a one-shot, Sysdig is watching for the execution of who. Use Sysdig\u2019s filtering to further isolate a specific run, e.g.:<\/p>\n

sysdig proc.name=who and proc.ppid=534<\/code><\/p>\n

This watches for a who that\u2019s about to be run in a shell that you\u2019ve determined to have PID of 534.<\/td>\n<\/tr>\n

Trace only when certain\/specific system calls are made<\/td>\nstrace -e open who<\/code><\/p>\n

strace -e trace=open,read who<\/code><\/td>\n

sysdig evt.type=open and proc.name=who<\/code><\/p>\n

sysdig \"evt.type in (open,read) and proc.name=who\"<\/code><\/td>\n

<\/td>\n<\/tr>\n
Save a trace to a file<\/td>\nstrace -o output.txt who<\/code><\/td>\nsysdig -w output.scap proc.name=who<\/code><\/td>\nWith strace, the file produced contains the same text you\u2019d have viewed on the screen if run interactively. With Sysdig, you get a raw, re-usable capture file, such that you can view the text output with:<\/p>\n

sysdig -r output.scap<\/code><\/p>\n

You could also use this as the basis to apply filters or any other Sysdig functionality you want to apply as you revisit the original events.<\/td>\n<\/tr>\n

Watch a running process with PID=1363<\/td>\nstrace -p 1363<\/code><\/td>\nsysdig proc.pid=1363<\/code><\/td>\n<\/td>\n<\/tr>\n
Print a timestamp for each output line of the trace<\/td>\nstrace -t who<\/code><\/td>\nsysdig proc.name=who<\/code><\/td>\nSysdig prints timestamps by default.<\/td>\n<\/tr>\n
Print relative time for system calls<\/td>\nstrace -r who<\/code><\/td>\nsysdig -tD proc.name=who<\/code><\/td>\nSysdig offers several more ways to represent timestamps via the -t option.<\/td>\n<\/tr>\n
Generate batch statistics reports of system calls<\/td>\nstrace -c who<\/code><\/td>\nsysdig -w output.scap proc.name=who
\n# Now run the \u201cwho\u201d separately<\/code>For one-shot batch text reports:
\nsysdig -r output.scap -c topscalls -c topscalls_time<\/code><\/p>\n

Or for an interactive report that allows for further drill-down:
\ncsysdig -r output.scap -v syscalls<\/code><\/td>\n

Sysdig\u2019s default behavior is more optimized for the case of presenting event data as it happens rather than \u201cbatch\u201d reporting. This is why the Sysdig equivalent is done in two steps here.<\/td>\n<\/tr>\n
Generate live, per-second statistics reports of system calls for running process with PID=1363<\/td>\nN\/A<\/td>\ncsysdig -v syscalls proc.pid=1363<\/code><\/td>\nWhile strace can show individual events as they happen live, or provide a single batch report for the execution of a command, csysdig\u2019s views provide a unique ability to show live, periodic reports<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

htop<\/h3>\n

Since htop is a live, interactive, curses-style tool, we\u2019ll compare it to the live, interactive, curses-style csysdig.<\/p>\n

For starters, both tools use the same approach of navigating the live table via Up\/Down\/Left\/Right arrows and also PgUp\/PgDn. For operations that affect a single process (killing, renicing, etc.) it is assumed you\u2019ve used these controls to first highlight a particular process.<\/p>\n

\n
\n\n\n\n\n\n<\/colgroup>\n\n\n\n\n\n\n\n\n\n\n\n
\n
Operation<\/div>\n<\/th>\n
\n
htop<\/div>\n<\/th>\n
\n
csysdig<\/div>\n<\/th>\n
\n
Note<\/div>\n<\/th>\n<\/tr>\n<\/thead>\n
Change sort order based on a column of the table<\/td>\nPress\u00a0F6<\/code>,\u00a0<<\/code>, or\u00a0><\/code>\u00a0and then select a column by name, or
\nPress\u00a0M<\/code>,\u00a0P<\/code>, or\u00a0T<\/code>to sort by Memory, Processor Usage, or Time
\nPress\u00a0I<\/code>\u00a0to invert the sort order<\/td>\n		Press\u00a0F9<\/code>\u00a0or\u00a0><\/code>\u00a0and then select a column by name, or<\/p>\n

Press\u00a0shift<1-9><\/code>\u00a0to sort by any column\u00a0n<\/code>, and press repeatedly to invert sort order, or<\/p>\n

Mouse-click on a column header<\/td>\n

<\/td>\n<\/tr>\n
Kill a process<\/td>\nPressF9<\/code>\u00a0or\u00a0k<\/code><\/td>\nPressk<\/code><\/td>\n<\/td>\n<\/tr>\n
Renice a process<\/td>\nPress\u00a0F7<\/code>\u00a0or\u00a0]<\/code>\u00a0to reduce the nice value by 1
\nPress\u00a0F8<\/code>\u00a0or\u00a0[<\/code>\u00a0to increase the nice value by 1<\/td>\n		Press\u00a0]<\/code>\u00a0to reduce the nice value by 1
\nPress\u00a0[<\/code>\u00a0to increase the nice value by 1<\/td>\n		This illustrates how easy it is to customize Sysdig. I noticed when first writing this article that csysdig was missing a couple minor features like this, so I used the opportunity to learn how easy it is to write\/modify Chisels, then put up my improvements as a\u00a0Pull Request<\/a>. You can do the same!<\/td>\n<\/tr>\n
Display only processes started by a user named “phil”<\/td>\nPress\u00a0u<\/code>, then
\nSelect the user name\u00a0phil<\/code>\u00a0from the list<\/td>\n		Launch as:
\ncsysdig user.name=phil<\/code>Or mouse-click\u00a0Filter:<\/code>\u00a0from within\u00a0csysdig<\/code>\u00a0at the top of the default Processes view, then append\u00a0and user.name=phil<\/code>\u00a0to the current filter text<\/td>\n		<\/td>\n<\/tr>\n
Change the output refresh interval to once every 5 seconds<\/td>\nLaunch as:
\nhtop -d 50<\/code><\/td>\n		Launch as:
\ncsysdig -d 5000<\/code><\/td>\n		As you can see, htop works in units of tenths-of-a-second, while csysdig works in milliseconds.<\/td>\n<\/tr>\n
Start a system call trace on a process<\/td>\nPress\u00a0s<\/code>\u00a0to start an\u00a0strace<\/code><\/td>\nPress\u00a0F6<\/code>\u00a0to start a\u00a0sysdig<\/code><\/td>\n<\/td>\n<\/tr>\n
List open files for a process<\/td>\nPress\u00a0l<\/code>\u00a0to run a one-time\u00a0lsof<\/code><\/td>\nPress\u00a0f<\/code>\u00a0to run a one-time lsof Or to see real-time, updating reports of files\/directories used by a process, drill down to a specific process by pressing\u00a0Enter<\/code>, then press\u00a0F2<\/code>\u00a0and select a View such as\u00a0Files<\/code>,\u00a0File Opens List<\/code>, or\u00a0Directories<\/code>.<\/td>\nSee the Note above for \u201cRenice a process\u201d about how the one-time\u00a0lsof<\/code>\u00a0was recently added as an enhancement.<\/td>\n<\/tr>\n
Follow a process, such that it remains highlighted even as its order in the list changes<\/td>\nPress\u00a0F<\/code><\/td>\nDefault behavior is to always follow the highlighted process<\/td>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

lsof<\/h3>\n
\n
\n\n\n\n\n\n<\/colgroup>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
Operation<\/div>\n<\/th>\n
\n
lsof<\/div>\n<\/th>\n
\n
csysdig<\/div>\n<\/th>\n
\n
Note<\/div>\n<\/th>\n<\/tr>\n<\/thead>\n
List all open files belonging to all active processes<\/td>\nlsof<\/code><\/td>\nsysdig -c lsof<\/code><\/td>\n<\/td>\n<\/tr>\n
List processes that have opened the specific file \/var\/log\/syslog<\/td>\nlsof \/var\/log\/syslog<\/code><\/td>\nsysdig -c lsof \"fd.name=\/var\/log\/syslog\"<\/code><\/td>\n<\/td>\n<\/tr>\n
List processes that have opened files under the directory \/var\/log<\/td>\nlsof +d \/var\/log<\/code><\/td>\nsysdig -c lsof \"fd.directory=\/var\/log\"<\/code><\/td>\n<\/td>\n<\/tr>\n
List files opened by processes named \u201csshd\u201d<\/td>\nlsof -c sshd<\/code><\/td>\nsysdig -c lsof \"proc.name=sshd\"<\/code><\/td>\n<\/td>\n<\/tr>\n
List files opened by a specific user named \u201cphil\u201d<\/td>\nlsof -u phil<\/code><\/td>\nsysdig -c lsof \"user.name=phil\"<\/code><\/td>\n<\/td>\n<\/tr>\n
List files opened by everyone except for the user named \u201cphil\u201d<\/td>\nlsof -u ^phil<\/code><\/td>\nsysdig -c lsof \"user.name!=phil\"<\/code><\/td>\n<\/td>\n<\/tr>\n
List all open files for a specific process with PID=1081<\/td>\nlsof -p 1081<\/code><\/td>\nsysdig -c lsof \"proc.pid=1081\"<\/code><\/td>\n<\/td>\n<\/tr>\n
List all files opened by user “phil” or a process named “sshd” (OR logic)<\/td>\nlsof -u phil -c sshd<\/code><\/td>\nsysdig -c lsof \"'user.name=phil or proc.name=sshd'\"<\/code><\/td>\nNote the use of two layers of quotes with the Sysdig filter.<\/td>\n<\/tr>\n
List all files opened by an “sshd” process for user “phil” (AND logic)<\/td>\nlsof -u phil -c sshd -a<\/code><\/td>\nsysdig -c lsof \"'user.name=phil and proc.name=sshd'\"<\/code><\/td>\nNote the use of two layers of quotes with the Sysdig filter.<\/td>\n<\/tr>\n
Observe repeating reports of open files based on live activity<\/td>\nEnable repeat mode with one of:
\nlsof -r<\/code>
\nlsof +r<\/code><\/td>\n		Similar live data can be obtained with a live\/interactive csysdig view, launched like so:
\ncsysdig -v files<\/code>
\ncsysdig -v file_opens<\/code><\/td>\n		<\/td>\n<\/tr>\n
List all network connections<\/td>\nlsof -i<\/code><\/td>\nsysdig -c lsof \"fd.type=ipv4\"<\/code><\/td>\n<\/td>\n<\/tr>\n
List network connections in use by a specific process with PID=1014<\/td>\nlsof -i -a -p 1014<\/code><\/td>\nsysdig -c lsof \"'fd.type=ipv4 and proc.pid=1014'\"<\/code><\/td>\nNote the use of two layers of quotes with the Sysdig filter.<\/td>\n<\/tr>\n
List processes that are listening on port 22<\/td>\nlsof -i :22<\/code><\/td>\nsysdig -c lsof \"'fd.port=22 and fd.is_server=true'\"<\/code><\/td>\nNote the use of two layers of quotes with the Sysdig filter.<\/td>\n<\/tr>\n
List all TCP or UDP connections<\/td>\nlsof -i tcp<\/code><\/p>\n

lsof -i udp<\/code><\/td>\n

sysdig -c lsof \"fd.l4proto=tcp\"<\/code><\/p>\n

sysdig -c lsof \"fd.l4proto=udp\"<\/code><\/td>\n

<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

tcpdump<\/h3>\n

tcpdump is focused entirely on network traffic, while network traffic is only a subset of what Sysdig covers. Many tcpdump use cases involve filtering, and tcpdump uses network-specific\u00a0BPF filters<\/a>, whereas Sysdig uses its own broader\u00a0Sysdig filtering<\/a>. The two approaches look similar in many ways, but you\u2019ll want to look at the docs for each side-by-side as you progress to more advanced filtering needs. Also, since in Linuxeverything is a file<\/a>, you\u2019ll notice the Sysdig filtering examples below all leverage a \u201cnetwork-connections-via-file-descriptors\u201d approach.<\/p>\n

\n
\n\n\n\n\n\n<\/colgroup>\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
Operation<\/div>\n<\/th>\n
\n
tcpdump<\/div>\n<\/th>\n
\n
csysdig<\/div>\n<\/th>\n
\n
Note<\/div>\n<\/th>\n<\/tr>\n<\/thead>\n
Capture packets from a particular interface eth0 (192.168.10.119)<\/td>\ntcpdump -i eth0<\/code><\/td>\nsysdig fd.ip=192.168.10.119<\/code><\/td>\nSysdig does not currently have filtering based on named interfaces, but the equivalent via IP address is shown here.<\/td>\n<\/tr>\n
Capture only 100 packets<\/td>\ntcpdump -c 100<\/code><\/td>\nsysdig -n 100 fd.type=ipv4<\/code><\/td>\n<\/td>\n<\/tr>\n
Display captured packets in ASCII<\/td>\ntcpdump -A<\/code><\/td>\nsysdig -A fd.type=ipv4<\/code><\/td>\n<\/td>\n<\/tr>\n
Display captured packets in HEX and ASCII<\/td>\ntcpdump -XX<\/code><\/td>\nsysdig -X fd.type=ipv4<\/code><\/td>\n<\/td>\n<\/tr>\n
Capture packet data, writing it into into a file<\/td>\ntcpdump -w saved.pcap<\/code><\/td>\nsysdig -w saved.scap fd.type=ipv4<\/code><\/td>\nThe Sysdig file format is capable of holding event data for much more than just network packets (e.g. system calls).<\/td>\n<\/tr>\n
Read back saved packet data from a file<\/td>\ntcpdump -r saved.pcap<\/code><\/td>\nsysdig -r saved.scap<\/code><\/td>\n<\/td>\n<\/tr>\n
Capture only packets longer\/smaller than 1024 bytes<\/td>\ntcpdump greater 1024<\/code><\/p>\n

tcpdump less 1024<\/code><\/td>\n

sysdig \"fd.type=ipv4 and evt.buflen > 1024\"<\/code><\/p>\n

sysdig \"fd.type=ipv4 and evt.buflen < 1024\"<\/code><\/td>\n

The\u00a0greater\/less<\/code>\u00a0options in tcpdump reference overall packet length whereas\u00a0evt.buflen<\/code>\u00a0in Sysdig is relative to payload size.<\/td>\n<\/tr>\n
Capture only UDP or TCP packets<\/td>\ntcpdump udp<\/code><\/p>\n

tcpdump tcp<\/code><\/td>\n

sysdig fd.l4proto=udp<\/code><\/p>\n

sysdig fd.l4proto=tcp<\/code><\/td>\n

Note that we don\u2019t need to explicitly include\u00a0fd.type=ipv4<\/code>since we\u2019re using other network-only filters here.<\/td>\n<\/tr>\n
Capture only packets going to\/from a particular port<\/td>\ntcpdump port 22<\/code><\/td>\nsysdig fd.port=22<\/code><\/td>\nNote that we don\u2019t need to explicitly include\u00a0fd.type=ipv4<\/code>since we\u2019re using other network-only filters here.<\/td>\n<\/tr>\n
Capture packets for a particular destination IP and port<\/td>\ntcpdump dst 54.165.81.189 and port 6666<\/code><\/td>\nsysdig fd.rip=54.165.81.189 and fd.port=6666<\/code><\/td>\nNote that we don\u2019t need to explicitly include\u00a0fd.type=ipv4<\/code>since we\u2019re using other network-only filters here.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

iftop<\/h3>\n

Since iftop is a live, interactive, curses-style tool, we\u2019ll compare it to the live, interactive, curses-style csysdig. Also, like tcpdump, iftop uses BPF filters. See the previous intro to the section on tcpdump for more detail about filtering differences.<\/p>\n

\n
\n\n\n\n\n\n<\/colgroup>\n\n\n\n\n\n\n\n\n\n\n\n
\n
Operation<\/div>\n<\/th>\n
\n
iftop<\/div>\n<\/th>\n
\n
csysdig<\/div>\n<\/th>\n
\n
Note<\/div>\n<\/th>\n<\/tr>\n<\/thead>\n
Display a table of current bandwidth usage between pairs of hosts<\/td>\niftop<\/code><\/td>\nLaunch as:
\ncsysdig -v connections<\/code>Or press\u00a0F2<\/code>\u00a0from within\u00a0csysdig<\/code>\u00a0to change the View, then up-arrow to select\u00a0Connections<\/code><\/td>\n		By default iftop watches just the first interface it finds, whereas by default csysdig watches traffic across the entire host.<\/td>\n<\/tr>\n
Turn on display of network ports<\/td>\nLaunch as:
\niftop -P<\/code>Or press\u00a0p<\/code>\u00a0from within\u00a0iftop<\/code><\/td>\n		Default behavior is to always display ports<\/td>\n<\/td>\n<\/tr>\n
Observe traffic for just the eth0 interface (192.168.10.119)<\/td>\nLaunch as:
\niftop -i eth0<\/code><\/td>\n		Launch as:
\ncsysdig -v connections fd.ip=192.168.10.119<\/code>Or mouse-click on\u00a0Filter:<\/code>\u00a0from within\u00a0csysdig<\/code>, then append and\u00a0fd.ip=192.168.10.119<\/code>\u00a0to the existing filter text<\/td>\n		sysdig\/csysdig do not currently have filtering based on named interfaces, but the equivalent via IP address is shown here.<\/td>\n<\/tr>\n
Resolve DNS names<\/td>\nPress\u00a0n<\/code>\u00a0from within\u00a0iftop<\/code>\u00a0to toggle resolution for all hosts shown<\/td>\nPress\u00a0n<\/code>\u00a0from within\u00a0csysdig<\/code>\u00a0to run\u00a0nslookup<\/code>\u00a0on the currently-highlighted remote host<\/td>\n<\/td>\n<\/tr>\n
Change sort order based on a column of the table<\/td>\nPress < to sort by source Press > to sort by destination<\/td>\nPress\u00a0F9<\/code>\u00a0or\u00a0><\/code>\u00a0and then select a column by name, or<\/p>\n

Press\u00a0shift <1-9><\/code>\u00a0to sort by any column\u00a0n<\/code>, and press repeatedly to invert sort order, or<\/p>\n

Mouse-click on a column header<\/td>\n

<\/td>\n<\/tr>\n
Filter to show only traffic going to\/from IP address 54.84.222.1<\/td>\nLaunch as:
\niftop -f \"host 54.84.222.1\"<\/code><\/td>\n		Launch as:
\ncsysdig -v connections fd.ip=54.84.222.1<\/code>Or mouse-click on\u00a0Filter:<\/code>\u00a0from within csysdig, then append\u00a0and fd.ip=54.84.22.1<\/code>\u00a0to the existing filter text<\/td>\n		<\/td>\n<\/tr>\n
Pause the display<\/td>\nPress\u00a0P<\/code><\/td>\nPress\u00a0p<\/code><\/td>\n<\/td>\n<\/tr>\n
Scroll the display<\/td>\nPress\u00a0j<\/code>\u00a0to scroll up
\nPress\u00a0k<\/code>\u00a0to scroll down<\/td>\n		Press\u00a0Up\/Down\/Left\/Right<\/code>\u00a0arrows or\u00a0PgUp\/PgDn<\/code>\u00a0to scroll through the table<\/td>\nsysdig\/csysdig go well beyond scrolling through a single-table, since you can drill down into the Connections View to see data in other groupings such as per-container or per-thread.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

==<\/p>\n

\u53c2\u8003\u94fe\u63a5\uff1a<\/h5>\n