SystemTap 3.0

SystemTap Beginners Guide

Introduction to SystemTap

Edition 3.0

Red Hat, Inc.

Don Domingo

Engineering Services and Operations Content Services

[ddomingo@redhat.com](https://www.oschina.net/action/GoToLink?url=mailto%3Addomingo%40redhat.com)

William Cohen

Engineering Services and Operations Performance Tools

[wcohen@redhat.com](https://www.oschina.net/action/GoToLink?url=mailto%3Awcohen%40redhat.com)

Copyright © 2013 Red Hat, Inc

Legal Notice

This documentation is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License version 2 as published by the Free Software Foundation.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

For more details see the file COPYING in the source distribution of Linux.

Abstract

This guide provides basic instructions on how to use SystemTap to monitor different subsystems of a Linux system in finer detail.

Preface

1. Document Conventions

1.1. Typographic Conventions

1.2. Pull-quote Conventions

1.3. Notes and Warnings

2. We Need Feedback!

1. Introduction

1.1. Documentation Goals

1.2. SystemTap Capabilities

1.3. Limitations of SystemTap

2. Using SystemTap

2.1. Installation and Setup

2.1.1. Installing SystemTap

2.1.2. Installing Required Kernel Information Packages Manually

2.1.3. Initial Testing

2.2. Generating Instrumentation for Other Computers

2.3. Running SystemTap Scripts

2.3.1. SystemTap Flight Recorder Mode

3. Understanding How SystemTap Works

3.1. Architecture

3.2. SystemTap Scripts

3.2.1. Event

3.2.2. SystemTap Handler/Body

3.3. Basic SystemTap Handler Constructs

3.3.1. Variables

3.3.2. Target Variables

3.3.3. Conditional Statements

3.3.4. Command-Line Arguments

3.4. Associative Arrays

3.5. Array Operations in SystemTap

3.5.1. Assigning an Associated Value

3.5.2. Reading Values From Arrays

3.5.3. Incrementing Associated Values

3.5.4. Processing Multiple Elements in an Array

3.5.5. Clearing/Deleting Arrays and Array Elements

3.5.6. Using Arrays in Conditional Statements

3.5.7. Computing for Statistical Aggregates

3.6. Tapsets

4. User-space Probing

4.1. User-Space Events

4.2. Accessing User-Space Target Variables

4.3. User-Space Stack Backtraces

5. Useful SystemTap Scripts

5.1. Network

5.1.1. Network Profiling

5.1.2. Tracing Functions Called in Network Socket Code

5.1.3. Monitoring Incoming TCP Connections

5.1.4. Monitoring TCP Packets

5.1.5. Monitoring Network Packets Drops in Kernel

5.2. Disk

5.2.1. Summarizing Disk Read/Write Traffic

5.2.2. Tracking I/O Time For Each File Read or Write

5.2.3. Track Cumulative IO

5.2.4. I/O Monitoring (By Device)

5.2.5. Monitoring Reads and Writes to a File

5.2.6. Monitoring Changes to File Attributes

5.2.7. Periodically Print I/O Block Time

5.3. Profiling

5.3.1. Counting Function Calls Made

5.3.2. Call Graph Tracing

5.3.3. Determining Time Spent in Kernel and User Space

5.3.4. Monitoring Polling Applications

5.3.5. Tracking Most Frequently Used System Calls

5.3.6. Tracking System Call Volume Per Process

5.4. Identifying Contended User-Space Locks

6. Understanding SystemTap Errors

6.1. Parse and Semantic Errors

6.2. Runtime Errors and Warnings

7. References

A. Revision History

Index