Scrolls
3 min read
On this page

IO & File Operations

CLI tools live on file operations: reading input, writing output, processing files, walking directories. Go's standard library provides everything you need. The os package handles files, bufio handles buffered IO, filepath handles cross-platform paths, and io connects them all through the Reader/Writer interfaces.

Reading Files

os.ReadFile: Small Files

For files that fit comfortably in memory:

data, err := os.ReadFile("config.json")
if err != nil {
    return fmt.Errorf("reading config: %w", err)
}
// data is []byte

os.Open + bufio.Scanner: Line by Line

For large files or when you need to process line by line:

f, err := os.Open("access.log")
if err != nil {
    return fmt.Errorf("opening log: %w", err)
}
defer f.Close()

scanner := bufio.NewScanner(f)
lineNum := 0
for scanner.Scan() {
    lineNum++
    line := scanner.Text() // string, without newline
    if strings.Contains(line, "ERROR") {
        fmt.Printf("Line %d: %s\n", lineNum, line)
    }
}
if err := scanner.Err(); err != nil {
    return fmt.Errorf("scanning log: %w", err)
}

Handling Long Lines

The default scanner buffer is 64KB. For files with longer lines:

scanner := bufio.NewScanner(f)
scanner.Buffer(make([]byte, 1024*1024), 1024*1024) // 1MB buffer

Reading with a Specific Delimiter

bufio.Scanner defaults to splitting on newlines. Use a custom split function for other delimiters:

scanner := bufio.NewScanner(f)
scanner.Split(bufio.ScanWords) // split on whitespace

for scanner.Scan() {
    word := scanner.Text()
    fmt.Println(word)
}

Writing Files

os.WriteFile: Simple Writes

data := []byte("Hello, World!\n")
err := os.WriteFile("output.txt", data, 0644)
if err != nil {
    return fmt.Errorf("writing file: %w", err)
}

The 0644 is the file permission: owner read/write, group and others read-only.

os.Create + Buffered Writing

For larger outputs or when you build content incrementally:

f, err := os.Create("report.csv")
if err != nil {
    return fmt.Errorf("creating file: %w", err)
}
defer f.Close()

w := bufio.NewWriter(f)
defer w.Flush() // important: flush buffered data

for _, record := range records {
    fmt.Fprintf(w, "%s,%d,%f\n", record.Name, record.Count, record.Value)
}

Always call Flush on a bufio.Writer. Without it, the last chunk of data stays in the buffer and never reaches the file.

Appending to Files

f, err := os.OpenFile("app.log", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
    return err
}
defer f.Close()

fmt.Fprintf(f, "%s: %s\n", time.Now().Format(time.RFC3339), message)

filepath: Cross-Platform Paths

Never build file paths with string concatenation. Use filepath:

// Join path components (uses correct separator for OS)
path := filepath.Join("data", "users", "config.json")
// Linux: "data/users/config.json"
// Windows: "data\users\config.json"

// Get directory and filename
dir := filepath.Dir("/home/user/docs/report.pdf")   // "/home/user/docs"
base := filepath.Base("/home/user/docs/report.pdf")  // "report.pdf"
ext := filepath.Ext("report.pdf")                     // ".pdf"

// Remove extension
name := strings.TrimSuffix("report.pdf", filepath.Ext("report.pdf")) // "report"

// Resolve to absolute path
abs, err := filepath.Abs("../config.json")

// Clean up messy paths
clean := filepath.Clean("./data/../data/./users/") // "data/users"

Working with stdin & stdout

CLI tools should work with pipes. Read from os.Stdin, write to os.Stdout:

func main() {
    inputFile := flag.String("input", "-", "input file (- for stdin)")
    flag.Parse()

    var reader io.Reader
    if *inputFile == "-" {
        reader = os.Stdin
    } else {
        f, err := os.Open(*inputFile)
        if err != nil {
            fmt.Fprintf(os.Stderr, "error: %v\n", err)
            os.Exit(1)
        }
        defer f.Close()
        reader = f
    }

    // Process input from either file or stdin
    scanner := bufio.NewScanner(reader)
    for scanner.Scan() {
        processLine(scanner.Text())
    }
}

This lets your tool work both ways:

cat data.txt | mytool
mytool -input data.txt

Always write errors to os.Stderr, not os.Stdout. This keeps stdout clean for piping:

fmt.Fprintln(os.Stderr, "warning: skipping malformed line")

Temp Files & Directories

Use os.CreateTemp and os.MkdirTemp for temporary storage:

// Temp file in system temp dir
tmpFile, err := os.CreateTemp("", "myapp-*.json")
if err != nil {
    return err
}
defer os.Remove(tmpFile.Name()) // clean up when done
defer tmpFile.Close()

fmt.Fprintln(tmpFile, `{"status": "processing"}`)

The * in the pattern is replaced with a random string. The first argument is the directory (empty string means system default).

// Temp directory
tmpDir, err := os.MkdirTemp("", "myapp-")
if err != nil {
    return err
}
defer os.RemoveAll(tmpDir) // clean up entire directory

outputPath := filepath.Join(tmpDir, "result.csv")

Walking Directory Trees

filepath.WalkDir

func findGoFiles(root string) ([]string, error) {
    var files []string
    err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err // permission error, etc.
        }
        if d.IsDir() && d.Name() == "vendor" {
            return filepath.SkipDir // skip vendor directory
        }
        if filepath.Ext(path) == ".go" {
            files = append(files, path)
        }
        return nil
    })
    return files, err
}

os.ReadDir: One Level

When you only need the immediate children:

entries, err := os.ReadDir(".")
if err != nil {
    return err
}
for _, entry := range entries {
    info, _ := entry.Info()
    fmt.Printf("%s  %8d  %s\n",
        info.Mode(), info.Size(), entry.Name(),
    )
}

filepath.Glob: Pattern Matching

matches, err := filepath.Glob("data/*.csv")
// matches: ["data/users.csv", "data/orders.csv"]

File Permissions

Go uses Unix-style permission bits:

// Owner: read/write, Group: read, Others: read
os.WriteFile("config.json", data, 0644)

// Owner: read/write/execute, Group: read/execute, Others: read/execute
os.Mkdir("bin", 0755)

// Owner only
os.WriteFile("secrets.env", data, 0600)

Check if a file exists:

if _, err := os.Stat("config.json"); os.IsNotExist(err) {
    fmt.Println("config file not found")
}

Copying Files

Go has no os.Copy. Use io.Copy with readers and writers:

func copyFile(src, dst string) error {
    source, err := os.Open(src)
    if err != nil {
        return err
    }
    defer source.Close()

    dest, err := os.Create(dst)
    if err != nil {
        return err
    }
    defer dest.Close()

    _, err = io.Copy(dest, source)
    return err
}

io.Copy streams data without loading the entire file into memory.

Common Pitfalls

  • Not closing files. Always defer f.Close() immediately after opening. Leaked file descriptors crash long-running programs.
  • Forgetting to flush bufio.Writer. Buffered writers hold data in memory. Without Flush, the last chunk is lost.
  • Building paths with string concatenation. "dir/" + filename breaks on Windows. Always use filepath.Join.
  • Reading entire large files into memory. Use bufio.Scanner or io.Copy for large files. os.ReadFile is only for files that fit comfortably in RAM.
  • Writing errors to stdout. Errors on stdout corrupt piped output. Always use os.Stderr for error messages.
  • Not handling scanner buffer overflow. Lines longer than 64KB silently fail. Set a larger buffer for files with long lines.
  • Using os.Remove on temp directories. Use os.RemoveAll for directories.

Key Takeaways

  • os.ReadFile and os.WriteFile handle small files. bufio.Scanner and bufio.Writer handle large files.
  • Always use filepath.Join for cross-platform path construction.
  • CLI tools should read from stdin by default and write errors to stderr.
  • Use os.CreateTemp and os.MkdirTemp for temporary files, always cleaning up with defer.
  • filepath.WalkDir traverses directory trees. filepath.SkipDir prunes branches.
  • io.Copy streams data between any Reader and Writer without loading everything into memory.
  • Always close files, flush buffers, and clean up temporary resources.