I/O & Strings
The io package defines the interfaces that connect everything in Go. Files, network connections, HTTP bodies, compression streams, and buffers all speak the same language: io.Reader and io.Writer. The string and byte manipulation packages build on these interfaces to provide efficient text processing. The standard library is your framework.
io.Reader & io.Writer
These two interfaces are the foundation of Go's I/O system.
type Reader interface {
Read(p []byte) (n int, err error)
}
type Writer interface {
Write(p []byte) (n int, err error)
}
Any type that implements Read can be a data source. Any type that implements Write can be a data destination. This uniformity means a function written for one data source works with all of them.
func WordCount(r io.Reader) (int, error) {
scanner := bufio.NewScanner(r)
scanner.Split(bufio.ScanWords)
count := 0
for scanner.Scan() {
count++
}
return count, scanner.Err()
}
func main() {
// Count words in a string
r := strings.NewReader("the quick brown fox jumps over the lazy dog")
n, _ := WordCount(r)
fmt.Println("Words:", n)
// Count words in a file — same function
f, _ := os.Open("essay.txt")
defer f.Close()
n, _ = WordCount(f)
fmt.Println("Words:", n)
}
Words: 9
Words: 2847
strings.Builder for Efficient Concatenation
String concatenation with + creates a new string on every operation. For building strings incrementally, strings.Builder is far more efficient.
func BuildReport(items []Item) string {
var sb strings.Builder
sb.WriteString("Report\n")
sb.WriteString("======\n\n")
for _, item := range items {
fmt.Fprintf(&sb, "%-20s $%.2f\n", item.Name, item.Price)
}
fmt.Fprintf(&sb, "\nTotal items: %d\n", len(items))
return sb.String()
}
strings.Builder implements io.Writer, so you can pass it to fmt.Fprintf and any other function that accepts a writer.
func BenchmarkConcat(b *testing.B) {
for i := 0; i < b.N; i++ {
s := ""
for j := 0; j < 1000; j++ {
s += "x"
}
}
}
func BenchmarkBuilder(b *testing.B) {
for i := 0; i < b.N; i++ {
var sb strings.Builder
for j := 0; j < 1000; j++ {
sb.WriteString("x")
}
_ = sb.String()
}
}
BenchmarkConcat-8 1234 812345 ns/op 530048 B/op 999 allocs/op
BenchmarkBuilder-8 234567 5123 ns/op 3072 B/op 6 allocs/op
The builder is over 100 times faster for this case.
bytes.Buffer
bytes.Buffer is similar to strings.Builder but works with byte slices and implements both io.Reader and io.Writer. Use it when you need to read and write, or when you are working with bytes rather than strings.
func CompressData(data []byte) ([]byte, error) {
var buf bytes.Buffer
w := gzip.NewWriter(&buf)
_, err := w.Write(data)
if err != nil {
return nil, err
}
if err := w.Close(); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
bytes.Buffer also works well as a test double for any io.Writer or io.Reader.
func TestLogger(t *testing.T) {
var buf bytes.Buffer
logger := NewLogger(&buf)
logger.Info("test message")
output := buf.String()
if !strings.Contains(output, "test message") {
t.Errorf("log output missing message: %s", output)
}
}
fmt for Formatting
The fmt package provides formatted I/O. The key functions form a pattern.
// Print to stdout
fmt.Println("Hello, world")
fmt.Printf("Name: %s, Age: %d\n", name, age)
// Format to a string
s := fmt.Sprintf("error at line %d: %s", line, msg)
// Write to any io.Writer
fmt.Fprintf(w, "HTTP %d %s\n", code, status)
Common format verbs:
fmt.Printf("%s", "string") // string
fmt.Printf("%d", 42) // integer
fmt.Printf("%f", 3.14) // float
fmt.Printf("%.2f", 3.14159) // float with precision
fmt.Printf("%v", myStruct) // default format
fmt.Printf("%+v", myStruct) // with field names
fmt.Printf("%#v", myStruct) // Go syntax representation
fmt.Printf("%T", myStruct) // type name
fmt.Printf("%q", "has \"quotes\"") // quoted string
fmt.Printf("%x", []byte("abc")) // hex encoding
strconv for Type Conversion
The strconv package converts between strings and basic types.
// String to int
n, err := strconv.Atoi("42")
// Int to string
s := strconv.Itoa(42)
// String to float
f, err := strconv.ParseFloat("3.14", 64)
// String to bool
b, err := strconv.ParseBool("true")
// Int to string with specific base
s = strconv.FormatInt(255, 16) // "ff"
Use strconv instead of fmt.Sprintf for simple conversions — it is faster because it avoids the reflection overhead of the fmt package.
// Slower
s := fmt.Sprintf("%d", n)
// Faster
s := strconv.Itoa(n)
Composing Readers & Writers
The real power of Go's I/O system is composition. Readers and writers wrap each other like layers.
io.Copy
io.Copy connects a reader to a writer, streaming data without loading it all into memory.
func DownloadFile(url string, filepath string) error {
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
f, err := os.Create(filepath)
if err != nil {
return err
}
defer f.Close()
_, err = io.Copy(f, resp.Body)
return err
}
io.TeeReader
io.TeeReader creates a reader that writes everything it reads to a writer. It is like the Unix tee command.
func HashAndStore(r io.Reader, w io.Writer) (string, error) {
h := sha256.New()
tee := io.TeeReader(r, h)
// Copy data to the writer; the hash is computed as a side effect
_, err := io.Copy(w, tee)
if err != nil {
return "", err
}
return hex.EncodeToString(h.Sum(nil)), nil
}
io.MultiWriter
io.MultiWriter creates a writer that duplicates writes to multiple destinations.
func SetupLogging(logFile string) (io.Writer, error) {
f, err := os.OpenFile(logFile, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
return nil, err
}
// Write to both stdout and the log file
multi := io.MultiWriter(os.Stdout, f)
return multi, nil
}
func main() {
w, err := SetupLogging("app.log")
if err != nil {
log.Fatal(err)
}
fmt.Fprintln(w, "Application started")
}
Layered I/O
You can stack readers and writers to build complex pipelines.
func ReadCompressedJSON(r io.Reader, v any) error {
// Layer 1: decompress
gz, err := gzip.NewReader(r)
if err != nil {
return err
}
defer gz.Close()
// Layer 2: buffer for performance
buffered := bufio.NewReader(gz)
// Layer 3: decode JSON
return json.NewDecoder(buffered).Decode(v)
}
This function reads gzip-compressed JSON from any io.Reader — a file, an HTTP response body, a byte buffer, or a network connection.
io.ReadAll & io.LimitReader
io.ReadAll reads everything from a reader into a byte slice. Use it when you need the entire contents in memory.
body, err := io.ReadAll(resp.Body)
For untrusted input, combine it with io.LimitReader to prevent memory exhaustion.
// Read at most 1 MB
limited := io.LimitReader(resp.Body, 1<<20)
body, err := io.ReadAll(limited)
Common Pitfalls
- Using string concatenation in loops. Use
strings.Builderorbytes.Bufferinstead of+=in a loop. The performance difference grows with the number of iterations. - Reading everything into memory. Use
io.Copyfor streaming when you do not need the entire content at once. A large file streamed throughio.Copyuses kilobytes; read withio.ReadAllit uses the file's entire size. - Not closing readers. Readers that wrap resources (files, HTTP bodies, gzip streams) must be closed. Use
deferimmediately after creation. - Ignoring the return values of Write.
Writereturns the number of bytes written and an error. Check both, especially when writing to network connections or files. - Using fmt.Sprintf for simple conversions.
strconv.Itoaandstrconv.FormatFloatare faster thanfmt.Sprintffor converting numbers to strings.
Key Takeaways
io.Readerandio.Writerare the universal interfaces for data flow in Go.- Use
strings.Builderfor building strings andbytes.Bufferfor working with byte data. fmthandles formatted output;strconvhandles type conversions more efficiently.- Compose readers and writers with
io.Copy,io.TeeReader, andio.MultiWriter. - Layer I/O (compression, buffering, encoding) by wrapping readers inside readers.
- The standard library's I/O primitives replace the need for most third-party streaming libraries.