Binary Data and Buffer

In JavaScript, strings are encoded as UTF-16. When dealing with raw system data streams, network packets, or file uploads, you must process raw binary byte arrays. Node.js provides the global Buffer class to manage memory outside the V8 heap.

1. What is a Buffer?

A Buffer represents a fixed-size chunk of memory allocated outside the V8 engine. Once created, a Buffer size cannot be changed. It is an array-like structure where each element is a single byte (a value between 0 and 255, which is represented in hexadecimal format).

2. Creating Buffers

Node.js offers three primary methods to allocate and instantiate Buffers:

• Buffer.alloc(size): Allocates a zero-filled Buffer of a specified size. This is secure because it wipes any historical data in memory.
• Buffer.allocUnsafe(size): Allocates a Buffer without zero-filling. It is faster but could leak sensitive historical data in memory.
• Buffer.from(data): Instantiates a Buffer from an array, string, or another Buffer.

// Allocate a clean 10-byte buffer
const cleanBuf = Buffer.alloc(10);
console.log(cleanBuf); // Outputs: <Buffer 00 00 00 00 00 00 00 00 00 00>

// Allocate an unsafe 10-byte buffer (fast but dirty)
const unsafeBuf = Buffer.allocUnsafe(10);
console.log(unsafeBuf); // May contain raw fragments of old memory

// Create a buffer from a string
const stringBuf = Buffer.from("Hello Node", "utf8");
console.log(stringBuf); // Outputs: <Buffer 48 65 6c 6c 6f 20 4e 6f 64 65>

3. Buffer Encodings and Decoding

You can transform Buffers back into human-readable strings using different encoding formats, such as UTF-8, Base64 (used for data transfer/URLs), and Hex (hexadecimal string representation):

// Create a buffer from a UTF-8 string
const buf = Buffer.from("TypeScript is cool", "utf8");

// Convert buffer back to UTF-8 string
console.log(buf.toString("utf8")); // Outputs: TypeScript is cool

// Convert buffer to Base64
const base64Str = buf.toString("base64");
console.log("Base64 String:", base64Str);

// Convert buffer to Hex
const hexStr = buf.toString("hex");
console.log("Hex String:", hexStr);

// Restore a buffer from a Base64 string
const decodedBuf = Buffer.from(base64Str, "base64");
console.log(decodedBuf.toString("utf8")); // Outputs: TypeScript is cool

4. Writing and Modifying Buffers

Because Buffers represent fixed-length arrays in memory, writing beyond their capacity will truncate the input data:

const buf = Buffer.alloc(8); // Allocate 8 bytes

// Write string to buffer
const bytesWritten = buf.write("JavaScript", "utf8");
console.log(`Bytes written: ${bytesWritten}`); // Outputs: 8 (truncated!)
console.log(buf.toString("utf8")); // Outputs: JavaScri (exactly 8 bytes)