Notes on Calvin Dsouza

Logs in Distributed Systems

Sat, 26 Aug 2023 15:16:26 +0530

Introduction

A few years ago, logs had a simple meaning to me: data written to standard output on the terminal or a file to monitor an application and assist in debugging in case of a crash. This definition of logs was confined to mostly human-readable application logs. Although essential, this view of logs was limited.

So what are logs in distributed systems?
In the context of distributed and data systems, logs are an append-only sequence of records written to a file, ordered by time, often in a non-human-readable format, and sometimes compressed for better performance. These simple yet powerful append-only files allow an easy way to record an ordered sequence of immutable records and store them reliably. Now, let’s explore how these logs are utilized in distributed systems.

Serialization in Protocol Buffers

Sun, 20 Aug 2023 14:47:17 +0530

In the world of distributed systems and microservices, we often want to communicate with different services hosted on different servers in various geographical locations. To do this, we need to serialize the payload (in-memory structure to bytes) at the sender’s end and send it over the network. Then it’s deserialized (bytes to in-memory structure) at the receiver’s end. We want to make this process as fast and efficient as possible. So how can we achieve this, and what are our options?

Understanding Transaction Isolation in DynamoDB: From Concepts to Conflicts

Fri, 11 Aug 2023 20:00:33 +0530

Isolation in DynamoDB transactions or any data system for that matter is often an overlooked property but it is very important to understand, as concurrent transactions are almost unavoidable when working on distributed systems.

But what is isolation in relation to a database system? Isolation is a property of a database system that specifies how and when any change made by an operation becomes visible to another operation that is happening concurrently.