OpenTelemetry in Action – Identifying Database Dependencies

Any organization can use microservices to increase agility. They address critical factors like team autonomy, time to market, cost-effective scaling, load scaling, and avoid complete outages. One of the biggest hurdles organizations face when attempting to break down monolithic applications into microservices is identifying their database dependencies.

Data sharing can be difficult and time-consuming. Databases don’t allow you to specify what can be shared and what cannot. You might accidentally break the functionality of another microservice by changing a schema in order to serve one microservice better.

It can be difficult to find the data owner or locate the business logic behind manipulating the data.

This article will discuss how OpenTelemetry can be used to identify components that share the same databases and objects such as tables.

Observability and OpenTelemetry: Basics

Let’s discuss OpenTelemetry and observability before we start building our demo app.

What makes an application highly visible?

If a system’s internal state can be inferred from its output at any time, it is considered highly observable. An example of this is a mobile app that can be observed and interact with multiple services. It can then reconstruct the transaction that results in an error response, so developers can determine the root cause.

A observable application collects the following information about every transaction:

1. Logs – Recording individual events that make up a transaction.
2. Metrics – Record of the aggregates of all events that make up a transaction.
3. Traces : A record of latency operations that helps identify bottlenecks during transactions.

All three pieces of information are required to reconstruct the system’s current state at any given moment.

What is OpenTelemetry?

OpenTelemetry generates logs and metrics in an integrated manner. OpenTelemetry is a standard for capturing observability data. There are several components to the OpenTelemetry data modeling.

Attributes

OpenTelemetry data structures are made up of attributes. These are key-value pair. OpenTelemetry defines the attributes that any component (e.g., an SQL client or an HTTP Request) can specify.

Events

Events can be described as a time stamp and a collection of attributes. An event can be recorded with details like messages and exception details.

Context

A context is a collection of attributes that are shared by a group of events. There are two types. The resource defines the location for events. Static contextual They don’t lose their value after an application executable is started. The name, version or library name are examples.

Dynamic contextual or span defines the active operation that contains an event. When the operation is executed, the value of span attributes changes. The span attributes are used to indicate the start time, HTTP response status code and the path of an HTTP request.

The context must be passed to all associated services in a distributed transaction. The context is used by the receiver service to create new spans in such cases. The distributed trace is a trace that crosses the service boundaries. This process involves the transfer of context to other services.

Logs

Logs are events that occur only when resources are used. This is an example of a log that is emitted when a program begins up.

Traces

A graph of operations can be used to organize events. A trace is a graph that shows the events associated with a transaction.

Metrics

A single event could occur multiple times in an application. Or its value may change. A metric is an event that has a value. It can either be the number of related events or a computation of the event’s value. A metric could be the system memory event. Its attributes include usage and utilization.

The documentation provides more information about OpenTelemetry concepts.

OpenTelemetry is used to identify database dependencies

OpenTelemetry is a standard for capturing attributes by applications. GitHub provides the specification for attributes that should cover a database client call. Many languages offer out-of-the box instrumentation libraries that collect data for database operations.

This demo will make use of the .NET SQLClient instrumentation to OpenTelemetry, in coordination with lightstep for analysis and storage.

Let’s look at the architecture of the demo app to see the path the telemetry takes to get to Lightstep. Because traces are sufficient to determine the relationship between monolithic databases and their components, we will only be focusing on them. For complete visibility, however, enterprise applications should also generate relevant logs and metrics.

To emit observability signals, we’ll first instrument our monolithic application using the OpenTelemetry SDK. For.NET applications instrumentation is a manual process. However, automatic instrumentation can be used for Java and Golang applications.

The SDK includes the OpenTelemetry Protocol Exporter (OTLP). We can use the exporter to send data directly into a telemetry service. OpenTelemetry platforms like Jaeger or Lightstep combine the traces to provide insights.

When the SDK is integrated, various parts of your app, such as the ASP.NET Core Request handler and SQL Client start automatically producing traces with relevant data. To enrich the information, your code may generate additional traces. OpenTelemetry is based upon the System.Diagnostics types. * Namespace as follows

1. System.Diagnostics.ActivitySource represents an OpenTelemetry tracer responsible for producing Spans.
2. System.Diagnostics.Activity represents a Span.
3. The AddTag function allows you to add attributes to a span. You can also add baggage to a span using the AddBaggage function. The baggage is then transported to the child activities. Other services might have the W3C header.

After you have instrumented your application, you can either run automated tests or let users use it to examine all interactions between the application’s database and the application.

Demonstration

Let’s make a monolithic employee management service (EMS) that is ASP.NET core minimal API. These are the endpoints of our API:

1. POST: Record the hours spent by employees on a project.
2. GET /ems/billing/employeeId: Fetches an employee’s hours on different projects.
3. POST /ems/payroll/add : Adds an employee in the payroll.
4. GET /ems/payroll/employeeId: Fetches the payroll data for the employee.

The monolith serves two distinct domains, billing and payroll. Complex monoliths might not have such dependencies, so segregating them may require code refactoring. You can easily uncouple these dependencies by simply studying them.
The following GitHub repository contains the complete source code for the EMS application.

read more https://thecloudblog.net/post/opentelemetry-in-action-identifying-database-dependencies/