Product

Elastic Common Schema .NET library and integrations released

By
Stuart Cam

The Elastic Common Schema (ECS) defines a common set of fields for ingesting data into Elasticsearch. A common schema helps you correlate data from sources like logs and metrics or IT operations analytics and security analytics. Further information on ECS can be found in the official Elastic documentation, GitHub repository, or the Introducing Elastic Common Schema article.

This blog post is to announce the release of the ECS .NET library — a full C# representation of ECS using .NET types. This library forms a reliable and correct basis for integrations with Elasticsearch, that use both Microsoft .NET and ECS. These types can be used as-is, in conjunction with the official .NET clients for Elasticsearch, or as a foundation for other integrations.

We have also shipped integrations for Elastic APM Logging with Serilog and NLog, vanilla Serilog, and for BenchmarkDotnet.

There are a number of NuGet packages available for ECS version 1.4.0:

  • Elastic.CommonSchema.Serilog — Formats a Serilog log message into a JSON representation that can be indexed into Elasticsearch.
  • Elastic.Apm.SerilogEnricher — Adds transaction id and trace id to every Serilog log message that is created during a transaction. This works in conjunction with the Elastic.CommonSchema.Serilog package and forms a solution to distributed tracing with Serilog.
  • Elastic.Apm.NLog — Introduces two special placeholder variables (ElasticApmTraceId, ElasticApmTransactionId) for use within your NLog templates.
  • Elastic.CommonSchema.BenchmarkDotNetExporter — An exporter for BenchmarkDotnet that can index benchmarking results directly into Elasticsearch, which can be helpful for detecting code-related performance problems over time.
  • Elastic.CommonSchema — Foundational project that contains a full C# representation of ECS, used by the other integrations listed above.

Check out the Elastic Common Schema .NET GitHub repository for further information.

These packages are discussed in further detail below.

Elastic.CommonSchema.Serilog

This package includes EcsTextFormatter, a Serilog ITextFormatter implementation that formats a log message into a JSON representation that can be indexed into Elasticsearch, taking advantage of ECS features.

To use, simply configure the Serilog logger to use the EcsTextFormatter formatter:

var logger = new LoggerConfiguration()
                .WriteTo.Console(new EcsTextFormatter())
                .CreateLogger();

In the code snippet above the new EcsTextFormatter() method argument enables the custom text formatter and instructs Serilog to format the event as ECS-compatible JSON. The sample above uses the Console sink, but you are free to use any sink of your choice, perhaps consider using a filesystem sink and Elastic Filebeat for durable and reliable ingestion.

An example of the output from the snippet above is given below:

{
  "@timestamp": "2019-11-22T14:59:02.5903135+11:00",
  "log.level": "Information",
  "message": "Log message",
  "ecs": {
    "version": "1.4.0"
  },
  "event": {
    "severity": 0,
    "timezone": "AUS Eastern Standard Time",
    "created": "2019-11-22T14:59:02.5903135+11:00"
  },
  "log": {
    "logger": "Elastic.CommonSchema.Serilog"
  }
}

The EcsTextFormatter is also compatible with popular Serilog enrichers, and will include this information in the written JSON:

LoggerConfiguration = LoggerConfiguration
  .Enrich.WithThreadId()
  .Enrich.WithThreadName()
  .Enrich.WithMachineName()
  .Enrich.WithProcessId()
  .Enrich.WithProcessName()
  .Enrich.WithEnvironmentUserName();

Download the package from NuGet, or browse the source code on GitHub.

Elastic.Apm.SerilogEnricher

This Serilog enricher adds the transaction id and trace id to every log event that is created during a transaction. This works in conjunction with the Elastic.CommonSchema.Serilog package and forms a solution to distributed tracing with Serilog.

To use, simply configure the logger to use the Enrich.WithElasticApmCorrelationInfo() enricher:

var logger = new LoggerConfiguration()
   .Enrich.WithElasticApmCorrelationInfo()
   .WriteTo.Console(outputTemplate: "[{ElasticApmTraceId} {ElasticApmTransactionId}] {Message:lj} {NewLine}{Exception}")
   .CreateLogger();

In the code snippet above, Enrich.WithElasticApmCorrelationInfo() enables the enricher for this logger, which will set two additional properties for log lines that are created during a transaction:

  • ElasticApmTransactionId
  • ElasticApmTraceId

These two properties are printed to the Console using the outputTemplate parameter, of course they can be used with any sink and as suggested above you could consider using a filesystem sink and Elastic Filebeat for durable and reliable ingestion. This enricher is also compatible with the Elastic.CommonSchema.Serilog package.

The inclusion and configuration of the Elastic.Apm.SerilogEnricher assembly enables a rich navigation experience within Kibana, between the Logging and APM user interfaces, as demonstrated below:

Logs and traces

The prerequisite for this to work is a configured Elastic .NET APM Agent. If the agent is not configured the enricher won't add anything to the logs.

Download the package from NuGet, or browse the source code on GitHub.

Elastic.Apm.NLog

Introduces two special placeholder variables (ElasticApmTraceId, ElasticApmTransactionId), which can be used in your NLog templates. The intention is that this package will work in conjunction with a future Elastic.CommonSchema.NLog package and form a solution to distributed tracing with NLog.

var target = new MemoryTarget();
target.Layout = "${ElasticApmTraceId}|${ElasticApmTransactionId}|${message}";
Agent.Tracer.CaptureTransaction("TestTransaction", "Test", t =>
{
	traceId = "trace-id";
	transactionId = "transaction-id";
	logger.Debug("InTransaction");
});
// Logged message will be in format of `trace-id|transation-id|InTransaction`
// or `||InTransaction` if the place holders are not available

The above snippet allows you to add the following placeholders in your NLog templates:

  • ElasticApmTraceId
  • ElasticApmTransactionId

These placeholders will be replaced with the appropriate Elastic APM variables if available.

The prerequisite for this to work is a configured Elastic .NET APM agent. If the agent is not configured the enricher won't add anything to the logs.

Download the package from NuGet, or browse the source code on GitHub.

Elastic.CommonSchema.BenchmarkDotNetExporter

An exporter for BenchmarkDotnet that can index benchmarking result output directly into Elasticsearch, this can be helpful to detect performance problems in changing code bases over time.

var options = new ElasticsearchBenchmarkExporterOptions(url)
{
	GitBranch = "externally-provided-branch",
	GitCommitMessage = "externally provided git commit message",
	GitRepositoryIdentifier = "repository"
};
var exporter = new ElasticsearchBenchmarkExporter(options);

var config = CreateDefaultConfig().With(exporter);
BenchmarkRunner.Run(typeof(Md5VsSha256), config);

The code snippet above configures the ElasticsearchBenchmarkExporter with the supplied ElasticsearchBenchmarkExporterOptions. It is possible to configure the exporter to use Elastic Cloud as follows:

var options = new ElasticsearchBenchmarkExporterOptions(url)
{
	CloudId = "CLOUD_ID_HERE"
};

Example _source from a search in Elasticsearch after a benchmark run:

{
  "_index":"benchmark-dotnet-2020-01-01",
  "_type":"_doc",
  "_id":"pfFAh28B14pBZI_VO098",
  "_score":1.0,
  "_source":{
    "agent":{
      "git":{
        "branch_name":"externally-provided-branch",
        "commit_message":"externally provided git commit message",
        "repository":"repository"
      },
      "language":{
        "jit_info":"RyuJIT",
        "dot_net_sdk_version":"3.0.101",
        "benchmark_dot_net_caption":"BenchmarkDotNet",
        "has_ryu_jit":true,
        "build_configuration":"RELEASE",
        "benchmark_dot_net_version":"0.12.0",
        "version":".NET Core 3.0.1 (CoreCLR 4.700.19.47502, CoreFX 4.700.19.51008)"
      },
      "type":"Elastic.CommonSchema.BenchmarkDotNetExporter",
      "version":"1.0.0+7cedae2aaa06092ea253155279b835cee6160b3a"
    },
    "os":{
      "name":"Linux",
      "version":"ubuntu 18.10",
      "platform":"unix"
    },
    "message":null,
    "benchmark":{
      "q1":3632.625,
      "lower_outliers":[],
      "q3":5047.625,
      "confidence_interval":{
        "margin":14613.282591693971,
        "level":12,
        "mean":4123.291666666667,
        "lower":-10489.990925027305,
        "n":3,
        "standard_error":462.4594877151704
      },
      "percentiles":{
        "p0":3632.625,
        "p67":4151.345,
        "p25":3661.125,
        "p100":5047.625,
        "p90":4776.025000000001,
        "p80":4504.425,
        "p50":3689.625,
        "p85":4640.225,
        "p95":4911.825
      },
      "memory":{
        "bytes_allocated_per_operation":112,
        "total_operations":4,
        "gen2_collections":0,
        "gen1_collections":0,
        "gen0_collections":0
      },
      "max":5047.625,
      "interquartile_range":1415,
      "all_outliers":[],
      "upper_fence":7170.125,
      "standard_deviation":801.0033291649501,
      "kurtosis":0.6666666666666661,
      "n":3,
      "standard_error":462.4594877151704,
      "min":3632.625,
      "median":3689.625,
      "upper_outliers":[],
      "variance":641606.3333333333,
      "mean":4123.291666666667,
      "lower_fence":1510.125,
      "skewness":0.3827086238595402
    },
    "@timestamp":"2020-01-08T22:22:10.7917398+00:00",
    "host":{
      "hardware_timer_kind":"Unknown",
      "physical_processor_count":1,
      "logical_core_count":12,
      "in_docker":false,
      "processor_name":"Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz",
      "chronometer_frequency_hertz":1000000000,
      "has_attached_debugger":false,
      "physical_core_count":6,
      "architecture":"X64"
    },
    "log.level":null,
    "event":{
      "duration":1385324200,
      "measurement_stages":[
        {
          "operations":2,
          "iteration_mode":"Overhead",
          "iteration_stage":"Jitting"
        },
        {
          "operations":2,
          "iteration_mode":"Workload",
          "iteration_stage":"Jitting"
        },
        {
          "operations":4,
          "iteration_mode":"Overhead",
          "iteration_stage":"Warmup"
        },
        {
          "operations":4,
          "iteration_mode":"Overhead",
          "iteration_stage":"Actual"
        },
        {
          "operations":4,
          "iteration_mode":"Workload",
          "iteration_stage":"Warmup"
        },
        {
          "operations":4,
          "iteration_mode":"Workload",
          "iteration_stage":"Actual"
        },
        {
          "operations":4,
          "iteration_mode":"Workload",
          "iteration_stage":"Result"
        }
      ],
      "job_config":{
        "run_time":".NET Core 3.0",
        "jit":"Default",
        "launch":{
          "unroll_factor":2,
          "max_iteration_count":0,
          "launch_count":1,
          "iteration_count":3,
          "run_strategy":"Throughput",
          "iteration_time_in_milliseconds":0,
          "warm_count":3,
          "max_warmup_iteration_count":0,
          "invocation_count":4,
          "min_warmup_iteration_count":0,
          "min_iteration_count":0
        },
        "id":"ShortRun",
        "gc":{
          "heap_affinitize_mask":0,
          "server":false,
          "no_affinitize":false,
          "allow_very_large_objects":false,
          "retain_vm":false,
          "cpu_groups":false,
          "concurrent":false,
          "heap_count":0,
          "force":false
        },
        "platform":"AnyCpu"
      },
      "original":"Md5VsSha256.Sha256: ShortRun(Runtime=.NET Core 3.0, InvocationCount=4, IterationCount=3, LaunchCount=1, UnrollFactor=2, WarmupCount=3) [N=1000]",
      "method":"Elastic.CommonSchema.BenchmarkDotNetExporter.IntegrationTests.Md5VsSha256.Sha256(N: 1000)",
      "module":"Elastic.CommonSchema.BenchmarkDotNetExporter.IntegrationTests",
      "description":"Sha256",
      "action":"Sha256",
      "category":"Elastic.CommonSchema.BenchmarkDotNetExporter.IntegrationTests.Md5VsSha256-20200108-232208",
      "type":"Md5VsSha256",
      "parameters":"N=1000",
      "repetitions":{
        "measured":4,
        "warmup":4
      }
    }
  }
}

Download the package from NuGet, or browse the source code on GitHub.

Elastic.CommonSchema

Foundational project that contains a full C# representation of ECS. This package is used by the other packages listed above, and helps form a reliable and correct basis for integrations into Elasticsearch, that use both Microsoft .NET and ECS.

The intention of this package is to provide an accurate and up-to-date representation of ECS that is useful for integrations. Using this package ensures that, as a library developer, you are using the full potential of ECS and have a decent upgrade and versioning pathway through NuGet.

The types are annotated with the corresponding DataMember attributes, enabling out-of-the-box serialization support with the official clients.

Client installation

In this example, we will also install the Elasticsearch.net Low Level Client and use this to perform the HTTP communications with our Elasticsearch server.

PM> Install-Package Elasticsearch.Net

Connecting to Elasticsearch

var node = new Uri("http://localhost:9200");
var config = new ConnectionConfiguration(node);
var lowLevelClient = new ElasticLowLevelClient(config);

Creating an index template

Now we need to put an index template, so that any new indices that match our configured index name pattern are to use the ECS template.

We ship with different index templates for different major versions of Elasticsearch within the Elastic.CommonSchema.Elasticsearch namespace.

// We are using Elasticsearch version 7.4.0, let's use a 7 version index template
var template = Elastic.CommonSchema.Elasticsearch.IndexTemplates.GetIndexTemplateForElasticsearch7("ecs-*");

// Send the template to the Elasticsearch server
var templateResponse = lowLevelClient.Indices.PutTemplateForAll<StringResponse>(
	"ecs-template", 
	template);

// Check everything was successful
Debug.Assert(templateResponse.Success);

Now that we have applied the index template, any indices that match the pattern ecs-* will use ECS.

NOTE: We only need to apply the index template once. You can check to see if the index template exists using the Index template exists API, and if it doesn't, create it.

Creating and indexing an ECS event manually

Creating a new ECS event is as simple as newing up an instance:

var ecsEvent = new Base
{
    Timestamp = DateTimeOffset.Parse("2019-10-23T19:44:38.485Z"),
    Dns = new Dns
    {
        Id = "23666",
        OpCode = "QUERY",
        Type = "answer",
        Question = new DnsQuestion
        {
             Name   = "www.example.com",
             Type = "A",
             Class = "IN",
             RegisteredDomain = "example.com"
        },
        HeaderFlags = new [] { "RD", "RA" },
        ResponseCode = "NOERROR",
        ResolvedIp = new [] { "10.0.190.47", "10.0.190.117" },
        Answers = new []
        {
            new DnsAnswers
            {
                Data = "10.0.190.47",
                Name = "www.example.com",
                Type = "A",
                Class = "IN",
                Ttl = 59
            },
            new DnsAnswers
            {
                Data = "10.0.190.117",
                Name = "www.example.com",
                Type = "A",
                Class = "IN",
                Ttl = 59
            }
        }
    },
    Network = new Network
    {
        Type = "ipv4",
        Transport = "udp",
        Protocol = "dns",
        Direction = "outbound",
        CommunityId = "1:19beef+RWVW9+BEEF/Q45VFU+2Y=",
        Bytes = 126
    },
    Source = new Source
    {
        Ip = "192.168.86.26",
        Port = 5785,
        Bytes = 31
    },
    Destination = new Destination
    {
        Ip = "8.8.4.4",
        Port = 53,
        Bytes = 95
    },
    Client = new Client
    {
        Ip = "192.168.86.26",
        Port = 5785,
        Bytes = 31
    },
    Server = new Server
    {
        Ip = "8.8.4.4",
        Port = 53,
        Bytes = 95
    },
    Event = new Event
    {
        Duration = 122433000,
        Start = DateTimeOffset.Parse("2019-10-23T19:44:38.485Z"),
        End = DateTimeOffset.Parse("2019-10-23T19:44:38.607Z"),
        Kind = "event",
        Category = "network_traffic"
    },
    Ecs = new Ecs
    {
        Version = "1.2.0"
    },
    Metadata = new Dictionary<string, object>
    {
        { "client", "ecs-dotnet" }
    }
};

This can then be indexed into Elasticsearch:

var indexResponse = lowLevelClient.Index<StringResponse>(index,PostData.Serializable(ecsEvent));

// Check everything was successful
Debug.Assert(indexResponse.Success);

Congratulations, you are now using the Elastic Common Schema!

A note on the Metadata property

The C# Base type includes a property called Metadata with the signature:

/// <summary>
/// Container for additional metadata against this event.
/// </summary>
[DataMember(Name = "_metadata")]
public IDictionary<string, object> Metadata { get; set; }

This property is not part of the ECS specification, but is included as a means to index supplementary information.

Advanced metadata storage

In instances where using the IDictionary<string, object> Metadata property is not sufficient, or there is a clearer definition of the structure of the ECS-compatible document you would like to index, it is possible to subclass the Base object and provide your own property definitions.

The Elastic.CommonSchema.BenchmarkDotNetExporter project takes this approach, in the Domain source directory, where the BenchmarkDocument subclasses Base.

Versioning and compatibility

The version of the Elastic.CommonSchema package matches the published ECS version, with the same corresponding branch names:

The version numbers of the NuGet package must match the exact version of ECS used within Elasticsearch. Attempting to use mismatched versions, for example a NuGet package with version 1.4.0 against an Elasticsearch index configured to use an ECS template with version 1.3.0, will result in indexing and data problems.

In closing

The goal of ECS is to enable and encourage users of Elasticsearch to normalize their event data, so that they can better analyze, visualize, and correlate the data represented in their events. Using Elastic Common Schema as the basis for your indexed information also enables some rich out-of-the-box visualisations and navigation in Kibana.

Using the ECS .NET assembly ensures that you are using the full potential of ECS and that you have an upgrade path using NuGet.

Give the new Elastic Common Schema .NET integrations a try in your own cluster, or spin up a 14-day free trial of the Elasticsearch Service on Elastic Cloud. And if you run into any problems or have any questions, reach out on the Discuss forums or on the GitHub issue page.