Logging in .NET – Elastic Search, Kibana and Serilog

I’ve been using log4net in the past and I found it quite useful as it is ready to use out of the box. In my last workplace, we are using SPLUNK and its amazing as I’m able to troubleshoot production issue by looking at the trend and activities. You can do query based and filtering the log and build a pretty dashboard. Downside of it is the cost for Splunk is expensive (i don’t think its for the mainstream user or small business)

So I’ve found another logging mechanism/storage/tool which is amazing!! It is called Elastic Search and its open source (well there are different subscriptions level for better supports). Pretty much Elastic Search is the engine for search and analytics

How about the GUI/Dashboard?Yes you can use Kibana. It is an open source data visualization platform that allows you to interact with data

Ok, so lets say if I have a .NET, how do I write my log to Elastic Search?You can use SeriLog. It will allow you to log structured event data to your log and you can use Serilog Elastic Search sink to integrate with Elastic Search

Serilog has different sink providers that allow you to store your log externally (beside file) including SPLUNK

I will talk more about Serilog separately in different post, stay tune!

Automapper – Dynamic and Generic Mapping

In Automapper, we normally have 1 to 1 mapping defined but I have a case whereas the incoming stream as a json payload which then I cast it as a dynamic (using JObject parse) and in one of the property within the payload it defined which object that it needs to cast into. Lets take a look at the sample below

Input
Json payload to create a city

{
    "requestId": "C4910016-C30D-415C-89D3-D08D724429A6",
    "messageType": "CITY_CREATED",
    "categoryName": "categoryA",
    "metadata": {
        "city": "sydney",
        "state": "NSW",
        "postcode": "2000",
        "country": "australia"
    }
}

at the same time we can also have a Json payload to create a staff

{
  "requestId":"C4910016-C30D-415C-89D3-D08D724429A6",
  "messageType": "STAFF_CREATED",
  "categoryName": "categoryB",
  "staffDetail": {
    "name": "fransiscus",
    "dateOfBirth": "01/01/1950"
  },
  "location" : {
    "cityId" : "1"
  }
}

So what we are doing in here, all the message will go into payload property (it can contain any object) and we add some extra information/header/metadata on the parent level
Desired Outputs

{
    "messageType": "CITY_CREATED",
    "payload": {
        "city": "sydney",
        "state": "NSW",
        "postcode": "2000",
        "country": "australia"
    },
    "provider": "abc",
    "providerRequestId": "C4910016-C30D-415C-89D3-D08D724429A6",
    "receivedAt": "2015-09-30T23:53:58.6118521Z",
    "lastUpdated": "2015-09-30T23:53:58.6128283Z",
    "lastUpdater": "Transformer",
    "attempt": 0
}
{
    "messageType": "STAFF_CREATED",
    "payload": {
        "staffName": "fransiscus",
        "dateOfBirth": "01/01/1950",
        "cityId": "1"
    },
    "provider": "abc",
    "providerRequestId": "C4910016-C30D-415C-89D3-D08D724429A6",
    "receivedAt": "2015-09-30T23:53:58.6118521Z",
    "lastUpdated": "2015-09-30T23:53:58.6128283Z",
    "lastUpdater": "Transformer",
    "attempt": 0
}

To map this to a concrete class 1:1 mapping is straight forward and easy. The problem here is that the “messageType” is the one that decided which object that it should be

Automapper Configuration:

1. POCO object

abstract class that stores all the metadata

public abstract class Metadata
    {
        public string MessageType { get; set; }

        public string Provider { get; set; }

        public string ProviderRequestId { get; set; }

        public DateTime ReceivedAt { get; set; }

        public DateTime LastUpdated { get; set; }

        public string LastUpdater { get; set; }

        public int Attempt { get; set; }

        public List<string> Errors { get; set; } 
    }
 public class City
    {
        public string CityName { get; set; }
        public string State { get; set; }
        public string PostCode { get; set; }
        public string Country { get; set; }
    }
public class StaffDetail 
    {
        public string Name { get; set; }
        public string DateOfBirth { get; set; }
        public int CityId { get; set; }
    }
public class Message<T> : Metadata where T : class
    {
        public T Payload { get; set; }
    }

2. Lets create a TypeConverter for the base class which is Metadata and from this converter it will return the derived class

public class MetadataTypeConverter : TypeConverter<dynamic, Metadata>
    {
protected override Metadata ConvertCore(dynamic source)
        {
            Metadata metadata;

            var type = (string)source.messageType.Value;

            switch (type)
            {
                case "STAFF_CREATED":
                    metadata = new Message<StaffDetail> { Payload = Mapper.Map<dynamic, StaffDetail>(source) };
                    break;
                case "CITY_CREATED":
                    metadata = new Message<City> { Payload = Mapper.Map<dynamic, City>(source) };
                    break;

                default: throw new Exception(string.Format("no mapping defined for {0}", source.messageType.Value));
            }

            metadata.ProviderRequestId = source.requestId;
            metadata.Topic = string.Format("{0}.{1}.pregame", 
                producerTopicName,
                source.categoryName ?? source.competition.categoryName);
            metadata.Provider = "My Provider";
            metadata.MessageType = source.messageType;
            metadata.ReceivedAt = DateTime.UtcNow;
            metadata.LastUpdated = DateTime.UtcNow;
            metadata.LastUpdater = "Transformer";
            metadata.Attempt = 0;

            return metadata;
        }
    }

3. Lets create a TypeConverter for the derived class which are Staff and City

public class CityTypeConverter : TypeConverter<dynamic, City>
    {
        protected override City ConvertCore(dynamic source)
        {
            City city = new City();
            city.CityName = source.metadata.city;
            city.State = source.metadata.state;
            city.Postcode = source.metadata.postcode;
            city.Country = source.metadata.country;

            return city;
        }
    }
 public class StaffDetailTypeConverter : TypeConverter<dynamic, StaffDetail>
    {
        protected override StaffDetail ConvertCore(dynamic source)
        {
            StaffDetail staffdetail = new StaffDetail();
            staffdetail.Name = source.staffDetail.name;
            staffdetail.DateOfBirth = source.staffDetail.dateOfBirth;
            staffdetail.CityId = source.location.cityId;

            return staffdetail;
        }
    }

3. Define the Automapper mapping in the configuration

public class WhafflMessageMapping : Profile
    {
        public override string ProfileName
        {
            get
            {
                return this.GetType().Name;
            }
        }

        protected override void Configure()
        {
            this.CreateMap()
                .ConvertUsing(new MetadataTypeConverter());

            this.CreateMap()
                .ConvertUsing(new StaffDetailTypeConverter());

            this.CreateMap()
                .ConvertUsing(new CityTypeConverter());
        }

        private Metadata BuildWhafflMessage(dynamic context)
        {
            var type = ((string)context.messageType.Value);

            switch (type)
            {
                case "STAFF_CREATED":
                    return new Message { Payload = Mapper.Map(context) };
                case "CITY_CREATED:
                    return new Message { Payload = Mapper.Map(context) };

                default: throw new Exception(string.Format("no mapping defined for {0}", context.messageType.Value));
            }

        }
    }

Yield keyword in .NET

I believe some of you already know about this but for me I never used it. Yield keyword has been existed since .NET 2.0 so I decided to look up of what it does and try to understand it

Based on MSDN

Yield is used in an iterator block to provide a value to the enumerator object or to signal the end of iteration, it takes one of the following form

Based on my understanding

Yield is a concatenation for a collection, or in SQL we normally use UNION

Yield break; is used to exit from the concatenation (remember it is not used to skip !)

One practical sample that I can think of is to get the enumerable of exception from inner exception (e.g stack trace)

sample code

Code Snippet
  1. class Program
  2.     {
  3.         ///<summary>
  4.         /// simple function to return IEnumerable of integer
  5.         ///</summary>
  6.         ///<returns></returns>
  7.         private static IEnumerable<int> GetIntegers()
  8.         {
  9.             for (int i = 0; i <= 10; i++)
  10.                 yield return i;
  11.         }
  12.         ///<summary>
  13.         /// simple function to return collection of class
  14.         ///</summary>
  15.         ///<returns></returns>
  16.         private static IEnumerable<MyClass> GetMyNumbers()
  17.         {
  18.             for (int i = 0; i <= 10; i++)
  19.                 if (i > 5)
  20.                     yield break;
  21.                 else
  22.                     yield return new MyClass() { Number = i };
  23.         }
  24.         internal class MyClass
  25.         {
  26.             public int Number { get; set; }
  27.             public string PrintNumber
  28.             {
  29.                 get {
  30.                     return “This is no “ + Number.ToString();
  31.                 }
  32.             }
  33.         }
  34.         static void Main(string[] args)
  35.         {
  36.             Console.WriteLine(“Simple array of integer”);
  37.             foreach (var number in GetIntegers())
  38.                 Console.WriteLine(number.ToString());
  39.             Console.WriteLine();
  40.             Console.WriteLine(“Collection of classes”);
  41.             foreach (var myclass in GetMyNumbers())
  42.                 Console.WriteLine(myclass.PrintNumber);
  43.             Console.ReadLine();
  44.         }
  45.     }

Output

Simple array of an integer
0
1
2
3
4
5
6
7
8
9
10Collection of classes
This is no 0
This is no 1
This is no 2
This is no 3
This is no 4
This is no 5

Remapping svc extension in IIS 7 extension handlers

If you open your WCF/ svc file in your localhost through browser and it just display blank page then what might happen is you don’t have handler associated with svc extension

To verify it, go to your IIS and go to IIS Handler Mappings and try to find an svc extension entry, but if you couldn’t find extension SVC exists on the list, then you can remap/re-add the svc extension by running

C:\Windows\Microsoft.NET\Framework64\v3.0\Windows Communication Foundation\ServiceModelReg.exe –i

*if you are still using 32 bit then you just need to replace Framework64 with Framework