How to Handle Missing Values

In any organization Data Analysis is important in order to improve the business performance and to sort out different problems associated with the business. To analyse the situation first thing you need is historical data (relevant to the problem). Once you have data, its need to be in proper format so it can be easily analyse. There may be number of problems you have to face while bringing the data into proper format. Missing Data is most common problem that comes up during the data analysis process, especially when you get some feed back from customers to analyse the situation and due to some reason they are not interested to answer all the questions. During the Data Preparation phase (Data Mining Phases) you have to sort out these things to bring the data into proper format. There are two methods that are widely used to come up such situation and helps in well formed decision making in the end of analysis. One is “Avoid the missing data” and other is “Data Imputation” Many data cleansing methods had been developed and incorporated into analysis softwares (SPSS) to handle these problems.

Avoid Missing data:

This is the easiest way to handle the missing data. Delete all those records that are not completely filled. For example you send feedback form to 1000 customers, and out of 1000 only 800 feedback forms are completely filled. So easiest way is to avoid 200 forms and only consider rest of 800 feedback forms for analysis purposes.                                              

               Some time it may happen for a particular attribute or column you get less response as compare to other attributes. For example out of 1000 responders, 500 don’t like to answer question 6th. So in this case attribute corresponding to 6th number question can be discarded from the analysis.

                                                     Main advantage of this method is, it is not time consuming and same time it is very easy to follow. But there are many drawback associated with this method. Avoiding record or feedback results in losing some information. If the sample data size is large avoiding some records or attributes may not effect the results, but still you need to keep in mind you are losing something.

Data imputation:

Data imputation is another method of handling missing values. By using this method we try to fill missing values in the in the records and attributes. This method is quite useful because by following this method we can make sure we have all the information from responders. There are number of methods to fill the missing values few are given below.

Case Substitution

Mean Substitution

Hot Deck Imputation

Nearest Neighbour Imputation

Case Substitution:

By using this method we can replace the missing value with historical value from similar cases. We can not use value from current sample for case substitution; it must be from previous observations.

Mean Substitution:

This is quite simple method. In this method we can simply replace the missing value with the mean value for that particular attribute. But this method can not be applied for categorical data. This method is only useful for column or attributes imputation not for row imputation.

Hot Deck Imputation:

In this method, missing value is filled with value that comes from similar cases or records in the current sample. That means if two records are quite similar and in one record, value for some attribute is missing then we can fill the value from other similar record.

Cold Deck Imputation:

In this method we can replace missing value with single fixed value. Value must be from external source, which mean it can not be from current sample. Fixed value means if in particular attribute we are missing 5 values, then in all cases we have to fill same value.

 Next Article: Population Sampling and Need of Sampling

Mandeep Singh

www.mandeepvba.blogspot.com