While working on a flex connector (csv based) this past week, I came across a situation whereby we had a date field that would sometimes be empty. Now of course for string and long values this is not an issue, but if the date string is empty the agent will through an error each time, complaining about an unparsable date. The log file we were processing is huge, so if we let it run as, it turns over a new log file every couple of seconds because of the date issue.
So how do we get around this .... ?
The first step in overcoming this issue is to assign the date field as a string. We then have a function that will allow us to convert a string to a TimeStamp, and if NULL return nothing. The function is called __createOptionalTimeStampFromString. The issue that I had was that the value was not NULL but rather a blank, hence the agent was still complaining, as it thought it was receiving a value.
To overcome this I used a combination of functions
event.deviceCustomDate1=__createOptionalTimeStampFromString(__ifThenElse(timestamp_token,"","2000-01-01 01:01:01 +0000",timestamp_token),yyyy-MM-dd HH:mm:ss Z).
Essentially what this does is check to see if the string is blank. If it is, then it sets the value to 2000-01-01 01:01:01 +0000, else it returns the actual value. Then it takes that string value and converts it to a date. We also specify the format for that date token.
Note: This should only be used for CustomDate1 / 2 and FlexDate 1 / 2. You can run into some serious issues with partition management if you start assigning dates in the past / future to endtime.