AWS Streaming with Kinesis Firehose

I want to create a log delivery stream from the EC2 instance and stream it using firehose with source kinesis.

I install the kinesis agent on windows and it's running fine;

Here is my appsetting.json file inside AWSKinesisTap folder

{  "Sources": [   {    "Id": "SyslogDirectorySource",    "SourceType": "DirectorySource",    "Directory": "C:\\LogSource\\",    "FileNameFilter": "*.log",    "RecordParser": "SysLog",    "TimeZoneKind": "UTC",    "InitialPosition": "Bookmark"   }  ],  "Sinks": [  {    "Id": "KinesisStreamSink",    "SinkType": "KinesisStream",    "StreamName": "SyslogKinesisDataStream",    "Region": "us-east-1"   }   ],   "Pipes": [   {    "Id": "SyslogDS2KSSink",    "SourceRef": "SyslogDirectorySource",    "SinkRef": "KinesisStreamSink"    }   ]  }                      

AWSKinisisTap log from ProgramData

 2021-03-06 13:15:06.0816 Amazon.KinesisTap.Hosting.LogManager INFO Config file changed.   2021-03-06 13:15:09.4376 SyslogDirectorySource INFO DirectorySource id    SyslogDirectorySource watching directory C:\LogSource\ with filter *.log stopped.  2021-03-06 13:15:09.4376 Amazon.KinesisTap.Hosting.LogManager INFO Performance counter sink  stopped.  2021-03-06 13:15:09.7072 KinesisStreamSink INFO KinesisStreamSink id KinesisStreamSink for StreamName firehose stopped.  2021-03-06 13:15:09.7072 Amazon.KinesisTap.Hosting.LogManager INFO Log manager stopped.  2021-03-06 13:15:10.1990 Amazon.KinesisTap.Hosting.LogManager INFO Registered factory Amazon.KinesisTap.AWS.AWSEventSinkFactory.   2021-03-06 13:15:10.1990 Amazon.KinesisTap.Hosting.LogManager INFO Registered factory Amazon.KinesisTap.Windows.PerformanceCounterSinkFactory.   2021-03-06 13:15:11.1375 Amazon.KinesisTap.Hosting.LogManager INFO Registered factory Amazon.KinesisTap.Core.DirectorySourceFactory.   2021-03-06 13:15:11.1375 Amazon.KinesisTap.Hosting.LogManager INFO Registered factory Amazon.KinesisTap.ExchangeSource.ExchangeSourceFactory.   2021-03-06 13:15:11.1375 Amazon.KinesisTap.Hosting.LogManager INFO Registered factory Amazon.KinesisTap.Uls.UlsSourceFactory.  2021-03-06 13:15:11.1375 Amazon.KinesisTap.Hosting.LogManager INFO Registered factory Amazon.KinesisTap.Windows.WindowsSourceFactory.  2021-03-06 13:15:11.6037 Amazon.KinesisTap.Hosting.LogManager INFO Registered factory Amazon.KinesisTap.Core.Pipes.PipeFactory.  2021-03-06 13:15:12.0783 Amazon.KinesisTap.Hosting.LogManager INFO Registered factory Amazon.KinesisTap.AWS.CredentialProvider.ProfileRefreshingCredentialsFactory.  2021-03-06 13:15:12.5418 Amazon.KinesisTap.Hosting.LogManager INFO Registered factory Amazon.KinesisTap.AutoUpdate.AutoUpdateFactory.  2021-03-06 13:15:12.5418 Amazon.KinesisTap.Hosting.LogManager INFO Performance counter sink  started.    2021-03-06 13:15:12.5522 KinesisStreamSink INFO KinesisStreamSink id KinesisStreamSink for StreamName SyslogKinesisDataStream started.    2021-03-06 13:15:12.5522 Amazon.KinesisTap.Hosting.LogManager INFO Connected source SyslogDirectorySource to sink KinesisStreamSink     2021-03-06 13:15:12.5522 SyslogDirectorySource INFO DirectorySource id SyslogDirectorySource watching directory C:\LogSource\ with filter *.log started.  

Datastream from AWS

Firehose delivery stream

I have chosen kinesis stream from the data source and give full access in IAM roles. Data destination is an s3 bucket.

The problem is I am not getting any log files into AWS, no activity in the firehose, and no file in S3. What would be the problem?

Recent error

2021-03-07 19:16:36.5337 EventLogSink ERROR KinesisFirehoseSink EventLogSink client exception after 35 attempts: Amazon.KinesisFirehose.Model.InvalidArgumentException: This operation is not permitted on KinesisStreamAsSource delivery stream type. ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (400) Bad Request. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.

The previous problem is gone after adding roles, now I have new problems

System.NullReferenceException: Object reference not set to an instance of an object

https://stackoverflow.com/questions/66509201/aws-streaming-with-kinesis-firehose March 07, 2021 at 02:24AM