How to emit native JSON from VPC Flow Logs to Kinesis Firehose without Lambda?

When publishing VPC Flow Logs directly to Amazon Data Firehose, the flow log data is delivered to the Firehose delivery stream in plain text format, not JSON. Unfortunately, there isn't a native way to emit VPC Flow Logs as JSON directly to Firehose without additional processing.

However, Firehose itself offers built-in data transformation capabilities that can help you convert the plain text flow logs to JSON format without requiring a separate Lambda function. Firehose includes features like data transformation, compression, and encryption that can enhance the efficiency of your flow log processing pipeline.

For your use case of forwarding to Rapid7 via HTTP Endpoint, you have a few options:

1. Use Firehose's built-in data transformation feature, which can apply transformations to your data before delivery to the destination. This is integrated into the Firehose service itself and doesn't require you to manage separate Lambda functions or pay for additional compute resources beyond the standard Firehose transformation charges.

2. If your SIEM can parse the plain text format of VPC Flow Logs, you could deliver them as-is and let Rapid7 handle the parsing on their end, avoiding any transformation costs entirely.

3. Consider using a custom log format when creating your flow log. While the data will still be in plain text format when delivered to Firehose, you can structure the fields in a way that makes downstream parsing more efficient.

The most cost-effective approach would be to leverage Firehose's native transformation capabilities rather than introducing CloudWatch Logs or standalone Lambda functions into your architecture. This keeps your pipeline simple while still allowing you to transform the data format as needed for your SIEM integration.
Sources
Create a flow log that publishes to Amazon Data Firehose - Amazon Virtual Private Cloud
Publish flow logs to Amazon Data Firehose - Amazon Virtual Private Cloud

The short answer: Firehose has no built-in text-to-JSON parser, and no amount of configuration will add one. But your problem is probably smaller than it looks, because there's a layer here worth separating.

What actually arrives at your HTTP endpoint

Firehose's HTTP endpoint delivery already wraps records in a JSON envelope:

{
  "requestId": "...",
  "timestamp": 1234567890,
  "records": [
    { "data": "<base64-encoded record>" }
  ]
}

So Rapid7 is receiving JSON. What's inside data, once base64-decoded, is the space-delimited flow log line — because AWS documents that when publishing to Firehose, flow log data is delivered in plain text format. That's the flow log record format itself, not something Firehose is doing to it.

Worth confirming what Rapid7 actually receives before engineering around this. If their parser handles the standard VPC Flow Log format, you may already be done.

Why the other Firehose features don't help

Record format conversion (JSON → Parquet/ORC) requires JSON input. It converts out of JSON, not into it.

Dynamic partitioning with inline JQ also operates on JSON input, and only produces partition keys — it doesn't rewrite the payload.

Both assume the record is already structured. Yours isn't.

The recommendation: let the SIEM parse

This is what every documented integration does. Splunk's is instructive in the opposite direction — flow logs going to Splunk need a Lambda to convert to raw format, because Splunk's parser expects the native flow log line. The SIEM side is where field extraction belongs.

Rapid7 InsightIDR has a VPC Flow Logs event source with a parser for the standard format. Configure that rather than reshaping upstream. Concretely:

• Use the AWS default log record format (version 2) rather than a custom field selection, unless Rapid7 documents support for custom ordering. Custom formats change field positions and will break a positional parser.
• Point Rapid7 at its flow-log parser, not a generic JSON one.

This costs nothing, adds no moving parts, and is the pattern AWS's own partner integrations use.

If Rapid7 genuinely requires keyed JSON

Then you need a transformation step somewhere, and the realistic options are:

Lambda transformation on Firehose. You've ruled this out, but worth sizing rather than assuming. These functions are tiny and run on batched records, not per-record — for typical flow log volumes the monthly cost is usually small relative to Firehose ingestion itself. Worth calculating before discarding.

Flow Logs → S3 in Parquet, Rapid7 pulls from S3. VPC Flow Logs to S3 supports Parquet natively, which is structured and self-describing. This drops Firehose, Lambda, and CloudWatch entirely. The trade-off is batch delivery rather than streaming, and it depends on Rapid7 supporting S3 collection — which most SIEMs do.

A collector in front of Rapid7. If you already run a Rapid7 collector, have it parse there rather than in AWS.

What I'd check first

Send a test batch and look at what Rapid7 does with it. If fields extract correctly, the question is moot. If they don't, the specific failure tells you whether you need full JSON or just a format tweak — and that's a much cheaper thing to learn than building a pipeline around an assumption.