SORACOM Developers

Getting Started

Using the SORACOM Funnel Kinesis Firehose adapter to aggregate data on the cloud (using SDK)

This guide explains how to use the SORACOM SDK and SORACOM Funnel adapter for Amazon Kinesis Firehose adapter to aggregate data sent from a device on Amazon S3. To do this using the SORACOM User Console, see Using the SORACOM Funnel Kinesis Firehose adapter to aggregate data on the cloud (User Console Version).

For general information about SORACOM Funnel, please refer to SORACOM Funnel service overview. For more detailed technical specifications, see SORACOM Funnel function details.


Preface

SORACOM Funnel is a cloud resource adapter that lets you forward data directly from a device to a specific cloud service.

Simply indicating a resource on a cloud service allows you to directly aggregate data from your device. While the forwarded destination is not a generic protocol as in SORACOM Beam, SORACOM Funnel lets you directly access leading cloud services with minimal time and effort.

Funnel


Amazon Kinesis Firehose setup

  1. First, select your preferred Amazon Web Services (AWS) region. In this example, we select US (Oregon).

Funnel

  1. Select Kinesis from the AWS console.

Funnel

  1. Click Go to Kinesis Firehose.

Funnel

  1. From the Amazon Kinesis Firehose administration screen, select Create Stream.

Funnel

  1. Select Amazon S3 as the Destination and my-kinesis-firehose01 as the Delivery stream name. Then select S3 bucket.

Funnel

  1. Select the IAM role used by the Kinesis Firehose service. Here we select Firehose delivery IAM role.

Funnel

  1. If the screen below appears, allow access to the IAM role.

Funnel

  1. Confirm the settings and click Create Delivery Stream to create a Delivery Stream.

Funnel

  1. Create the IAM users that will access Kinesis Firehose. On the AWS IAM Management Console, select an existing or create a new user and attach the policy. We attach Full Access.

The Amazon Kinesis Firehose cloud service is now set up and ready for use.


SORACOM Funnel setup

SORACOM Funnel setup consists of two steps:

First, please make sure that the CLI (SDK) is up to date.

$ gem soracom version

You should be using “Soracom API tool v1.1.0” or later.

Please refer to SORACOM SDK for Ruby for introduction method and setting.

Register credentials

The authentication information of various cloud services accessed with Funnel uses the credential store function.

In the case of AWS, save your AWS Credential information in the following format. (In this case, save it as my-aws-credentials.json file name.)

{
  "type": "aws-credentials",
  "credentials": {
    "accessKeyId": "AAAAAAAAAAAAAAAAAAAA",
    "secretAccessKey": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  },
  "description": "Kinesis Firehose"
}

Execute the following command to register the AWS Credential above.

$ soracom credentials create-credentials --credentials-id=my-aws-credentials --credentials-body=file://my-aws-credentials.json
{
  "description": "Kinesis Firehose",
  "type": "aws-credentials",
  "credentials": {
    "accessKeyId": "AAAAAAAAAAAAAAAAAAAA"
  },
  "credentialsId": "my-aws-credentials",
  "createDateTime": 1452790292,
  "updateDateTime": 1452790292,
  "lastUsedDateTime": null
}

The following files specified in the above command represent the AWS Credential information file created earlier.

Cloud service setup

Apply the Funnel setting (cloud service specification) to the Air SIM group.

Please make the following file for setting. (In this case, save as file name kinesis_firehose.json.)

{
  "destination": {
    "provider": "aws",
    "service": "firehose",
    "resourceUrl": "https://firehose.us-west-2.amazonaws.com/<Delivery stream name>"
  },
  "credentialsId": "my-aws-credentials",
  "enabled": true
}

For <Delivery stream name>, specify the name of Kinesis Firehose you created. In this guide, we created “my-kinesis-firehose”.

Create the target group, specify the group ID, and apply this setting. (Partial responses of the following commands are shown.)

$ soracom group update-configuration --group-id XXXXXXXXXXXXXXXXXXXXXX --namespace SoracomFunnel --params file://kinesis_firehose.json
{
  "destination": {
    "provider": "aws",
    "service": "firehose",
    "resourceUrl": "https://firehose.us-west-2.amazonaws.com/my-kinesis-firehose"
  },
  "credentialsId": "my-aws-credentials",
  "enabled": true
}

For other cloud service settings, please check Group configuration details.

This completes the setting.


Using the Kinesis Firehose adapter to aggregate data on the cloud

We now send some data. Remember to assign the SORACOM Air SIM card to the group you previously created.

Send a request to the Funnel endpoint. The endpoints are as follows:

Send over TCP

$ nc funnel.soracom.io 23080
{"message":"Hello SORACOM Funnel via TCP!"} [Enter]
200
[Ctrl+C]
{"operatorId": "OP0000000000", "timestamp": 1452791551499, "destination": {"resourceUrl": "https://firehose.us-west-2.amazonaws.com/my-kinesis-firehose", "service": "firehose", "provider": "aws"}, "credentialsId": "my-aws-credentials", "payloads": {"message":"Hello SORACOM Funnel via TCP!"}, "sourceProtocol": "tcp", "imsi": "440XXXXXXXXXXXX"}

Send over UDP

$ nc -u funnel.soracom.io 23080
{"message":"Hello SORACOM Funnel via UDP!"} [Enter]
200
[Ctrl+C]
{"operatorId": "OP0000000000", "timestamp": 1452791551499, "destination": {"resourceUrl": "https://firehose.us-west-2.amazonaws.com/my-kinesis-firehose", "service": "firehose", "provider": "aws"}, "credentialsId": "my-aws-credentials", "payloads": {"message":"Hello SORACOM Funnel via UDP!"}, "sourceProtocol": "udp", "imsi": "440XXXXXXXXXXXX"}

Send over HTTP

~$ curl -vX POST http://funnel.soracom.io -d '{"message":"Hello SORACOM Funnel via HTTP!"}' -H Content-Type:application/json
* Rebuilt URL to: http://funnel.soracom.io/
*   Trying 100.127.65.43...
* Connected to funnel.soracom.io (100.127.65.43) port 80 (#0)
> POST / HTTP/1.1
> Host: funnel.soracom.io
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Type:application/json
> Content-Length: 44
>
* upload completely sent off: 44 out of 44 bytes
< HTTP/1.1 204 No Content
< Date: Thu, 14 Jan 2016 17:18:46 GMT
< Connection: keep-alive
<
* Connection #0 to host funnel.soracom.io left intact
{"operatorId": "OP0000000000", "timestamp": 1452791551499, "destination": {"resourceUrl": "https://kinesis.ap-northeast-1.amazonaws.com/my-kinesis-firehose", "service": "firehose", "provider": "aws"}, "credentialsId": "my-aws-credentials", "payloads": {"message":"Hello SORACOM Funnel via UDP!"}, "sourceProtocol": "udp", "imsi": "440XXXXXXXXXXXX"}
{"operatorId": "OP0000000000", "timestamp": 1453210349875, "destination": {"resourceUrl": "https://firehose.us-west-2.amazonaws.com/my-kinesis-firehose", "service": "firehose", "provider": "aws"}, "credentialsId": "my-aws-credentials", "payloads": {"message": "Hello SORACOM Funnel via HTTP!"}, "sourceProtocol": "http", "imsi": "440XXXXXXXXXXX"}

Check aggregated data

Check the S3 Bucket previously specified as the Kinesis Firehose Destination. Check that a file has been created as below.

Funnel

{"operatorId": "OP0026965167", "timestamp": 1453217753433, "destination": {"resourceUrl": "https://firehose.us-west-2.amazonaws.com/my-kinesis-firehose", "service": "firehose", "provider": "aws"}, "credentialsId": "my-aws-credentials", "payloads": {"message":"Hello SORACOM Funnel via UDP!"}, "sourceProtocol": "tcp", "imsi": "440XXXXXXXXXXXX"}
{"operatorId": "OP0026965167", "timestamp": 1453217957353, "destination": {"resourceUrl": "https://firehose.us-west-2.amazonaws.com/my-kinesis-firehose", "service": "firehose", "provider": "aws"}, "credentialsId": "my-aws-credentials", "payloads": {"message":"Hello SORACOM Funnel via HTTP!"}, "sourceProtocol": "http", "imsi": "440XXXXXXXXXXXX"}

That’s it! With Funnel, there is no need to load cloud service SDKs or maintain cloud service credentials on your individual devices.

Instead, just specify the cloud service credentials and resource URL to directly send data from devices to your preferred cloud service. This lets quickly build a system connecting your devices to cloud services with minimal effort and minimal use of device memory and power.

We will continue to add new features to SORACOM Funnel, so if you have any requests or recommendations, please let us know!

Getting Started

SORACOM Air for Cellular

SORACOM Air for Sigfox

SORACOM Beam

SORACOM Canal/Direct/Door

SORACOM Endorse

SORACOM Funnel

SORACOM Gate

SORACOM Harvest

SORACOM Inventory

SORACOM Junction

SORACOM Krypton

SORACOM Lagoon

Service Detail

Developer Tools

pagetop