Get Running with Arcadia Instant and KSQL

Published on April 23, 2018

This guide is meant to provide an overview of how to get the Arcadia Data demo environment up and running for Confluent’s KSQL system and Apache Kafka®.

  1. PREREQUISITES
  2. DOWNLOAD AND RUN ARCADIA INSTANT
  3. RUN KAFKA + KSQL
  4. CONNECT ARCADIA INSTANT TO KSQL
  5. SHUTDOWN
  6. RESOURCES

PREREQUISITES

Get Docker Installed

Make sure you have a recent version of Docker installed. You can download Docker Community Edition here.

Note: If you are running a version of Windows not supported by Docker CE for Windows (e.g., Windows 10 Home), then you need to use Docker Toolbox, which also needs Oracle VirtualBox installed.

We’ll be assuming the hostname for the Docker instances is the same as localhost depending on your configuration you may need a different hostname. For those of you using Docker Toolbox, you need to use the IP address of the running Docker image (more information later).

Increase the Resource Allocation for Docker

You may need to dedicate more memory and cores than the default when running KSQL and Kafka in Docker.

Advanced Docker

If you are using Docker Toolbox, you should update these settings in VirtualBox.

DOWNLOAD AND RUN ARCADIA INSTANT

Arcadia Instant is available for free download here. Request the download link, download it, and then install it on your computer. The installation process is straightforward, but if you’d like some guidance, check out this knowledge base article, Getting Started 1: Exploring Arcadia Instant.

Arcadia Instant LoginPlease register upon first use of Arcadia Instant.

RUN KAFKA + KSQL

You can skip the steps below if you already have a running KSQL rest server to connect to.

Grab the docker-compose Config Files

We’ve created a sample stack based off of the standard KSQL clickstream demo included in GitHub. The main difference is the addition of a service that runs a KSQL server and creates and loads the required streams and tables.

First open up a terminal window. (Docker Toolbox users should use the Docker Quickstart Terminal in your Start menu.)

Then create a separate working directory called “quickstart” and download the docker-compose.yml file using the following commands:

mkdir quickstart
cd quickstart
curl -o docker-compose.yml http://get.arcadiadata.com/ksql/docker-compose.yml

This may take a few minutes as the Docker images are downloaded. Then you can start up all the services:

docker-compose up -d

You can verify the services are running by typing:

docker-compose ps

Create the Initial Schemas

Now let’s build the schemas for the tables and streams we will soon query:

First enter the KSQL CLI:

docker-compose exec ksql-rest ksql-cli remote http://localhost:8082

Once in the KSQL CLI, run the schema creation script using the command:

RUN SCRIPT '/tmp/clickstream-schema.sql';

Now you can exit the KSQL CLI:

exit

You can now navigate your web browser to http://localhost:8082/index.html to access the KSQL UI. If you type in the URL in your browser, be sure to include “index.html” in the URL or else you will get a connection error. You can click ‘Show Topics’ or ‘Show Streams’ to see the schema objects just created.

Note: Docker Toolbox users should run “docker-machine ip” to get the Docker image IP address, and use that IP address instead of “localhost” in the URL above.

KSQL

CONNECT ARCADIA INSTANT TO KSQL

Configuring the Connection

In your Arcadia Instant window, navigate to the DATA tab then click on the NEW CONNECTION button. Fill out the dialog box with the following information:

Connection type: Confluent KSQL
Connection name: <provide a name for this connection>
Hostname or IP address: localhost
Port #: 8082
Username: admin
Password: admin

As mentioned earlier, Docker Toolbox users should use the IP address of the Docker image instead of “localhost.”

You can test the connection by clicking on the TEST button. Once you verify the connection, you can save the connection.

Arcadia Data Connection

Browse the Catalog

Click on Connection Explorer to get a view into all the existing topics, tables, and streams that have been created for that data connection.

Arcadia Connection Explorer

SHUTDOWN

To clean up your Docker environment, simply shut down the running services associated with this setup. Go to the “quickstart” directory that you created earlier, and then run:

docker-compose down

You can verify the services are down by running:

docker-compose ps

You can restart your services any time by going to the “quickstart” directory and running:

docker-compose up -d

NOTE: You will need to rerun the schema creation steps each time you restart the Docker containers

RESOURCES

Community Forum

This setup is supported by the community. You can ask questions or help others by visiting https://alexandria.arcadiadata.com.

Documentation

General Arcadia Data documentation is available at: http://documentation.arcadiadata.com.

Documentation for Arcadia Instant is available at: http://docs.arcadiadata.com.

Guides and Demos

A Day in the Life of a Business Analyst Using Streaming Analytics article

A Day in the Life of a Business Analyst and Streaming Analytics video 

Arcadia Data / Confluent KSQL demo

Arcadia Data Telco demo

Arcadia Enterprise 4.2 Kafka-KSQL Integration