This guide is meant to provide an overview of how to get the Arcadia Data demo environment up and running for Confluent’s KSQL system and Apache Kafka®.
- DOWNLOAD AND RUN ARCADIA INSTANT
- RUN KAFKA + KSQL
- CONNECT ARCADIA INSTANT TO KSQL
Get Docker Installed
Make sure you have a recent version of Docker installed. You can download Docker Community Edition here.
We’ll be assuming the hostname for the Docker instances is the same as localhost depending on your configuration you may need a different hostname. For those of you using Docker Toolbox, you need to use the IP address of the running Docker image (more information later).
Increase the Resource Allocation for Docker
You may need to dedicate more memory and cores than the default when running KSQL and Kafka in Docker.
If you are using Docker Toolbox, you should update these settings in VirtualBox.
DOWNLOAD AND RUN ARCADIA INSTANT
Arcadia Instant is available for free download here. Request the download link, download it, and then install it on your computer. The installation process is straightforward, but if you’d like some guidance, check out this knowledge base article, Getting Started 1: Exploring Arcadia Instant.
Please register upon first use of Arcadia Instant.
RUN KAFKA + KSQL
You can skip the steps below if you already have a running KSQL rest server to connect to.
Grab the docker-compose Config Files
We’ve created a sample stack based off of the standard KSQL clickstream demo included in GitHub. The main difference is the addition of a service that runs a KSQL server and creates and loads the required streams and tables.
First open up a terminal window. (Docker Toolbox users should use the Docker Quickstart Terminal in your Start menu.)
Then create a separate working directory called “quickstart” and download the docker-compose.yml file using the following commands:
curl -o docker-compose.yml http://get.arcadiadata.com/ksql/docker-compose.yml
This may take a few minutes as the Docker images are downloaded. Then you can start up all the services:
docker-compose up -d
You can verify the services are running by typing:
The system is ready for the next step when the “quickstart_ksql-datagen-clickstream_codes_1” and “quickstart_ksql-datagen-clickstream_users_1” containers show a status of “Exit 0” (as shown below) and the other containers have a status of “Up” so keep checking the status with the command above.
Note: Sometimes the “quickstart_ksql-datagen-clickstream_1” and the “quickstart_ksql-datagen-pageviews_1“ containers also exit as shown below. To resolve this, rerun the “docker-compose up -d” command shown above, and check the status of the system again with the “docker-compose ps” command.
Create the Initial Schemas
Now let’s build the schemas for the tables and streams we will soon query:
First enter the KSQL CLI:
docker-compose exec ksql-rest ksql-cli remote http://localhost:8082
Once in the KSQL CLI, run the schema creation script using the command:
RUN SCRIPT '/tmp/clickstream-schema.sql';
Now you can exit the KSQL CLI:
You can now navigate your web browser to http://localhost:8082/index.html to access the KSQL UI. If you type in the URL in your browser, be sure to include “index.html” in the URL or else you will get a connection error. You can click ‘Show Topics’ or ‘Show Streams’ to see the schema objects just created.
Note: Docker Toolbox users should run “docker-machine ip” to get the Docker image IP address, and use that IP address instead of “localhost” in the URL above.
CONNECT ARCADIA INSTANT TO KSQL
Configuring the Connection
In your Arcadia Instant window, navigate to the DATA tab then click on the NEW CONNECTION button. Fill out the dialog box with the following information:
Connection type: Confluent KSQL
Connection name: <provide a name for this connection>
Hostname or IP address: localhost
Port #: 8082
As mentioned earlier, Docker Toolbox users should use the IP address of the Docker image instead of “localhost.”
You can test the connection by clicking on the TEST button. Once you verify the connection, you can save the connection.
Browse the Catalog
Click on Connection Explorer to get a view into all the existing topics, tables, and streams that have been created for that data connection.
To clean up your Docker environment, simply shut down the running services associated with this setup. Go to the “quickstart” directory that you created earlier, and then run:
You can verify the services are down by running:
You can restart your services any time by going to the “quickstart” directory and running:
docker-compose up -d
Note: You will need to rerun the schema creation steps each time you restart the Docker containers (i.e., rerun the steps in the “Create the Initial Schemas” section).
This setup is supported by the community. You can ask questions or help others by visiting https://alexandria.arcadiadata.com.
General Arcadia Data documentation is available at: http://documentation.arcadiadata.com.
Documentation for Arcadia Instant is available at: http://docs.arcadiadata.com.