Live Crystal Shardbox development with Johannes Müller (straight-shoota)

Live Crystal Shardbox development with Johannes Müller (straight-shoota)

The resulting Pull Request

• 0:00 Welcome, a bit of background.

• 1:10 Johannes introduces himself. he works at Manas Tech

• 3:30 Gabor introduces himself.

• 6:00 Gabor is trying to explain what does Shardbox do.

• 14:00 Explaining the Shardbox catalog.

• 20:55 Explaining the Shardbox core.

• 22:00 Schema overview

• 39:30 Explaining about the code of Shardbox core

• 48:25 Setting up the development environment - PostgreSQL in Docker

• 58:50 Install Crystal dependencies for the Shardbox core (till 1:14:30)

• 1:16:20 Build the worker

• 1:22:40 Start setting up the Shardbox web application

• 1:29:15 We have Shardbox running on my computer!

• 1:33:50 Change the Jinja templates to add links to the licenses

• 1:36:45 Change the search route to accept the new parameter

• 1:57:00 Showing the query on the search result page

• 2:03:00 Commit the changes, send pull-request.

• 2:09:10 Plans for Shardbox

Schema overview is in this file: db/schema-overview.svg in the shardbox-core repository.

The catalog directory in the catalog repository contains the manually mapped shards.

PostgreSQL in Docker

Start running the Docker container:

docker run -v$(pwd):/opt --name shardbox -e POSTGRES_PASSWORD=Secret -d postgres

Connect to the container:

docker exec -it shardbox bash

In the container:

cd /opt
apt-get update
apt-get install make
PGUSER=postgres TEST_DATABASE_URL=postgres://postgres:Secret@localhost/shardbox_test make test_db

At the end of the session working on sharbox you can stop the container:

docker stop shardbox

Packages we had to install (on the Ubuntu host)

sudo apt-get install libsass-dev
sudo apt-get install postgresql-client
sudo apt-get install libgit2-dev

Get the IP address of the Docker container:

docker inspect shardbox

Use that IP address in the following commands:

We checked if we can access the database using psql:

psql postgres://postgres:Secret@172.17.0.2/shardbox_test

Running the tests of shardbox-core:

export TEST_DATABASE_URL=postgres://postgres:Secret@172.17.0.2/shardbox_test
export SHARDS_OPTS=--ignore-crystal-version
# make test # this did not work
crystal spec

Build the worker

make build

Setting up for development database:

export DATABASE_URL=postgres://postgres:Secret@172.17.0.2/shardbox_test

Create a Personal access token with a name that you can easily recognize, e.g. "Shardbox Development Token" using the following rights:

repo
   x public_repo

user
   x read:user
   x user:email

export GITHUB_TOKEN=...
./bin/worker
./bin/worker import_catalog ../shardbox-catalog/catalog/

Importing the whole catalog can take a lot of time so we copied a single file from the catalog and ran on it. I would think a flag to limit the number of shards to be imported would help here.

mkdir catalog/
cp ../shardbox-catalog/catalog/Web_Servers.yml catalog/
./bin/worker import_catalog catalog/

One can also update the metrics:

./bin/worker update_metrics

Building the web applicaton in shardbox-web

SHARDS_OPTS=--ignore-crystal-version make build

Only the web application:

SHARDS_OPTS=--ignore-crystal-version make bin/app

Run the web application

./bin/app

Format the source code:

crystal tool format