Live Crystal Shardbox development with Johannes Müller (straight-shoota)

The resulting Pull Request

Schema overview is in this file: db/schema-overview.svg in the shardbox-core repository.

The catalog directory in the catalog repository contains the manually mapped shards.

PostgreSQL in Docker

Start running the Docker container:

docker run -v$(pwd):/opt --name shardbox -e POSTGRES_PASSWORD=Secret -d postgres

Connect to the container:

docker exec -it shardbox bash

In the container:

cd /opt
apt-get update
apt-get install make
PGUSER=postgres TEST_DATABASE_URL=postgres://postgres:Secret@localhost/shardbox_test make test_db

At the end of the session working on sharbox you can stop the container:

docker stop shardbox

Packages we had to install (on the Ubuntu host)

sudo apt-get install libsass-dev
sudo apt-get install postgresql-client
sudo apt-get install libgit2-dev

Get the IP address of the Docker container:

docker inspect shardbox

Use that IP address in the following commands:

We checked if we can access the database using psql:

psql postgres://postgres:Secret@

Running the tests of shardbox-core:

export TEST_DATABASE_URL=postgres://postgres:Secret@
export SHARDS_OPTS=--ignore-crystal-version
# make test # this did not work
crystal spec

Build the worker

make build

Setting up for development database:

export DATABASE_URL=postgres://postgres:Secret@

Create a Personal access token with a name that you can easily recognize, e.g. "Shardbox Development Token" using the following rights:

   x public_repo

   x read:user
   x user:email

export GITHUB_TOKEN=...
./bin/worker import_catalog ../shardbox-catalog/catalog/

Importing the whole catalog can take a lot of time so we copied a single file from the catalog and ran on it. I would think a flag to limit the number of shards to be imported would help here.

mkdir catalog/
cp ../shardbox-catalog/catalog/Web_Servers.yml catalog/
./bin/worker import_catalog catalog/

One can also update the metrics:

./bin/worker update_metrics

Building the web applicaton in shardbox-web

SHARDS_OPTS=--ignore-crystal-version make build

Only the web application:

SHARDS_OPTS=--ignore-crystal-version make bin/app

Run the web application


Format the source code:

crystal tool format