Live Crystal Shardbox development with Johannes Müller (straight-shoota)
- 0:00 Welcome, a bit of background.
- 1:10 Johannes introduces himself. he works at Manas Tech
- 3:30 Gabor introduces himself.
- 6:00 Gabor is trying to explain what does Shardbox do.
- 14:00 Explaining the Shardbox catalog.
- 20:55 Explaining the Shardbox core.
- 22:00 Schema overview
- 39:30 Explaining about the code of Shardbox core
- 48:25 Setting up the development environment - PostgreSQL in Docker
- 58:50 Install Crystal dependencies for the Shardbox core (till 1:14:30)
- 1:16:20 Build the worker
- 1:22:40 Start setting up the Shardbox web application
- 1:29:15 We have Shardbox running on my computer!
- 1:33:50 Change the Jinja templates to add links to the licenses
- 1:36:45 Change the search route to accept the new parameter
- 1:57:00 Showing the query on the search result page
- 2:03:00 Commit the changes, send pull-request.
- 2:09:10 Plans for Shardbox
Schema overview is in this file: db/schema-overview.svg in the shardbox-core repository.
The catalog directory in the catalog repository contains the manually mapped shards.
PostgreSQL in Docker
Start running the Docker container:
docker run -v$(pwd):/opt --name shardbox -e POSTGRES_PASSWORD=Secret -d postgres
Connect to the container:
docker exec -it shardbox bash
In the container:
cd /opt apt-get update apt-get install make PGUSER=postgres TEST_DATABASE_URL=postgres://postgres:Secret@localhost/shardbox_test make test_db
At the end of the session working on sharbox you can stop the container:
docker stop shardbox
Packages we had to install (on the Ubuntu host)
sudo apt-get install libsass-dev sudo apt-get install postgresql-client sudo apt-get install libgit2-dev
Get the IP address of the Docker container:
docker inspect shardbox
Use that IP address in the following commands:
We checked if we can access the database using psql:
Running the tests of shardbox-core:
export TEST_DATABASE_URL=postgres://postgres:Secret@172.17.0.2/shardbox_test export SHARDS_OPTS=--ignore-crystal-version # make test # this did not work crystal spec
Build the worker
Setting up for development database:
Create a Personal access token with a name that you can easily recognize, e.g. "Shardbox Development Token" using the following rights:
repo x public_repo user x read:user x user:email
export GITHUB_TOKEN=... ./bin/worker ./bin/worker import_catalog ../shardbox-catalog/catalog/
Importing the whole catalog can take a lot of time so we copied a single file from the catalog and ran on it. I would think a flag to limit the number of shards to be imported would help here.
mkdir catalog/ cp ../shardbox-catalog/catalog/Web_Servers.yml catalog/ ./bin/worker import_catalog catalog/
One can also update the metrics:
Building the web applicaton in shardbox-web
SHARDS_OPTS=--ignore-crystal-version make build
Only the web application:
SHARDS_OPTS=--ignore-crystal-version make bin/app
Run the web application
Format the source code:
crystal tool format
Published on 2021-06-23