For GPN19 I prepared a second talk on Documentation with any Editor . The talk was based on a previous one from Froscon 13, but the pipeline tooling changed.
This time there was a technical issue during the recording and so there are only the slides available, but you can still watch the video of the Froscon talk: Froscon 13: Documentation with any Editor
All scripts and the complete pipeline definition can be found in the GPN19-documentation Gitlab Repository .
tl;dr
I use Asciidoctor for documentation for years. It’s just easy to write, has more options than Markdown and I create all kinds of output for colleagues, customers or the web, just from one source. Converting the source to HTML, PDF, EPUB or several other formats, can be done on the command line of any Operating system, in Docker containers or delivery pipelines. I decided to use the Asciidoctor Docker container with Gitlab CI/CD . One of the advantages of Asciidoctor is that it can be easily put into a version control system like Git. I used Git for storing it since the early beginning, so the step to use a pipeline in combination was easy.
But why do I use Gitlab instead of other git software and Jenkins ? To be honest, I think it’s easier to start with Gitlab, because it’s completely integrated. Gitlab is available on premises (Docker container, Kubernetes, bare server) and public on https://gitlab.com .
Goal
As the title of the talk mentions, the main advantage to use a pipeline with Asciidoctor is, that you can edit your documents with the editor of your choice (I often use even VIM in a termux session on my mobile phone or tablet). Conversion happens after commit and push to the git repository.
Advantage
I can be sure that always the latest version is available, changing something (typos, additional information) doesn’t need any copy & paste to whatever other systems.
Building the pipeline and documents
Directory structure and documents
.
├── docker-asciidoctor
│ └── Dockerfile
├── documents-personal
│ └── basic-example.adoc
├── documents-work
│ ├── _attributes.asciidoc
│ ├── basic-example.adoc
│ ├── configfiles
│ │ └── customization
│ │ └── themes
│ │ └── Theme
│ │ ├── applications
│ │ │ └── blogs.css
│ │ └── custom.css
│ ├── example.adoc
│ ├── images
│ ├── include
│ │ ├── network2.adoc
│ │ └── network.adoc
│ ├── main-example.adoc
│ ├── more.asciidoc
│ ├── _variables-linux.asciidoc
│ ├── _variables-project.asciidoc
│ └── _variables-win.asciidoc
├── images
│ ├── diag-4e35056b3ac38736b7ce541f3f9bcced.png
│ ├── icons
│ │ ├── caution.png
│ │ ├── example.png
│ │ ├── home.png
│ │ ├── important.png
│ │ ├── LICENSE
│ │ ├── next.png
│ │ ├── note.png
│ │ ├── prev.png
│ │ ├── README.adoc
│ │ ├── tip.png
│ │ ├── up.png
│ │ └── warning.png
│ ├── logo.png
│ ├── png.png
│ └── test.png
├── LICENSE
├── pdftheme
│ ├── logo.png
│ ├── personal-theme.yml
│ └── work-theme.yml
├── presentations
│ └── slidedeck.adoc
├── public
│ ├── html-personal
│ ├── html-work
│ ├── images
│ ├── pdf-personal
│ ├── pdf-work
│ └── presentations
├── README.adoc
├── scripts
│ ├── create-index.sh
│ ├── html-conversion.sh
│ ├── html-work-conversion.sh
│ ├── pdf-conversion.sh
│ ├── pdf-work-conversion.sh
│ ├── reveal-conversion.sh
│ └── scripts
└── wiki-articles
Git submodule pointing to https://github.com/asciidoctor/docker-asciidoctor
Some example files to include in Asciidoctor
images folder used with all asciidoc files
pdftheme for company logo on output
some scripts for conversion and generating an overview of all documentation
Output folder which is published to Gitlab pages
The idea was to put documents to documents-work and it will generate html5 with side toc and pdf with a company logo on the title page. The pdf theme adds a logo, author, pages and copyright to the headers and footers of each page. The personal documents just used the default themes.
The folder structure is already prepared for confluence wiki and epub, but it’s not implemented until now.
Pipeline definition
Just add .gitlab-ci.yml to your repository and Gitlab will run your defined pipeline.
Build and test the Docker container
I decided to build the container directly in the Gitlab project, run some tests and put it into the included registry. Running Docker in Docker needs some work on the on premises deployments of Gitlab, but it works out of the box on the public one.
I add the original Github repository as a submodule to my project.
git add submodule https://github.com/asciidoctor/docker-asciidoctor
gitlab-ci.yml for building the container and tests
image: docker:git
services:
- docker:dind
variables:
CONTAINER_TEST_IMAGE: registry.gitlab.com/stoeps/$CI_PROJECT_NAME:$CI_BUILD_REF_NAME
CONTAINER_RELEASE_IMAGE: registry.gitlab.com/stoeps/$CI_PROJECT_NAME:latest
before_script:
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com
- git submodule sync --recursive
- git submodule update --init --recursive
stages:
- build
- test
- release
build:
stage: build
script:
- docker build -t $CONTAINER_TEST_IMAGE docker-asciidoctor
- docker push $CONTAINER_TEST_IMAGE
only:
changes:
- docker-asciidoctor/Dockerfile
refs:
- master
variables:
- $BUILDCONTAINER
test 1:
stage: test
script:
- echo "Run tests here"
- docker run -t --rm $CONTAINER_TEST_IMAGE asciidoctor -v | grep "Asciidoctor"
only:
changes:
- docker-asciidoctor/Dockerfile
refs:
- master
variables:
- $BUILDCONTAINER
test 2:
stage: test
script:
- docker run -t --rm $CONTAINER_TEST_IMAGE asciidoctor-revealjs -v
only:
changes:
- docker-asciidoctor/Dockerfile
refs:
- master
variables:
- $BUILDCONTAINER
release-image:
stage: release
script:
- echo "Tag Image and push to registry"
- docker pull $CONTAINER_TEST_IMAGE
- docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE
- docker push $CONTAINER_RELEASE_IMAGE
only:
changes:
- docker-asciidoctor/Dockerfile
refs:
- master
variables:
- $BUILDCONTAINER
Use docker:git image (work with git repository)
dind → Docker in Docker
Define Variables (Test image and release image name)
Run this scripts before the build starts (login in to registry, pull submodule)
3 Stages used in the pipeline (build, test and release)
Job build in stage build
Run only when
Dockerfileis newer, in branchmasterand when the$BUILDCONTAINERvariable is trueJob test1 in stage test
Job release image in stage release
Get test image from registry
Tag as release image
Push to registry
So the image build runs only when a variable SBUILDCONTAINER is set to true. I did this to save time, because we don’t need to build the container image multiple times. I decided to create a scheduler for the weekend, which sets the variable and is scheduled all week.
And even then, it needs to be a new Dockerfile to run this.
A successful build looks like this:
So we see our three stages and the different jobs. Splitting into stages has some advantages. First of all, jobs are running parallel in one stage, when a stage (or job) is not successful, the following stages do not run.
Build the output documents
I decided to run the built of the documents separately, so not the best when you want to save build time, but easier to mount the scripts, images, and documents as volumes into the container. After a successful build, the output is copied to the folder public, a script runs to build an overview html page and then it’s published to Gitlab pages.
I use the script for an overview because I couldn’t find a way to enable directory index for the Gitlab pages.
To do so, I added to additional stages (Conversion and Deploy)
After a successful build, the output is copied to the folder public, a script runs to build an overview html page and then it’s published to Gitlab pages.
I use the script for an overview because I couldn’t find a way to enable directory index for the Gitlab pages.
...
stages:
- build
- test
- release
- conversion
- deploy
...
pdf:personal:
stage: conversion
script:
- echo "Start Asciidoctor conversion"
- echo $CONTAINER_IMAGE:$CI_COMMIT_SHA
- docker run -t --rm -v $(pwd)/documents-personal:/documents/ -v $(pwd)/images:/images -v $(pwd)/scripts:/scripts $CONTAINER_RELEASE_IMAGE /scripts/pdf-conversion.sh
- cp documents-personal/*.pdf public/pdf-personal
artifacts:
name: "pdf-personal-$CI_COMMIT_TAG"
paths:
- public/pdf-personal
expire_in: 2 hours
except:
variables:
- $BUILDCONTAINER
pdf:work:
stage: conversion
script:
- echo "Start Asciidoctor conversion"
- echo $CONTAINER_IMAGE:$CI_COMMIT_SHA
- docker run -t --rm -v $(pwd)/documents-work:/documents/ -v $(pwd)/images:/images -v $(pwd)/scripts:/scripts -v $(pwd)/pdftheme:/pdftheme/ $CONTAINER_RELEASE_IMAGE /scripts/pdf-work-conversion.sh
- cp documents-work/*.pdf public/pdf-work
artifacts:
name: "pdf-work-$CI_COMMIT_TAG"
paths:
- public/pdf-work
expire_in: 2 hours
except:
variables:
- $BUILDCONTAINER
html:
stage: conversion
script:
- echo "Start Asciidoctor conversion"
- echo $CONTAINER_IMAGE:$CI_COMMIT_SHA
- docker run -t --rm -v $(pwd)/documents-work:/documents/ -v $(pwd)/images:/images -v $(pwd)/scripts:/scripts $CONTAINER_RELEASE_IMAGE /scripts/html-work-conversion.sh
- cp documents-work/*.html public/html-work
- docker run -t --rm -v $(pwd)/documents-personal:/documents/ -v $(pwd)/images:/images -v $(pwd)/scripts:/scripts $CONTAINER_RELEASE_IMAGE /scripts/html-conversion.sh
- cp documents-personal/*.html public/html-personal
artifacts:
name: "html-$CI_COMMIT_TAG"
paths:
- public/html-work
- public/html-personal
- images
expire_in: 2 hours
except:
variables:
- $BUILDCONTAINER
reveal:
stage: conversion
script:
- echo "Start Asciidoctor conversion"
- echo $CONTAINER_IMAGE:$CI_COMMIT_SHA
- docker run -t --rm -v $(pwd)/presentations:/documents/ -v $(pwd)/images:/images -v $(pwd)/scripts:/scripts $CONTAINER_RELEASE_IMAGE /scripts/reveal-conversion.sh
- cp presentations/*.html public/presentations
artifacts:
name: "html-$CI_COMMIT_TAG"
paths:
- public/presentations
- images
expire_in: 2 hours
except:
variables:
- $BUILDCONTAINER
pages:
stage: deploy
dependencies:
- html
- reveal
- pdf:personal
- pdf:work
script:
- cp -arvf images/* public/images/
- sh scripts/create-index.sh
- chmod +r public -R
artifacts:
paths:
- public
expire_in: 1 hour
only:
refs:
- master # this job will affect only the 'master' branch
except:
variables:
- $BUILDCONTAINER
Job
pdf:personalin stage conversiondocker command, mounting three volumes into the container and convert to pdf then
Copy resulting documents to public
Save artifacts, so the outputs of the conversion are saved
This time this runs always, except when
$BUILDCONTAINERis trueThe pages job has the other jobs from stage conversion as dependencies, so their artifacts can be used
build an overview of all converted documents
Gitlab Pages
When we now look at https://stoeps.gitlab.io/gpn19-documentation , we see the built overview page.
Check https://stoeps.gitlab.io/gpn19-documentation/pdf-work/main-example.pdf for an overview of a converted PDF file with title page, logo, plantuml conversion and so on. The source documents can be found in the repository itself with the same name (but extension .adoc).
