Jenkins
Jenkins is an automation server. It allows for all kinds of automations. It is primarily used for Build automation, Continuous Integration, and Continuous Deployment.
Installations
- Install Jenkins on Ubuntu (using Vagrant)
- Vagrant for Jenkins on Ubuntu
- Jenkins in Docker
Examples
- Jenkins Pipeline - Hello World (pipeline, agent, stages, stage, stpes, echo)
- Jenkins Pipeline: running external programs with sh or bat, returnStdout, trim
- Jenkins Pipeline: Send e-mail notifications
- Jenkins Pipeline: Add some text to the job using manager.addShortText
- Jenkins CLI: create node
- Jenkins Pipeline BuildUser plugin
- Jenkins Pipeline - set and use environment variables
- Jenkins Pipeline: git checkout using reference to speed up cloning large repositories
- Jenkins report the name of the stage that failed (STAGE_NAME)
- Jenkins Pipeline: triggers from Version Control Systems (pollSCM)
- How to set the job number and description for a Jenkinsfile in a Jenkins Pipeline? (currentBuild, displayName, description)
- Jenkins Pipeline: read a file, write a file - readFile, writeFile
- Separate jobs for development and production currentBuild, projectName
- Jenkins pipeline: get current user (currentBuild, getBuildCauses)
- Jenkins parameters
- Arbitrary code execution in the shell (sh, parameters)
- Jenkins pipeline: parallel stages
- Jenkins Pipeline: Collect exit code from external commands (sh, bat, returnStatus)
- Jenkins pipeline: Get previous build status - currentBuild, getPreviousBuild
- Jenkins pipeline: interactive input during process (input)
- Jenkins pipeline: List agents by name or by label
- Jenkins pipeline: add badges
- Report failures.
- Send alerts
- Collect test coverage data.
- Jenkins slides
- Jenkins printing Unicode characters
Run external code, capture output
script {
v = sh(script: 'echo " 42"; echo', returnStdout: true).trim()
echo v
echo "a${v}b"
}
bat for windows.
catch and print error in jenkins
jenkins exception try - catch
https://fraaargh.wordpress.com/2018/06/20/get-a-jobs-console-logfile-from-a-jenkins-pipeline/
pipeline {
agent none
stages {
stage ('Catch crash') {
agent { label 'master'}
steps {
echo "before crash"
script {
try {
sh 'exit 1'
} catch (err) {
echo "exception caught, going on"
println err // java.lang.ClassCastException:
org.jenkinsci.plugins.workflow.steps.EchoStep.message expects class java.lang.String but received
class hudson.AbortException
}
}
echo "after crash"
}
}
stage ('Continue after crash') {
agent { label 'master'}
steps {
echo "stage after crash"
}
}
}
}
try {
//sh "ls -l no_such"
a = 10
b = 0
c = a/b
}
catch(Exception ex) {
//currentBuild.result = 'FAILURE'
println("exception")
println(ex) // hudson.AbortException: script returned exit code 2
println(ex.toString())
println(ex.getMessage())
println(ex.getStackTrace())
}
dir and tmp are problematic
stages {
stage ('Run external exe') {
agent { label 'master'}
steps {
sh 'pwd'
dir('/tmp/gabor') {
echo "inside"
sh 'pwd'
//sh 'sudo ./do-something.py'
}
sh 'pwd'
//sh "sudo sh -c 'cd /tmp; ./do-something.py; cd -'"
}
}
examples/jenkins/mkdir_exception.txt
java.io.IOException: Failed to mkdirs: /tmp@tmp/durable-e569697c
at hudson.FilePath.mkdirs(FilePath.java:1170)
at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.<init>(FileMonitori
ngTask.java:156)
at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java
:198)
at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java
:190)
at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:111)
at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:86)
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep
.java:182)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
at sun.reflect.GeneratedMethodAccessor1989.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterc
eptor.java:133)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
at WorkflowScript.run(WorkflowScript:16)
Jenkins / Groovy - define functions and call them with parameters
def report(status) {
println "status=${status}"
}
and call them
report("text")
Environment variables on Linux
sh 'printenv' sh 'env'
git Backup
Other
echo bat(returnStdout: true, script: 'set')
build(job: 'RevertServerAutomationCloud', parameters: [
string(name: 'VM_SNAPSHOT', value: 'CleanDb')
])
how to include one jenkinsfile in another one?
how to avoid repetititon?
stage('Revert agent 100')
{
steps
{
}
}
stage('Revert agent 102')
{
steps
{
}
}
how do try - catch and repeat interact?
vSphere buildStep: [$class: 'RevertToSnapshot', snapshotName: "${params.VM_SNAPSHOT}", vm: "${params.VM_NAME}"], serverName: '192.168.1.1'
httpRequest authentication: 'df8-b86d-3272', consoleLogResponseBody: true, httpMode: 'POST', ignoreSslErrors: true, responseHandle: 'NONE', url:
"http://192.168.1.1:8080/computer/${params.AGENT_NAME}/doDisconnect?offlineMessage=bye", validResponseCodes: '100:404'
Active Choices Parameter
examples/jenkins/array_list.txt
try {
List<String> subnets = new ArrayList<String>()
def subnetsRaw = "gcloud compute networks subnets list --project=${GCE_PROJECT} --network=corp-development --format=(NAME)".execute().text
for (subnet in subnetsRaw.split()) {
subnets.add(subnet)
}
return subnets
} catch (Exception e) {
print e
print "There was a problem fetching the artifacts"
}
Options
options {
ansiColor('xterm')
timestamps()
}
Scripts
Scripts not permitted to use method groovy.lang.GroovyObject invokeMethod java.lang.String java.lang.Object (org.jenkinsci.plugins.workflow.cps.EnvActionImpl keys). Administrators can decide whether to approve or reject this signature.
archiveArtifacts can be called multiple times
archiveArtifacts artifacts: 'mydata.json', onlyIfSuccessful: true writeJSON(file: 'otherfile.log', json: data, pretty: 4) archiveArtifacts artifacts: '*.log', onlyIfSuccessful: true
Environment variable values must either be single quoted, double quoted, or function calls.
They cannot be earlier defined environment variables or parameter values. We can however overcome this limitation by calling a function and passing the values to it.
examples/jenkins/exceptions.jenkinsfile
pipeline {
agent none
options {
ansiColor('xterm')
timestamps()
}
parameters {
string(name: 'machine', defaultValue: 'asos', description: 'Some text input')
string(name: 'size', defaultValue: '23', description: 'Some number input')
}
environment {
answer = 42
// machine_name = params.machine // -> Environment variable values must either be single quoted, double quoted, or function calls.
machine_name = set_machine_name(params.machine)
}
stages {
stage('try') {
agent { label 'master' }
steps {
script {
sh "hostname"
print("params.machine=${params.machine}")
print("params.size=${params.size}")
print("env.answer=${env.answer}")
print("env.machine_name=${env.machine_name}")
}
}
}
stage('try-agent') {
agent { label 'jenkins-simple-agent' }
steps {
script {
sh "hostname"
}
}
}
}
}
def set_machine_name(value) {
return value
}
Jenkins environment
Even if there is an exception in the environment section Jenkins will still run the "success" part of the post section. Same problem if there is an exception on one of the functions.
To overcome this we create an environment variable as the last step in the environment section and then we check that variable using
if (! binding.hasVariable('environment_is_set')) { error("Environment failed to set properly") }
That does not help in case there is an exception in the functions.
http_request
response = httpRequest discovery_url
println response
config_str = response.getContent()
for (item in config.sources) {
item.value
item.key
Sending e-mail problem fixed
https://stackoverflow.com/questions/20188456/how-to-change-the-security-type-from-ssl-to-tls-in-jenkins
* Sending e-mail In Manage Jenkins - Configure System in the Extended mail section set the SMTP: smtp.office365.com Domain name: @company.com Advanced: Use SMTP Authentication: + User Name: cicdserver@company.com Password: SMTP port: 587 E-mail notification section: SMTP server: smtp.office365.com Default user e-mail suffix: @company.com Advanced User Name: cicdserver@company.com Password: SMTP port: 587
Shut down Jenkins (via the Windows services) Open the file: C:\Program Files (x86)\Jenkins\jenkins.xml and change the arguments line to be:
examples/jenkins/arguments.txt
<arguments>-Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Dmail.smtp.starttls.enable=true -jar "%BASE%\jenkins.war" --httpPort=8080 --webroot="%BASE%\war"</arguments>
(specifically add: -Dmail.smtp.starttls.enable=true ) Then start Jenkins again.
Client was not authenticated to send anonymous mail Error sending to the following VALID addresses
examples/jenkins/variables.Jenkinsfile
pipeline {
agent none
stages {
stage('try') {
agent { label 'master' }
steps {
script {
//some_strange_name = null
//print(some_strange_name)
if (true) {
print("creating variable")
some_strange_name = 1
}
print(some_strange_name)
if (binding.hasVariable('some_strange_name')) {
print("has some_strange_name")
print(some_strange_name)
} else {
print("DOES NOT have some_strange_name")
}
}
}
}
stage('try again') {
agent { label 'master' }
steps {
script {
if (binding.hasVariable('some_strange_name')) {
print("has some_strange_name")
print(some_strange_name)
} else {
print("DOES NOT have some_strange_name")
}
}
}
}
}
}
Skip steps
Jenkins pipeline stop early with success How to indicate that a job is successful pipeline conditional step stage
examples/jenkins/skip_step.Jenkinsfile
pipeline {
agent none
options {
ansiColor('xterm')
timestamps()
}
parameters {
booleanParam(defaultValue: false, description: 'Checkbox', name: 'yesno')
}
stages {
stage('first') {
agent { label 'master' }
steps {
script {
println("first before")
println(params.yesno)
done = true
return
println("first after")
}
}
}
stage('second') {
agent { label 'master' }
when {
expression {
return ! done;
}
}
steps {
script {
println("second")
}
}
}
}
}
examples/jenkins/worker_job.Jenkinsfile
import java.text.SimpleDateFormat
pipeline {
agent none
options {
ansiColor('xterm')
timestamps()
}
parameters {
string(name: 'machine', defaultValue: '', description: 'Name of the machine')
string(name: 'size', defaultValue: '23', description: 'The size')
choice(choices: ['Mercury', 'Venus', 'Earth', 'Mars'], description: 'Pick a planet', name: 'planet')
//}
stages {
stage('try') {
agent { label 'master' }
steps {
script {
sh "hostname"
def data = readJSON text: '{}'
data.name = "test-529" as String
data.date = new java.text.SimpleDateFormat('yyyyMMddHHmmss').format(new Date())
writeJSON(file: 'mydata.json', json: data, pretty: 4)
archiveArtifacts artifacts: 'mydata.json', onlyIfSuccessful: true
//error("Fail after first artifact")
writeJSON(file: 'otherfile.log', json: data, pretty: 4)
archiveArtifacts artifacts: '*.log', onlyIfSuccessful: true
}
}
}
}
}
jenkins sh commnad fails - jenkins stops
examples/jenkins/sh_fails.jenkinsfile
pipeline {
agent { label 'master' }
stages {
stage('only') {
steps {
println("one")
sh "ls -l"
println("two")
sh "ls -l no_such"
println("three")
}
}
}
}
Repository branch filter for Git
It will start multiple jobs if more than one branch was pushed out. It is triggered even if there were no new commits in the branch that was pushed out
By default it will run every branch matching the filter, even if that branch was last changed 2 years ago. This can be a problem if for some reason your have hundreds or thousands of historical branches.
:^origin/(work|dev)-.*
You can limit this by selecting another option and setting the ancestry date to limit how many days you are ready to go back in time.
- Strategy for choosing what to build
- Choosing strategy: Ancestry
- Maximum Age of Commit: 1
In a pipeline
examples/jenkins/branch_filter_age_limit.jenkinsfile
checkout([
$class: 'GitSCM',
branches: [[name: '*/master']],
doGenerateSubmoduleConfigurations: false,
extensions: [[
$class: 'BuildChooserSetting',
buildChooser: [$class: 'AncestryBuildChooser', ancestorCommitSha1: '', maximumAgeInDays: 1]
]],
submoduleCfg: [],
userRemoteConfigs: [[]]
])
Jenkins Pipeline code reuse
https://cleverbuilder.com/articles/jenkins-shared-library/ https://jenkins.io/doc/book/pipeline/shared-libraries/
Exceptions
When you encounter one of those 40-lines long Java stack-traces, look for WorkflowScript to locate the source of the problem.
examples/jenkins/exception_catching.jenkinsfile
pipeline {
agent { label 'master' }
stages {
stage('only') {
steps {
script {
println("one")
sh "ls -l"
println("two")
try {
//sh "ls -l no_such"
a = 10
b = 0
c = a/b
}
catch(Exception ex) {
//currentBuild.result = 'FAILURE'
println("exception")
println(ex) // hudson.AbortException: script returned exit code 2
println(ex.toString())
println(ex.getMessage())
println(ex.getStackTrace())
}
//} catch(ArrayIndexOutOfBoundsException ex) {
//println("Catching the Array out of Bounds exception");
//}catch(Exception ex) {
println("three")
is_it_the_answer(42)
echo "try again"
is_it_the_answer(23)
}
}
}
stage('second') {
steps {
script {
// if (manager.logContains("three")) {
// manager.addWarningBadge("tres")
// } else {
// manager.addWarningBadge("nem harom")
// }
}
}
}
}
}
def is_it_the_answer(n) {
if (n == 42) {
return 'yes'
}
throw new Exception('Nope')
}
try {
is_it_the_answer(23)
} catch (err) {
print(err)
print(err.getMessage())
}
Jenkins parse console output
The logContains can parse the log created be the previous stages (but not the current stage) It can also be used in a post-action.
"manager" is org.jvnet.hudson.plugins.groovypostbuild.GroovyPostbuildRecorder$BadgeManager@41fe3861
examples/jenkins/parse_console_output.jenkinsfile
stage('second') {
steps {
script {
if (manager.logContains("three")) {
manager.addWarningBadge("drei")
} else {
manager.addWarningBadge("kein drei")
}
}
}
}
print("A disk image was created: zorg-1552278641")
def matcher = manager.getLogMatcher(/.* (zorg-\d+).*/)
print(matcher)
if (matcher?.matches()) {
def image_name = matcher.group(1)
print(image_name)
manager.addShortText(image_name)
}
readJSON
jenkins build step
https://jenkins.io/doc/pipeline/steps/pipeline-build-step/
jenkins collect status from invoked jobs
examples/jenkins/status_from_build.Jenkinsfile
pipeline {
agent { label 'master' }
stages {
stage('only') {
steps {
script {
println("BEFORE")
results = []
try {
def result = build job: "experiment-subproject", wait: true
println(result) // org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper
println(result.getId())
} catch(err) {
println("ERROR caught")
println(err) // hudson.AbortException: experiment-subproject #4 completed with status FAILURE (propagate: false to ignore)
// https://javadoc.jenkins-ci.org/hudson/AbortException.html
println(err.getMessage())
println(err.getStackTrace())
println(err.getCause())
println(err.getLocalizedMessage())
println(err.toString())
}
println("AFTER")
def res = build job: "experiment-subproject", wait: false
println(res) // null
rep = build job: "experiment-subproject", wait: true, propagate: false
println(rep) // org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper
println(rep.getId())
println(rep.getResult()) // FAILURE
println(rep.getDurationString())
results.push(rep)
//sh "ls -l"
//println("two")
//} catch(ArrayIndexOutOfBoundsException ex) {
//println("Catching the Array out of Bounds exception");
//}catch(Exception ex) {
//println("three")
//print("A disk image was created: jenkins-agent-1554 and then more")
//is_it_the_answer(42)
//echo "try again"
//try {
// is_it_the_answer(23)
//} catch (err) {
// print(err)
// print(err.getMessage())
//}
}
}
}
stage('second') {
steps {
script {
echo "hi"
println(rep.getId())
println(rep.getResult()) // FAILURE
println(rep.getProjectName())
println(rep.getDisplayName())
def res = build job: "experiment-subproject", wait: true, propagate: false
results.push(res)
// if (manager.logContains("three")) {
// manager.addWarningBadge("tres")
// } else {
// manager.addWarningBadge("nem harom")
// }
//def image_name = ''
//def matcher = manager.getLogMatcher(/.* (jenkins-agent-(\d+)).*/)
//print(matcher)
//if (matcher?.matches()) {
// image_name = matcher.group(1)
// println(image_name)
// manager.addShortText(image_name)
// println(image_name.getClass())
//}
//do_sg(image_name)
}
}
}
stage ('gitz') {
agent { label 'build-small' }
steps {
script {
results.each {
println(it)
println(it.getId())
println(it.getResult()) // FAILURE
println(it.getProjectName())
println(it.getAbsoluteUrl())
}
}
}
}
}
}
def is_it_the_answer(n) {
if (n == 42) {
return 'yes'
}
throw new Exception('Nope')
}
def do_sg(name) {
print("do_sg with $name")
}
links
https://jenkins.io/doc/pipeline/steps/
https://jenkins.io/doc/pipeline/steps/pipeline-build-step/#-build-%20build%20a%20job
https://jenkins.io/doc/pipeline/steps/pipeline-input-step/
script {
manager.addShortText("\n${params.cluserName}", "black", "yellow", "0px", "white")
manager.addShortText("${params.clusterId}-${params.command}", "black", "lightgreen", "0px", "white")
manager.addShortText("${params.services}", "black", "AliceBlue", "0px", "white")
if (params.usage) {
manager.addShortText("${params.usage}", "DimGray", "AliceBlue", "0px", "white")
}
}
Elapsed time
examples/jenkins/elapsed_time.Jenkinsfile
import groovy.time.TimeCategory
import groovy.time.TimeDuration
pipeline {
agent { label 'master' }
stages {
stage('first') {
steps {
script {
start = new Date()
echo "in first"
}
}
}
stage('second') {
steps {
script {
echo "in second"
}
}
}
}
post {
always {
echo "in post always"
script {
def stop = new Date()
TimeDuration td = TimeCategory.minus( stop, start )
println("Elapsed time: $td")
}
}
failure {
echo "in post failure"
}
}
}
Serialize regex
After a regex matching Jenkins sometimes wants to serialize the object, but it cannot do it and thus is raises an exception. (I have not yet figured out when does this happen.)
examples/jenkins/serialize_regex.Jenkinsfile
pipeline {
agent { label 'master' }
stages {
stage('first') {
steps {
script {
echo "in first"
def text = """This is some long test
with more than 1
rows.
"""
def match = (text =~ /\bsome\b/)
println("still first")
check_me()
}
}
}
stage('second') {
steps {
script {
echo "in second"
}
}
}
}
post {
always {
echo "in post always"
}
failure {
echo "in post failure"
}
}
}
def check_me() {
println("in check")
def text = """This is some long test
with more than 1
rows.
"""
def match = (text =~ /\bsome\b/)
}
Send mail
emailext (
subject: "Test by Gabor",
body: "Test from Gabor",
to: 'foo@bar.com',
from: 'jenkins-noreply@example.com',
listId: 'gabor-experimental',
)
Jenkins - check if host is accessible
examples/jenkins/ping_host.jenkinsfile
// catch exception
// stop process early reporting failure
pipeline {
agent { label 'master' }
parameters {
string(name: 'hostname', defaultValue: 'gabor-dev', description: 'Hostname or IP address')
}
stages {
stage('first') {
steps {
script {
echo "Checking $params.hostname"
try {
sh("ping -c 1 $params.hostname")
} catch(Exception e) {
println("Exception: ${e}")
error("Could not find host $params.hostname, is it on?")
return
}
echo "still in first"
}
}
}
stage('second') {
steps {
script {
echo "in second"
}
}
}
}
post {
always {
echo "in post always"
}
failure {
echo "in post failure"
}
}
}
last started stage
// last_started = env.STAGE_NAME
Early stop
jenkins - skip rest of the current stage, early end of current stage - stop the whole job reporting error
return // skipping rest of the current stage, running next stage
error("Some error message") // raise an exception stop the whole job
currentBuild
echo currentBuild.number // failed expecting number
println(currentBuild.number) // works
println("Number: ${currentBuild.number}")
println("Result: ${currentBuild.result}")
println("Display name: ${currentBuild.displayName}")
during the steps the result is null in the post section it is already 'SUCCESS' or 'FAILURE'
returnStatus instead of stopping the job
the exit code
def res2 = sh(script: "pwd", returnStatus: true)
Sample code
def errors = ''
def res = 'SUCCESS'
def res1 = sh(script: "xyz", returnStatus: true)
println("res1 ${res1}")
if (res1 !=0) {
res = 'FAILURE'
errors += "xyz"
}
def res2 = sh(script: "abc", returnStatus: true)
println("res2: ${res2}")
if (res2 !=0) {
res = 'FAILURE'
errors += "abc"
}
def res3 = sh(script: "pwd", returnStatus: true)
println("res3: ${res3}")
if (res3 !=0) {
res = 'FAILURE'
errors += "pwd"
}
//if (params.expected == 'Success') {
// sh "pwd"
//} else {
// sh "xyz"
//}
if (res == 'FAILURE') {
error(errors)
jenkins error, we need to clean the regex variables
def output = sh(...)
def version = (output =~ /Version\s*:\s*([\d.]+)/)
def build_number = (output =~ /Build number\s*:\s*([\d]+)/)
currentBuild.description = "${version[0][1]} - ${build_number[0][1]}<br>"
version = ""
build_number = ""
Optional artifacts
post {
always {
script {
archiveArtifacts artifacts: 'screenshots/*', fingerprint: true, allowEmptyArchive: true
}
}
}
multiple cron with parameter
examples/jenkins/multiple_cron.Jenkinsfile
// multiple cron with parameter
// There is a plugin and this experiment
triggers {
cron("35,36 * * * *")
}
stages {
stage('Check disk usage') {
steps {
script {
echo "--------------"
echo "HELLO $BUILD_NUMBER"
echo "--------------"
String[] ENVS = ["env1", "env3"]
def ENV_COUNT = ENVS.length
def x = BUILD_NUMBER.toInteger() % ENV_COUNT
echo x.toString()
echo ENVS[x]
Published on 2018-03-24