Skip to content

Development Process⚓︎

This is a development process we are tying to standardize within the team and encourage ourselves to follow in most cases.

The Swagger page⚓︎

WFL ships with a Swagger UI that documents all available endpoints. It's available at path <host>/swagger.

For local access, see accessing Swagger Locally.

Development Setup⚓︎

Clojure development feels very different from Scala and Java development. It even differs markedly from development in other dynamic languages such as Python or Ruby.

Get a demonstration from someone familiar with Clojure development before you spend too much time trying to figure things out on your own.

Find a local Cursive user for guidance if you like IntelliJ. Olivia Kotsopoulos knows how to use it.

Cursive licences are available here. If none are available, free non-commercial licenses are suitable for open-source development.

The steps for getting this project set up with very recent versions of IntelliJ differ from Cursive's docs:

Tip

Run make prebuild before launching IntelliJ as it sets up all libraries and derived resources and sources: bash make TARGET=prebuild -jN

  1. Outside of IntelliJ, clone the repo.
  2. Now inside of IntelliJ, import the project.
  3. Use the Project Structure window (Help -> Find Action -> Project Structure) to set a JDK as the Project SDK

There is also a Calva plugin for Visual Studio Code.

Tom Lyons hacks Clojure in Emacs using CIDER and nREPL. CIDER is not trivial to set up, but not especially difficult if you are used to Emacs. (I can help if CIDER gives you trouble.)

Process⚓︎

We base feature branches off develop, make pull requests, ask for reviews and merge back to develop on Github.

For the release process, please refer to the release guide.

  1. Clone the repo git@github.com:broadinstitute/wfl.git

  2. Start from the latest copy of the remote develop git checkout develop git pull origin develop

  3. Create a feature branch

    It is highly recommended that you follow the naming convention shown below so JIRA could pick up the branch and link it to our JIRA board. git checkout -b tbl/GH-666-feature-branch-something

  4. Start your work, add and commit your changes git add "README.md" git commit -m "Update the readme file."

  5. [Optional] Rebase onto latest develop if you want to get updates git checkout develop git pull origin develop --ff git checkout tbl/GH-666-feature-branch-something git rebase develop

    alternatively, you could use the following commands without switching branches: git checkout tbl/GH-666-feature-branch-something git fetch origin develop git merge develop

  6. Push branch to Github in the early stage of your development (recommended): git push --set-upstream origin tbl/GH-666-feature-branch-something

  7. Create the pull request on Github UI. Be sure to fill out the PR description following the PR template instructions.

    • If the PR is still in development, make sure use the dropdown menu and choose Create draft pull request

    • If the PR is ready for review, click Create pull request.

  8. Look for reviewer(s) in the team.

  9. Address reviewer comments with more commits.

  10. Receive approval from reviewers.

  11. Merge the PR.

Development Tips⚓︎

Here are some tips for WFL development.

Some of this advice might help when testing Liquibase migration or other changes that affect WFL's Postgres database.

setting up a local Postgres⚓︎

You can test against a local Postgres before running Liquibase or SQL against a shared database in gotc-dev or gasp production.

  1. Install Postgres locally. You need version 11 because that is what Google's hosted service supports, and there are differences in the SQL syntax.

    brew install postgresql@11

  2. Start Postgres.

    pg_ctl -D /usr/local/var/postgresql@11 start

    Tip

    It might be useful to set up some an alias for postgres if you are using zsh, for example: alias pq="pg_ctl -D /usr/local/var/postgresql@11" thus you could use pq start or pq stop to easily spin up and turn down the db.

  3. [Optional] Create wfl DB.

    If you see errors like this when launching a local WFL server or applying liquibase updates:

    FATAL: database "wfl" does not exist

    You should do as instructed within your terminal:

    createdb wfl

    Or to recreate an existing wfl DB:

    dropdb wfl createdb wfl

You are now free to launch a local WFL server pointing to your local DB.

Assuming that WFL_POSTGRES_URL in (wfl.environment/defaults) is set to point at a running local Postgres (e.g. jdbc:postgresql:wfl), running ./ops/server.sh (or however you launch a local WFL server) will connect the server to that running local Postgres.

Now any changes to WFL state will affect only your local database. That includes running Liquibase, so don't forget to reset :debug to env before deploying your changes after merging a PR.

migrating a database⚓︎

To change WFL's Postgres database schema, add a changeset XML file in the database/changesets directory. Name the file for a recent or the current date followed by something describing the change. That will ensure that the changesets list in the order in which they apply. Note that the id and logicalFilePath attributes are derived from the changeset's file name. Then add the changeset file to the database/changlog.xml file.

Test the changes against a local scratch database. See the next section for suggestions.

debugging JDBC SQL⚓︎

Something seems to swallow SQL exceptions raised by Postgres and the JDBC library. Wrap suspect clojure.java.jdbc calls in wfl.util/do-or-nil to ensure that any exceptions show up in the server logs.

debugging API specs⚓︎

If an API references an undefined spec, HTTP requests and responses might silently fail or the Swagger page will fail to render. Check the clojure.spec.alpha/defs in wfl.api.routes for typos before tearing your hair out.

accessing Swagger locally⚓︎

First, start a local WFL server.

shell ./ops/server.sh

To view the rendered Swagger page: shell open http://localhost:3000/swagger

debugging Liquibase locally⚓︎

Running liquibase update: bash liquibase --classpath=$(clojure -Spath) \ --url=jdbc:postgresql:wfl \ --changeLogFile=database/changelog.xml \ --username=$USER update For the above, the username and password need to be correct for the target environment.

If you're running a local server with the postgres command above, you don't need a password and can omit it.

Otherwise, you may be able to find this data in the Vault entry for the environment's server -- resources/wfl/environments.clj has some environments if you've built locally. You can use --password=$ENV_SOMETHING to supply it.

Tip

It is more convenient to use the following alias to migrate the database schema from within the api directory: clojure -M:liquibase if you are working with a local database.

Override ENVIRONMENT variables for local development⚓︎

WFL uses src/wfl/api/environment.clj to read and process environment variables. Most of the variables have their default values, which can be overwritten for development purposes. For example, if we want to run system tests in parallel against a local WFL instance, use below under api/ directory:

shell WFL_WFL_URL=http://localhost:3000 clojure -M:parallel-test wfl.system.v1-endpoint-test

REPL testing with fixtures.⚓︎

Now that we're using fixtures, and so on, in our tests, it is no longer good enough to run deftest vars as functions. Running a test like this (test-something) does not set up the necessary fixtures.

However, clojure.test/test-vars can run a test with all the surrounding clojure.test mechanism in place. It takes a vector of vars like this.

clojure (comment (test-vars [#'test-something]))

No tests found⚓︎

When trying to run tests in the command line, you may see the test suite exit prematurely -- but successfully -- with a warning indicating that no tests were found, and thus no tests were run.

shell $ make api TARGET=check export CPCACHE=/Users/okotsopo/wfl/api/.cpcache; \ clojure -M:test unit | \ tee /Users/okotsopo/wfl/derived/api/unit.log ... WARNING: No tests were found, make sure :test-paths and :ns-patterns are configured correctly in tests.edn. api unit finished on Thu Jun 24 15:03:33 EDT 2021 ...

This may indicate a compilation error in code not compiled as part of the build, e.g. tests. Linting can help expose any such errors.

Tip

By default, linting will halt on the first thrown exception, requiring further linting after fixing until the process succeeds.

```shell $ make api TARGET=lint ... == Eastwood 0.4.2 Clojure 1.10.3 JVM 11.0.10 == Directories scanned for source files:

src test ... == Linting wfl.system.cdc-covid19-surveillance-demo == Exception thrown during phase :analyze+eval of linting namespace wfl.system.cdc-covid19-surveillance-demo Got exception with extra ex-data: msg='No such var: covid' (keys dat)=(:form :file :end-column :column :line :end-line) (:form dat)= ^{:line 101} covid/rename-gather

ExceptionInfo No such var: covid ... An exception was thrown while analyzing namespace wfl.system.cdc-covid19-surveillance-demo Lint results may be incomplete. If there are compilation errors in your code, try fixing those. If not, check above for info on the exception.

Stopped analyzing namespaces after wfl.system.cdc-covid19-surveillance-demo due to exception thrown. 28 namespaces left unanalyzed.

If you wish to force continuation of linting after an exception in one namespace, make the option map key :continue-on-exception have the value true. ... ```

Back to top